1

I recently learnt some basic devops and set my server on Digitalocean droplet with travis CI. As there is no travis deploy provider supports for Digitalocean, I have to write my own deploy script. However I have more than 50000 files i.e near about 1 GB of data. I tried scp, ftp, rsync but I feel it takes a lot of time.

What are the transfer tools that is used in enterprise level ? Do they also use RSYNC or SCP. Or there are other tools that is better than these which are being used by medium level enterprises which I am not aware of.

Apart from it I have another question is there any deployment script/tool that can parallely transfer my files over ssh ?

Improve this question

2 Answers 2

Reset to default
4

Try LFTP that has commands to mirror or parallel copy. LFTP can run over many protocols.

The likely problem you have is that you are transfering a lot of data from a build service to a single vm over the Internet. Enterprises use the same tools and protocols but they pay for more bandwidth else ensure that their builds and deploy happens within the same cloud provider. They then deploy to many servers in parallel using standard protocols like SCP with tools like Ansible. The networks within a single cloud provider (Azure, AWS, GCP) are prioritised. This means of you want fat deployment bandwidth, and to run on Digital Ocean, you should build on Digital Ocean.

It may be the case that a very large amount of the 50k of files dont change on each build. So what you really want is a tool that looks at what files are already deployed. We can expect such tools to use file name, timestamps, file sizes and checksums to check whether a file needs uploading. If your build is stateless and creates 1G of fresh files each time their timestamps are going to be newer that the identical files already deployed. LFTP has an "ignore time stamp" option to avoid transfers of newer but identical files. See this question and answer https://serverfault.com/a/311670

Improve this answer
2
  • If enterprises are just using plain old SCP they tend to do deployments using something like Ansible Tower or some "platform" not because it does file transfer faster but because it has role based access control and can hide the ssh keys/passwords from the dev team and keep a full audit trail of what was done. I don't equate "enterprise devops software" with "better technology" as it all uses established protocols or is actually based on opensource like tower. I equate "enterprise devops software" with "costs a lot of money to adds a load of controls and reports to keep management happy".
    – simbo1905
    Commented Dec 8, 2018 at 16:08
  • A build service with a "freemuim" business model would go bankrupt very fast if it didn’t throttle bandwidth for free customers and prioritise paying customers. Likewise allowing unlimited bandwidth for “basic tier” hosting would put a hosting company out of business as anyone paying for “premium hosting” wouldn’t see good network speeds. This means “cheap” or “free” means “just good enough for typical users”. Trying to deploy 50k files or 1G isn’t what a typical user needs so it’s not surprising that it “isn’t good enough". Enterprises don't use free or basic services they pay for premium.
    – simbo1905
    Commented Dec 9, 2018 at 8:25
1

Not sure this helps you directly, but one option is to build the application within your CI build and push the (versioned) binaries/packages into an artifact repositorie like Sonatype Nexus or JFrog Artifactory. Your build would then, in a later step pull these binaries from the target machine/from within the droplet.

There are many packaging formats, depending on your technology stack, like e.g. NPM packages, jars, nuget packages, gems, Docker images and so on. Typically, you can also define your own package formats.

As I said, this is not a direct answer to the question about "enterprise level file transfer tools", but it is a common way to decouple building and pushing the application and pulling/deploying it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.