I'm experiencing much slower traffic for the exact same files when downloading (or uploading) them via the AWS website vs. downloading them via the CLI.
Is there a known reason for this?
Is there something that can be done about it?
DevOps Stack Exchange is a question and answer site for software engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure. It only takes a minute to sign up.Sign up to join this community
Try reducing the number of concurrent connections used by awscli to 1 using this command:
aws configure set default.s3.max_concurrent_requests 1
You could be experiencing an issue with the number of concurrent connections that the awscli is opening, if you are using the
aws s3 commands (not
In the cli user guide under the Managing Objects section you will see that all high-level upload commands on an S3 Bucket beyond a large object threshold (awscli default 8MB) will use a multipart upload.
I couldn't find anything about the use of concurrent connections when using the web console, but looking at the output of
lsof I could see my browser was only opening a single connection for a ~500MB file download, whereas using
aws cp 10 concurrent connections were being opened. So this would appear to be the key difference in behaviour - at least with my browser and test case.
Running a quick, dirty test with the default (10) connections and then 1 concurrent connection, my download completed in the following times:
So it seems the overhead of the extra connections can sometimes cause slower downloads/uploads over my network, rather than making them faster, which could be what you are experiencing.
You could also try tweaking some of the other defaults around how multipart operations are handled which may yield better performance over your network, listed here: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html.