1

I'm experiencing much slower traffic for the exact same files when downloading (or uploading) them via the AWS website vs. downloading them via the CLI.

Is there a known reason for this?

Is there something that can be done about it?

6
  • Which one is slower? How large is the file? Are the cli and the browser both running on your local machine? – Michael - sqlbot Aug 19 '18 at 18:26
  • It’s slower in the console. I’ve checked with files ranging from 10Kb to 150Mb. Tried with chrome and firefox. Both in my machine. – Uberhumus Aug 19 '18 at 21:27
  • There's no easily-explainable "standard" reason for that. Every interaction with S3 takes place using the same APIs and same endpoints, whether you are using the console via a web browser or the CLI or one of the SDKs. S3 does support parallel (byte range splitting) uploads (files > 5MB) and downloads (any size). A browser download will never do parallel, but a browser upload sometimes does, and the CLI... I'm unsure of its standard behavior. I suspect that's the most likely explanation, unless you are using a web proxy behind your browser that has performance issues. – Michael - sqlbot Aug 19 '18 at 21:49
  • If it’s of any use, this is reliably repeated in our LAN, from any computer for over a week. – Uberhumus Aug 19 '18 at 21:51
  • I suspect your LAN may have a web proxy. Sometimes, a telltale sign of this is to visit a guaranteed-invalid domain such as www.invalid.invalid or invalid.example.com and the proxy may reveal itself in the error message, which will not be a standard browser error. – Michael - sqlbot Aug 19 '18 at 21:56
1

TL;DR Try reducing the number of concurrent connections used by awscli to 1 using this command: aws configure set default.s3.max_concurrent_requests 1


You could be experiencing an issue with the number of concurrent connections that the awscli is opening, if you are using the aws s3 commands (not aws s3api).

In the cli user guide under the Managing Objects section you will see that all high-level upload commands on an S3 Bucket beyond a large object threshold (awscli default 8MB) will use a multipart upload.

I couldn't find anything about the use of concurrent connections when using the web console, but looking at the output of lsof I could see my browser was only opening a single connection for a ~500MB file download, whereas using aws cp 10 concurrent connections were being opened. So this would appear to be the key difference in behaviour - at least with my browser and test case.

Running a quick, dirty test with the default (10) connections and then 1 concurrent connection, my download completed in the following times: s3.max_concurrent_requests=10 8m22.965s s3.max_concurrent_requests=1 6m47.772s

So it seems the overhead of the extra connections can sometimes cause slower downloads/uploads over my network, rather than making them faster, which could be what you are experiencing.

You could also try tweaking some of the other defaults around how multipart operations are handled which may yield better performance over your network, listed here: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html.

1
  • It works better in the CLI, not worse. – Uberhumus Aug 22 '18 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.