Geek Answers Handbook

Throttling S3 teams with aws cli

I run a backup script using the AWS CLI to execute the S3 sync command every night on my MediaTemple server. This has been working smoothly for several months, but I updated my Plesk installation and now every night when the backup script starts, MediaTemple shuts down my server due to overuse. The limits that I seem to cross are as follows:

RESOURCE INFO: Packets per second limit: 35000 Packets per second detected: 42229.11667000000306870788 Bytes per second limit: 50000000 Bytes per second detected: 61801446.10000000149011611938

They also include a network snapshot when it takes the server offline, which includes many open connections to Amazon's IP addresses (9 during the snapshot).

Is there anything I can do to throttle AWS connections? It is advisable that I look for an option in the AWS API (although I did not see anything useful in the documentation), but a ban on what I can do at my end to manage connections at the network level?

+9
amazon-s3 amazon-web-services aws-cli
source share
4 answers

As a result, I used Trickle and corked the download and upload speeds at a speed of 20,000 kb / s. This allows me to use the existing script without much change (all I had to do was add a trickle call to the top of the command).

Also, it seems that bandwidth limitation has been added as a problem for the AWS CLI, so hopefully this will not be a problem for people if it is implemented.

+2
source share

AWS CLI S3 transfer commands (including synchronization) have the following parameters:

  • max_concurrent_requests -
    • Default: 10
    • The maximum number of simultaneous requests.
  • multipart_threshold -
    • Default: 8 MB
    • The size threshold that the CLI uses for multipart transfer of individual files.
  • multipart_chunksize -
    • Default: 8 MB
    • When using multipart transfer, this is the block size that the CLI uses for multi-page transfers of individual files.

This is not as granular as throttling packets per second, but it seems that setting a lower value for the simultaneous request and lowering both the multipart threshold and chunksize will help. If the values ​​you inserted are close to average, I would start with these values ​​and adjust until you reliably exceed the limits:

 $ aws configure set default.s3.max_concurrent_requests 8 $ aws configure set default.s3.multipart_threshold 6MB $ aws configure set default.s3.multipart_chunksize 6MB
+6
source share

Besides changing the default maximum connections and fragment size, which are already mentioned, you can also set max_bandwidth. This is very effective when loading large individual files.

aws configure set default.s3.max_bandwidth 50MB/s

+3
source share

If you cannot do the chore with an aws s3 command like me, you can use:

sudo apt-get install pv (or yum install pv) pv -L 1M local_filename 2>/dev/null | aws s3 cp - s3://bucket_name/remote_filename

where -L 1M limits the bandwidth to 1 M / s and dashes after cp indicate stdin

Note: awscli from apt-get is too old to support stdin input, you need to update it via pip

+1
source share

More articles:


All Articles