I put a large number of small items on S3 using the REST API. The average payload is ~ 10 bytes.
Elements all move in one bucket and have randomized names (i.e. there is no lexicographic order)
From EC2 I succeeded at a speed of 4-500 per second. I use a thread pool of 96 threads, with 64 TCP connections.
I sometimes get HTTP 500, but have not yet received 503 - it means that the client slows down the speed of requests.
Does anyone know what I can really achieve? I know that the pipe between EC2 and S3 can control the bandwidth of 20 MB / s, so I hope to do a little better.
source
share