All geek questions in one place

Slow gear S3 & # 8594; EC2. Incorrect code?

I use this script with PHP5-CLI to download a file from Amazon S3 to an EC2 instance, but it is rather slow (1 minute for 160 MB, about 2 MB / s). Presumably at least 10 MB / s should be possible. Am I doing something wrong in the code?

require 'aws-sdk/sdk.class.php'; $s3 = new AmazonS3(array( 'key' => '********', 'secret' => '******************' )); $s3->get_object('mahbucket', 'filename.tar.gz', array('fileDownload' => 'downloaded.tar.gz'));

The bucket is in eu-west-1 , and the instance is in eu-west-1a

( Crossroads posted on AWS forums )

+4
source share
1 answer

Your script looks great, but I ran into similar problems a while ago that I could not fix ...

My new approach uses s3cmd to transfer files between S3 buckets and my EC2 instances. You just need to configure it once, and then you can run it through PHP. It is also more secure because you do not need to store your AWS credentials inside your script.

In my opinion, the speed is fine:

 user@mothership :~/s3# s3cmd put test.bin s3://data.example.com/test.bin test.bin -> s3://data.example.com/test.bin [part 1 of 13, 15MB] 15728640 of 15728640 100% in 0s 16.39 MB/s done test.bin -> s3://data.example.com/test.bin [part 2 of 13, 15MB] 15728640 of 15728640 100% in 0s 15.55 MB/s done test.bin -> s3://data.example.com/test.bin [part 3 of 13, 15MB] 15728640 of 15728640 100% in 0s 16.18 MB/s done test.bin -> s3://data.example.com/test.bin [part 4 of 13, 15MB] 15728640 of 15728640 100% in 0s 17.32 MB/s done test.bin -> s3://data.example.com/test.bin [part 5 of 13, 15MB] 15728640 of 15728640 100% in 0s 18.87 MB/s done test.bin -> s3://data.example.com/test.bin [part 6 of 13, 15MB] 15728640 of 15728640 100% in 0s 16.58 MB/s done test.bin -> s3://data.example.com/test.bin [part 7 of 13, 15MB] 15728640 of 15728640 100% in 0s 16.29 MB/s done test.bin -> s3://data.example.com/test.bin [part 8 of 13, 15MB] 15728640 of 15728640 100% in 0s 16.95 MB/s done test.bin -> s3://data.example.com/test.bin [part 9 of 13, 15MB] 15728640 of 15728640 100% in 0s 15.33 MB/s done test.bin -> s3://data.example.com/test.bin [part 10 of 13, 15MB] 15728640 of 15728640 100% in 0s 17.01 MB/s done test.bin -> s3://data.example.com/test.bin [part 11 of 13, 15MB] 15728640 of 15728640 100% in 0s 15.36 MB/s done test.bin -> s3://data.example.com/test.bin [part 12 of 13, 15MB] 15728640 of 15728640 100% in 0s 16.56 MB/s done test.bin -> s3://data.example.com/test.bin [part 13 of 13, 10MB] 11256320 of 11256320 100% in 0s 14.15 MB/s done user@mothership :~/s3# s3cmd get s3://data.example.com/test.bin test.bin.new s3://data.example.com/test.bin -> test.bin.new [1 of 1] 200000000 of 200000000 100% in 11s 16.20 MB/s done

This article may also be of interest: Network Performance in Amazon EC2 and Amazon S3

+2
source

Source: https://habr.com/ru/post/1416144/

More articles:


All Articles