Geek Answers Handbook

Aws s3 putObject vs sync

I need to upload a large file to aws s3 bucket. every 10 minutes my code removes the old file from the source directory and generates a new file. The file size is about 500 MB. Now I used the s3.putObject () method to load each file after creation. I also heard about aws s3 sync. his coming with aws-cli. It was used to upload files to the s3 bucket.

I used aws-sdk for node.js to download s3. aws-sdk for node.js does not contain the s3-sync method. s3-sync is better than s3.putObject () ?. I need a faster download.

+5
source share
1 answer

There is always more than a way to do things, so to upload a file to an S3 bucket you can:

  • use aws CLI and run aws s3 cp ...
  • use aws CLI and run aws s3api put-object ...
  • use aws SDK (your language of choice)

you can also use the sync method, but there is no need to synchronize the whole directory for one file, and as a rule, when searching for the best performance, it is better to run several cp instances to benefit from multi-threaded vs sync mono -thread.

basically all of these methods are wrappers for aws S3 API calls. From amazon doc

Making REST API calls directly from your code can be cumbersome. It requires that you write the necessary code to calculate a valid signature to authenticate your requests. Instead, we recommend the following options:

  • Use the AWS SDK to submit your requests (see Sample Code and Library). With this option, you do not need to write code to calculate the signature for request authentication, as SDK clients authenticate your requests using the access keys that you provide. Unless you have a good reason, you should always use the AWS SDK .
  • Use the AWS CLI to invoke the Amazon S3 API. For information on configuring AWS CLI and Amazon S3 command examples, see the following sections: Configure AWS CLI in the Amazon Simple Storage Service Developer's Guide. Use Amazon S3 with the AWS Command Line Interface in the AWS Command Line Interface User Guide.

Amazon recommends using the SDK. In the end, I think it's really a question of what you like best and how you integrate this piece of code into the rest of your program. For a one-time action, I always go to the CLI.

While performance, using one or the other, will not make a difference, since they are again a wrapper for invoking the AWS API. To optimize the transfer, you should look at the aws s3 transfer acceleration and see if you can enable it

+1
source

More articles:


All Articles