AWS S3 Client for Linux with Multipage Download

Question

Which amazon s3 client do you use on linux with multi-page loading? I have 6 GB of ZIP files to download, and s3curl is not possible due to the maximum limit of only 5 GB.

Thanks. James

Answer 1

I use S3 Tools , it will automatically use the multi-page download function for files larger than 15 MB for all PUT commands:

Multipart is enabled by default and downloads for files larger than 15MB. You can set this limit at 5 MB (Amazon limit) with -multipart-chunk-size-mb = 5 or any other value between 5 and 5120 MB

After installation and configuration, simply run the following command:

~$ s3cmd put largefile.zip s3://bucketname/largefile.zip 

Alternatively, you can simply use split from the command line in the zip file:

 split -b1024m largefile.zip largefile.zip- 

and recombine your file system later using:

 cat largefile.zip-* > largefile.zip 

If you choose the second option, you may want to save the MD5 file hashes before downloading so that you can check the integrity of the archive when it is recombined later.

Answer 2

The official AWS command line interface supports multi-page loading. (He uses the boto successor botocore under the hood):

AWS Command Line Interface (CLI) is the one-stop tool for managing your AWS services. With just one download and configuration tool, you can manage multiple AWS services from the command line and automate them using scripts.

Besides this unified approach to all AWS APIs, it also adds a new set of simple files to efficiently transfer files to and from Amazon S3 with features similar to the well-known Unix commands, for example:

• ls - List of S3 objects and common prefixes under the prefix or all S3 codes.
• cp - Copy a local file or S3 object to another location locally or on S3.
• sync - Syncs S3 directories and prefixes.
• ...

So cp would be enough for the option used, but be sure to check sync , it is especially effective for many common scenarios (and the type implies cp depending on the arguments).

Answer 3

The boto library includes an s3 command-line tool called s3put , which can handle multi-page loading of large files.

Answer 4

Personally, I created the python s3upload.py file with a simple function to upload large files using boto and multipart upload.

Now every time I need to upload a large file, I simply run the command as follows:

 python s3upload.py bucketname extremely_large_file.txt 

More information and function code can be found here .

Answer 5

You can install the S3 bucket in the file system .

Answer 6

You can watch the FTP / Amazon S3 / Glacier CrossFTP client .

Answer 7

I just started using s4cmd , and processed the 50 gigabyte file just fine