How to make 10,000 files in S3 public

Question

I have a folder in a bucket with 10,000 files. There seems to be no way to download them and publish them right away. So I downloaded them all, they are private, and I need to make them publicly available.

I tried the aws console, it just gives an error (works fine with folders with fewer files).

I tried to use the S3 organization in Firefox, the same thing.

Is there some kind of software or some script that I can run to make it all publicly available?

Answer 1

You can create a bucket policy (see the example below) that gives access to all files in the bucket. Bucket policy can be added to the bucket through the AWS console.

{ "Id": "...", "Statement": [ { "Sid": "...", "Action": [ "s3:GetObject" ], "Effect": "Allow", "Resource": "arn:aws:s3:::bucket/*", "Principal": { "AWS": [ "*" ] } } ] } 

Also look at the following policy generator tool provided by Amazon.

http://awspolicygen.s3.amazonaws.com/policygen.html

Answer 2

If you are downloading for the first time, you can install publicly accessible files for downloading on the command line:

 aws s3 sync . s3://my-bucket/path --acl public-read 

As described in Using high-level s3 commands with AWS command-line interface

Unfortunately, ACLs apply only when downloading files. It (in my testing) does not apply ACLs to already uploaded files.

If you want to update existing objects, you could synchronize the recycle bin with yourself before, but it seems to have stopped working.

[Doesn't work anymore] This can be done from the command line:

 aws s3 sync s3://my-bucket/path s3://my-bucket/path --acl public-read 

(Thus, this no longer answers the question, but leaves the answer for reference, as it was before.)

Answer 3

I had to change several hundred thousand objects. I ran an instance of EC2 to run it, which speeds up its work. First you want to install the aws-sdk gem.

Here is the code:

 require 'rubygems' require 'aws-sdk' # Change this stuff. AWS.config({ :access_key_id => 'YOURS_HERE', :secret_access_key => 'YOURS_HERE', }) bucket_name = 'YOUR_BUCKET_NAME' s3 = AWS::S3.new() bucket = s3.buckets[bucket_name] bucket.objects.each do |object| puts object.key object.acl = :public_read end 

Answer 4

I had the same problem, the solution from @DanielVonFange is deprecated as there is no new version of the SDK.

Adding a code snippet that works for me right now with the AWS Ruby SDK:

 require 'aws-sdk' Aws.config.update({ region: 'REGION_CODE_HERE', credentials: Aws::Credentials.new( 'ACCESS_KEY_ID_HERE', 'SECRET_ACCESS_KEY_HERE' ) }) bucket_name = 'BUCKET_NAME_HERE' s3 = Aws::S3::Resource.new s3.bucket(bucket_name).objects.each do |object| puts object.key object.acl.put({ acl: 'public-read' }) end 

Answer 5

I just wanted to add that with the new S3 console you can select your folders and select “ Make public to make all the files inside the folders public. It works as a background task, so it must process any number of files.

Answer 6

Take a look at BucketExplorer , it manages mass operations very well and is a reliable S3 client.

Answer 7

If it were necessary, but the number of files made it slow down in serial mode. So I wrote a script that does this on the iron.io IronWorker service . Their 500 free hours of calculations per month is enough to handle even large buckets (and if you exceed what the price is reasonable). Since it runs in parallel, it ends in less than a minute for the 32,000 objects that I had. I also believe that their servers run on EC2, so communication between the job and S3 is fast.

No one can use my script for their needs.

Answer 8

You think they will publicly read the default behavior, right? :) I shared your frustration at creating a custom API for interacting with S3 with a C # solution. Here is a snippet that downloads an S3 object and sets it for public access by default:

 public void Put(string bucketName, string id, byte[] bytes, string contentType, S3ACLType acl) { string uri = String.Format("https://{0}/{1}", BASE_SERVICE_URL, bucketName.ToLower()); DreamMessage msg = DreamMessage.Ok(MimeType.BINARY, bytes); msg.Headers[DreamHeaders.CONTENT_TYPE] = contentType; msg.Headers[DreamHeaders.EXPECT] = "100-continue"; msg.Headers[AWS_ACL_HEADER] = ToACLString(acl); try { Plug s3Client = Plug.New(uri).WithPreHandler(S3AuthenticationHeader); s3Client.At(id).Put(msg); } catch (Exception ex) { throw new ApplicationException(String.Format("S3 upload error: {0}", ex.Message)); } } 

The ToACLString (acl) function returns a public read , BASE_SERVICE_URL s3.amazonaws.com , and the constant is AWS_ACL_HEADER x-amz-acl . The plug and DreamMessage stuff is likely to look weird to you as we use the Dream framework to optimize our HTTP messages. Essentially, we do http PUT with the specified headers and a special header signature for aws specifications (see this page in aws docs for examples on how to create an authorization header).

To modify existing ACLs with 1000 objects, you can write a script, but it may be easier to use the GUI tool to fix the immediate problem. The best I've used so far is a company called cloudberry for S3; they seem to have a free 15-day trial for at least one of their products. I just checked that it will allow you to select several objects at once and set their ACLs for sharing via the context menu. Enjoy the cloud!

Answer 9

Using cli:

aws s3 ls s3://bucket-name --recursive > all_files.txt && grep.jpg all_files.txt > files.txt && cat files.txt | awk '{cmd="aws s3api put-object-acl --acl public-read --bucket bucket-name --key "$4;system(cmd)}'