Track progress downloads on S3 using Ruby aws-sdk

Question

First, I know that there are many questions that look like this in SO. Last week I read the most, if not all of them. But I still can’t do this job for me.

I am developing a Ruby on Rails application that allows users to upload MP3 files to Amazon S3. The download itself works fine, but a progress bar will greatly improve the user's experience on the website.

I use the aws-sdk gem, which is official from Amazon. I looked everywhere in my documentation for callbacks during the boot process, but I could not find anything.

Files are downloaded one at a time directly to S3, so it does not need to be loaded into memory. No need to upload multiple files.

I decided that I might need to use jQuery to do this work, and I'm fine with that. I found this very promising: https://github.com/blueimp/jQuery-File-Upload And I even tried to follow the example here: https://github.com/ncri/s3_uploader_example

But I just couldn't get this to work for me.

The documentation for aws-sdk also BRIEFLY describes streaming downloads with a block:

obj.write do |buffer, bytes| # writing fewer than the requested number of bytes to the buffer # will cause write to stop yielding to the block end 

But this is hardly useful. How to write to the buffer? I tried a few intuitive options that would always lead to timeouts. And how would I even upgrade my browser based buffering?

Is there a better or simpler solution for this?

Thanks in advance. I would appreciate any help on this.

Answer 1

The "buffer" object obtained by passing the block to #write is an instance of StringIO. You can write to the buffer using #write or # <<. Here is an example that uses a block form to upload a file.

 file = File.open('/path/to/file', 'r') obj = s3.buckets['my-bucket'].objects['object-key'] obj.write(:content_length => file.size) do |buffer, bytes| buffer.write(file.read(bytes)) # you could do some interesting things here to track progress end file.close 

Answer 2

After reading the AWS gem source code, I adapted (or basically copy) the multi-page loading method to provide current progress based on how many pieces were loaded

 s3 = AWS::S3.new.buckets['your_bucket'] file = File.open(filepath, 'r', encoding: 'BINARY') file_to_upload = "#{s3_dir}/#{filename}" upload_progress = 0 opts = { content_type: mime_type, cache_control: 'max-age=31536000', estimated_content_length: file.size, } part_size = self.compute_part_size(opts) parts_number = (file.size.to_f / part_size).ceil.to_i obj = s3.objects[file_to_upload] begin obj.multipart_upload(opts) do |upload| until file.eof? do break if (abort_upload = upload.aborted?) upload.add_part(file.read(part_size)) upload_progress += 1.0/parts_number # Yields the Float progress and the String filepath from the # current file that being uploaded yield(upload_progress, upload) if block_given? end end end 

The compute_part_size method is defined here , and I changed it to this:

 def compute_part_size options max_parts = 10000 min_size = 5242880 #5 MB estimated_size = options[:estimated_content_length] [(estimated_size.to_f / max_parts).ceil, min_size].max.to_i end 

This code has been tested on Ruby 2.0.0p0.