Geek Answers Handbook

Paperclip, Delayed Job, S3, Heroku - design for deferred processing of files with uploaded users: db or s3?

I need design feedback for uploading and deferring file processing using heroku, clip, pending work and, if necessary, s3. Parts of this were discussed elsewhere, but I could not find a complete discussion anywhere.

Description of the task:

  • Download the file (using paperclip to s3 / db on heroku). The file must be private, as it contains sensitive data.
  • Queue file for processing (deferred task)
  • The job starts in the queue.
  • The file is extracted (from s3 / db), and processing ends
  • File deleted (from s3 / db)

Since I am using a delayed job, I need to decide between saving the file to the database or to s3. I assume that saving the file to the web server is out of the question, since I use the hero and delayed work. Uploading files to s3 takes a lot of time. But storing files in db is more expensive. Ideally, we would like the processing to complete as quickly as possible.

What is the most common design pattern? Store files on s3? Store files in db? Any recommended gems used to extract and process files stored in s3 (aws-s3? S3?)?

+5
source share
2 answers

Heroku - 30 (), s3 .

carrierwave (rawncasts ) paperclip, , , , carrierwave_direct s3, .

Delayed_job (railscasts - delayed_job) s3 , .

gem :

gem 'delayed_job'

gem "aws-s3", :require => 'aws/s3'

gem 'fog'

gem 'carrierwave'

gem 'carrierwave_direct'

fog gem - . AWS how-to, .

(, , , )

def create
    @asset = Asset.new(:description => params[:description], :user_id => session[:id], :question_id => @question.id)
    if @asset.save && @asset.update_attributes(:file_name => sanitize_filename(params[:uploadfile].original_filename, @asset.id))
        AWS::S3::S3Object.store(sanitize_filename(params[:uploadfile].original_filename, @asset.id), params[:uploadfile].read, 'bucket_name', :access => :private, :content_type => params[:uploadfile].content_type)
            if object.content_length.to_i < @question.emailatt.to_i.megabytes && object.content_length.to_i < 5.megabytes
                url = AWS::S3::S3Object.url_for(sanitize_filename(params[:uploadfile].original_filename, @asset.id), 'bucket_name')
                if @asset.update_attributes(:download_link => 1)
                    if Usermailer.delay({:run_at => 5.minutes.from_now}).attachment_user_mailer_download_notification(@asset, @question)
                        process_attachment_user_mailer_download(params[:uploadfile], @asset.id, 24.hours.from_now, @question.id)
                        flash[:notice] = "Thank you for the upload, we will notify this posts author"
                    end
                end
            end
    else
        @asset.destroy
        flash[:notice] = "There was an error in processing your upload, please try again"
        redirect_to(:controller => "questions", :action => "show", :id => @question.id)
    end
end


private

    def sanitize_filename(file_name, id)
        just_filename = File.basename(file_name)
        just_filename.sub(/[^\w\.\-]/,'_')
        new_id = id.to_s
        new_filename = "#{new_id}" + just_filename
    end

    def delete_process(uploadfile, asset_id, time, question_id)
        asset = Asset.find(:first, :conditions => ["id = ?", asset_id])
        if delete_file(uploadfile, asset_id, time) && asset.destroy
            redirect_to(:controller => "questions", :action => "show", :id => question_id)
        end
    end


def process_attachment_user_mailer_download(uploadfile, asset_id, time, question_id)
        asset = Asset.find(:first, :conditions => ["id = ?", asset_id])
        if delete_file(uploadfile, asset_id, time) && @asset.delay({:run_at => time}).update_attributes(:download_link => 0)
            redirect_to(:controller => "questions", :action => "show", :id => question_id)
        end
    end

    #S3 METHODS FOR CREATE ACTION

    #deletes the uploaded file from s3
    def delete_file(uploadfile, asset_id, time)
        AWS::S3::S3Object.delay({:run_at => time}).delete(sanitize_filename(uploadfile.original_filename, asset_id), 'bucket_name')
    end

, ( , Rails). , , . , .

+4

:

. .

+3

More articles:


All Articles