Geek Answers Handbook

How to run multiple nokogiri screenshots at once

I have a website that requires using Nokogiri on different sites to retrieve data. This process starts as a background job using the delayed_job gem. However, it takes about 3-4 seconds per page to start, since it must pause and wait for other websites to respond. I am currently just running them, basically speaking

Websites.all.each do |website|
  # screen scrape
end

I would like to execute them in batches, and not one at a time, so I do not need to wait for a server response from each site (it can take up to 20 seconds in case).

What would be the best way to rubies or rails?

Thanks for your help in advance.

+5
source share
3

. Railscasts.

, .

spawn, , !

, :

:

 spawn do
    #execute your code here :)
 end

http://railscasts.com/episodes/171-delayed-job

https://github.com/tra/spawn

+2

, Typhoeus, HTTP-.

blawg Nokogiri, .

-, .

+5

EventMachine, - . , em-http-request, mutliple HTTP- , .

em-http-request github docs:

EventMachine.run {
  http1 = EventMachine::HttpRequest.new('http://google.com/').get
  http2 = EventMachine::HttpRequest.new('http://yahoo.com/').get

  http1.callback { }
  http2.callback { } 
end

,

callbacks = []
Websites.all.each do |website|
    callbacks << EventMachine::HttpRequest.new(website.url).get
end

callbacks.each do |http|
    http.callback { }
end

rails -, EventMachine:

bundle exec rails server thin

eventmachine em-http-request. !

+2

More articles:


All Articles