How to make concurrent HTTP requests in Heroku?

Question

I am creating a Ruby on Rails application that accesses 6-7 APIs, captures information from them based on user input, compares and displays results for users (information is not stored in the database). I will use Heroku to deploy the application. I would like these HTTP requests to access the APIs in parallel so that the response time is better and not executed sequentially. What do you think is the best way to achieve this in Geroku?

Thanks so much for any suggestions!

Answer 1

If you want to really fulfill server side requests (tfe javascript solution is a good idea), your best bet would be to use EventMachine . Using EventMachine provides an easy way to do non-blocking IO.

Also check out the EM-Synchrony driver suite with Ruby 1.9 fiber support (including HTTP).

All you need to do for a non-blocking HTTP request is something like:

require "em-synchrony" require "em-synchrony/em-http" EM.synchrony do concurrency = 2 urls = ['http://url.1.com', 'http://url2.com'] # iterator will execute async blocks until completion, .each, .inject also work! results = EM::Synchrony::Iterator.new(urls, concurrency).map do |url, iter| # fire async requests, on completion advance the iterator http = EventMachine::HttpRequest.new(url).aget http.callback { iter.return(http) } http.errback { iter.return(http) } end p results # all completed requests EventMachine.stop end 

Good luck!

Answer 2

You can always make client-side requests using Javascript. Then you can not only run them in parallel, but you don’t even need to travel around the world to your own server.

Answer 3

I have not tried to parallelize such requests. But I tried parallel to the hero, it works like a charm! This is my simple blog post about this.

http://olemortenamundsen.wordpress.com/2010/10/17/spawning-multiple-threads-at-heroku-using-parallel/

Answer 4

Look at creating each request as a background job: http://blog.heroku.com/archives/2009/7/15/background_jobs_with_dj_on_heroku/

The more Workers you buy from Heroku, the more background jobs you can handle at the same time, leaving Dynos to serve your users.