Geek Answers Handbook

Working with API restrictions?

I have an application configured to make scheduled calls to several APIs once a day. This works very well, but I know that some of the APIs I call (like Twitter) have a speed limit. Since the number of calls I make is constantly growing, can anyone recommend a way to throttle my calls so that I can send packets at x per hour / minute, etc.?

I found Gutton Ratelimit , does anyone use this and is it good? Are there others I should look at?

+7
source share
4 answers

If you use any background worker to make API calls, you can transfer the task, which will be performed in the next time interval, when the speed limits were reset.

class TwitterWorker include Sidekiq::Worker def perform(status_id) status = Twitter.status(status_id) # ... rescue Twitter::Error::TooManyRequests # Reschedule the query to be performed in the next time slot TwitterWorker.perform_in(15.minutes, status_id) end end

There is no scientific solution, although, for example, there is a risk that the request may be rescheduled each time if you try to make much more API calls per day than the speed limit allows. But until then, something can easily do the trick!

+4
source

Another solution is to buy proxies that allow you to send requests with different IP addresses.

Use standard http lib http://ruby-doc.org/stdlib-2.0/libdoc/net/http/rdoc/Net/HTTP.html#method-c-Proxy

I'm not sure that you will not be blocked, but it may be worth a try. A randomly selected IP should increase your limits.

0
source

If you do not make concurrent requests, there are not many.

  • Indicate how many delays you need per request
  • Check the time before the request, subtract from the time after the request and sleep rest.

With simultaneous requests you can be more accurate, I once wrote about it here

0
source

I know this is an old question, but wanted to mention something if it helps others with the same question.

If the work can be queued for tasks using resque, then you can use the just released gem, which pauses the queue when hit in rate_limit, and pauses it after a while.

https://github.com/pavoni/resque-rate_limited_queue

0
source

More articles:


All Articles