Delayed HTTP Avi Call Throttling

Question

I am trying to implement some chokes in our REST API. A typical approach, after a certain threshold, block the request (with a response of 403 or 429 ). However, I saw one api that adds a delay to the response .

As you call the API, we will look at your average calls per second (s / s) over the previous five-minute period. Here is what will happen:

• in 3c / s and add a 2 second delay

• for 5c / s and add a 4 second delay

• for 7c / s and add a 5 second delay

From the client’s point of view, I see that this is better than returning an error. The worst thing that can happen is that you will slow down.

I am wondering how this can be achieved without adversely affecting the application server . To add these delays, the server needs to leave the request open, as a result of which it will delay more and more request processors, which means that it has less ability to enter new requests.

What is the best way to accomplish this? (i.e. is this something that can be done on the web server / load balancer so that the application server is not adversely affected? a throttling layer that can be added for this purpose?)

We use Django / Tastypie, but the question is more about the architecture / conceptual level.

Answer 1

If you use a synchronous application server, which is the most common installation for Django applications (for example, the machine gunner by default --worker-class sync ), then adding such a delay in the application will really have a very bad impact on performance. The worker processing the delayed request will be blocked for the delay period.

But you can use an asynchronous application server (e.g. gunicorn with '- worker class gevent`), and then the overhead should be negligible. A worker who processes delayed requests can process other requests during the delay.

Doing this in a reverse proxy might be the best option as it allows you to easily and flexibly configure the policy. There is an external nginx module for such a thing .