Geek Answers Handbook

Cancel HTTP Request

I am migrating one of my projects from C # and it is difficult for me to solve the multithreading problem in Python. The problem is with the long-lived HTTP request that is expected (the request will respond when a specific event occurs on the server). Here is the summary:

I am sending a request using urllib2 in a separate thread. When the request returns or expires, the main thread is notified. It works great. However, there are times when I need to cancel this outstanding request and switch to a different URL. There are four solutions that I can consider:

  • Cancel an outstanding request. C # has WebRequest.Abort() , which I can call cross-thread to abort the request. Python urllib2.Request seems to be a pure data class, in this case only store information about the request; responses are not related to Request objects. Therefore, I cannot do this.
  • Interrupt the stream. C # has Thread.Interrupt() that will raise a ThreadInterruptedException in the thread if it is in the idle state, or the next time it enters that state. (Waiting for a monitor and file / socket input / output are both standby states). Python does not seem to have anything comparable; There seems to be no way to wake up a thread that blocks on I / O.
  • Set a low timeout in the request. In the timeout, check the “aborted” flag. If this is not correct, restart the request.
  • As in the case of option 3, add the “interrupted” flag to the state object, so that when the request finally ends one way or another, the thread knows that the answer is no longer needed and just turns off.

Options 3 and 4 are apparently the only ones supported by Python, but option 3 is a terrible solution, and 4 will support an open connection that I don't need. I hope to become a good networker and close this connection when I no longer need it. Is there a way to actually cancel an outstanding request anyway?

+4
source share
4 answers

Consider using gevent . Gevent uses non-black collaborative units called green. Greenlets can "block" on IO, which actually means "go to sleep until the IO is ready." You may have a green requester, which owns the socket, and the main green, which decides when to interrupt. When you want to interrupt and switch URLs, the main greens will kill the requester greens. The initiator requests the received exception, closes the socket / urllib2 request, and starts operation.

Edited to add: Gevent is not compatible with threads, so be careful with this. You will either have to use gevent all the way, or threads all the way. However, the threads in python seem to be lame due to the GIL .

+2
source

Like Spike Gronim, answer, but even harder.

Consider rewriting this in a twisted form. You probably want to subclass twisted.web.http.HTTPClient , in particular by implementing handleResponsePart to interact with the client (or handleResponseEnd if you do not need to see it before the end of the response). To close the connection earlier, you simply call the loseConnection method in the client protocol.

+1
source

Maybe this "killable thread" snippet can be useful if you have no other choice. But I would be of the same opinion as Spike Gronim and recommend using gevent .

0
source

I found this question using Google, and used Spike Gronim's answer to come up with:

 from gevent import monkey monkey.patch_all() import gevent import requests def post(*args, **kwargs): if 'stop_event' in kwargs: stop_event = kwargs['stop_event'] del kwargs['stop_event'] else: stop_event = None req = gevent.spawn(requests.post, *args, **kwargs) while req.value is None: req.join(timeout=0.1) if stop_event and stop_event.is_set(): req.kill() break return req.value

I thought it could be useful for other people.

It works the same as a regular request.post, but accepts the optional keyword argument 'stop_event'. This is threading.Event. The request will terminate if stop_event is set.

Use with caution, because if it does not wait for a connection or connection, it can block the GIL (as mentioned). It (gevent) seems compatible with streams these days (via a monkey patch).

0
source

More articles:


All Articles