Geek Answers Handbook

Python: httplib.CannotSendRequest when nesting SimpleXMLRPCServers streams

I occasionally get an httplib.CannotSendRequest exception when using the SimpleXMLRPCServers chain that use SocketServer.ThreadingMixin.

What I mean by "chaining" is this:

I have a client script that uses xmlrpclib to call a function on SimpleXMLRPCServer. This server, in turn, calls another SimpleXMLRPCServer. I understand how that sounds, but there are good reasons why this architecture was chosen, and I see no reason why this should not be possible.

(testclient)client_script ---calls--> (middleserver)SimpleXMLRPCServer ---calls---> (finalserver)SimpleXMLRPCServer --- does something
  • If I do not use SocketServer.ThreadingMixin, this problem does not occur (but I need the requests to be multithreaded, so this does not help.)
  • If I have only one level of services (i.e. only a script client that calls the destination server directly), this does not happen.

I was able to reproduce the problem in a simple test code below. There are three fragments:

finalserver:

 import SocketServer import time from SimpleXMLRPCServer import SimpleXMLRPCServer from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass # Create server server = AsyncXMLRPCServer(('', 9999), SimpleXMLRPCRequestHandler) server.register_introspection_functions() def waste_time(): time.sleep(10) return True server.register_function(waste_time, 'waste_time') server.serve_forever()

middleserver:

 import SocketServer from SimpleXMLRPCServer import SimpleXMLRPCServer from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler import xmlrpclib class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass # Create server server = AsyncXMLRPCServer(('', 8888), SimpleXMLRPCRequestHandler) server.register_introspection_functions() s = xmlrpclib.ServerProxy('http://localhost:9999') def call_waste(): s.waste_time() return True server.register_function(call_waste, 'call_waste') server.serve_forever()

testclient:

 import xmlrpclib s = xmlrpclib.ServerProxy('http://localhost:8888') print s.call_waste()

To reproduce, use the following steps:

  • Run python finalserver.py
  • Run python middleserver.py
  • Run python testclient.py
  • While (3) is still running, run another python instance testclient.py

Quite often (almost every time) you will get the error below the first attempt to perform step 4. Interestingly, if you try again to perform step (4) again, the error will not occur.

 Traceback (most recent call last): File "testclient.py", line 6, in <module> print s.call_waste() File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request return self.parse_response(response) File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response return u.close() File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close raise Fault(**self._stack[0]) xmlrpclib.Fault: <Fault 1: "<class 'httplib.CannotSendRequest'>:">

The Internet seems to be saying that this exception can be caused by multiple httplib.HTTPConnection.request calls without intermediate getresponse calls. However, the Internet does not discuss this in the context of SimpleXMLRPCServer. Any pointers towards resolving the httplib.CannotSendRequest problem will be appreciated.

==================================================== =========================================== ANSWER:

OK, I'm a little stupid. I think I was looking at the code for a lengthy period of time when I missed the obvious solution, looking at me in the face (quite literally, because the answer is actually in the question itself.)

Basically, CannotSendRequest occurs when httplib.HTTPConnection is interrupted by an intermediate request operation. Each httplib.HTTPConnection.request must be connected to a .getresponse () call. If this connection is interrupted by another request operation, the second request will result in a CannotSendRequest error. So:

 connection = httplib.HTTPConnection(...) connection.request(...) connection.request(...)

will fail because you have two requests for the same connection before calling getresponse.

Linking this to my question:

  • the only place in the three programs where such connections are made is serverproxy calls.
  • the problem only occurs during streaming, so this is probably a race condition.
  • The only place that uses server proxy sharing is in the middleserver.py file

The solution then obviously is to have each thread create its own server server. The fixed version of middleserver is lower and it works:

 import SocketServer from SimpleXMLRPCServer import SimpleXMLRPCServer from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler import xmlrpclib class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass # Create server server = AsyncXMLRPCServer(('', 8888), SimpleXMLRPCRequestHandler) server.register_introspection_functions() def call_waste(): # Each call to this function creates its own serverproxy. # If this function is called by concurrent threads, each thread # will safely have its own serverproxy. s = xmlrpclib.ServerProxy('http://localhost:9999') s.waste_time() return True server.register_function(call_waste, 'call_waste') server.serve_forever()

Since this version causes each thread to have its own xmlrpclib.serverproxy, there is no risk of the same serverproxy instance calling HTTPConnection.request more than once in a row. Programs work as intended.

Excuse for troubling.

+7
source share
1 answer

OK, I'm a little stupid. I think I was looking at the code to delay the period of time when I missed the obvious solution, looking at me in the face (quite literally, because the answer is actually in the question itself.)

Basically, CannotSendRequest occurs when httplib.HTTPConnection is interrupted by an intermediate request operation. Basically, every httplib.HTTPConnection.request should be connected to a .getresponse () call. If this connection is interrupted by another request operation, the second request will result in a CannotSendRequest error. So:

 connection = httplib.HTTPConnection(...) connection.request(...) connection.request(...)

will fail because you have two requests for the same connection before calling getresponse.

Linking this to my question:

  • the only place in the three programs where such connections are made is serverproxy calls.
  • the problem only occurs during streaming, so this is probably a race condition.
  • The only place that uses server proxy sharing is in the middleserver.py file

The solution then obviously is to have each thread create its own server server. The fixed version of middleserver is lower and it works:

 import SocketServer from SimpleXMLRPCServer import SimpleXMLRPCServer from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler import xmlrpclib class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass # Create server server = AsyncXMLRPCServer(('', 8888), SimpleXMLRPCRequestHandler) server.register_introspection_functions() def call_waste(): # Each call to this function creates its own serverproxy. # If this function is called by concurrent threads, each thread # will safely have its own serverproxy. s = xmlrpclib.ServerProxy('http://localhost:9999') s.waste_time() return True server.register_function(call_waste, 'call_waste') server.serve_forever()

Since this version leads to the fact that each thread has its own xmlrpclib.serverproxy, there is no risk that the server proxy calls HTTPConnection.request more than once in a row. Programs work as intended.

Excuse for troubling.

+11
source

More articles:


All Articles