Consecutive consecutive multiple deferment in twisted

Question

Currently, I'm still new to the whirlwind, and it annoyed me.

I am sending a sequence of commands over TCP and expect a response from the lineRecieved reader, which may take a few seconds to process and arrive, so I wrapped it in pending. The first pending job is fine, but the second one starts because the first one is still being processed, which leads to garbage, because the endpoint can only process one command at a time. Expected behavior on an asysc system, but not what I need. If I have one or two commands, I could use the deferedChain handler for processing, but since I have potentially dozens of commands to run sequentially, I am afraid that this will quickly turn into irreplaceable spaghetti.

What is the clean way to do this?

Thank you very much

Code example

def connectionMade(self): self.fire_def('command1') print'fire command 2' self.fire_def('command2')#Fires when command one is running def fire_def(self,request): d = self.getInfo(request) d.addCallback(self.print_result) return d def print_result(result): print result def getInfo(self,request): print 'sending', request self.d = defer.Deferred() self.sendLine(request) return self.d def lineReceived(self, line): line = line.strip() self.buffer.append(line) if self.d is None: return if 'result_I_want' in self.buffer: print 'Firing Callback' self.d.callback(self.buffer) 

Answer 1

The code in your question only knows how to track one Deferred . If the application code calls getInfo twice without enough time to complete the first action with the result, it will violate its own internal tracking state:

 def getInfo(self,request): print 'sending', request self.d = defer.Deferred() self.sendLine(request) return self.d d_foo = getInfo(foo) d_bar = getInfo(bar) 

In this sequence, d_foo and d_bar are different instances of Deferred . However, the second time getInfo called getInfo value of the self.d attribute changes from d_foo to d_bar . d_foo Deferred is lost. Later, when `lineReceived starts:

 def lineReceived(self, line): line = line.strip() self.buffer.append(line) if self.d is None: return if 'result_I_want' in self.buffer: print 'Firing Callback' self.d.callback(self.buffer) 

self.d d_bar , although the string is probably the answer to the foo request. This means that d_bar will receive a response for foo and d_foo will never receive any response at all.

To fix this problem, it can help keep the list (or queue) of Deferred instances in the protocol. Add to it when a new request for information is created, exit it when a response is received. (I’m not sure which protocol you are implementing, so I don’t know how you decide how many lines are enough to make up the answer. If the protocol does not determine this, then it is broken and you may want to switch to a better protocol.)

If you fix this, the answers will at least be delivered to different Deferred instances.

You also described a problem with a forced sequential operation. There are several ways to interpret this. One way is to interpret it as meaning that you want one request to be “outstanding” on the network at a time. In other words, you do not want getInfo send new request lines until lineReceived response data to Deferred returned by the previous call to getInfo .

In this case, a delayed chain is just a thing. Despite the fact that you have N “Delays” when you introduce this sequential restriction, you actually have a series of 2 “Delays”. You have a Delay, which is executed earlier, and a Delay, which should be launched only after the previous one has its result. You expand this to N, and then consider that later Deferred will be an earlier Deferred in the new pair, and the third Deferred will become new later than Deferred.

Or in another way, if you have D1, D2, D3 and D4, you link them like this:

 D2 is chained to D1 and only runs when D1 is complete D3 is chained to D2 and only runs when D2 is complete D4 is chained to D3 and only runs when D3 is complete 

However, while this may work, it is actually not the easiest way to implement serialization. Instead, I suggest that you explicitly start the work in getInfo and explicitly disable it in lineReceived :.

 def _sendRequest(self, d, request): print 'sending', request self.d = d self.sendLine(request) def getInfo(self,request): if self.d is None: d = defer.Deferred() self._sendRequest(d, request) return d else: queued_d = defer.Deferred() self._requests.append((request, queued_d)) return queued_d def lineReceived(self, line): line = line.strip() self.buffer.append(line) if self.d is None: return if 'result_I_want' in self.buffer: print 'Firing Callback' now_d = self.d self.d = None buffer = self.buffer self.buffer = [] if self._requests: request, queued_d = self._requests.pop(0) self._sendRequest(queued_d, request) now_d.callback(buffer) 

Note that in lineReceived code will make sure everything is in a consistent state before the line now_d.callback(buffer) . This is a subtle but important point. There may be callbacks on now_d that affect the protocol — for example, by calling getInfo again. It is important that the protocol is in a consistent state before running this code, otherwise it will get confused - perhaps by sending requests out of order or by requesting new requests when they are actually sent. This is an example of securing your code from re-entry. This is not an idea that is unique to Twisted-enabled programs, but because people most often associate the idea of ​​reinstalling with a multi-threaded program, people often miss the opportunity when writing Twisted-based code.

Answer 2

Basically, you return those pending from each other if you want them to be executed one by one.

So you want d2 to be executed only after d1 is executed, fine, return d2 from the d1 callback.

In other words, in your example, you will need to call command2 somewhere near the end of the callback to command1.