Some OVERLAPS using WSASend are not returned in a timely manner using GetQueuedCompletionStatus?

Question

Background: I use CreateIoCompletionPort, WSASend / Recv and GetQueuedCompletionStatus to override the io socket on my server. For flow control, when you send to a client, I only allow a few WSASend () when all pending OVERLAPs have exited IOCP.

Problem. Recently, there have been cases when OVERLAP does not return to IOCP. The thread calling GetQueuedCompletionStatus does not receive them, and they remain in my local pending queue. I checked that the client is receiving data from the socket and the socket is connected. When WSASend () was called, no errors were returned. OVERLAP simply never returns without an external stimulus, for example:

• Disconnecting a socket from a client or server immediately allows the GetQueuedCompletionStatus stream to retrieve OVERLAPs
• When making additional WSASend () calls, it sometimes takes several before all OVERLAPs suddenly exit the queue.

Question: Has anyone seen this type of behavior? Any ideas on what causes this?

Thanks Jeffrey

Answer 1

WSASend() may not complete in a timely manner if the TCP window is full. In this case, the stack cannot send more data, so your WSASend() waiting and your completion will not happen until the TCP stack can send more data.

If you have a protocol between your client and a server that does not have flow control built into the protocol itself. And you yourself do not perform write-based flow control, and just send the data as fast as your server can send, then you can get to the point where either the network or your client cannot keep up, and TCP flow control starts (when the TCP window fills up). If you continue to simply disconnect data asynchronously with additional calls to WSASend() , then in the end you will go your way through all the unloaded memory on the machine and at that moment all bets will be disabled (the probability that the driver may lead to a blue screen).

So, in the end, improvements from overlapping sockets can and sometimes can take longer than you can expect. In your example, I expect the terminations you get when you close the socket are all the failures?

I talk about this a little more on my blog; here: http://www.lenholgate.com/blog/2008/07/write-completion-flow-control.html and here: http://www.serverframework.com/asynchronousevents/2011/06/tcp-flow- control-and-asynchronous-writes.html