Non-Blocking TCP Buffer Issues

Question

I think I have a problem. I have two TCP applications connected to each other that use Winsock I / O completion ports to send / receive data (non-blocking sockets).

Everything works fine until a data packet occurs. The sender sends invalid / invalid data.

I allocate buffers that I push onto the stack, and if I understand correctly, it is wrong to do this, because these buffers should remain when I sent them, until I receive a “write completely” notification from IOCP.

Take this for example:

void some_function() { char cBuff[1024]; // filling cBuff with some data WSASend(...); // sending cBuff, non-blocking mode // filling cBuff with other data WSASend(...); // again, sending cBuff // ..... and so forth! } 

If I understand correctly, each of these WSASend () calls must have its own unique buffer, and this buffer can only be reused after sending is complete.
Right?

Now, what strategies can be implemented to support a large package of such buffers, how should I process them, how can I avoid a performance penalty, etc.?

And if I use buffers, this means that I have to copy the data that will be sent from the source buffer to the temporary one, so I would set SO_SNDBUF for each socket to zero, so the system will not copy what I already copied. Are you with me? Please let me know if I do not understand.

Answer 1

Take a look at boost::asio . Asynchronous IO is his specialty (as the name suggests). This is a fairly mature library, which is now located in Boost starting at 1.35. Many people use it in production for very intensive networking. There is a wealth of examples in the documentation.

One thing is certain - take buffers very seriously.

Edit:

The basic idea behind processing queue input spikes.

• Create, say, three linked lists of pre-allocated buffers - one for free buffers, one for the data to be processed (received), one for the data to be sent.
• Each time you need to send something, remove the buffer from the free list (select a new one if the free list is empty), fill in the data, put them in the list to be sent.
• Every time you need to get something - take a buffer from the free list, as described above, give it the procedure for receiving IO.
• Periodically select buffers from the queue, send them to send the procedure.
• Upon completion of sending (built-in or asynchronous) - return them to the free list.
• Upon completion of reception - put the buffer in the list to be processed.
• Ask your business routine to pick up buffers in the list to be processed.

Then the packages will fill this input queue until you can process them. You might want to limit the size of the queue to avoid purging all the memory.

Answer 2

I do not think it is a good idea to make a second dispatch before the first transfer is completed.

Likewise, I do not think it is a good idea to change the buffer before sending is complete.

I would be inclined to store data in some kind of queue. One thread may continue to add data to the queue. The second thread can run in a loop. Make the package and wait for it to complete. If there is more data, send another one, otherwise wait for more data.

You will need a critical section (or some of them) in order to share the queue between the streams well and, possibly, the event or semaphore for the sending stream, in order to wait if there is no ready data.

Answer 3

Now, what strategies can be implemented to support a large package of such buffers, how should I process them, how can I avoid a performance penalty, etc. ??

It's hard to know the answer without knowing more about your particular design. In general, I would not support my own "buffer bag" and instead used the OS built into the buffer bag - a bunch.

But anyway, what I would do in the general case is an interface for callers of your code that reflect what WSASend does for overlapping i / o. For example, suppose you provide an interface for sending a specific structure:

 struct Foo { int x; int y; }; // foo will be consumed by SendFoo, and deallocated, don't use it after this call void SendFoo(Foo* foo); 

I would require SendFoo users to allocate a Foo instance with a new one and tell them that after calling SendFoo, the memory no longer “belongs” to its code, and therefore they should not use it.

You can enhance this even further with a little trickery:

 // After this operation the resultant foo ptr will no longer point to // memory passed to SendFoo void SendFoo(Foo*& foo); 

This allows the SendFoo body to send the memory address to WSASend, but change the pointer passed to NULL, separating the connection between the calling code and their memory. Of course, you cannot know exactly what the caller is doing with this address; they may have a copy elsewhere.

This interface also provides that with each WSASend one block of memory will be used. You are really stepping into more than dangerous territory, trying to split one buffer between two WSASend calls.