NOTE. . This question has been ported to serverfault.com .
I would like to help find the reason for the decrease in data transfer speed in my application.
I have 12 embedded systems and a Linux server. Embedded systems send data to the server via TCP via Ethernet through the switch. The following is a TCP StreamGraph made from capturing Wireshark traffic from a single board.
As you can see, data transfer occurs at approximately 5.8 Mbps to approximately 0.25 seconds. It is as fast as I can expect the embedded system to work. After that, delays are inserted into the transfer. Below is a close-up of the chart:
The linear ladder curve below marked ACK shows how much data was ACKed by the server at any given time. The corresponding curve, designated as RWIN, shows how much buffer space there will be on datapc. The smaller vertical segments indicated by SENT DATA represent the actual packets sent.
At point A, the server stores data as fast as it is sent, but then the server does not send any messages for 23 ms. The embedded system is allowed to send to RWIN without waiting for the ACK, but it does not, because it must support the transmitted data until it is canceled (in case of retransmission), and the send buffer space is limited.
Then, at point B, all received ACKed data at once, and normal verification and sending resume for 2.5 ms before another pause occurs.
Wireshark was captured from another PC that was connected to the switch port, which was configured to mirror all data sent and received to the port to which the embedded system was connected.
The Linux server launches a Java application that processes the data and stores it on disk. It shows no signs of CPU overrun. Operating system - Ubuntu Server 12.04 with default network settings.
I see that I could probably benefit from allocating more send buffer space in the embedded system to match the amount of space to receive on the Linux server, but this is not a limiting factor here.
My questions:
- What could be the reason that the Linux server suspends the ACK, even if it is obviously able to get everything as good as possible?
- How can I debug this?