TCP keep-alive connects after TCP null window and mistakenly closes the connection

Question

We observe how this pattern is often found between two RHEL 6 units that transmit data over a TCP connection. The client issues TCP Window Full, after 0.2 seconds the client sends TCP Keep-Alives, to which the server responds with answers that have the correct form. However, the client is not satisfied with this and continues to send TCP Keep-Alives until it finally closes the connection to RST after almost 9 seconds.

This is despite the fact that RHEL blocks have a TCP Keep-Alive configuration by default:

net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_intvl = 75 

... which states that this should happen only before 2 hours of silence. Am I reading PCAP incorrectly (corresponding packages are available on request)?

Below is a screenshot of the Wireshark template with my own notes in the middle.

Answer 1

In fact, these keep-alive packets are not used to support TCP! They are used to detect window size updates.

Wireshark treats them as keep-alive packets only because these packets look like keep-alive packets.

A TCP keepalive packet is simply an ACK with a sequence number set one less than the current sequence number for the connection.

(We assume that ip 10.120.67.113 refers to host A, 10.120.67.132 refers to host B.) In packet No. 244911 A acks seq 24507484. In the next packet (No. 2491212) B send seq 24507483 (24507484-1)).

Why are there so many keep-alive packages, what are they used for?

A sends data to B, and B responds with a zero-window size to inform A that it is temporarily no longer able to receive data. To ensure that A knows when B can receive data again, A sends a keep-alive packet again and again B with a persistence timer , B responds to A with its window size information (In our case, window size B was always zero )

And normal TCP exponential rollback is used when calculating a constant timer. So, we can see that A sends its first “keep-alive” packet after 0.2 s, sends its second packet after 0.4 s, the third goes after 0.8, the intermediate goes after 1.6 s ...

This phenomenon is related to TCP flow control .

Answer 2

The source and destination IP addresses in packets coming from the client do not match the destination and source IP addresses in the response packets, which indicates that NAT is running between the boxes. It is also important to understand where the packets were captured. It is likely that capturing the packet on the client itself will help to understand the problem.

Note that a client can generate a TCP keepalive if it does not receive a data packet for two hours or more. In accordance with RFC 1122, the client repeats the keepalive if it does not receive a response from the partner. It eventually shuts down after a continuous reuse failure.

NAT devices typically implement connection caching to maintain the state of current connections. If the connection size reaches the limit, NAT devices drop old connections to serve new connections. It can also lead to such a scenario.

This packet capture indicates that there is a high probability that the packets do not reach the client, so it will be useful to capture packets on the client machine.

Answer 3

I read the trace a little differently: The sender sends more data than the receiver can process, and receives a zero-delay response. The sender sends window probes (not keepalives, this will be suitable for this soon), and the application refuses after 10 seconds without any changes and closes the connection, reset indicates that data is being sent in the TCP buffer. If an application uses a large block entry in a socket, it may not have seen progress for more than 10 seconds observed in tcpdump.

If this is a direct connection (without proxies, etc.), the most likely reason is that receiving a receiving stop (or slower than the sender and sending data)

Answer 4

It seems to me that package number 249522 caused the application to 10.120.67.113 to abort the connection. All window probes receive a zero-window response from .132 (no payload), and then .132 sends a (unsolicited) packet 249522 with 63 bytes (and still shows 0 window). The PSH flag assumes that this is 63 bytes - this is all the data written by the application on .132. Then .113 responds with RST in the same millisecond. I can’t think of why the TCP stack sent RST right after receiving the data (serial numbers are correct). In my opinion, it is almost certain that the .113 application decided to abandon the .132 sent based on the 63-byte message.