Identify asymmetric network delays

Question

Imagine that you have many clustered servers on many hosts in a heterogeneous network environment, so the connections between the servers can have completely different delays and bandwidth. You want to build a map of connections between servers by transferring data between them.

Of course, this map may become obsolete over time as the network topology changes, but now ignore these difficulties and assume that the network is relatively static.

Given the delays between nodes in this host graph, bandwidth calculation is a relative simple time exercise. However, I have more problems with delays. To get round-trip time, there is a simple time to check the return path from the local host to the remote host - both synchronization events (start, stop) occur on the local host.

What if I want one-way time under the assumption that latency is not equal in both directions? Assuming the clocks on different nodes are not exactly synchronized (at least their error is the same magnitude as the delays associated with it) - how can I calculate a one-way delay?

In a related question, is asymmetric latency (where is the connection faster in direction than the other), common in practice? For what reasons / hardware configurations? Of course, I know about asymmetric bandwidth scenarios, especially on last-mile consumer lines such as DSL and Cable, but I'm not sure about the delay.

Added by:. After reviewing the comment below, the second part of the question is probably better on serverfault .

Answer 1

As far as I know, asymmetric delays - especially the asymmetry of the "last mile" - cannot be automatically determined, because any network protocol for time synchronization is equally affected by the same asymmetry, so you don’t have a reference point to evaluate asymmetry.

If each endpoint had, for example, its own GPS watch, then you will have a reference point for work.

In Rapid Measurement of LogP Parameters for Messaging Platforms , the authors note that delay measurement requires clock synchronization external to the measured system. (Highlighting in bold is shown in italics in the source text.)

Asymmetric latency can only be measured by sending a message with a time stamp of t _s and allowing the recipient to receive a delay from t _r - t _s , where t _r is the reception time. This requires clock synchronization between the sender and the recipient. Without external clock synchronization (for example, using GPS receivers or specialized software such as network time protocol , NTP), the clock can only be synchronized to the granularity of roundtrip time between two hosts [10], which is useless for measuring network latency.

None of the network algorithms (for example, NTP) will eliminate the problems of communication with the last mile, since each entrance to the algorithm itself will evenly obey the performance characteristics of the last mile and, therefore, is not "external" in the above sense. (I am sure that you can build evidence, but I don’t have time to build it right now.)

Answer 2

To solve this problem, there is a project called One-Way Ping (OWAMP). Activity can be seen in LKML to add high-resolution timestamps to incoming packets ( SO_TIMESTAMP , SO_TIMESTAMPNS , etc.) to help compute these statistics.

http://www.internet2.edu/performance/owamp/

There is even a Java version:

http://www.av.it.pt/jowamp/

Please note that packet timing really needs hardware support, and many modern next-generation network cards offer only millisecond resolution, which may not be synchronized with the host clock. The DDK has MSDN articles on host clock and NIC clock synchronization that demonstrate potential problems. TSC timestamps in nanoseconds are problematic due to major differences and may require the Nehalem architecture to work properly at the required resolutions.

http://msdn.microsoft.com/en-us/library/ff552492(v=VS.85).aspx

Answer 3

You can measure the asymmetric delay in a link by sending packets of different sizes to a port that returns a packet of a fixed size, for example, send some udp packets to a port that responds with an icmp error message. The icmp error message is always the same size, but you can adjust the size of the udp packet to send.

see http://www.cs.columbia.edu/techreports/cucs-009-99.pdf

Answer 4

In the absence of a synchronized clock, asymmetry cannot be measured, as proven in a 2011 article, "Fundamental limits on synchronizing clocks over networks".

https://www.researchgate.net/publication/224183858_Fundamental_Limits_on_Synchronizing_Clocks_Over_Networks