Why does my Linux prio tc not improve network latency?

Question

I am building a Linux embedded real-time application that has a lot of network traffic. Of the set of traffic, two connections are time critical. One is the input, and the other is the output. My application requires that this traffic take precedence over other, time-independent traffic.

Two things cost me:

• Minimize the number of remote packets due to congestion on these two connections.
• Minimize the delay through the device (input to output) on these two connectors.

I came (somewhat!) To speed up Linux traffic management, and I understand that it applies primarily to outgoing traffic, since the remote device is responsible for the priority of the data transferred to it. I set up my application as a real-time process and worked on problems related to what priority to run it.

Now I'm starting tc setup. For my test case, here is what I use:

tc qdisc add dev eth0 root handle 1: prio bands 3 priomap 2 2 2 2 2 2 2 0 2 2 2 2 2 2 2 2 tc qdisc add dev eth0 parent 1:1 handle 10: pfifo tc qdisc add dev eth0 parent 1:2 handle 20: pfifo tc qdisc add dev eth0 parent 1:3 handle 30: pfifo 

Basically I say: send all priority 7 traffic to range 0 and all other traffic through lane 2. As soon as I have this simple test work, I will do a better job with other traffic.

First let me check my expectations: I expect that any traffic that has priority 7 should always go outside before the traffic has any other priority. This should lead to the fact that the delay on such traffic will be relatively not affected by other traffic on the box, no? My mtu is set to 1500, and through the interface I get about 10 MB / s. The maximum additional latency in range 0 caused by band 2 traffic is one packet (<= 1500 bytes) or 150 us (1500 bytes / 10 MB / s = 150 us).

Here is my test setup:

Two Linux boxes. Box 1, which runs a TCP server that uses echos input. Box 2 connects to the first, sends packets over TCP and measures the delay (time is sent for the received time).

I use the same tc setting for Linux boxes.

In applications (both on the server and on the client), I set SO_PRIORITY to the socket as follows:

 int so_priority = 7; setsockopt(m_socket.native(), SOL_SOCKET, SO_PRIORITY, &so_priority, sizeof(so_priority)); 

I use tc to verify that my traffic is in range 0, and all other traffic is in range 2:

 tc -s qdisc ls dev eth0 

Here rub: When there is no other traffic, I see delays in the range of 500 us. When I have other traffic (for example, an scp job copying a 100 MB file), latencies jump to 10+ ms. What is really strange is that NONE OF THE WORK I DO NOT ACCEPT. In fact, if I change lanes (so that all my traffic goes through lower priority 2, and the other traffic passes through lane 1), I don’t see the difference in delay.

What I expected is that when there is other traffic on the network, I would see an increase in latency of about 150, not 10 ms! By the way, I checked that loading a field with other processes with priority not in real time does not affect the delay, as well as traffic on other interfaces.

Another point of note: if I drop mtu to 500 bytes, the latency decreases to about 5 ms. Nevertheless, this is an order of magnitude worse than the unloaded case. Also - why does changing mtu affect it so much, but using tc to set the priority queue has no effect ???

Why doesn't tc help me? What am I missing?

Thanks!

Eric

Answer 1

You did not say anything about the rest of your network, but I assume that you are queued up on the router, which usually has long queues to optimize bandwidth. The best way to fix this is to transfer the priority queue to the shaper with bandwidth just below your upstream bandwidth. This way, your packets with high priority will be queued inside your window, and not on the external router, allowing your high priority packets to jump into the queue before the queue, as you expect.

Answer 2

The prio object will simply send the highest priority packet available at the time the packets were sent (usually, as soon as the previous packet was sent if there are no packets waiting to be released).

Your test is based on the fact that the packets were queued by the corresponding software processes on each computer, and the received packets were received from the port on each computer.

Any scheduling delays that affect the time that a process receives on any computer can affect the ability of a process to place a message in a queue or to retrieve and process a message from a queue. It looks like you have loaded at least one of the machines to verify this, but my experience is that loading the machine will definitely affect the measured delay like this (not microseconds in milliseconds), so it might be worth repeating this with loading both machines with priority tasks.

Another thing to check is the timestamp you use to measure the delay — the time the echo message is actually received on the client machine or the time that your program processes it. If the latter, then you measure not only the network latency, but also the time between the received message and your program, getting a piece of the processor and reaching the point where you check the time - see http://wiki.wireshark.org/Timestamps .

As an aside, I do not think that you can get a guaranteed second-level micro-session reaction without real time, as a mechanism. On the other hand, if your application is VoIP, then you will usually be fine until a delay of 200 milliseconds.

Answer 3

Have you tried to capture packets and check if the TOS value of the IP header has been changed?

to use SO_PRIORITY you will need linux 2.6.39 or later.

you should change IP_TOS instead.

you must install:

 int iptos_precedence = 0xc0; if (setsockopt(sock_fd, IPPROTO_IP, IP_TOS, &iptos_precedence, sizeof(iptos_precedence)) < 0) { //print errno (or something like that) }