The metrics you are looking for exist in /proc/schedstat .
The format of this file is described in sched-stats.txt in the kernel source code. In particular, you need the cpu<N> lines:
CPU statistics -------------- cpu<N> 1 2 3 4 5 6 7 8 9 First field is a sched_yield() statistic: 1) # of times sched_yield() was called Next three are schedule() statistics: 2) This field is a legacy array expiration count field used in the O(1) scheduler. We kept it for ABI compatibility, but it is always set to zero. 3) # of times schedule() was called 4) # of times schedule() left the processor idle Next two are try_to_wake_up() statistics: 5) # of times try_to_wake_up() was called 6) # of times try_to_wake_up() was called to wake up the local cpu Next three are statistics describing scheduling latency: 7) sum of all time spent running by tasks on this processor (in jiffies) 8) sum of all time spent waiting to run by tasks on this processor (in jiffies) 9) # of timeslices run on this cpu
In particular, field 8. To find the length of the execution queue, you must:
- Observe field 8 for each processor and record the value.
- Wait for some interval.
- Review field 8 for each CPU again and calculate how much the value has increased.
- Dividing this difference by the length of the expected time interval (in jiffies), according to Little Law , we get the average length of the scheduler's execution queue per interval.
Unfortunately, I do not know of any utility for automating this process, which is usually installed or even packaged into a Linux distribution. I have not used it, but the kernel documentation suggests http://eaglet.rain.com/rick/linux/schedstat/v12/latency.c , which unfortunately refers to a domain that can no longer be resolved. Fortunately, it is available on the way back of the car .
Why not sar or vmstat ?
These tools report the number of processes currently running. Of course, if this number exceeds the number of processors, some of them must wait. However, processes can still be expected, even when the number of processes is less than the number of processors, for a number of reasons:
- A process can be assigned to a specific processor.
- The scheduler may decide to plan the process on a specific CPU for better use of the cache or for reasons of NUMA optimization.
- The scheduler can deliberately idle the CPU to allow more time for a competing process with a higher priority on another CPU that uses the same execution core (hyper-threading optimization).
- Hardware interrupts can only be processed on specific processors for various hardware and software reasons.
Moreover, the number of running processes is determined only in one instant. In many cases, this number can fluctuate quickly, and there may be a conflict between the moments of the metric sample.
These things mean that the number of processes executed minus the number of processors is not a reliable indicator of processor conflict.