Nuclear-free core, isolcpus, nohz_full and rcu_nocbs

Question

I have the addition of "isolcpus = 3 nohz_full = 3 rcu_nocbs = 3" in grub.conf on RedHat 7.1, kernel: linux 3.10.0-229 kernel and as per http://www.breakage.org/2013/11/ 15 / nohz_fullgodmode / I also run the following command:

cat /sys/bus/workqueue/devices/writeback/cpumask f echo 1 > /sys/bus/workqueue/devices/writeback/cpumask cat /sys/bus/workqueue/devices/writeback/numa 1 echo 0 > /sys/bus/workqueue/devices/writeback/numa 

There are only 4 processor cores in the box, I run the following shell:

 watch -d 'cat /proc/interrupts' 

It looks like work perfectly, only cpu0 Local timer interrupts have 2000 in 2 seconds, else cpu 1 to cpu 3 has less than 10 in 2 seconds.

and then I test the following source:

 void *Thread2(void *param) { pthread_detach(pthread_self()); while( 1 ){ sleep( 100000 ) ; } } void *Thread1(void *param) { pthread_detach(pthread_self()); while( 1 ){ ; } } int main(int argc, char** argv) { pthread_t tid ; pthread_create(&tid , NULL, Thread1, (void*)(long)3); pthread_create(&tid , NULL, Thread2, (void*)(long)3); while( 1 ) sleep( 5 ) ; } 

and run it:

 taskset -c 3 ./x1.exe 

see output in:

 watch -d 'cat /proc/interrupts' 

this time, cpu 3 gets 10 ~ 30 local timer interrupts in 2 seconds, looks great, then I'm trying to start 2 thread1:

 pthread_create(&tid , NULL, Thread1, (void*)(long)3); pthread_create(&tid , NULL, Thread1, (void*)(long)3); 

then run it again:

 taskset -c 3 ./x1.exe 

then I see that kernel 3 has the same local timer interrupts with kernel 0, which is 2000 interrupts in 2 seconds.

May I ask why 2 very busy threads 1 will call core 3? much more timer interruptions ?! why did this happen ?! and how to change it, if possible ?!

Answer 1

In the second case, Kernel must schedule tasks with 2 cpu bindings on core 3, and setting dynamic ticks is applicable only when there is only one task to be performed. I thought SCHED_FIFO would stop these interrupts (and so I started responding), but this is not yet implemented according to https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt

You cannot change this behavior except for scheduling threads on different CPUs. You can always crack the kernel to achieve what you need.