NOTE We have several close votes for this post. I don’t know exactly why, since no one has commented, but this question is about how our custom Java written server and its socket read calls can interact with Windows and make this high-level CPU usage, so I think this is the right forum to post this question, not the sysadmin scope.
I have a custom Java server that serves TCP connections from several different types (some websites, some of them), and one of our clients has problems with periodic periods with high CPU + high system CPU.
We could reproduce this on our test server, which runs on Windows Server 2008 on vmware.
When switching from one server to another or during other network events that cause several clients (40-80) to restore their connections at the same time, we see a high processor load, including a very high system CPU (> 50% on average with spikes over 70% ) These clients usually support TCP sockets on the server, so during the transition to another resource we will see that 80-160 sockets will be reconnected and serviced.
We used kernrate for the profile during one of these periods and have difficulty understanding the results.
Kernrate reports a low idle and high kernel time, as we expected during this period:
P0 K 0:00:39.405 (55.9%) U 0:00:27.424 (38.9%) I 0:00:03.603 ( 5.1%) DPC
0:00:01.528 ( 2.2%) Interrupt 0:00:00.327 ( 0.5%)
Interrupts= 114880, Interrupt Rate= 1631/sec.
P1 K 0:00:37.596 (53.4%) U 0:00:29.281 (41.6%) I 0:00:03.556 ( 5.0%) DPC
0:00:00.078 ( 0.1%) Interrupt 0:00:00.624 ( 0.9%)
Interrupts= 96016, Interrupt Rate= 1363/sec.
TOTAL K 0:01:17.002 (54.7%) U 0:00:56.706 (40.3%) I 0:00:07.160 ( 5.1%) DPC
0:00:01.606 ( 1.1%) Interrupt 0:00:00.951 ( 0.7%)
Total Interrupts= 210896, Total Interrupt Rate= 2994/sec.
Total Profile Time = 70434 msec
, , 200/, , , 70- 800/. Java 3 , ( ). , .
BytesStart BytesStop BytesDiff.
Available Physical Memory , 6296776704, 6266650624, -30126080
Available Pagefile(s) , 12591370240, 12577845248, -13524992
Available Virtual , 8796052869120, 8796048171008, -4698112
Available Extended Virtual , 0, 0, 0
Committed Memory Bytes , 305979392, 303964160, -2015232
Non Paged Pool Usage Bytes , 81997824, 82333696, 335872
Paged Pool Usage Bytes , 164028416, 164057088, 28672
Paged Pool Available Bytes , 4147101696, 4149116928, 2015232
Free System PTEs , 33556099, 33556099, 0
- -. , :
Total Avg. Rate
Context Switches , 462496, 6566/sec.
System Calls , 835593, 11863/sec.
Page Faults , 32814, 466/sec.
I/O Read Operations , 747, 11/sec.
I/O Write Operations , 3792, 54/sec.
I/O Other Operations , 27565, 391/sec.
I/O Read Bytes , 382146, 512/ I/O
I/O Write Bytes , 684128, 180/ I/O
I/O Other Bytes , 890365, 32/ I/O
Kernrate , NTOSKRNL, 10 :
OutputResults: KernelModuleCount = 131
Percentage in the following table is based on the Total Hits for the Kernel
ProfileTime 12806 hits, 65536 events per hit --------
Module Hits msec %Total Events/Sec
NTOSKRNL 10561 70429 82 % 9827282
AMDPPM 1001 70429 7 % 931456
HAL 701 70429 5 % 652298
VMXNET3N61X64 351 70428 2 % 326619
WIN32K 68 70428 0 % 63276
TCPIP 35 70428 0 % 32568
AFD 19 70428 0 % 17680
NETIO 17 70429 0 % 15818
VM3DMP 11 70428 0 % 10235
NTFS 6 70429 0 % 5583
NDIS 6 70429 0 % 5583
LSI_SAS 6 70429 0 % 5583
FLTMGR 6 70429 0 % 5583
VMMOUSE 5 70429 0 % 4652
I8042PRT 5 70429 0 % 4652
VMCI 3 70429 0 % 2791
CDD 2 70428 0 % 1861
DXGMMS1 1 70428 0 % 930
VMHGFS 1 70429 0 % 930
NSIPROXY 1 70428 0 % 930
NTOSKRNL :
ProfileTime 10561 hits, 65536 events per hit --------
Module Hits msec %Total Events/Sec
KeSynchronizeExecution 2485 70429 23 % 2312356
ExReleaseRundownProtectionCacheAwareEx 2099 70429 19 % 1953173
IoGetPagingIoPriority 1561 70429 14 % 1452550
memmove 1006 70429 9 % 936108
RtlDelete 952 70429 9 % 885860
memset 448 70429 4 % 416875
ExfAcquirePushLockExclusive 326 70429 3 % 303351
PoStartNextPowerIrp 280 70429 2 % 260547
RtlFindClearBits 239 70429 2 % 222395
KeBugCheckEx 199 70429 1 % 185174
KeWaitForMultipleObjects 92 70429 0 % 85608
FsRtlTeardownPerFileContexts 67 70429 0 % 62345
KeSetTimer 62 70429 0 % 57692
NtWaitForSingleObject 59 70429 0 % 54901
wcsncat_s 58 70429 0 % 53970
, , - , - (< 500/sec, )? KeSynchronizeExecution, ExReleaseRundownProtectionCacheAwareEx IoGetPagingIoPriority, ?
, , , .