JBoss / HotSpot JVM crash

We have a web application built on Jboss that has been in production for several years; in the past 18 months he has never fallen. However, in the last two days, jvm crashed 4 times.

When the JVM crashes, we get an error report that I am trying to decrypt. I looked at several other crash reports (on this site and others), but none of them resemble ours; and because I am new to reading them, I cannot say what I am missing.

[Edited crash reports to add additional information]

I had four crash reports ... I will send 2 of them (with libraries except brevity) if this helps diagnose the problem. None of the 4 looks exactly the same. I send only 2, because on the body there is a limit of 30000 char.

CRASH REPORT # 1 :

# # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0x0000002a, pid=7674, tid=2159016848 # # Java VM: Java HotSpot(TM) Server VM (1.5.0_20-b02 mixed mode) # Problematic frame: # C 0x0000002a # --------------- THREAD --------------- Current thread (0x80bbff08): VMThread [id=7679] siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0000002a Registers: EAX=0x0c0c0c0c, EBX=0xb7b35810, ECX=0x8130b9c8, EDX=0x9df81dd0 ESP=0x80afebbc, EBP=0x80afebd8, ESI=0xb7b26780, EDI=0xb7b44c2c EIP=0x0000002a, CR2=0x0000002a, EFLAGS=0x00010206 Top of Stack: (sp=0x80afebbc) 0x80afebbc: b796e7e2 8130b9c8 9df81dd0 b7b33d00 0x80afebcc: b796e7aa b7b35810 b7b44c2c 80afec68 0x80afebdc: b79e0b6e b7b44c2c 00000000 80b2f980 0x80afebec: 80afec08 006d9b7b 007c4120 00028098 0x80afebfc: 80afec10 00028098 80b2f980 80afec38 0x80afec0c: b79a167a 00000000 00000001 00000001 0x80afec1c: 00000000 00000000 000aa873 000f4240 0x80afec2c: 00000000 80b2f980 09a07ea0 80afec68 Instructions: (pc=0x0000002a) 0x0000001a: [error occurred during error reporting, step 100, id 0xb] Stack: [0x80a7f000,0x80b00000), sp=0x80afebbc, free space=510k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C 0x0000002a V [libjvm.so+0x474b6e] V [libjvm.so+0x473e9f] V [libjvm.so+0x473b07] V [libjvm.so+0x51fd4a] V [libjvm.so+0x51f236] V [libjvm.so+0x51e847] V [libjvm.so+0x51eab5] V [libjvm.so+0x51e620] V [libjvm.so+0x43bc78] C [libpthread.so.0+0x550b] VM_Operation (0x7a47dbe0): parallel gc system gc, mode: safepoint, requested by thread 0x7ab234d8 **[processes excerpted for brevity]** Other Threads: =>0x80bbff08 VMThread [id=7679] 0x80bd18a0 WatcherThread [id=7687] VM state:at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event]) [0x09a0a6a0/0x09a0a6c8] Threads_lock - owner thread: 0x80bbff08 [0x09a0aaf0/0x09a0ab08] Heap_lock - owner thread: 0x7ab234d8 Heap PSYoungGen total 54464K, used 398K [0xad860000, 0xb1060000, 0xb1140000) eden space 54016K, 0% used [0xad860000,0xad860000,0xb0d20000) from space 448K, 88% used [0xb0d20000,0xb0d83a40,0xb0d90000) to space 1664K, 0% used [0xb0ec0000,0xb0ec0000,0xb1060000) PSOldGen total 466048K, used 245327K [0x91140000, 0xad860000, 0xad860000) object space 466048K, 52% used [0x91140000,0xa00d3d20,0xad860000) PSPermGen total 107776K, used 67420K [0x81140000, 0x87a80000, 0x91140000) object space 107776K, 62% used [0x81140000,0x853173f8,0x87a80000) **[dynamic libraries exceprted]** VM Arguments: jvm_args: -Dprogram.name=run.sh -Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/jboss/jboss-5.1.0.GA/lib/endorsed java_command: org.jboss.Main -c livingdiary Launcher Type: SUN_STANDARD Environment Variables: JAVA_HOME=/usr/java/jdk JRE_HOME=/usr/java/jdk/jre PATH=/usr/kerberos/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/jboss/bin:/opt/jboss/jboss-5.1.0.GA/bin LD_LIBRARY_PATH=/usr/java/jdk1.5.0_20/jre/lib/i386/server:/usr/java/jdk1.5.0_20/jre/lib/i386:/usr/java/jdk1.5.0_20/jre/../lib/i386 SHELL=/bin/bash Signal Handlers: SIGSEGV: [libjvm.so+0x517270], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGBUS: [libjvm.so+0x517270], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGFPE: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGPIPE: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGILL: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGUSR2: [libjvm.so+0x43c4f0], sa_mask[0]=0x00000004, sa_flags=0x10000004 SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGQUIT: [libjvm.so+0x43bf20], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGTERM: [libjvm.so+0x43bf20], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 --------------- SYSTEM --------------- OS:Fedora release 8 (Werewolf) uname:Linux 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 19:19:45 EST 2008 i686 libc:glibc 2.7 NPTL 2.7 rlimit: STACK 10240k, CORE 0k, NPROC 65536, NOFILE 1024, AS infinity load average:0.06 0.05 0.00 CPU:total 4 (cores per cpu 4, threads per core 1) family 6 model 7 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2 Memory: 4k page, physical 3358636k(327528k free), swap 511k(511k free) vm_info: Java HotSpot(TM) Server VM (1.5.0_20-b02) for linux-x86, built on Jun 18 2009 12:08:29 by java_re with gcc 3.2.1-7a (J2SE release) time: Fri May 13 01:49:23 2011 elapsed time: 5968 seconds 

CRASH REPORT # 2

 # # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0x006d7426, pid=9790, tid=1949301648 # # Java VM: Java HotSpot(TM) Server VM (1.5.0_20-b02 mixed mode) # Problematic frame: # C [libc.so.6+0x69426] # --------------- THREAD --------------- Current thread (0x7859f560): JavaThread "HDScanner" [_thread_in_native, id=9828] siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0000000c Registers: EAX=0x98ec2e90, EBX=0x007c2ff4, ECX=0x00000001, EDX=0x00000000 ESP=0x742fe718, EBP=0x742fe780, ESI=0x09ba30f0, EDI=0x09ba3150 EIP=0x006d7426, CR2=0x0000000c, EFLAGS=0x00210246 Top of Stack: (sp=0x742fe718) 0x742fe718: 00000050 00000000 0000000d 742fe650 0x742fe728: 742fe790 b77fbd91 09ba30f8 00000060 0x742fe738: 02400010 0079893c 7859f560 00000001 0x742fe748: 00000000 b77e027a 7859f560 007c4120 0x742fe758: 0000004c 742fe778 006d9b7b 007c4120 0x742fe768: 0000004d b754a058 7859f620 007c2ff4 0x742fe778: 007c4120 09ba30f8 742fe7b8 006db0f0 0x742fe788: 007c4120 09ba30f8 09ba30f8 b754a058 Instructions: (pc=0x006d7426) 0x006d7416: 44 17 04 01 0f 85 05 03 00 00 8b 57 08 8b 47 0c 0x006d7426: 39 7a 0c 0f 84 80 05 00 00 8b 83 d0 00 00 00 89 Stack: [0x7427f000,0x74300000), sp=0x742fe718, free space=509k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libc.so.6+0x69426] C [libc.so.6+0x6d0f0] cfree+0x90 C [libjava.so+0x1162f] JNU_ReleaseStringPlatformChars+0x1f C [libjava.so+0x15ae8] Java_java_io_UnixFileSystem_getBooleanAttributes0+0xc8 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J java.io.UnixFileSystem.getBooleanAttributes0(Ljava/io/File;)I J org.jboss.virtual.plugins.context.file.FileHandler.exists()Z J org.jboss.virtual.plugins.context.file.FileHandler.createChildHandler(Ljava/lang/String;)Lorg/jboss/virtual/spi/VirtualFileHandler; J org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(Ljava/lang/String;)Lorg/jboss/virtual/spi/VirtualFileHandler; J org.jboss.virtual.plugins.context.file.FileHandler.getChild(Ljava/lang/String;)Lorg/jboss/virtual/spi/VirtualFileHandler; J org.jboss.virtual.VirtualFile.getChild(Ljava/lang/String;)Lorg/jboss/virtual/VirtualFile; J org.jboss.deployers.vfs.spi.structure.modified.UpdateDeleteVisitor.doVisit(Lorg/jboss/virtual/VirtualFile;)V J org.jboss.deployers.vfs.spi.structure.modified.SynchVisitor.visit(Lorg/jboss/virtual/VirtualFile;)V J org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(Lorg/jboss/virtual/spi/VirtualFileHandler;)V J org.jboss.virtual.plugins.context.AbstractVFSContext.visit(Lorg/jboss/virtual/spi/VirtualFileHandler;Lorg/jboss/virtual/spi/VirtualFileHandlerVisitor;ZZZZLorg/jboss/virtual/VirtualFileFilter;)V J org.jboss.virtual.plugins.context.AbstractVFSContext.visit(Lorg/jboss/virtual/spi/VirtualFileHandler;Lorg/jboss/virtual/spi/VirtualFileHandlerVisitor;ZZZZLorg/jboss/virtual/VirtualFileFilter;)V J org.jboss.virtual.plugins.context.AbstractVFSContext.visit(Lorg/jboss/virtual/spi/VirtualFileHandler;Lorg/jboss/virtual/spi/VirtualFileHandlerVisitor;)V J org.jboss.virtual.VFS.visit(Lorg/jboss/virtual/VirtualFile;Lorg/jboss/virtual/VirtualFileVisitor;)V v ~I2CAdapter j org.jboss.virtual.VirtualFile.visit(Lorg/jboss/virtual/VirtualFileVisitor;)V+13 j org.jboss.deployers.vfs.spi.structure.modified.SynchWrapperModificationChecker.hasStructureBeenModifed(Lorg/jboss/virtual/VirtualFile;Lorg/jboss/deployers/vfs/spi/structure/VFSDeploymentContext;)Z+58 j org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(Lorg/jboss/deployers/vfs/spi/structure/VFSDeploymentContext;Z)Z+90 v ~C2IAdapter J org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(Lorg/jboss/virtual/VirtualFile;)Z J org.jboss.system.server.profileservice.repository.HotDeploymentRepository.getModifiedDeployments()Ljava/util/Collection; v ~I2CAdapter j org.jboss.system.server.profile.repository.AbstractProfile.getModifiedDeployments()Ljava/util/Collection;+15 v ~C2IAdapter J org.jboss.system.server.profileservice.hotdeploy.HDScanner.scan()V v ~I2CAdapter j org.jboss.system.server.profileservice.hotdeploy.HDScanner.run()V+1 j java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4 j java.util.concurrent.FutureTask$Sync.innerRunAndReset()Z+22 j java.util.concurrent.FutureTask.runAndReset()Z+4 j java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Ljava/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask;)Z+1 j java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic()V+1 j java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V+8 j java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V+43 v ~C2IAdapter J java.util.concurrent.ThreadPoolExecutor$Worker.run()V v ~OSRAdapter j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub -------------- PROCESS --------------- Java Threads: ( => current thread ) 0x77192cd8 JavaThread "http-127.0.0.1-28080-20" daemon [_thread_blocked, id=28750] 0x7219d8b8 JavaThread "WorkManager(2)-17" daemon [_thread_blocked, id=25933] 0x0c90bc98 JavaThread "WorkManager(2)-16" daemon [_thread_blocked, id=25932] 0x73156008 JavaThread "WorkManager(2)-15" daemon [_thread_blocked, id=25895] 0x0b16ff58 JavaThread "WorkManager(2)-14" daemon [_thread_blocked, id=25894] 0x730058a0 JavaThread "WorkManager(2)-13" daemon [_thread_blocked, id=25609] 0x09415a90 JavaThread "WorkManager(2)-12" daemon [_thread_blocked, id=25608] 0x74064110 JavaThread "http-127.0.0.1-28080-19" daemon [_thread_blocked, id=25000] 0x72d62ee0 JavaThread "http-127.0.0.1-28080-18" daemon [_thread_blocked, id=24949] 0x73dfcc28 JavaThread "http-127.0.0.1-28080-17" daemon [_thread_blocked, id=24948] 0x73077f18 JavaThread "http-127.0.0.1-28080-16" daemon [_thread_blocked, id=23122] 0x74aac3a8 JavaThread "http-127.0.0.1-28080-15" daemon [_thread_blocked, id=17793] 0x71bdede8 JavaThread "WorkManager(2)-11" daemon [_thread_blocked, id=9050] 0x0a961a60 JavaThread "WorkManager(2)-10" daemon [_thread_blocked, id=9048] 0x72b5fef8 JavaThread "http-127.0.0.1-28080-14" daemon [_thread_blocked, id=7880] 0x74aadf58 JavaThread "WorkManager(2)-9" daemon [_thread_blocked, id=4977] 0x74a72b20 JavaThread "WorkManager(2)-8" daemon [_thread_in_native, id=4960] 0x0da25ec8 JavaThread "WorkManager(2)-7" daemon [_thread_blocked, id=4958] 0x11c024a0 JavaThread "ConnectionValidator" daemon [_thread_blocked, id=24055] 0x7f7b0198 JavaThread "JBoss System Threads(1)-10" daemon [_thread_blocked, id=23943] 0x77094fb8 JavaThread "WorkManager(2)-6" daemon [_thread_blocked, id=23942] 0x115b5fd8 JavaThread "WorkManager(2)-5" daemon [_thread_blocked, id=23941] 0x6edfa668 JavaThread "JBoss System Threads(1)-9" daemon [_thread_blocked, id=23929] 0x7a94be80 JavaThread "JBoss System Threads(1)-8" daemon [_thread_blocked, id=20996] 0x7215f9c0 JavaThread "http-127.0.0.1-28080-13" daemon [_thread_blocked, id=20954] 0x79dfde68 JavaThread "JBoss System Threads(1)-7" daemon [_thread_blocked, id=19144] 0x7d240cd8 JavaThread "JBoss System Threads(1)-6" daemon [_thread_blocked, id=13260] 0x70e85de0 JavaThread "JBoss System Threads(1)-5" daemon [_thread_blocked, id=12405] 0x772fd448 JavaThread "http-127.0.0.1-28080-12" daemon [_thread_blocked, id=12404] 0x730785b8 JavaThread "http-127.0.0.1-28080-11" daemon [_thread_blocked, id=11301] 0x731502d0 JavaThread "http-127.0.0.1-28080-10" daemon [_thread_blocked, id=11138] 0x7d27ac88 JavaThread "JBoss System Threads(1)-4" daemon [_thread_blocked, id=10273] 0x715a7ba8 JavaThread "http-127.0.0.1-28080-9" daemon [_thread_blocked, id=10270] 0x1327a9f0 JavaThread "Timer-3" daemon [_thread_blocked, id=9943] 0x6dee6250 JavaThread "JBoss System Threads(1)-3" daemon [_thread_blocked, id=9939] 0x73fac440 JavaThread "http-127.0.0.1-28080-8" daemon [_thread_blocked, id=9938] 0x740905b8 JavaThread "http-127.0.0.1-28080-7" daemon [_thread_blocked, id=9937] 0x73007570 JavaThread "http-127.0.0.1-28080-6" daemon [_thread_blocked, id=9936] 0x71b2cd30 JavaThread "http-127.0.0.1-28080-5" daemon [_thread_blocked, id=9935] 0x740f7d68 JavaThread "http-127.0.0.1-28080-4" daemon [_thread_blocked, id=9933] 0x11cfa830 JavaThread "http-127.0.0.1-28080-3" daemon [_thread_blocked, id=9932] 0x0d009408 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=9913] 0x0aad7a20 JavaThread "File Reaper" daemon [_thread_blocked, id=9912] 0x11bff2b8 JavaThread "http-127.0.0.1-28080-2" daemon [_thread_blocked, id=9911] 0x0b856a48 JavaThread "http-127.0.0.1-28080-1" daemon [_thread_blocked, id=9909] 0x7dcdf8d0 JavaThread "ajp-127.0.0.1-28009-Acceptor-0" daemon [_thread_in_native, id=9883] 0x78c99c80 JavaThread "http-127.0.0.1-28080-Acceptor-0" daemon [_thread_in_native, id=9882] 0x78693c70 JavaThread "JBossLifeThread" [_thread_blocked, id=9881] 0x792f6f78 JavaThread "timerFactory" [_thread_blocked, id=9880] 0x78738ce0 JavaThread "timerFactory" [_thread_blocked, id=9879] 0x78694b90 JavaThread "Connection Consumer for dest JBossQueue[chatHistoryArchiver] id=3" daemon [_thread_blocked, id=9876] 0x777948b0 JavaThread "Connection Consumer for dest JBossQueue[mailMessageInternalDelivery] id=1" daemon [_thread_blocked, id=9874] 0x771c1d80 JavaThread "Connection Consumer for dest JBossQueue[mailMessageSMTPDelivery] id=2" daemon [_thread_blocked, id=9873] 0x08ac5f78 JavaThread "Timer-2" daemon [_thread_blocked, id=9867] 0x725eba80 JavaThread "WorkManager(2)-4" daemon [_thread_blocked, id=9866] 0x731227e0 JavaThread "WorkManager(2)-3" daemon [_thread_blocked, id=9865] 0x0f39b070 JavaThread "WorkManager(2)-2" daemon [_thread_blocked, id=9864] 0x09a31fc0 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=9862] 0x0a8ee368 JavaThread "IdleRemover" daemon [_thread_blocked, id=9861] 0x0d6130a0 JavaThread "Thread-16" daemon [_thread_blocked, id=9856] 0x0d36c178 JavaThread "JCA PoolFiller" [_thread_blocked, id=9855] 0x72dadad0 JavaThread "HSQLDB Timer @858c07" daemon [_thread_blocked, id=9854] 0x72fd1920 JavaThread "DefaultQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=9853] 0x7336ee60 JavaThread "DefaultQuartzScheduler_Worker-9" [_thread_blocked, id=9852] 0x7375e560 JavaThread "DefaultQuartzScheduler_Worker-8" [_thread_blocked, id=9851] 0x77ffae50 JavaThread "DefaultQuartzScheduler_Worker-7" [_thread_blocked, id=9850] 0x77ffacc8 JavaThread "DefaultQuartzScheduler_Worker-6" [_thread_blocked, id=9849] 0x735b5df8 JavaThread "DefaultQuartzScheduler_Worker-5" [_thread_blocked, id=9848] 0x0bdb85b0 JavaThread "DefaultQuartzScheduler_Worker-4" [_thread_blocked, id=9847] 0x0a4d4bf0 JavaThread "DefaultQuartzScheduler_Worker-3" [_thread_blocked, id=9846] 0x090d3208 JavaThread "DefaultQuartzScheduler_Worker-2" [_thread_blocked, id=9845] 0x0c1b9d08 JavaThread "DefaultQuartzScheduler_Worker-1" [_thread_blocked, id=9844] 0x09089938 JavaThread "DefaultQuartzScheduler_Worker-0" [_thread_blocked, id=9843] 0x0a8b1228 JavaThread "WorkManager(2)-1" daemon [_thread_blocked, id=9842] 0x0cfb05a0 JavaThread "AOPListner" daemon [_thread_blocked, id=9841] 0x0da31c10 JavaThread "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon [_thread_blocked, id=9840] 0x7376f478 JavaThread "PooledInvokerAcceptor#0-4445" [_thread_in_native, id=9839] 0x738fec80 JavaThread "RMI TCP Accept-4444" daemon [_thread_in_native, id=9838] 0x7386f278 JavaThread "Thread-13" [_thread_blocked, id=9837] 0x739fdd38 JavaThread "Listener:4712" daemon [_thread_in_native, id=9836] 0x7dcdeb50 JavaThread "Thread-12" daemon [_thread_blocked, id=9834] 0x758f8aa8 JavaThread "Thread-11" daemon [_thread_blocked, id=9833] 0x78b61ba8 JavaThread "Thread-10" daemon [_thread_blocked, id=9832] 0x74548198 JavaThread "Listener:4713" daemon [_thread_in_native, id=9831] 0x78ff8b08 JavaThread "AcceptorThread[ServerSocket[addr=/127.0.0.1,port=0,localport=4446]]" [_thread_in_native, id=9830] 0x7759edb8 JavaThread "ServerSocketRefresh" daemon [_thread_blocked, id=9829] =>0x7859f560 JavaThread "HDScanner" [_thread_in_native, id=9828] 0x78190340 JavaThread "AcceptorThread[ServerSocket[addr=/127.0.0.1,port=0,localport=3873]]" [_thread_in_native, id=9827] 0x78694df8 JavaThread "ServerSocketRefresh" daemon [_thread_blocked, id=9826] 0x788f7910 JavaThread "secondaryServerSocketThread[0]" daemon [_thread_in_native, id=9825] 0x78341100 JavaThread "AcceptorThread[ServerSocket[addr=/127.0.0.1,port=0,localport=4457]]" [_thread_in_native, id=9824] 0x78bf81e0 JavaThread "ServerSocketRefresh" daemon [_thread_blocked, id=9823] 0x09787878 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=9820] 0x768ee010 JavaThread "RMI RenewClean-[127.0.0.1:59098]" daemon [_thread_blocked, id=9818] 0x7972ff40 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=9816] 0x790f9558 JavaThread "RMI TCP Accept-1090" daemon [_thread_in_native, id=9815] 0x7f3301b8 JavaThread "SubscriptionWatchDog" [_thread_blocked, id=9814] 0x09550498 JavaThread "JBoss System Threads(1)-2" daemon [_thread_in_native, id=9813] 0x0b58ef60 JavaThread "GC Daemon" daemon [_thread_blocked, id=9812] 0x09dfebf0 JavaThread "RMI Reaper" [_thread_blocked, id=9811] 0x0b59e7e8 JavaThread "Timer-1" daemon [_thread_blocked, id=9810] 0x08947368 JavaThread "RMI TCP Accept-21098" daemon [_thread_in_native, id=9809] 0x0b87ecf0 JavaThread "JBoss System Threads(1)-1" daemon [_thread_in_native, id=9808] 0x08947e18 JavaThread "Timer-Log4jService" daemon [_thread_blocked, id=9807] 0x7f78a070 JavaThread "Timer-0" daemon [_thread_blocked, id=9805] 0x80bf78b0 JavaThread "DestroyJavaVM" [_thread_blocked, id=9790] 0x80bcfe10 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=9802] 0x80bcea30 JavaThread "CompilerThread1" daemon [_thread_blocked, id=9801] 0x80bcdab0 JavaThread "CompilerThread0" daemon [_thread_blocked, id=9800] 0x80bccaf0 JavaThread "AdapterThread" daemon [_thread_blocked, id=9799] 0x80bcbcc8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=9798] 0x80bc2848 JavaThread "Finalizer" daemon [_thread_blocked, id=9797] 0x80bc2318 JavaThread "Reference Handler" daemon [_thread_blocked, id=9796] Other Threads: 0x80bbff08 VMThread [id=9795] 0x80bd19b8 WatcherThread [id=9803] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap PSYoungGen total 41472K, used 4100K [0xad860000, 0xb0740000, 0xb1140000) eden space 35328K, 1% used [0xad860000,0xad90cb38,0xafae0000) from space 6144K, 55% used [0xb0140000,0xb0494728,0xb0740000) to space 6336K, 0% used [0xafae0000,0xafae0000,0xb0110000) PSOldGen total 466048K, used 214068K [0x91140000, 0xad860000, 0xad860000) object space 466048K, 45% used [0x91140000,0x9e24d278,0xad860000) PSPermGen total 71168K, used 70964K [0x81140000, 0x856c0000, 0x91140000) object space 71168K, 99% used [0x81140000,0x8568d1f0,0x856c0000) **[dynamic libraries excerpted]** VM Arguments: jvm_args: -Dprogram.name=run.sh -Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true - Djava.endorsed.dirs=/opt/jboss/jboss-5.1.0.GA/lib/endorsed java_command: org.jboss.Main -c livingdiary Launcher Type: SUN_STANDARD Environment Variables: JAVA_HOME=/usr/java/jdk JRE_HOME=/usr/java/jdk/jre PATH=/usr/kerberos/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/jboss/bin:/opt/jboss/jboss-5.1.0.GA/bin LD_LIBRARY_PATH=/usr/java/jdk1.5.0_20/jre/lib/i386/server:/usr/java/jdk1.5.0_20/jre/lib/i386:/usr/java/jdk1.5.0_20/jre/../lib/i386 SHELL=/bin/bash Signal Handlers: SIGSEGV: [libjvm.so+0x517270], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGBUS: [libjvm.so+0x517270], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGFPE: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGPIPE: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGILL: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGUSR2: [libjvm.so+0x43c4f0], sa_mask[0]=0x00000000, sa_flags=0x10000004 SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGQUIT: [libjvm.so+0x43bf20], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGTERM: [libjvm.so+0x43bf20], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 --------------- SYSTEM --------------- OS:Fedora release 8 (Werewolf) uname:Linux 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 19:19:45 EST 2008 i686 libc:glibc 2.7 NPTL 2.7 rlimit: STACK 10240k, CORE 0k, NPROC 65536, NOFILE 1024, AS infinity load average:0.01 0.02 0.00 CPU:total 4 (cores per cpu 4, threads per core 1) family 6 model 7 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2 Memory: 4k page, physical 3358636k(383020k free), swap 511k(511k free) vm_info: Java HotSpot(TM) Server VM (1.5.0_20-b02) for linux-x86, built on Jun 18 2009 12:08:29 by java_re with gcc 3.2.1-7a (J2SE release) time: Wed May 11 07:00:36 2011 elapsed time: 853441 seconds 

Other things I did:
1. I added -Xcheck: jni to the JVM option to see if there is any problem with the native library. 2. Jboss debugging has been deployed to find out if I can capture more information before the JVM dies.

Thanks so much if you have any advice!

UPDATE LAST NIGHT

So, the server was on a couple of days ... and then crashed again for the fifth time. I added a new question with a new report: https://stackoverflow.com/questions/6005976/jboss-hotspot-jvm-crashing-again

UPDATE AFTER H / W DIAGNOSTICS WORK Just received the following response from GoDaddy after running the diagnostics:

Thank you for contacting Server Support for your xxxxx server.

We performed a hardware test against your server and found that the chassis failed and needs to be replaced. We tried to replace the chassis with the same model but it is not available. Therefore, we will need to change the chassis using a different model, but this will require a server that will be re-provisioned with the operating system not loading with other configuration equipment. Re-provisioning erase all data on the server and reinstall the operating system with a simple control panel. For us, continue to replace the chassis, we need your explicit permission to resubmit your server to the name.

+1
source share
2 answers

Are crash reports always the same? Do you have stack traces? those. bit in a crash led by

--------------- PROCESS ---------------

This is a vague tbh dump. One option is quirky RAM, as it seems to have died when a safe place to collect garbage has been reached. Have you checked the hardware diagnostics on the box?

+1
source

I agree with Matt, that sounds like a hardware issue. I saw three of your crash reports, and they all fail in different parts. One, mainly when executing VM code. The second is in native code. And the third (in the question, which you should not open, but added here), in standard Java code.

Of course, if you have several production machines, and the problem is different, then I would seriously think about buying a subscription and real support.

+1
source

Source: https://habr.com/ru/post/1416276/


All Articles