How to use HotSpot DTrace probes on SmartOS?

Question

On Mac OS X, I can find HotSpot probes to run Java programs by doing:

cody.mello@ashur ~ (1) % sudo dtrace -ln 'hotspot*:::' Password: Invalid connection: com.apple.coresymbolicationd ID PROVIDER MODULE FUNCTION NAME 165084 hotspot46 libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-clinit 165085 hotspot46 libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-concurrent 165086 hotspot46 libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-end 165087 hotspot46 libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-erroneous 165088 hotspot46 libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-error 165089 hotspot46 libjvm.dylib _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread [instanceKlass::initialize_impl(instanceKlassHandle, Thread*)] class-initialization-recursive ... 

But if I create a simple Java program and run it on SmartOS:

 cody@101901c9-6d66-ea32-fe42-f1fbebd4bf99 ~ % cat Loop.java class Loop { public static void main(String[] args) throws InterruptedException { while (true) { Thread.sleep(5000); } } } cody@101901c9-6d66-ea32-fe42-f1fbebd4bf99 ~ % javac Loop.java cody@101901c9-6d66- ea32-fe42-f1fbebd4bf99 ~ % java Loop 

I can not find any probes:

 cody@101901c9-6d66-ea32-fe42-f1fbebd4bf99 ~ (255) % pfexec dtrace -ln 'hotspot*:::' ID PROVIDER MODULE FUNCTION NAME dtrace: failed to match hotspot*:::: No probe matches description 

Is there anything special I need to do to see them?

Answer 1

The problem is that on SmartOS (and other lighting options), as well as on their own Solaris siblings, the DTrace module in the JVM is lazily loaded (i.e. DOF compiled with -x lazyload ). As a result, DTrace sensors do not load until explicit activation. There are two ways to handle this. First, you can say that DTrace itself included specific test objects, forcing the target process to load its probes. This requires (at least) the identifier of the target process; to resolve this in the example presented in the question, it would be something like this:

 % pfexec dtrace -ln 'hotspot*$target:::' -p `pgrep -fn "java Loop"` 

This will pick up the USDT hotspot (and hotspot_jni ) testers, but still does not work using the jstack() action on a machine filled with unsuspecting Java processes. (That is, it works when you want to use USDT probes on a well-known process, and not when you want to use the ustack helper profile for all Java processes.) If this is the problem you care about, in the lighting options (SmartOS, OmniOS, etc.) .d.), you can effectively cancel the lazy loading of DTrace probes (and the stack assistant) using the audit library dedicated to the task. This library - /usr/lib/dtrace/libdtrace_forceload.so and its 64-bit version, /usr/lib/dtrace/64/libdtrace_forceload.so - effectively forces DTrace probes to load when the process starts, providing you with USDT probes and the jstack() action jstack() for all such processes. To do this, for 32-bit JVMs, start java using the LD_AUDIT_32 environment LD_AUDIT_32 :

 export LD_AUDIT_32=/usr/lib/dtrace/libdtrace_forceload.so 

For 64-bit JVMs:

 export LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so