JDK comes with a bunch of handy tools. It’s good to
know about them. In this post we will take a look at
jps
, jstack
, jmap
and jhat
.
jps
jps
is a JVM counterpart to unix ps
command.
By default jps
prints process PID and
the name of the main class
or the name of a jar file if the
application was started using java -jar
option.
$ jps
54177 Jps
54173 App
54452 app.jar
But it can be more talkative.
-l
option adds a package name to the
main class name and a full/relative path to the jar filename.
-m
option will print command line arguments passed to the program.
$ jps -lm
54355 pl.marcinchwedczuk.app.App arg1 arg2 arg3
54452 build/libs/app.jar arg1 arg2 arg3
54458 jdk.jcmd/sun.tools.jps.Jps -lm
To print JVM switches we use -v
option:
$ jps -v
54654 app.jar -Xmx32m -Xms32m
54657 Jps -Dapplication.home=... -Xms8m ...
jstack
jstack PID
can be used to print current stack traces of
all threads running in java process.
You can also print stack traces from a core dump file.
The output of jstack
command is often referred to
as a thread dump.
Thread dumps are invaluable resources when it comes to debugging
nasty deadlocks that show up only on the production servers.
On the other hand the number of threads in a serious java application
can be overwhelming. To make the most of thread dumps, you
need to give threads meaningful names. You should give names
(via Thread::setName
) at least to the threads that you create
yourself and
you should also supply a “thread naming” ThreadFactory
when creating
new thread pools (e.g. newFixedThreadPool(int nThreads, ThreadFactory threadFactory)
).
Let’s create a simple app that deadlocks and see what jstack
will print:
public static void main(String[] args) throws InterruptedException {
Lock l1 = new ReentrantLock();
Lock l2 = new ReentrantLock();
var t1 = new Thread(() -> {
Thread.currentThread().setName("AppThread#1");
l1.lock();
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
l2.lock();
});
var t2 = new Thread(() -> {
Thread.currentThread().setName("AppThread#2");
l2.lock();
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
l1.lock();
});
t1.start(); t2.start();
t1.join(); t2.join();
}
EDIT: I was tired when I wrote this code. It will deadlock
both threads
say 99% of time but not always. Instead of sleep
I should use
CountDownLatch
. I leave the code as it is as I don’t want to regenerate
thread dumps but I wanted to point out this problem.
For readability I had to shorten jstack
output.
$ jstack `jps | grep App | cut -d ' ' -f 1`
"main" #1 prio=5 os_prio=31 cpu=46.25ms elapsed=85.17s tid=0x00007fb623810800 nid=0x1803 in Object.wait() [0x000070000027a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@14.0.1/Native Method)
...
at java.lang.Thread.join(java.base@14.0.1/Thread.java:1371)
at pl.marcinchwedczuk.bzzz.App.main(App.java:37)
"AppThread#1" #13 prio=5 os_prio=31 cpu=1.30ms elapsed=85.12s tid=0x00007fb622031000 nid=0x9b03 waiting on condition [0x00007000014b3000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@14.0.1/Native Method)
...
at java.util.concurrent.locks.ReentrantLock.lock(java.base@14.0.1/ReentrantLock.java:322)
at pl.marcinchwedczuk.bzzz.App.lambda$main$0(App.java:22)
at pl.marcinchwedczuk.bzzz.App$$Lambda$1/0x0000000800b65840.run(Unknown Source)
at java.lang.Thread.run(java.base@14.0.1/Thread.java:832)
...
Found one Java-level deadlock:
=============================
"AppThread#1":
waiting for ownable synchronizer 0x00000007ffd92998, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "AppThread#2"
"AppThread#2":
waiting for ownable synchronizer 0x00000007ffd92968, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "AppThread#1"
...
Notice that JVM was able to detect the deadlock, saving us a hours
of debugging. If you have a heisenbug you may consider running
jstack
periodically and searching its output for Found * deadlock
lines.
Now let’s see how we can extract thread dumps for a core dump. And for that we need a core dump.
Here I will describe how to make a core dump on macOS 10.15
(this is based on this article)
First we need to execute ulimit -c unlimited
in the shell
to remove the file size limit on created core-dump files.
A simple crashing hello world C program can create about 2GB core dump,
Java cores can have sizes of 10GB or more.
Then we need to set appropriate
permissions for /cores
directory:
$ sudo chmod 1777 /cores
# Test if we have enough permissions
$ echo 1 > /cores/test
TIP: 1
in 1777
is for sticky bit.
If a directory has this bit set then
only the owner of a file contained in that directory
can remove or rename that file.
If we additionally create a file with 700
permissions
then nobody beyond us will be able to change or
remove the file.
Then in the same shell in which we executed ulimit -c unlimited
we have to run a java application and in a new
terminal window we need to send SIGSEGV
to that app:
$ kill -SIGSEGV PID
After being hit by SIGSEGV
Java should crash with a message:
$ java -jar build/libs/bzzz.jar
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fff6f4cfdfa, pid=55303, tid=775
...
# Core dump will be written. Default location: /cores/core.56128
...
It may take a while to write 10GB+ file on disk so be patient.
Now for some reason I was not able to take a core dump from
official Oracle distribution of JDK. When I used OpenJDK
build everything worked perfectly. Now when I switched to OpenJDK
I have to use OpenJDKs jstack
to analyze the core dump.
$ export PATH=/usr/local/Cellar/openjdk/14.0.1/bin:$PATH
$ jhsdb jstack --core /cores/core.56128 \
--exe /usr/local/Cellar/openjdk/14.0.1/bin/java
"main" #1 prio=5 tid=0x00007fb4c300c000 nid=0x1d03 in Object.wait() [0x00007000027db000]
java.lang.Thread.State: WAITING (on object monitor)
JavaThread state: _thread_blocked
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- waiting on <0x000000061fe55d88> (a java.lang.Thread)
- java.lang.Thread.join(long) @bci=72, line=1303 (Interpreted frame)
- locked <0x000000061fe55d88> (a java.lang.Thread)
- java.lang.Thread.join() @bci=2, line=1371 (Interpreted frame)
- pl.marcinchwedczuk.bzzz.App.main(java.lang.String[]) @bci=57, line=37 (Interpreted frame)
...
For some reason in OpenJDK 14 the command jstack /path/to/java coredump
did not work. Instead I have to use a new tool introduced in JDK9
called jhsdb
. Anyway the result is the same, we managed to
get thread dumps from the core dump. Again the tool was smart enough
to point out the deadlock (not visible on attached listing).
OK lets cleanup our system and revert the settings:
ulimit -c 0
, rm /cores/*
and sudo chmod 1775 /cores
.
jmap
jmap
can be used to display various info about
Java process heap. For example we may take a heap
histogram, which tells us number of instances and
total memory size taken per class.
$ jmap -histo $PID | head
No dump file specified
num #instances #bytes class name (module)
-------------------------------------------------------
1: 965 2775128 [I (java.base@14.0.1)
2: 7555 399568 [B (java.base@14.0.1)
3: 7324 175776 java.lang.String (java.base@14.0.1)
4: 1295 160512 java.lang.Class (java.base@14.0.1)
5: 964 88712 [Ljava.lang.Object; (java.base@14.0.1)
6: 1872 59904 java.util.HashMap$Node (java.base@14.0.1)
But the real power of jmap
lies in its ability
to take heap dumps:
$ jmap -dump:live,format=b,file=heapdump $PID
Dumping heap to /path/to/heapdump ...
Heap dump file created [4264220 bytes in 0.018 secs]
Heapdumps can then be comfortably opened and analyzed in tools like Eclipse Memory Analyzer.
Heapdumps can be quite huge, if you want to move them between servers remember to first compress them to speed things up. They also contain sensitive data like AWS access keys and user passwords, so please keep them secure.
TIP: You can also generate heapdumps on
out of memory errors using -XX:+HeapDumpOnOutOfMemoryError
JVM switch.
EDIT: Since JDK9 the recommended way of generating heapdumps changed. You may want to see this article on InfoQ.
jhat
Now what if you don’t want/have permissions to install fancy heapdump analyzers?
Not all is lost, we may still use primitive but working jhat
.
Run jhat heapdump
to start jhat
HTTP server with basic
heapdump browser.
EDIT: Unfortunately jhat
was removed in JDK9.