Thread Dump

Thread Dump is a snapshot taken at a given time which provides you with a complete listing of all created Java Threads. When monitoring tool fail to prove the root cause of an application issue such as hugging thread and slow response etc., thread dump is helpful to identify the root cause.

I suggest you to read JVM: How to analyze Thread Dump before reading below techniques.

How to Take Thread Dump

$pbrun weblogic
$/app/jdk/jdk1.7.0_72/bin/jstack [pid] >> /app/tmp/td_1.txt

Recommend to take thread dump multiple times in 15 sec interval when the problem occur.

Thread Dump Analysis

Use Threadlogic.jar (or samurai.jar) to open the thread dump file
Sort by Thread Group
Look for executeThread that's running
Thread start with "at java.net.SocketInputStream.socketRead" means it's reading from other service (vision, dvs, vip, or other microservice)
Look at 3 thread dump together, to figure out which thread takes too long to complete.
Thread blocked by Monitors: This session show you which thread is blocked, but you can ignore weblogic.socket.Muxer (as stated at #7)
Ignore ExecuteThread weblogic.socket.Muxer.

ex: ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'
Ignore: anything that's not executeThread
In JVM startup parameters we set the min thread pool to 200, if it reach 200, it will double to 400. if more than 400, our JVM will crash.
```
-Dweblogic.SelfTuningThreadPoolSizeMin=200
-Dweblogic.SelfTuningThreadPoolSizeMax=400
```
Find long running thread (right click): this is to find stuck/block thread that running from all threadump file (run for too long) waiting thread is fine. See below image
Look for stuck thread (sometimes, this will show up at .out log file)
Look for block thread (will only show up at thread dump)
Look at fatal log

Find long running thread

Example 1

ExecuteThread: '204' for queue: 'weblogic.kernel.Default (self-tuning)'
locked <0xfffffffd62a40878> (a com.---.--.---.---.---.--IconicOrderImpl)

Example 2

Waiting for lockmanager server (issue)


Advisories: Too Many Threads  Web Application Bottleneck 
[](https://www.javacodegeeks.com/2012/03/jvm-how-to-analyze-thread-dump.html)
"[ACTIVE] ExecuteThread: '78' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000105597fc0 nid=0x8f in Object.wait() [0xffffffff52179000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)at java.lang.Object.wait(Native Method)
-waiting on <0xfffffffebc40fe38> (a atg.service.**lockmanager**.ClientLockEntry)
      at atg.service.lockmanager.ClientLockEntry.acquireWriteLock(ClientLockEntry.java:421)

Thread Dump example (identify which call the thread is making)

look for the bottom anything before - locked <0xfffffffd675fbd48> (a vzw.pos.lazy.LazyBean) is standard, you will see this in every thread dump
don't look at com.vzw.pos.mobile.common don't look at spring....
look at something start with v-w.pos & AFTER locked <0xfffffffd675fbd48> (a vzw.pos.lazy.LazyBean)

More Example On Next Page

Resources:

JVM: How to analyze Thread Dump

1. Thread Dump