Sunday, June 5, 2011

Poor man's approach in practice

One day we ran into problem when an application was randomly starting using more CPU resources as usual. It continued picking more and more CPU resources from time to time. Thread dump showed many threads though we couldn't understand wich of them is problematic...and I remembered about poor man's profiler and its approach: the stack that's all we need to!

Htop is awesome
At first we simply caught problematic thread using htop (you just need to remember about tree process view, type t-button, or anyway you need to see all forked java processes with its identifiers and CPU usage). We just obtained a java thread wich took more CPU than others. We got an identifier of native OS thread, let it be 3256, but what's next?

Jstack
I like jstack, it helps me all the time I deal with spontaneous problems. If you look thru the content of jstack output you'll see some thread identifiers and all of them are hexadecimal. Thus we converted our native thread id into hex representation 0xcb8. Lazy guys can use Integer.toHexString ;)

The final step
We searched by our hexadecimal string in jstack output (don't forget about leading zeros, therefore just search by value, i.e. cb8 in our case). Matched nid thread attribute value (suppose stands for native id) was our problematic thread with its stack. In fact we were getting infinite loop. The problem is localized and will be fixed soon.
Post a Comment
 
Blogged.com Technology Blogs