Tuesday, May 8, 2012

Using JMX to Investigate Queue Manager and Java on Linux

Recently, we have been involved in a rather large Queue Manager deployment running on a Linux environment.  During this process, we have been reviewing various tools and techniques to help analyze and detect issues in a remote production environment.  As part of this process, we discovered the value of JMX, VisualVM, and the tools associated with debugging Java applications.  Most of these practices are useful on both Linux and Windows deployments. 

As a bit of background, our Queue Manager for Sametime application is built using Java and a collection of Java libraries.  Queue Manager leverages the Sametime Java API for access to the core Sametime services, it uses the Domino libraries for accessing Lotus Notes/Domino services, and Queue Manager uses Spring as the main application framework.  So, at a high level, Queue Manager is a Java Spring application running under Tomcat. 

Typically, we deploy Queue Manager on a Windows OS.  However, during a recent deployment, we configured Queue Manager to run on a Linux server.  As part of this deployment, we wanted to remotely monitor the thread activity of the application, the CPU usage of the application, and identify any issues related to either thread usage or memory management.

While there are some very useful tools (typically command line) to help with this type of analysis, we particularly like JMX and VisualVM.  After enabling JMX on the server hosting Queue Manager, our application developers then connect to the JMX monitoring system using VisualVM.  This allows our developers to monitor the performance, health, thread usage, and memory usage in real-time - using a very nice graphical display.  The added advantage is that we can use VisualVM to both remotely monitor a variety of internal servers, as well as use VisualVM to review thread dumps and heap dumps (memory allocation dumps) from a remote production environment. 

Here is a screen shot of VisualVM monitoring an internal Linux instance of Queue Manager:

VisualVM is also very useful during extended periods of load testing and stress testing.  Our internal developers created a variety of load testing tools in order to simulate the various Queue Manager  users and systems.  Our load testing suite includes the ability to scale the number of inbound Sametime IM seekers (people looking for help), the number of experts in a queue, the number of web service requests to the system, as well as experts entering and leaving various queues.  During these load tests, VisualVM provides a real time display of the threads, memory, and CPU usage for the system.

In addition to high level monitoring, VisualVM enables the ability to sample (collect) data over a specific period of time.  So, during certain load testing cycles, we can enable VisualVM to sample the activity of the system and then we can review the data to inspect threads, CPU usage across threads, and memory allocation.

View CPU usage by thread:

One of the most useful tools in VisualVM is the ability to dump threads.  To create a thread dump, right click on the JMX instance and select Dump Threads...

Snapshots and System Sampling:

While the ability to capture a thread dump is useful, we have found the ability to save and export a snapshot of the system, using the sampling module, to be incredibly valuable. The 'sampler' area of VisualVM provides the ability to 'snapshot' the current system state and then export this information.
Navigate to the Sampler tab and enable sampling:

Click the Snapshot button

Export the snapshot

Save the snapshot as a file (so it can be shared)

The exported snapshot may then be transferred to another machine and opened using VisualVM.  The ability to save profile snapshots, transfer these snapshots, and then inspect the snapshots using VisualVM has proved extremely valuable.

We have found JMX and VisualVM to be invaluable tools during both our internal load testing and remote performance tuning and analysis efforts.  JMX is a standard component of the Java environment, and the addition of VisualVM provides a compelling, and powerful, collection of debugging and analysis tools.

Here are some helpful links that we have collected during this process:

VisualVM and JMX
Pretty good Spring presentation on JMX and other useful topics:

Getting started with VisualVM:

20 Linux System Monitoring Tools Every SysAdmin Should Know:
Tomcat Related Links:

Reviewing Tomcat Configuration and Tuning:

Presentation by Mark Thomas on Tomcat Tuning:

Mulesoft article on Tomcat performance tuning:

Checking for looping code: