Icon

If you want to test this on your own app - check out our Hands-On Memory Leak Detection Guide and download the 30 Day Free Trial.

Memory Leaks are painful – especially when they crash your production servers. But - how does one go about analyzing the leak that only happens in the production environment? Here is a story I was told while on-site with one of our clients. I asked: “Chris, tell me about a recent performance problem and how you solved it.”

He replied: “I’ve got a good one. We had these memory problems in production with memory increasing steadily and eventually crashing a JVM. First thing I asked my Ops guy is to capture a Java Heap Dump. His response was: ’Heap WHAT?’ So I started explaining how to take a Heap Dump on the JVMs. Two hours later he had the file on the production machines. Now it was time to transfer this file out of Prod on my Desk. This took about 20 minutes. When I opened the file it turned out to be corrupted. So – back to capturing a new dump and transferring it – another hour gone. The second dump actually opened in my analysis tool. Unfortunately it was a bit hard to tell what the memory leak was with just one dump. I would need multiple dumps and compare them to see which objects are actually growing over time. This procedure took several hours. At the end I knew my top classes based on instance count but I had no real answer on the actual leak.”

Improve the process from hours to minutes

He continued: “The good news came when I realized that dynaTrace runs on these Production Servers. I fired up the dynaTrace Client on my workstation and started a Trending Memory Dump on the Production Server. 10 Minutes later I had my answer and knew where the problem was.”

“The Trending Memory Dump feature automatically creates a series of memory dumps that are automatically transferred to the dynaTrace Server. The dynaTrace Server analyzes the instance counts across the different dumps and shows which objects are growing.

“Knowing the growing objects, I create a Full Memory Dump to analyze what is actually holding references to these objects, telling me which are responsible for filling up my Heap. Once I know which objects are responsible for the leak, I can create Memory Sensor Rules to also identify which methods actually allocate these instances. This tells my developers where in the code to optimize this behavior.

“This approach not only saves me a lot of time – I don’t need to worry about making any configuration changes to the JVM to capture a Heap Dump nor do I need any help to capture the Heap and transfer it over. I can do it all from my workstation without impacting the running environment. I also like the new feature of dynaTrace automatically triggering a Memory Dump when we either cross a certain memory threshold or when the JVM throws an OOM Exception. That gives me all the data I need to analyze new memory problems. Is that a good story for your Community?” (smile)

JSF State Caching as actual Root Cause of this problem

The problem turned out to be related to the JSF Object Caching mechanism to support the Back-Button feature in Web Applications. For every page a user visits JSF caches the current user state objects. In order to do that it has to create copies of these objects and keeps them in memory as long as the user’s session is live. The more pages a user visits the more instances this user holds in memory. In the case of this client they did not allow the user to move back to the previous page via the Back Button. So they disabled this JSF feature and, with that, limit the required memory per active user.

If you want to know more about how to optimize JSF State Caching without turning it off I recommend the following two blog posts: JSF Application Tuning Tips, JSF Tuning Discussion

More on dynaTrace Memory Diagnostics

If you are a dynaTrace user and you want to explore the possibilities of Memory Diagnostics check out the following dynaTrace Walkthroughs


Related articles

Selected Topics

Not Logged In? Customers and AJAX Edition Users Login with your Community Account