We had an Out Of Memory Exception that got thrown in our JVM and picked up by AppMon. However, the built-in Out of Memory Incident did not trigger, which makes me wonder what exactly triggers the incident?
Can someone provide any detail?
This is what shows in the Exceptions Dashlet, yet no OoM Incident:
Exception Class: java.lang.OutOfMemoryError
Message: Failed to create a thread: retVal -xxxxxxxx
Solved! Go to Solution.
Out of Memory errors in Java and .NET occur when the VM/CLR cannot create a requested object because it cannot allocate any more memory.
For more details check this link:
Cesar, I'm guessing that you really didn't run out of memory, but instead simply couldn't create a thread. The inability to create a thread can be caused by several reasons such as being out of heap or native space, hitting a thread limit, Insufficient resources on the OS side ( ulimits being reached), etc.
But to answer your question, that AppMon incident is raised when the heap runs out of memory.
It would be interesting to go back and look at the heap values during that period of time and see if heap really seemed to be exhausted, i'm guessing that heap was fine and the thread issue was caused by some other resource problem. Your appplication JVM logs might show additional insight into the root cause.
I'm making some assumptions in this response, so no guarantees, but if you find your heap was of healthy size during that period, that might validate my theory. Let us know.
You were right. The heap was only at %30 usage. The cause of the exception was that we had hit the process limit of the Linux server. No new threads could be spawned.
My question still remains as to whether there is a concrete definition of this Incident? I understand that it triggers when all the Heap space is used, but is it just looking for utilization to hit %100, or looking for a specific error to be thrown, etc.?
The AppMon Incident triggers off the JRE Exception java.lang.OutOfMemoryError: Java heap space
Some good doc on various OutOfMemory conditions: