AppMon is showing constant high Garbage collection for both hosts. So I basically went to that application-->agent group-->advanced-->unchecked Resource Exhausted Notifications.
Still I can see memory in total memory from Diagnose runtime.
I found this process in as documentation. Please suggest me if there is another way of doing.
Our Prod server is running out of memory.
Those are trending memory snapshots which are different from a full deep memory analysis snapshot which I believe is what you have unchecked from occurring when there is an out of memory event. These do not significantly impact the application as they are just counting objects as opposed to analyzing the entire state of memory. These are often scheduled so can you check your active tasks and see if you have this being created on a schedule anywhere? It looks like they are being created every 5 minutes.
I looked into Server.0.0.log but all I see is like
2017-02-25 12:13:34 INFO [MemoryDumpTask] Creating Memory Dump for agentinfo[sessionReference=LiveSession[systemId=Newlook DTE Energy_Prod], agentGroupReferenceKeyId=Newlook_Portal_Prod, agentMappingReferenceKeyId=Newlook_Portal_Prod, host='aix860q2' processId=11731122, mode=direct]
2017-02-25 12:13:39 WARNING [MemoryDumpTaskHandler] Error while postprocessing memory dump: com.dynatrace.diagnostics.core.analytics.memory.MemoryDumpTaskHandler postProcessing:416
com.dynatrace.diagnostics.core.datacenter.serapi.DiskQuotaExceededException: Session Writing is currently not allowed because the disk space configuration is violated
Hm, I'm not sure where else to check. It definitely seems to be being scheduled every 5 minutes but I don't see where that is being set, if no one else has any ideas you may want to just open a case with support. Since it is just a trending snapshot it shouldn't be a major issue affecting the application or anything like that which is good but that definitely is odd.
What technologie is on your servers ? Since we are migrating to IPAS/websphere AS was85, we see that on the GC Caused suspension time dashlet, is showing parallel data of the minor-GC values on the heap dashlet.
Also related to that, the incident "GC health of process 'intranetv85[xxxxxxxx]@xxxxxxxxx:0' is not ok."
So for every minor GC, we see data in GC Caused suspension. And incident is comming, dough even no suspension time is there.
The Memory Dump Action Plugin was creating the memory dumps, not the "create a memory snapshot upon out-of-memory notification" checkbox. Removing the Memory Dump actions from the incident that was firing seems to have eventually resolved the issue.