cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

can AppMon generate a .phd file (portable heap dump) on the app server where the agent lives?

Alejandro_soto
Inactive

Is this possible? We are trying to determine who put that file there since it caused a disk to fill up.

My understanding is that a memory snapshot by AppMon (heapdump) will be saved to the AppMon server... but what if the app server doesn't have write access to the dt server? would this stay in a /tmp drive?

Thanks,

10 REPLIES 10

gabriel_casella
Dynatrace Pro
Dynatrace Pro

Maybe this can be related to a network problem. If in the middle of the transmission from the agent to the "server", the connection drops, then what to do with that dump?

Joseph_Hoffman
Champion

Dynatrace dump file are not written to the file system of the agent, the dynaTrace dump data is streamed thru a collector to the dTServer. The AppServer is not the writing process so it does not need file access to the dT Server system.

Dynatrace does not create .phd files, something else must have created it.

Hey Joe,

How can this occur? If the file is not written to the file system, how does our memory dump contain all memory from a particular instance in time? What I mean is - say the dump is 4GB. Where does that live? To get a snapshot, we need all memory at one specific time, so where does that 4GB go? If in some magical way the 4GB is just hanging out while the agent is streaming it to the collector and the agent fails (goes offline, service crashes, whatever), then that would cause the memory dump to fail, correct?

Brett, it seems to me like I'm a very similar issue as to what you are describing...

Except that to my knowledge I never told the agent to take a dump, and the 'Create memory snapshot upon OOM incident' option in the sensor is turned off.

These are from the agent log:

2016-05-03 07:12:04 [00002e2f]
info [native] Control channel connected successfully

2016-05-03 07:14:34 [00002e2f]
severe [native] Exception in controller:
util::SocketConnection::receiveExact(), 70014, End of file found. Retrying every 10 seconds.

2016-05-03 07:14:34 [0000dd58]
info [native] Got ResourceExhaustedNotification: @^A^Ï, not creating Memory Dump

Hey, I tried to ping you but it doesn't appear you're online. Are these from your agent logs? Is this implying the agent, collector, or server doesn't have space? Based on what Joe said, if the agent is streaming to the collector, I don't know if the collector stores the dump in the middle and then sends it to the server or if the collector literally just sends the data through from a buffer.

Thanks for the clarification Joe, follow up questions:

Can the agent issue a command to the WAS JVM to create that heapdump?

According to what the WAS team told me, the JVM is generating these dumps when OOM state is reached. The reason they are ending up in /temp is because two files with the exact name are being created.

Could it be that the WAS node agent and the dynatrace agent both issue the same command and therefore two exact files are being created?

Thanks,

Alejandro_soto
Inactive

I opened a ticket with support and they confirmed that our agent cannot do this. The severe and warning log messages are normal and related to the OOM alert on the JVM. This is because the agent can no longer talk to the collector due to the OOM alert.

Thanks for letting us know. Can you share the case number?

c_schwarzbauer
Dynatrace Champion
Dynatrace Champion

just as a quick summary:

  • the Java Agent creates memory dumps via JVMTI
  • as Joe said: the Agent does not persist memory dumps on disk, but immediately sends them to Collector/Server (via a dedicated TCP channel)
  • as the support team said: the Agent cannot trigger any kind of other memory dump (.phd, .hprof, ...)

HTH, Christian