I've created the following incident rule:
The incident triggers (its listed in the incident dashboard) and the e-mail is sent, but CPU Sampling and Thread Dump actions are not executed (nothing shown in CPU Sampling/Threads dashlet).
CPU sampling works when started manually. The .NET agent log file contains nothing except the usual startup messages (no info [dotnet] Initializing CPU-Sampling message).
Any ideas what could go wrong?
Hi Dominik - what is the condition for this incident? If it triggered by
something like the application process being unresponsive or being in
such a bad state that the agent can't even do anything (it is impacted
as well) then it wouldn't be able to create and send back a thread dump.
Maybe to isolate the issue you could try setting up a test incident with the thread dump for a scenario when the agent and process are certainly healthy to see if it can create the thread dump in that case.
Thanks for the suggestion James - your are indeed right, we try to analyze a problem where our monitored .NET application becomes unresponsive ("Process not responding") which only happens on our productive system. Therefore I've created measures/BTs to trigger the incident once the Process.Responding property returns false. This means the main/GUI thread is blocked, but other threads are still executing (because we get logs during this time etc.). This worked on our development system just fine (a while(true) on the GUI thread was used to simulate the issue), so I'm not sure if this is really the problem. I'll try to use another trigger condition to see if that works.