I have an incident rule for a custom metric, watching over Glassfish HTTP Threads listener queues.
The incident is created as expected when the conditions are met, but the action to trigger Thread Dumps on that incident is not working.
I'd expect 10 thread dumps to be created when the incident is triggered, but nothing is captured.
Am I missing something?
I'm not familiar with the Glassfish HTTP Queue metric. Is this possibly a metric you're getting from an API (JMX?), and as such may not have an associated host where threads could be captured?
I would also suggest looking in the agent logs and seeing if there's any information which explains why thread dumps can't be captured.
I was able to reproduce your situation using the Current CPU Load metric (% value), by setting the threshold to 3%. Wasn't hard to trigger the incident and each time I got a thread dump. Perhaps as a test, setup a new incident on a different metric (CPU consumption) and see if you can trigger the dump. Be sure to use a metric that's directly coming from a JVM.
Let us know your results.
Glassfish HTTP Queue is a custom metric we collect using the JMX Plugin. It's not a standard JVM metric, but I'd expect the incident rule to apply just the same.
I never found any error messages from the agents related to the thread dump.
Besides, I configured another incident based on a metric from PurePath response time. That one seems to work fine and Thread Dumps are triggered by incidents.