I activated Apache modules sensors for an agent where we are regularly seeing delays within Apache (several seconds in Apache vs a few hundred ms in the corresponding servlet request processing, but I'm still not able to identify the source of the delay. I've tried with the default module exclusion list, without any exclusion, then excluding several modules manually, with and without modules aggregation turned on, but I can only see the delay in the elapsed time without seeing which module / processing is causing it in the PurePaths:
Is there a debug flag that would allow for the agent to collect everything (without the maximum limit of 20 modules) in terms of Apache module processing, without aggregation? Does the 20 module limit work by taking the first 20 modules, or does it take the top 20 in terms of execution times?
I will try to have a sysadmin looking at the system stats, because all I can see is peaks in IO write (of a few MB/sec), so maybe it's disk latency while logging (though 16s is quite the latency...), and I've also seen DNS query related issue with similar configurations (mod_proxy), but in that case I would expect the delay to be apparent in the module execution time).
This example is apparent sporadically during a specific load testing, but I've seen very similar situations in production or in other tests.
FYI: we've narrowed it down to disk IO contention (due to another process log level. I'm waiting on an answer from support to see if this could be displayed within Appmon, or if the time spent writing to log isn't exposed in any way to an external agent.