Context: an application exposing web-services, running on a Mule ESB 3.5, with JMX port open.
My purpose if to have a detailed insight on the thread status (busy, waiting, idle, park, etc.) and thread pools evolution (number of thread created/used) of the application inside a single Dynatrace Dashboard.
I expect to have several pools based on the number of connectors and flows defined in the application configuration. When monitoring with VisualVM, connecting on the JMX port, I can see them in the red rectangle:
I've been looking into the Dynatrace menus, trying to create "Custom JMX" measures, looking on Dynatrace community and MuleSoft forums but without any answer.
So is there a way to define measures and/or sensors in Dynatrace to retrieve such information for a Mule application?
Have you tried Create Measure/Server Side Performance/Agent based measures/Custom JMX/Custom JMX Measure, then click "Browse Metrics" and "Pick Agent" and selecting the agent in question? You should then see a tree of JMX measures appear on the left, which you can then traverse and use the "Test" button on any leaf node that interests you.
If the node returns a non-zero number, you should be able to create a measure on it and use it as you would any other measure from that point.
Thank you for your answer.
Yes I did try that actually, but couldn't find any interesting measure in the proposed tree.
I tried to create a measure based on "Started" (as shown in the screenshot) but this just gave me a miserable flat line as chart for my two agents.
I even tried to use some of the JBoss 5/6, Tomcat and WebSphere JMX measures, which looked more sophisticated, but this obviously didn't work.
I found surprising the fact that VisualVM can, without knowing about Mule application server, monitor the threads through JMX with greater details than Dynatrace does.
Would you have other suggestions?
I think "started" is just a 1/0 type measure.
Have you seached for "thread" to find where the thread status might be exposed?
I did some googling trying to learn the names of the thread measures in JMX for Mule, but had no luck and don't have an instrumented Mule instance right now I can play with.
Also, keep in mind we only deal with numeric attributes from JMX, and I am not sure if the thread states are numeric (and if so, how to translate them to the "state").
Indeed, "Started" does not give much information 🙂
I did search for "Thread" and got a couple of results. As you can see on the screenshot below, I tried some of them, but unfortunately only "Java Virtual Machine - Thread Count" could be applied on my Mule agents.
However, the produced graph does not give more information than the existing Java "Application Process" dashboard.
It simply gives the evolution of the total number of threads run by the Mule (without even distinguishing application and application server threads).
Regarding your comment, having a distinction between the different application pools of thread would already be very interesting. I consider that having the thread statuses is a "nice to have".
I have to say that I cannot think of anything else to try right now. Can you?
Thanks a lot for your help anyway.
Please consider that custom JMX measures can only query numeric values, so a status timeline of a specific thread (group) is not possible. The only way to see a thread status is to make a thread dump which only is a snapshot (no timeline possible).
Thank you for your suggestion.
As explained in my previous post, seeing the thread status is (or rather was) a 'nice-to-have'.
The main goal of my post was to find out how to monitor the different application thread pools. Please refer to the very first screenshot where I show (in the red rectangle) that both "vmConnector" and "httpConnector" have 2 receiver threads.
Regarding the thread dump, the problem is that it freezes the application during the dump, and sometimes even make the Mule reboot, which unacceptable for a production environment.
Actually this is not possible. Via JMX only numeric values (like the Thread count) can be evaluated. Something like grouping by name of a thread pool cannot be done when it is not exposed as MBean.
All right, fair enough, but can't we achieve this goal though another Dynatrace functionality, such as Sensor or something?
Would you have a suggestion of custom Dynatrace plugin to group retrieved data, or some Java code to expose a new MBean, that I could try be develop myself? If yes, any idea about the way to proceed?
I would start here:
If not, you can modify it: