cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Monitoring thread status and thread pools of a Mule ESB application

bruno_cats_exte
Newcomer

Hi,

Context: an application exposing web-services, running on a Mule ESB 3.5, with JMX port open.

My purpose if to have a detailed insight on the thread status (busy, waiting, idle, park, etc.) and thread pools evolution (number of thread created/used) of the application inside a single Dynatrace Dashboard.

I expect to have several pools based on the number of connectors and flows defined in the application configuration. When monitoring with VisualVM, connecting on the JMX port, I can see them in the red rectangle:

I've been looking into the Dynatrace menus, trying to create "Custom JMX" measures, looking on Dynatrace community and MuleSoft forums but without any answer.

So is there a way to define measures and/or sensors in Dynatrace to retrieve such information for a Mule application?

Many thanks.

Regards,

Bruno

14 REPLIES 14

dave_mauney
Dynatrace Champion
Dynatrace Champion

Hi Bruno,

Have you tried Create Measure/Server Side Performance/Agent based measures/Custom JMX/Custom JMX Measure, then click "Browse Metrics" and "Pick Agent" and selecting the agent in question? You should then see a tree of JMX measures appear on the left, which you can then traverse and use the "Test" button on any leaf node that interests you.

If the node returns a non-zero number, you should be able to create a measure on it and use it as you would any other measure from that point.

HTH,

dave

bruno_cats_exte
Newcomer

Hi Dave,

Thank you for your answer.

Yes I did try that actually, but couldn't find any interesting measure in the proposed tree.

I tried to create a measure based on "Started" (as shown in the screenshot) but this just gave me a miserable flat line as chart for my two agents.

I even tried to use some of the JBoss 5/6, Tomcat and WebSphere JMX measures, which looked more sophisticated, but this obviously didn't work.

I found surprising the fact that VisualVM can, without knowing about Mule application server, monitor the threads through JMX with greater details than Dynatrace does.

Would you have other suggestions?

Thank you.

Bruno

dave_mauney
Dynatrace Champion
Dynatrace Champion

Hi Bruno,

I think "started" is just a 1/0 type measure.

Have you seached for "thread" to find where the thread status might be exposed?

I did some googling trying to learn the names of the thread measures in JMX for Mule, but had no luck and don't have an instrumented Mule instance right now I can play with.

Also, keep in mind we only deal with numeric attributes from JMX, and I am not sure if the thread states are numeric (and if so, how to translate them to the "state").

HTH,

dave

bruno_cats_exte
Newcomer

Hi Dave,

Indeed, "Started" does not give much information 🙂

I did search for "Thread" and got a couple of results. As you can see on the screenshot below, I tried some of them, but unfortunately only "Java Virtual Machine - Thread Count" could be applied on my Mule agents.

However, the produced graph does not give more information than the existing Java "Application Process" dashboard.

It simply gives the evolution of the total number of threads run by the Mule (without even distinguishing application and application server threads).

Regarding your comment, having a distinction between the different application pools of thread would already be very interesting. I consider that having the thread statuses is a "nice to have".

I have to say that I cannot think of anything else to try right now. Can you?

Thanks a lot for your help anyway.

Best regards,

Bruno

dave_mauney
Dynatrace Champion
Dynatrace Champion

Hi Bruno,

I don't have any further ideas. Perhaps someone else on the forum will...

Thanks,

dave

@Kristof R. @Chris G. @Guido M. Please, any idea here?

harald_berger
Dynatrace Pro
Dynatrace Pro

Hi All,

Please consider that custom JMX measures can only query numeric values, so a status timeline of a specific thread (group) is not possible. The only way to see a thread status is to make a thread dump which only is a snapshot (no timeline possible).

Best

Harry

bruno_cats_exte
Newcomer

Hi Harry,

Thank you for your suggestion.

As explained in my previous post, seeing the thread status is (or rather was) a 'nice-to-have'.

The main goal of my post was to find out how to monitor the different application thread pools. Please refer to the very first screenshot where I show (in the red rectangle) that both "vmConnector" and "httpConnector" have 2 receiver threads.

Regarding the thread dump, the problem is that it freezes the application during the dump, and sometimes even make the Mule reboot, which unacceptable for a production environment.

Regards,

Bruno

harald_berger
Dynatrace Pro
Dynatrace Pro

Hi Bruno,

Actually this is not possible. Via JMX only numeric values (like the Thread count) can be evaluated. Something like grouping by name of a thread pool cannot be done when it is not exposed as MBean.

Sorry

Harry

bruno_cats_exte
Newcomer

Hi Harry,

All right, fair enough, but can't we achieve this goal though another Dynatrace functionality, such as Sensor or something?

Would you have a suggestion of custom Dynatrace plugin to group retrieved data, or some Java code to expose a new MBean, that I could try be develop myself? If yes, any idea about the way to proceed?

Regards,

Bruno

bruno_cats_exte
Newcomer

Hi All,

So can I implement a plugin to retrieve thread related data from the JVM, process it (by grouping and filtering) and finally display the result into Dynatrace?

Thank you,

Bruno

Hi Bruno,

I see no technical reason this could not be done...

dave

Hi Dave,

I have no idea of how this could be done. Could you share some references?

Thank you,

Bruno

Hi Bruno,

I would start here:

https://community.dynatrace.com/community/display/DL/JMX+Enhanced+Monitor+Plugin

It looks like it may already support gathering the measures that interest you.

If not, you can modify it:

https://community.dynatrace.com/community/display/DOCDT65/Developing+a+User+Plugin

HTH,

dave