Problems defining JMX plugin in Jetty / Mule ESB

[james_billingha]

I've got a Mule ESB application that Dynatrace needs to monitor, I see SaaS supports Mule ESB but there are a number of problems. Most pressing is the inability to pick out JMX metrics from Jetty/Mule ESB, which is probably down to me defining the plugin.json incorrectly.

>I have an issue defining the plugin.json, tried multiple ways of accessing the MBeans which unfortunately have "'s and a space in the "name" part of the MBean. Can be seen below for an example app in MuleESB ->

$>beans -d Mule.echo
#domain = Mule.echo:
Mule.echo:Application="application totals",type=org.mule.Statistics
Mule.echo:Flow="EchoFlow",type=org.mule.Statistics
Mule.echo:connector=connector.http.mule.default,name="endpoint.http.localhost.8084",service="EchoFlow",type=Endpoint
Mule.echo:name="EchoFlow",type=Flow
Mule.echo:name="_muleSystemModel(seda)",type=Model
Mule.echo:name="application totals",type=Application
Mule.echo:name="connector.http.mule.default",type=Connector
Mule.echo:name=AllStatistics,type=Statistics
Mule.echo:name=Configuration
Mule.echo:name=MuleContext
$>get -d Mule.echo -b Mule.echo:Application="application\ totals",type=org.mule.Statistics TotalProcessingTime
#mbean = Mule.echo:Application="application totals",type=org.mule.Statistics:
TotalProcessingTime = 5290;

My plugin.json looks like (Simple one to get just one value as a test) ->

{
  "version" : "0.001",
  "name" : "custom.jmx.muleesb",
  "type" : "JMX",
  "entity": "PROCESS_GROUP_INSTANCE",
  "configUI" : {
		"displayName": "MuleESB JMX"
	},
  "processTypes": [ 10, 12, 13, 16, 17, 18 ], 
  "metrics" : [
    {
      "timeseries" :
      {
        "key" : "TotalProcessingTime",
        "unit" : "Count",
        "dimensions" : [
          "rx_pid"
        ],
	"displayname": "Total Processing Time"
      },
      "source" : {
        "domain": "Mule.echo",
        "keyProperties": {
	  "type": "org.mule.Statistics",
	  "name": "application totals"
        },
        "allowAdditionalKeys": false,
        "attribute": "TotalProcessingTime",
        "calculateDelta": false,
        "calculateRate": false,
        "aggregation": "SUM"
      }
    }
  ],
  "ui" :
  {
    "keymetrics" : [
      {
        "key" : "TotalProcessingTime",
        "aggregation" : "avg",
        "mergeaggregation" : "sum",
        "displayname" : "Total Processing Time"
      }
    ],
    "keycharts" : [
      {
        "title" : "MuleESB Total Processing Time",
		"group" : "MuleESB metrics",
        "series" : [
          {
            "key" : "TotalProcessingTime",
            "displayname" : "Processing Time",
            "rightaxis" : true,
            "color" : "rgba(101,152,186,0.4)",
            "seriestype" : "bar",
            "mergeaggregation" : "sum"
          }
        ]
      }
    ],
    "charts" : [
      {
        "group" : "MuleESB metrics",
        "title" : "MuleESB Total Processing Time",
        "series" : [
          {
            "key" : "TotalProcessingTime",
            "displayname" : "Processing Time",
            "rightaxis" : true,
            "color" : "rgba(101,152,186,0.4)",
            "seriestype" : "bar"
          }
        ]
      }
    ]
  }
}

The issue seems to be with "name", it should be "\"application\ totals\"", but Dynatrace won't accept any escaped characters like that. I also tried using "name": "*" and that also fails ...

I say "fails", unfortunately unless I do something like put in "\\\"application\ totals\\\" and get an exception I get nothing in the logs at all for the monitored process. But what I do see is ->

2018-02-28 14:20:46.745 UTC [7e19f84a] info    [native] Updating plugins content from revision 1519827060897 to 1519827091624 (11 plugins)
2018-02-28 14:20:46.745 UTC [7e19f84a] info    [native]   Adding plugin custom.jmx.muleesb with version 0.001
2018-02-28 14:21:16.187 UTC [7e19f84a] info    [native] Updating plugins content from revision 1519827091624 to 1519827060897 (10 plugins)
2018-02-28 14:21:16.187 UTC [7e19f84a] info    [native]   Removing plugin custom.jmx.muleesb
2018-02-28 14:21:26.547 UTC [7e19f84a] info    [native] Updating plugins content from revision 1519827060897 to 1519827091624 (11 plugins)
2018-02-28 14:21:26.547 UTC [7e19f84a] info    [native]   Adding plugin custom.jmx.muleesb with version 0.001

It "adds" and "removes" the plugin over and over ...

I've got the plugin.json above working for a more conventionally named MBean in the same process. So "get -d Mule.echo -b Mule.echo:name=MuleContext FreeMemory" works in the above and I get a chart with free memory in it on the UI showing the rest of the config is correct. It just can't read the "application totals" name key as far as I can see.

Any ideas? I've tried escaping, using *, using unicode.... Nothing works and there is no debugging info!

[Julius_Loman · Accepted solution]

Did you try to use the JMX Browser when defining plugin? See https://www.dynatrace.com/news/blog/jmx-plugins-editor-now-available-eap/

[james_billingha]

Cheers, I've asked, but I assume that is a nice UI for defining the plugin that ultimately executes on the OneAgent the same as any other. It is there that the execution fails as it can't read an MBean with that name (I assume). But I'll give it a go, looks a lot more user friendly!

[Julius_Loman]

It is definitely worth trying it. If you can't see the mbean, did you look also in the agent java logs for the monitored process? For JMX plugins data will not be in the Oneagent plugin logs, but in the Oneagent Java logs, since it's a java process and JMX attributes retrieval is performed using the Java instrumentation. Oneagent plugin logs are for the Oneagent SDK plugins only.

[james_billingha]

Log I found the messages above in was in /opt/dynatrace/oneagent/log/java, named for the MuleESB process - ruxitagent_org.mule.module.reboot.MuleContainerBootstrap_16962.0.log

I assume that's the right one?

Thanks again!

[Julius_Loman]

That should be the correct one. Unfortunately for increasing debug log you need to open a support ticket.

[james_billingha]

@Julius's solution worked, the plugin.json it created had the quotes delimited but not the space. Thought I'd tried that combination but apparently not!

[james_billingha]

Might be useful for others instrumenting MuleESB JMX, given you need a JMX config per application this script auto-generates one for every application defined - https://github.com/JD-Billingham/Dynatrace_MuleESB_JMX

[jeppe_lindberg3]

@Michael L. this is worth looking into