Icon

Comments have been closed on this page. Please use AppMon & UEM Open Q & A forum for questions about this plugin.

Overview

The JMX Enhanced Monitor Plugin allows to gather MBeans metrics from different JMX servers, e.g. Pivotal CloudFoundry Ops Metrics Tool, JBoss, JVM, Weblogic, WebSphere, etc.

Version

1.0.10

Compatible with

Dynatrace 6.1+

Author

eugene.turetsky@dynatrace.com

License

dynaTrace BSD

Support Level

Not Supported

Download

com.dynatrace.diagnostics.plugin.jmx_1.0.10.jar

com.dynatrace.diagnostics.plugin.jmx_1.0.11.jar (use only for JBoss 5.1.0 GA)

Description

The JMX Enhanced Monitor Plugin allows to gather MBeans metrics from different JMX servers, e.g. Pivotal CloudFoundry Ops Metrics Tool, JBoss, JVM, Weblogic, WebSphere, etc.

Release History

08-22-2015 Initial release of version 1.0.0

08-29-2015 Release 1.0.1. Changes include:

  • Removed dependency on the Java SDK 'jar' utility. It is not longer used by the plugin and hence no need to install a full Java SDK 1.6+ on the Dynatrace Collector box.
  • Simplified layout of the 'mg' directory in the Metric Group Plugin directory: it has only 'build' directory with the Metric Group plugin jar file.

09-06-2015 Release 1.0.2. Changes include:

  • Added support for the JBoss EAP 6.x, AS 7.x application servers

10-11-2015 Release 1.0.3. Changes include:

  • Added better mapping of different measure units into measures for JVM.

09-19-2016 Release 1.0.5. Changes include:

  • Added support for iOS

10-27-2016 Release 1.0.6. Changes include:

  • Fixed serialization issue related to Tomcat 8.0.x & Liferay 7.0.x

11-04-2016 Release 1.0.7. Changes include:

  • Enabled support for the Custom JMX Service URL.

11-27-2016 Release 1.0.8. Changes include:

  • Fixed recovery of the plugin process after JMX server failure.

12-08-2016 Release 1.0.9. Changes include:

  • Enforced requirement that the 'Plugin Instance Id' parameter is explicitly set when the 'Is Custom JMX service URL?' parameter is checked.

12-14-2016 Release 1.0.10. Changes include:

  • Added the 'Maximum Number of Captured Metrics' parameter to the task portion of the plugin to avoid gathering data for large amount of measures which could affect performance of the DT Server. Default value is 1000.

01-23-2017 Release 1.0.11. Changes include:

  • This is a special build to support JBoss 5.1.0 GA version of the JBoss application server. The JMX support in the JBoss 5.1.0 GA application server has a well known bug which was fixed in the later versions of the JBoss 6.x/7.x/etc. application servers. There is a workaround for this bug which requires minor modifications of the run.conf.sh/run.conf.bat files depending on the platform where JBoss 5.1.0 GA is running. To get detailed instructions please contact eugene.turetsky@dynatrace.com.
    The following are screenshots from execution of the plugin on the JBoss 5.1.0 GA application server: example of available measures and example of gathered measures.

Installation

Import the Plugin into the Dynatrace Server via the Dynatrace Server Settings menu -> Plugins -> Install Plugin. For details how to do this please refer to the dynaTrace documentation.

Usage

The JMX Enhanced Monitor Plugin contains a task plugin and a monitor plugin. A task plugin creates a metric group plugin which will be used by the monitor plugin. To install plugin you need to perform the following steps:

  1. Import the JMX Enhanced Monitor plugin into the Dynatrace server.
  2. Configure a task which will create a metric group plugin that contains list of gathered MBeans and their metrics.
  3. Execute a task plugin. The metric group plugin jar file will be created.
  4. Import jar file created in the step 3 into the Dynatrace server.
  5. Configure a monitor plugin.
  6. Setup schedule for the configured monitor plugin and execute it.
  7. Setup multiple instances of the JMX monitor plugin to monitor multiple JMX servers

Configuration of the Task Plugin

The Task plugin of the JMX Enhanced Monitor Plugin has the following list of parameters:

  • Is Plugin a Task? Boolean indicator which should be checked for tasks and uncheck for monitors.
  • Plugin Instance Id: String which uniquely identifies the task instance. The monitor plugin (see the 'Configuration of the Monitor Plugin' section below) must have the same value of this parameter. If empty, the default value will be used, e.g. ‘<host>.<port>’. 
    Note: Plugin Instance Id must not be empty if the 'Is Custom JMX service URL?' parameter is checked.
  • Maximum Number of Captured Metrics: allows to avoid gathering data for large amount of measures which could affect performance of the DT Server. Default value is 1000.
  • Is Custom JMX service URL? If checked, a custom JMX service URL will be used. Default is unchecked.
  • JMX Host: host of the JMX server. Mandatory parameter.
  • JMX Port: port of the JMX server. Mandatory parameter.
  • URL Path: URL path string of the JMX service URL if default JMX service URL does not work. Optional parameter.
  • JMX Server Type: list of supported JMX Server types:
    • CloudFoundry Metrics Tool;
    • JBoss;
    • JVM;
    • Weblogic (coming);
    • WebSphere (coming).
  • Is Authentication On? If checked, requires user and password. Default is unchecked.
  • User: user id which will be used to connect to the JMX server.
  • Password: password of the above user.
  • Include MBeans patterns: patterns are built using following schema:
    • <MBean-domain>:<key1>=<value1>,<key2>=<value2>…,*

For example,

    • 'org.cloudfoundry:*' pattern (without single quotes) includes measures from all Mbeans from the 'org.cloudfoundry' domain. 
    • 'org.cloudfoundry:job=DEA,*' pattern (without single quotes) includes only measures related to the DEA job from the org.cloudfoundry domain.

This parameter is multiline parameter. Each line of this parameter should contain one and only one MBeans pattern. The Include MBeans patterns is optional parameter. If not set, all MBeans will be included.

The CloudFoundry MBean domain is 'org.cloudfoundry'. Examples of the keys' hierarchy and their values are (see next screenshot down):

    • deployment - untitled_dev;
    • job - CloudController, DEA,HM9000, MetronAgent, Router, collector, etcd, uaa;
    • index - 0;
    • ip - 10.244.0.134, 10.244.0.26, ...

  • Exclude MBeans patterns: patterns are built using following schema:
    • <MBean-domain>:<key1>=<value1>,<key2>=<value2>…,*

The exclude MBeans patterns have the same structure as the include MBeans patterns which are described in the previous section.

This parameter is multiline parameter. Each line of this parameter should contain one and only one MBeans pattern. The Include MBeans patterns is optional parameter. The Exclude MBeans patterns is optional parameter. If not set, all MBeans will be included.

  • Metric Group Plugin Directory: directory where the task plugin will build Metric Group plugin jar file according to MBeans gathered from the configured JMX server. This directory has the following structure: 

    • The ‘mg’ directory contains the Metric Group plugin jar file in the build directory. 

    • The ‘archive’ directory contains zipped files of the ‘mg’ directory. They contain zipped ‘mg’ directory tree taken before the new jar file was created. Naming conventions of zipped archives is:

      • mg_<metric-group-version>_YYYY-MM-DD_HH-mm-ss.SSS.zip, where YYYY-MM-DD_HH-mm-ss.SSS is a timestamp when archive was created.

There are up to 100 of zipped ‘mg’ files in the ‘archive’ directory. the oldest ones are getting purged, when number of files in the directory exceeds 100 files.

  • Version of the Metric Group plugin: version of the metric group plugin which is assigned to the plugin. It has format: N.N.N, where N is a number. For example, 1.0.0. This is mandatory parameter.

Screenshot below shows an example of the JMX Extended Monitor plugin configuration for a CloudFoundry task plugin:

Execute task plugin after its configuration is done. As a result, a metric group plugin jar file will be created in the ‘build’ subdirectory of the ‘mg’ directory (see above the Metric Group Plugin directory structure).

Import the Metric Group Plugin into the Dynatrace Server via the Dynatrace Server Settings menu -> Plugins -> Install Plugin. For details how to do this please refer to the dynaTrace documentation.

Next screenshot shows installed a CloudFoundry Metric Group plugin which was created by the task plugin:

Note:

  1. Version 1.0.0 of the plugin uses 'jar' command internally and hence required Java SDK 1.6+ installed on the Collector box. This dependency is removed in the 1.0.1 version of the plugin (see Release History for version 1.0.1 of the plugin).
  2. To avoid gathering data for large amount of measures which could affect performance of the DT Server, it is recommended to use Include/Exclude patterns when configuring the Task plugin.

When the Task plugin needs to be executed?

The task plugin creates metric groups with MBeans measures. It should be executed initially before configuring a monitor plugin and every time when your JMX domain on the JMX server has new MBeans which measures the JMX Enhanced Monitor plugin instance needs to gather. The task plugin is synchronizing list of measures of the monitored JMX Server with the list of measures presented in the Metric Group plugin.

If new MBeans are not changing list of MBeans which are monitored by the instance of the JMX Enhanced Monitor plugin then there is no need to re-run task plugin to synchronize measures for this instance of monitor plugin. For example, if new MBeans are matching exclusion patterns of the monitor plugin, there is no need to rebuild Metric Group plugin jar file because new measures will be excluded anyway.  

Configuration of the Monitor plugin

The Monitor plugin of the JMX Enhanced Monitor Plugin has the following list of parameters:

  • Is Plugin a Task? Boolean indicator which should be unchecked for monitors.
  • Plugin Instance Id: String which uniquely identifies the task instance. If empty, the default value will be used, e.g. ‘<host>.<port>’. It must have the same value as it was used in the task plugin when the Metric Group jar file was created.
  • Is Custom JMX service URL? If checked, a custom JMX service URL will be used. Default is unchecked.
  • JMX Port: port of the JMX server. Mandatory parameter.
  • URL Path: URL path string of the JMX service URL if default JMX service URL does not work. Optional parameter.
  • JMX Server Type: list of supported JMX Server types:
    • CloudFoundry Metrics Tool;
    • JVM;
    • WebSphere;
    • Weblogic;
    • JBoss.
  • Is Authentication On? If checked, requires user and password. Default is unchecked.
  • User: user id which will be used to connect to the JMX server.
  • Password: password of the above user.
  • Include MBeans patterns: patterns are built using following schema:
    • <MBean-domain>:<key1>=<value1>,<key2>=<value2>…,*

This parameter is multiline parameter. Each line of this parameter should contain one and only one MBeans pattern. The Include MBeans patterns is optional parameter.

  • Exclude MBeans patterns: patterns are built using following schema:
    • <MBean-domain>:<key1>=<value1>,<key2>=<value2>…,*

This parameter is multiline parameter. Each line of this parameter should contain one and only one MBeans pattern. The Include MBeans patterns is optional parameter. The Exclude MBeans patterns is optional parameter.

  • Hosts: host of the JMX server. Mandatory for any monitor parameter.

Screenshot below shows an example of the JMX Extended Monitor plugin configuration for a CloudFoundry monitor plugin:

Next screenshot shows measures available for the CloudFoundry monitor plugin:

Next screenshot shows detailed screen of the CloudFoundry monitor execution:

JVM configuration for JMX monitoring

In order to monitor JVM using MBeans, the JVM JMX communication needs to be enabled. Next screenshot shows example of a set of VM arguments which need to be set to enable JMX communication:

Next screenshot below shows example of the task plugin configuration for Java JVM:

Next screenshot shows that built by the task a JVM metric group plugin was imported into the Dynatrace server. It also shows previously imported CloudFoundry metric group plugin.

Next screenshot shows example of the JVM Monitor plugin configuration screen which monitors JVM MBeans:

Next screenshot shows measures available for the JVM monitor plugin:

Next screenshot shows detailed screen of the JVM execution plugin:

JBoss configuration for JMX monitoring

The JBoss application server is using remoting-jmx protocol for its JMX server. There are multiple domains that JBoss JMX server is exposing for monitoring and management. Example of the JBoss domains and hierarchy of their keys is depicted in the next screenshot:

Examples of JBoss domains: jboss.as, jboss.as.expr, jboss.jta, jboss.modules, jboss.msc, jboss.remoting.handler, jboss.ws, etc.

Keys of JBoss MBeans are: 

  • deployment: could be any deployed application, e.g. jboss-greeter.war.
  • subsystem: datasource, ejb3, jpa, web, etc.
  • statistics: jdbc, pool, EJBUserDao, etc.

Example of filter structure for the JBoss: <domain-name>:deployment=<deployment-name>,subsystem=<subsystem-name>,datasource=<datasource-name>,statistics=<statistics-name>, i.e.

  • jboss.as:deployment=jboss-greeter.war,* selects all attributes from all subsystems from the jboss-greeter.war;
  • jboss.as:deployment=jboss-greeter.war,subsystem=datasources,* selects all attributes for all datasources from the jboss-greeter.war;
  • jboss.as:deployment=jboss-greeter.war,subsystem=datasources,data-source="java:jboss/datasources/GreeterQuickstartDS",* selects all attributes for the GreeterQuickstartDS datasource.

Next screenshot contains example of jdbc attributes for the GreeterQuickstartDS datasource:

Next screenshot contains example of pool attributes for the GreeterQuickstartDS datasource:

Next screenshot depicts example of pool attributes of the GreeterQuickstartDS datasource received from the JBoss server:

Next screenshot depicts a configuration screen for the task plugin for the JBoss application server:

Next screenshot depicts a configuration screen for the monitor plugin for the JBoss application server:

Example of using multiple instances of the JMX Enhances Monitor plugin

The JMX Enhanced Monitor plugin is designed to monitor different JMX servers by different instances of the plugin. Next screenshot shows two instances of the JMX Monitor plugin which are monitoring CloudFoundry and JVM JMX servers respectively:

Next screenshot shows two instances of the JMX Monitor plugin tasks for CloudFoundry and JVM respectively:

Usage of the plugin is also described in the following document here.

  1. Anonymous (login to see details)

    Eugene,

     Just wondering if you have a ball park on when this will included Websphere

     

    Thanks

    John

    1. Anonymous (login to see details)

      Hi John,

      I don't have exact dates but WebSphere and Weblogic app servers are the last two pieces which are left to implement for this plugin. They are on my plate of things to do.

      Eugene.

  2. Anonymous (login to see details)

    I have a websphere system that I was able to run the task against (I set the type as JVM) and it did build the measures file successfully. I imported the measures file, but when I poll the system using the Monitor (Set as JVM type) I get all "null" values returned. The monitor says it executed successfully, but I get nothing but nulls. Is this because I used JVM instead of WebSphere?

    1. Anonymous (login to see details)

      Hi George,

      I am checking to make sure that you do not have problems with the plugin. To confirm that we are on the same page here, I just configured this plugin to gather JVM metrics from the JVM which is running WebSphere 8.5. It took me overall under 3 minutes, so, please pay attention to this post.

      The JMX Enhanced Monitor plugin created 14 Metric Groups with 105 metrics in total (!!!). Again, it was done automatically in under 3 minutes from starting to configure the JMX Task plugin to creation of a sample JVM threads usage dashboard which is shown below. Please note that no Jconsole or VisualVM required to configure the JMX Monitor plugin. Here are details: 

      • The JMX Task plugin created the following 14 Metric Groups:
        1. ClassLoading
        2. Compilation
        3. Garbage Collection
        4. MarkSweepCompact
        5. Memory
        6. MemoryPool.JIT Code Cache
        7. MemoryPool.JIT.Data Cache
        8. MemoryPool.Java Heap
        9. MemoryPool.Class Storage
        10. MemoryPool.Miscellaneous Non-Heap Storage
        11. MemoryLeakConfig.LeakConfig
        12. OperatingSystem
        13. Runtime
        14. Threading

      Please see the following screenshot for details:

      • Above 14 Metric Groups contained the following 105 measures:

                

      • Below are two screenshots which depict configuration details of the JMX Enhanced Task plugin. Please note that include and exclude MBeans patterns parameters are left blank, i.e. plugin gathers all available measures.

          

      • The Monitor plugin configuration screenshots are depicted below. Here also no include or exclude patterns were specified,so, all measures will be monitored.

          

      • The Thread usage dashlet displays thread usage of the WebSphere JVM:

      The JMX Enhanced Monitor plugin was configured for JVM which is running WebSphere. The WebSphere JMX metrics will be added to the plugin very soon (together with the Weblogic measures).

      Please let me know if you have any further questions.

      Eugene.

      1. Anonymous (login to see details)

        Hi Eugene,

        I tried to connect to a Websphere Portal Server to monitor the Cache status. Unfortunately I receive a "java.net.SocketTimeoutException: Read timed out". However the connection itselfs seems to be established (as per netstat). Is there some timeout that cuts the connection after a certain time? If so - is it possible to change it? Also I have to connect to a Websphere cell, that requires SSL connection for ORB connection. Is there any way to configure the plugin to use SSL?

        Thank you in advance

        Michael

        1. Anonymous (login to see details)

          Hi Michael,

          Before we move forward, could you please check if you can connect and perform remote monitoring of the JVM JMX server using JConsole or VisualVM? For SSL access with JConsole see this link.

          Please note that as of today the JMX Enhanced Monitor plugin supports the following JMX servers:

          • CloudFoundry Metrics Tool;
          • JBoss;
          • JVM;
          • Weblogic (coming);
          • WebSphere (coming).

          Eugene.

  3. Anonymous (login to see details)

    Hi George,

    Plugin should be able to gather JVM metrics. Could you please make sure that you've imported the generated metric group plugin jar file? Let's setup a webex session that I can look into your issue. Please provide me with few 1 hour time slots for the webex. Before the meeting, could you please send me the generated metric group plugin jar file?

    Eugene.

    1. Anonymous (login to see details)

      Thank you for your time and efforts. I re-created everything this morning and unfortunately got the same results. I run the task, import the generated jar, create a monitor, check to make sure the measures are there, run the monitor, and still nothing but "nulls". I'll email you offline of this thread and send you the jar file.

       

       

      1. Anonymous (login to see details)

        Hi George,

        Thank you for following up on my post. Please send me e-mail with the generated jar file and FINER log file of the Monitor plugin. Also, please send me the 'Measures' tab of the Configure JMX Enhanced Monitor Plugin screen. I'll look into this information. You should be getting all available JVM measures out of the box, the same way I got them.

        Please send me information directly at eugene.turetsky@dynatrace.com. 

        Eugene. 

  4. Anonymous (login to see details)

    I have a customer that wants to extract queue size numbers using JMX from Jboss AMQ. We can easily find that using the normal custom JMX measures, BUT the queue names changes every now and then. We would like to pull out automatically all of them,

    It seems like this solution can do that BUT that we have to first run the task to create the jar for the measures and would have to do that everytime the queue names change (meaning the key names), is that correct?

    1. Anonymous (login to see details)

      Hi Anders,

      This is correct: every time when you have new measures or renamed old measures the task plugin needs to be executed.

      Eugene.

  5. Anonymous (login to see details)

    Hi, Eugene, All

    I have a question: how this task plugin will work in multiple server / JVM environment ? 

    For an application there are 10 web logic servers and each server has a JVM running (total 10 JVMs)

    Assume we have to extract ‘JMS pending message count’ from each of these JVMs

     Scenario 1: Run JMX Enhanced Monitor plugin task (after selecting metrics):

                        This step has to be executed for each of the 10 Servers / JVM ?

                        One jar file will be generated for each of the Server / JVM, which have to be imported in Dynatrace server to run against a target server ?

                        Means plugin multiple jar files to be managed ?

       OR

    Scenario 2:  Run JMX Enhanced Monitor plugin task (after selecting metrics)

    1. JMX Enhanced Monitor plugin task to be executed only once for anyone of those 10 Server / JVM
    2. Import generated plugin jar
    3. Execute this jar for all the 10 Server / JVMs

    Thanks,

    Ankur

     

    1. Anonymous (login to see details)

      Hi Ankur,

      As of now, the JMX Enhanced Monitor plugin does not support Weblogic JMX servers. See list of supported JMX servers in the 'Configuration of the Task Plugin' section at the top of this page. We are planning to add support for Weblogic JMX servers as well as for WebSphere JMX servers but as of today this task has low priority.

      Eugene.

  6. Anonymous (login to see details)

    Hi Eugene,

     

    instead of Weblogic JMX server can this plugin be used for individual JVMs running on each of the weblogic servers ? 

    and if yes then can one generate plugin run against multiple JVMs (similar) on different server ? Or will have to create one plugin for each server ?

     

    thanks

    1. Anonymous (login to see details)

      Hi Ankur,

      The plugin should work with the Weblogic JVM, however, it is not recommended way of using it for Weblogic application server. You will need to configure separate instance of the plugin for individual JVM. 

      Eugene.

  7. Anonymous (login to see details)

    Can this be modified to support Kafka?

    1. Anonymous (login to see details)

  8. Anonymous (login to see details)

    Will Kafka work just by entering in the parameters included in this plugin or would this be a source code change to the plugin to accept Kafka?

    1. Anonymous (login to see details)

      Some code changes will be needed.

      1. Anonymous (login to see details)

        Is there additional documentation or information that can be provided for using this will Kafka?

        1. Anonymous (login to see details)

          Hi Heather,

          There is no documentation for the plugin's use with Kafka, however, plugin will work with Kafka when you use the Custom JMX Service URL option. The Custom JMX Service URL should be the same as you would use in the jconsole or VisualVM.

          Please let me know if you have any further questions. You can contact me directly at eugene.turetsky@dynatrace.com. 

           Eugene.

  9. Anonymous (login to see details)

    Hi, when trying to get JMX metrics from an ActiveMQ server we run into this error: setConfiguration method: Custom JMX URL is not implemented yet.

    Is there a plan to update the plugin with that seeing as there already is an option for it?

    1. Anonymous (login to see details)

      Hi Michael,

      Yes, we have plan to update the JMX Enhanced Monitor plugin. Please send me information about your customer at eugene.turetsky@dynatrace.com to prioritize update plugin request.

      Eugene. 

  10. Anonymous (login to see details)

    Hi Eugene,

    reading the comments and learning that the setup of the plugin only takes minutes I was excited to try the plugin on a  test environment. I followed the installation instructions to run the plugin as a task. But for some reason no jar is created in the desired directory.

    The logfile shows the following message:

     Expand source

    Do you have any idea how to work around the problem?

    René

     

    1. Anonymous (login to see details)

      Hi René,

      Please set log level of the plugin to FINER, reproduce your problem, and send me FINER log file at eugene.turetsky@dynatrace.com. Also, please provide the following information:

      • The DT Server log and the DT Collector log which hosts an instance of the JMX plugin;
      • The system profile of the JMX plugin instance;
      • Type and version of the JMX server that the JMX plugin is connecting to;
      • OS type and version that JMX server is running on.

      You are welcome to send me support archive which contains requested above information.

      Eugene.

  11. Anonymous (login to see details)

    Hello,

    Can I use this plugin to source and monitor ehcache statistics ?

    Thanks

    Rahul.

  12. Anonymous (login to see details)

    Hi Eugene Turetsky, we're currently trying to get this plugin setup for Redhat Fuse AMQ and are having some issues getting the Task plugin configured correctly. Is there anyone that can assist in correctly defining this Task plugin for AMQ?

  13. Anonymous (login to see details)

    Hi Eugene Turetsky,

    the plugin (version 1.0.10 on dynatrace 6.3.5.1050) is working fine with one exception: No matter what I set at "Maximum Number of Captured Metrics", the plugin processes 1000 metrics:

    execute method: total number of metrics = 2613, processed metrics = 1000, skipped metrics = 1613

    It looks as if the parameter is ignored.

    regards

      Oliver

  14. Anonymous (login to see details)

    Hi Oliver Günther,

    I'm going to have a look at this. Seems to be an easy bug to fix. I will notify once a new version is online.

    best regards,

    Reinhard

  15. Anonymous (login to see details)

    Oliver Günther, I checked out version 1.0.11 of this plugin and was not able to reproduce the issue.

    Would you be willing to upgrade to 1.0.11 and confirm, that the number of processed metrics is getting reported correctly?

    thanks,

    Reinhard

  16. Anonymous (login to see details)

    Hi all,

    We have disabled comments on this plugin page.

    Please use the AppMon & UEM Plugins forum for questions about this plugin.

    Sorry to interrupt ongoing discussions. Please re-post your last question in AppMon & UEM Plugins forum.