Icon

Comments have been closed on this page. Please use AppMon & UEM Open Q & A forum for questions about this plugin.

Overview

This monitor collects message flow statistics by connecting to the MQ Manager.

Plugin Details

Plug-In Versions

MessageFlowStatistics_2.0.0.jar

MessageFlowStatistics_2.0.1.jar

MessageFlowStatistics_2.0.2.jar

MessageFlowStatistics_2.0.3.jar

dynaTrace Versions

>= 5.5

Author

Asad Ali (asad.ali@dynatrace.com)

License

dynaTrace BSD

Support

Not Supported

Known Problems

 

Release History

2013-03-05 Initial Release

Provided Measures

The following statistics are collected for each message flow:

  • TOTAL_ELAPSED_TIME
  • ELAPSED_TIME_PER_MESSAGE
  • MAXIMUM_ELAPSED_TIME
  • MINIMUM_ELAPSED_TIME
  • TOTAL_CPU_TIME
  • CPU_TIME_PER_MESSAGE
  • MAXIMUM_CPU_TIME
  • MINIMUM_CPU_TIME
  • CPU_TIME_WAITING_FOR_INPUT_MESSAGE
  • ELAPSED_TIME_WAITING_FOR_INPUT_MESSAGE
  • TOTAL_INPUT_MESSAGES
  • TOTAL_SIZE_OF_INPUT_MESSAGES
  • MAX_SIZE_OF_INPUT_MESSAGES
  • MIN_SIZE_OF_INPUT_MESSAGES
  • NUM_THREADS_IN_POOL
  • TIMES_MAX_THREADS_REACHED
  • TOTAL_NUM_MQ_ERRORS
  • TOTAL_NUMBER_OF_MESSAGES_WITH_ERRRORS
  • TOTAL_NUMBER_OF_ERRORS_PROCESSING_MESSAGES
  • TOTAL_NUMBER_OF_TIMEOUTS_WAITING_FOR_REPLIES_TO_AGGREGATE_MESSAGES
  • TOTAL_NUMBER_OF_BACKOUTS
  • TOTAL_NUMBER_OF_COMMITS

Configuration

The plugin can be run on any collector. In order for this plugin to work, ensure that SYSTEM.ADMIN.SVRCONN type channel is created on the MQ Manager. This is the channel that the MQ Manager would publish the statistics information.

For this monitor to work, ensure that the message flow statistics are enabled. To enable it, run the following command on the broker:

Restart the broker after executing the above command.

For more information, go to
http://www-01.ibm.com/software/integration/wbimessagebroker/library

When configuring the monitor to run in the dynaTrace environment, use the following value for the entry Stats Subscription Topic:

This monitor works for IBM Message Broker 7.0 and higher. Additionally, for this monitor to work in dynaTrace, version 4.2.0.3170 and higher is required.

Installation

Import the Plugin into the dynaTrace Server. For details how to do this please refer to the dynaTrace documentation.

  1. Anonymous (login to see details)

     

    Does anyone have any good dashboards that can be used with this Monitor Plugin?

  2. Anonymous (login to see details)

    Just this one from the demo environment.

  3. Anonymous (login to see details)

    Has anyone encountered this error when using the plugin.

    +-----------------------------------------------------------------------------

    + dynaTrace Collector Copyright (C) 2004-2013 Compuware Corporation

    +-----------------------------------------------------------------------------

    + Version 5.5.0.5562 built Tue Nov 19 06:39:13 EST 2013

    + Platform: Windows Server 2008 R2 6.1, amd64 64bit

    + vm: Java HotSpot(TM) 64-Bit Server VM 1.6.0_45, Sun Microsystems Inc., mixed mode

    2014-09-03 17:00:28 SEVERE [MessageFlowStatisticsMonitor] 2033 error occured. This means that no data is being published on the channel.

    2014-09-03 17:00:48 SEVERE [MessageFlowStatisticsMonitor] 2033 error occured. This means that no data is being published on the channel.

    2014-09-03 17:01:08 SEVERE [MessageFlowStatisticsMonitor] 2033 error occured. This means that no data is being published on the channel.

    2014-09-03 17:01:28 SEVERE [MessageFlowStatisticsMonitor] 2033 error occured. This means that no data is being published on the channel.

    2014-09-03 17:01:48 SEVERE [MessageFlowStatisticsMonitor] 2033 error occured. This means that no data is being published on the channel.

  4. Anonymous (login to see details)

    Cannot download the plugin, can you help me

     

  5. Anonymous (login to see details)

  6. Anonymous (login to see details)

    Thanks, it was established the download.

  7. Anonymous (login to see details)

    does anyone know how to change the message flow statistics messages published to the queue to have an expiry?  The tool connected as an exclusive consumer and was not deleting the messages from the queue - I had to shut it off to prevent it from filling the disk space.  We are MQ 7.0.1.7 and MB 7.0.0.6

    1. Anonymous (login to see details)

      Hi Mike,

      The issue is that the client isn't destructively reading the stats messages so we really need a fix for that rather than setting the message expiry. Presently any message expiry would be set by the message producer - in this case that would be the broker when it generates the XML statistics message. There is a thread further down discussing the issue of the messages being left on the durable-consumer queue.

      Martin

  8. Anonymous (login to see details)

    Asking Mike's question again to see if there are any suggestions.

     

    does anyone know how to change the message flow statistics messages published to the queue to have an expiry?  The tool connected as an exclusive consumer and was not deleting the messages from the queue - I had to shut it off to prevent it from filling the disk space.  We are MQ 7.0.1.7 and MB 7.0.0.6

    1. Anonymous (login to see details)

      Hi John,

      See my response to Mike's question, hoping to work with Asad to get a resolution.

      Martin

  9. Anonymous (login to see details)

    John

    I looked at the expiry option and could not find any. I am not sure whether it is setting on your side that is causing the queue to fill up. I have 5-6 customers that are using this plugin without the queue filling up.

    1. Anonymous (login to see details)

      We have been unsuccessful in getting this to work without filling up the queue.  Would any of your customers be willing to have a call with us to clarify the needed settings?

    2. Anonymous (login to see details)

      Message expiry is set when sending a messages - in this case it would require the broker to set it. There has been some discussion with IBM about being able to set a default expiry at the queue level. I found this RFE but the detail seems to indicate z/OS.

      RFE: https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=21984

      Fix detail: http://www-01.ibm.com/support/docview.wss?uid=swg1PI50761

      I'm going to reach out to my IBM rep to see if it's available on other platforms but we need to fix the client so it does a destructive read rather than rely on message expiry.

      1. Anonymous (login to see details)

        Thanks for working on this Martin.  I see you are also in contact with Colby from our side - he'll be able to help you with any testing you want to do.

        1. Anonymous (login to see details)

          Hi Mike,

          IBM have told me that the functionality to set message expiry at the queue level is available on distributed platforms from v8.0.0.4 onwards but as stated before we need to fix the issue with the plugin as this would just be a workaround. That said - I do think the message expiry would be useful if we had a durable subscriber that didn't reconnect for a long period of time, presently the plugin creates a non-durable subscription so the queue manager won't store messages when the plugin is disconnected. I'm going to open a new discussion on that topic shortly as creating a durable subscriber would mean the stats will be kept when the plugin is disconnected for maintenance etc. this could be added as an option.

          Regards,

          Martin

        2. Anonymous (login to see details)

          Hi Mike,

          Asad has provided a fix for the messages being left on the durable queue in v2.0.3

          Martin

  10. Anonymous (login to see details)

    Here is what we are seeing. When using this monitor it will start to use a queue like SYSTEM.MANAGED.NDURABLE.55CA156020004825. This queue depth will keep rising for as long as the monitor is started. Also if you look under the /var/mqm/qmgrs/<QMGRNAME>/queues it creates a ghost queue that gets larger with every polling cycle until it fills up the file system and crashes the QMGR. 

    Another issue is that even when it is suspended, it stays on the channel which is probably so that it won't miss any subscriptions while it is down. If that is the case it acts like a non-durable sub but is really a durable one. 

    A better way to solve this may be to change the way the code is from a PUB/SUB type and just have the admin create the subscription and send it to a predefined local queue. Then your monitor can just read from the queue like any normal MQ application. 

    Hope this helps. 

    1. Anonymous (login to see details)

      Was there a fix for messages being left on the non-durable topic/underlying queue? I'm seeing the same issue.

      1. Anonymous (login to see details)

        Nobody has really looked at it since I guess I was the only one with the issue. I would be nice to get this fixed as it is unusable in the current state, 

        1. Anonymous (login to see details)

          Guys

          I am going to look at this plugin again. If I need to do a webex to understand the misbehavior of the plugin, is that possible?

          1. Anonymous (login to see details)

            I would be able to do one next week if you need to see it. 

          2. Anonymous (login to see details)

            Hi Asad, I probably won't be able to host a webex but happy to help where I can, I'll also try to get some time to look at the code.

            1. Anonymous (login to see details)

              Hi Asad,

               

              What are the get options?

              You haven't used the constants so it's hard to know what they are:

              MQGetMessageOptions gmo = new MQGetMessageOptions();

              gmo.options = 1;  <-- this line is redundant

              gmo.waitInterval = 20000;

              gmo.options = 33; <-- what does 33 do?

               

              Martin

               

          3. Anonymous (login to see details)

            Hi Asad,

            I have created a possible solution/idea that might shed some light, I replaced the actual integer values for the open and get options in your code (see updated code snippet below) with what I think are the IBM constants you are using; it will greatly improve the readability and supportability of the code if you use the constants. So if I have understood the options correctly - I don't think you destructively read the message after browsing.

            I think you need to do a subsequent read with this option:

            MQGMO_MSG_UNDER_CURSOR

            Get message under browse cursor.  

            This option causes the message pointed to by the browse cursor to be retrieved, regardless of the MQMO_* options specified in the MatchOptions field in MQGMO. The message is removed from the queue.  

            The message pointed to by the browse cursor is the one that was last retrieved using either the MQGMO_BROWSE_FIRST or the MQGMO_BROWSE_NEXT option.

             

            Suggestion:

            1. Anonymous (login to see details)

              Let me make these changes and send the updated plugin soon to you.

    2. Anonymous (login to see details)

      Hi Colby,

      Asad has provided a fix for the messages being left on the durable queue in v2.0.3

      Martin

  11. Anonymous (login to see details)

    Hey Asad,

    Do you know if this plugin supports OD400 queues on AS/400's?  When trying to implement on a AS/400 queue, we receive the following message in the log, showing that the dspmq command may be incorrect. Speaking with the AS/400 individual at my client, he stated it should be the dspmqm command instead.

    2015-12-07 15:51:10 SEVERE [MQMonitorPlugin@Websphere MQ Monitor - AS400_0] General error : Command "/qsys.lib/qmq.lib/dspmq " did not produce any output
    2015-12-07 15:55:25 SEVERE [MQMonitorPlugin@Websphere MQ Monitor - AS400_0] General error : Command "/qsys/qmq/dspmq " did not produce any output

    Determination of the AS/400 support will help us to plan the next action items for implementing this plugin whether it be to alter the plugin source code or to determine the issues with the plugin configuration.  Therefore, please let us know your thoughts on the AS/400 MQ support.

    Thanks,

     Josh

    1. Anonymous (login to see details)

      Which version of the plugin are you using? In this plugin, I am not making any calls to dspmq. An older version of this plugin would use dspmq but the latest version does not?

      1. Anonymous (login to see details)

        Hey Asad,

        We are using version 2.0.0 of the plugin which appears to be the most current version.  Perhaps the plugin is still trying to call dspmq due to the AS/400 interface or an incorrect log message is being printed?  

        I turned on Finer logging and it unfortunately it did not give too much additional information:

        2015-12-07 15:51:10 SEVERE [MQMonitorPlugin@Websphere MQ Monitor - AS400_0] General error : Command "/qsys.lib/qmq.lib/dspmq " did not produce any output
        2015-12-07 15:55:25 SEVERE [MQMonitorPlugin@Websphere MQ Monitor - AS400_0] General error : Command "/qsys/qmq/dspmq " did not produce any output
        2015-12-09 08:13:06 INFO [SSHConnectionMethod@Websphere MQ Monitor - AS400_0] Connection seems to be down, trying to reconnect..
        2015-12-09 08:13:09 SEVERE [MQMonitorPlugin@Websphere MQ Monitor - AS400_0] General error : Command "/qsys/qmq/dspmq " did not produce any output

        1. Anonymous (login to see details)

          Hey Asad,

          Any input on the above question?  We are hoping to get this plugin implemented within the next few days and would like to determine if the plugin is compatible with the AS/400.

          Thanks,

          Josh

  12. Anonymous (login to see details)

    Josh,

    The library is also wrong as it should be qmqm.lib instead of qmq.lib

  13. Anonymous (login to see details)

    Thanks Colby!  Will update the monitor configuration to include the correct library.

  14. Anonymous (login to see details)

    Asad -

    Just wanted to commend you on this plugin.  I implemented it today.  It's very easy to implement, the data is valuable, and our middleware admins really like it!

    Kudos!

  15. Anonymous (login to see details)

    Joel

    I am glad it is working out for you. Happy to help.

    Asad

  16. Anonymous (login to see details)

    Archive vs Snapshot accounting and statistics collection

    Hi Asad,

    I was thinking that it might be worth updating the installation notes to include instructions for using 'archive' mode as well as 'snapshot' mode for those that don't need 20second granularity. We are currently testing with 'archive' mode and the default broker stats interval of 60 minutes and it seems to be working out fine. I intend to alter the broker stats interval to either 5 or 10 minutes shortly to have more granularity but I don't need 20 seconds and I suspect a 5 or 10 minute interval will require less system resources.

    Martin

  17. Anonymous (login to see details)

    When configuring the series properties the unit for anything relating to 'message size' should be selectable in B, KB, MB or GB instead of it being just a number. Can this be modified?

    1. Anonymous (login to see details)

      Martin

      I will modify the plugin to show the data in B instead of a number. You can always choose to change it from B to KB,MB etc. when you plot it.

  18. Anonymous (login to see details)

    Asad,

    I used the new plugin 2.0.3 and it corrected the ghost queue issue, but brought up another issue.If I create more then one monitor to a different brokers, it will break the previous monitor. So it looks like only one Broker can be defined at a time per Dynatrace Profile. Can you take a look at this?

    1. Anonymous (login to see details)

      Hi Asad,

      I'm seeing the same behaviour, have you had a chance to look into it?

      Martin

  19. Anonymous (login to see details)

    Hi Asad, any idea when you might be able to look into the issue of multiple brokers?

  20. Anonymous (login to see details)

    Hello,
    I made some changes to the plugin to allow us to connect with more than one Broker.

    It is not fully tested yet (need to confirm return values), but if anyone would like to try it:
    https://github.com/smc87/dynaTrace-MessageFlowMonitor/releases/tag/v2.0.4-beta

     

     

     

     

    1. Anonymous (login to see details)

      Sean,

      I tested this out and I still see the same behavior. 

      I have 2 brokers on 2 different systems. When I started the second one, that is the only data that now shows up. The strange part is I have 2 monitors setup (broker1 and broker2). When I do a split by message flows, I see the flow that is on broker2 under the broker1 monitor. It is also under the broker2 monitor as well. I wouldn't expect these to cross monitors since they are on separate systems. 

      1. Anonymous (login to see details)

        Hi, Thanks for testing. Shame this is still the result. 

        I'll try and reproduce/fix with the help of our broker team. 

         

  21. Anonymous (login to see details)

    I took a look at our environment. We are definitely getting different flows for each monitor, and from those flows we are also showing different data.

    Can you attach some logs (preferably at finest log level).

     

    Thanks,

    Sean.

  22. Anonymous (login to see details)

    Sean, 

    I am doing some more test with removing all monitors and recreate them all to see if that matters at all. Can you send me a note on how your environment is setup?

    how many systems/flows/qmgrs and how your monitors are setup?

    I will try to replicate this in QA where I have more control and see what logs I can provide you.

    1. Anonymous (login to see details)

      ahh yes, it is probably worth making sure you remove all existing measure first to make sure they weren't created by old versions! good thought!

      I'm currently connected to 5 Queue managers (each on a seperate host) most have hundreds of flows.

       

      I have been able to confirm that flows unique to specfic queue managers are not appearing elsewhere for me. 

      Good luck, let me know how you get on! 

       

       

  23. Anonymous (login to see details)

    Sean,

    I did the same steps in the Non-Prod Profile, Dynatrace system and it is working as expected.

    Steps: No monitors were configured from the start.

    Created 1 monitor, turned on broker stats, turned on monitor and graphed it on a chart.

    After 10 minutes I created 2nd monitor with another broker and graphed it out. 

    Dynatrace versions: PROD: 6.2.8.1019     NonProd: 6.2.14.1009

    Next steps:

    In Production I am going to delete all monitors from this plugin and leave them that way until tomorrow to make sure things are cleared out. Tomorrow I will then recreate the monitors one by one and see if I get results as I see in QA. The only other thing I could think of which I can't see being an issue, is I already have a dashboard for all these results, would that in anyway have an effect? When testing I use a new chart to ensure the data.

    A side enhancement:

    Broker stats based on time are in microseconds and not milliseconds. This can be adjusted by using the scale once you create your charts, but for someone who doesn't know that they may think the plugin is not giving accurate results vs Broker Explorer. Is that something that can be converted before the metrics are shown so we don't have to scale it?

    1. Anonymous (login to see details)

      Just to double check. Did you mean nanoseconds? 

      There doesn't seem to be an option for microseconds? 

      1. Anonymous (login to see details)

        I know Dynatrace doesn't have an option for microseconds, but that is what the broker statistics provide. To get around this currently I use a scale of 0.0001 which makes it close to milliseconds. That is why I figured the plugin code itself would have to do the conversion before it hits the collector. If it is not doable, maybe just update the usage notes at the top so people are aware.

        Link to article on statistics in microseconds. 

        https://www.ibm.com/support/knowledgecenter/SSRLD6_7.3.0/kqi_userguide/attr_kqitacmf.html

        1. Anonymous (login to see details)

          ahh ok, makes perfect sense.

          I'll look into it. 

  24. Anonymous (login to see details)

    Great glad to hear it is working as expected. I cant forsee any issue with the different minor version of dT. I think its more likely that stale data from a previous version of the plugin is confusing things. Hopefully your recreation of monitors in prod will prove this. I dont think a dashboard should cause any issues they are just xml as far as i know. 

    Interested to hear if you do get it working. 

    I will take a look at the time unit.

    1. Anonymous (login to see details)

      Sean,

      I recreated it today and got the same results. One thing I did notice is that when you stop or delete the monitor, it does not get off the channel and leaves it's subscription out there as well. I think this needs to be handled to get it off the QMGR when the monitor is stopped or removed.

      With that I stopped the channel to force it off and remove all the subscriptions it had for me recreating it. It then started back with only one subscription shown but the behavior is still the same. The first monitor was started and showed his flows. As soon as the second monitor was started it showed it's flows, but the first monitor started showing the second monitors flows as well and the one valid flow no longer showed data.

      I am out of idea's on this so I am going to delete them both and then leave them down for a week or so and try this all again. I am out most of next week so it will be a while before I can do this again. In the mean time please look into getting the monitor off the channel when it is stopped/removed.

      Thanks

      1. Anonymous (login to see details)

        FYI we are going to look closer at the flow duplication that you mention to 100% confirm one way or the other if we are seeing the same issue. 

  25. Anonymous (login to see details)

    Oh thats a shame. We also noted the subscription being left in place but had not noticed this impacting us in anyway with regards to valid stat collection.
    I'm not sure how much free time I will have to look into how to close the connection gracefully. If i get a chance I will. 

    Have you tried recreating with completely new monitor names to ensure that everything is created from scratch? I'm not sure how to explain it working in PVT but not in Prod for you.  

    I have it working fine on 6.2.7.1063 (EDIT - upon further discussion we've decided we need to look more closely at the stats)

    1. Anonymous (login to see details)

      We created the monitors with different names and that shows the same behavior (bad). We then recreated the monitors and ran them manually and did not run on a schedule and they acted like we would expect (good). After a couple manual runs, we turned them back on a schedule and it broke again.

      For Non-Prod which was working the past 3 days it was running on an Agent collector which was showing the positive results. It was then changed over to use the Monitor collector and it failed. I then turned it back over to the agent collector and it started working again. So the root cause seems to be the collector. 

      Is there any known issues with that plugin overloading the collector?

      1. Anonymous (login to see details)

        Hi Colby,
        Thanks for your extensive testing and description of the issue, it has been very helpful. 

        Could you please test/double check the returned values across multiple brokers from this version:
        https://github.com/smc87/dynaTrace-MessageFlowMonitor/releases/tag/v2.0.5-beta

        Cheers,
        Sean. 

         

        p.s when you ran manually did you run all at once from the top level MQFlow monitor, or did you run each individually at a slightly different time? If the latter then I think it would support the theory I was running with that multithreading has been the issue (since thread concurrency issue wont be present if running manually at seperate times - or in theory if you have different non-clashing schedules) - either way, please try the new version. (smile) 

        1. Anonymous (login to see details)

          Hi Sean,

          Sorry for the delay. Version 2.0.5 is now working in production when we use the Agent collector instead of the monitor collector. I am not really sure why that is, but that was the last issue. So currently I have multiple brokers monitored on both test and production dynatrace. 

           

          1. Anonymous (login to see details)

            Hi Colby,
            v2.0.5 looks to fix the issue where flows were appearing in unexpected places.

            wrt your other point,...I'm not 100% sure I know what an agent/monitor collector is, are they just different instances of the same thing (a collector)?
            All of our collectors run on dedicated servers, which the agents then connect to, then we just create multiple instances of the collectors if required.

            Have you tried using another one of you monitor collectors, or perhaps tried creating a brand new one? Maybe there is an underlying issue with the collector instance? Is it only this plugin that fails on that instance? Has it been restarted? Anything interesting in the logs at fine or finer levels? 

            Thanks,
            Sean.

            1. Anonymous (login to see details)

              Sean,

              I am working with Colby, V2.0.5 fixed the issue.  The monitors are running a collector that is dedicated to run monitors and does not collect agent information.

               

            2. Anonymous (login to see details)

              Looks like we are good Sean, no outstanding issues with the stability of the plug-in. Thanks for all your help getting it to this point.

              1. Anonymous (login to see details)

                No problem, glad someone else found this bit of work helpful. (smile)


                wrt to your remaining issue with the "monitor collector" I think it's worth trying some of the steps above like restarts / log checks, although if fruitless, i'm pretty sure a new instance of your monitor collector would be the quickest route to resolution (if acceptable). Depending on if you want to do the digging into the root cause of the current issue. 

                 

            3. Anonymous (login to see details)

              Hi Sean,

              since version 2.0.5 is now working. Can you please merge it back into the Dynatrace GitHub-repo (https://github.com/Dynatrace/Dynatrace-Message-Flow-Statistics-Monitoring-Plugin) ?

              Thanks

              Ingo

               

               

              1. Anonymous (login to see details)

                Hi Ingo,
                Sure, I'll review the diffs when I get a chance.
                This could be a little while though.

  26. Anonymous (login to see details)

    Hi,

    Can I monitor Sonic Server (Sonic Client, Domain Manager and Sonic Broker) end to end using this plug-in. I suppose we don not have agents other than web sphere message broker. Request you to please confirm.

     

  27. Anonymous (login to see details)

    Hello Everyone, 

    Does anyone have any detail on the function of the Authenticate Boolean parameter?  

    1. Anonymous (login to see details)

      Carl

      The Authenticate boolean lets you define whether the MQ needs authentication to pull statistics data from it or not. If you enable this option, you have to provide userId and password to authenticate with MQ.

      1. Anonymous (login to see details)

        Superb, thank you for the quick reply Asad!