RabbitMQ metrics extension management/connection per host

[steven_v]

Hello,

We are utilizing the RabbitMQ (RMQ) Metrics Extension plugin in our SaaS environment for PCF. We have the RMQ nodes instrumented with Dynatrace and have access to those out-of-the-box metrics. We noticed that when we enable the RMQ Metrics Extension plugin (under Monitored Technologies) per RMQ node, we tend to lose the connection (or it looks like the connection is lost):

Is there any way we can:

1) monitor the status of the RMQ Metrics Extension on EACH RMQ node

2) automatically re-enable the RMQ Metrics Extension on EACH RMQ node

3) How to troubleshoot why the extension gets disabled/lost/disconnected

We are not enabling or plan on enabling RMQ Metrics Extension at the Global level to avoid impact to any RMQ services and/or apps using the services. We are practicing an "opt-in" model hence enabling the extension per RMQ NODE/Host. 

Hope that makes sense!

Thanks,

Steven

[bradley_danyo]

Hi Steven,

If I'm understanding your question correctly, you should be able to click on the use host configuration button from your screenshot to monitor each node.  You could also look at the extensions API for some automation.  Hopefully this helps.

Regards

[steven_v]

Hi Bradley,

So we do that and get the extended metrics, but for some reason, after our RMQ software version is upgraded, this is what we see when we open the RMQ Host Settings --> Monitored Technologies --> RabbitMQ

[bradley_danyo]

What version of RMQ are you using?  Also, what does it look like in the global settings?  If you go to Settings --> Monitored Technologies --> RabbitMQ?  It looks like you're using the global settings rather than the individual host settings.

[steven_v]

RMQ version: https://docs.pivotal.io/rabbitmq-cf/2-0/releases.html#2-0-6

Settings --> Monitored Technologies --> RMQ : 

you'll notice that we do NOT have it enabled at the Global Level Settings (hence the "250 RMQ hosts need configuration"). 

The screenshot earlier was from an individual RMQ host that we did have enabled at some point but no longer is connected. You can see that I am in the RMQ Host settings itself.

[bradley_danyo]

Ok thanks for the info.  It looks like you'll need to select the use host configuration button from your second screenshot and fill out the info below (user, password, port, etc.).  Can you show me what the RabbitMQ process running on that host looks like in Dynatrace?  Just a screenshot of the process overview page like this: 

[steven_v]

So we entered the creds using the Host --> monitoring overview --> RMQ Extension

Here's the process page:

[bradley_danyo]

Are these containerized?  There may be an extra level of configuration needed if the OneAgent is also running within a container but  its just a matter of setting up the necessary rules so the networking can go from OneAgentPlugin process within OneAgent container --Out To Host--> routing rule on host --back into app container--> Access to RabbitMQ.  You could try setting up some kind of routing table rule on the host to redirect from localhost:<anyPort#> to the specific IP and port of the container that is running RabbitMQ.

[steven_v]

Hi Bradley,

Since we are running in a virtual CF environment, these RabbitMQ VMs are often recreated and destroyed and do not come back up with the same VM-ID or IP, hence Dynatrace is not able to re-enable the Dynatrace RabbitMQ Extension. 

Normally the same IP address is re-assigned to the same VM (including a replacement VM).

However, recently there has been technology enhancement called "BOSH Hot Swap VM" where a new VM is created while the old VM is still running then once the new one is online a quick network re-routing take place to send traffic to the new VM before the old is deleted. In that special scenario then the IP address change because the new VM has to have its own IP address to make this works and therefore cannot be assigned the old IP.