How can I do overall OpenStack monitoring?

[kohei-saito]

Hi,

I'd like to monitor overall and comprehensive OpenStack environment, that is, what I want to do is monitoring all of Computer node, Controller node and VMs.

• My understanding:About monitoring of OpenStack VMs
  I understand that I need to install OneAgent into each VM.
  Then if I want to monitor application on a VM into code level as full-stack monitoring, I should restart application and after that I can do it.

• My understanding:About monitoring of Computer Nodes and Controller Nodes
  I understand that I need to build Security Gateway and modify some files of OpenStack and the dynatrace settings about OpenStack.
  I'm sure that this setting is related with OpenStack API.

• Question:Is it necessary to install OneAgent into Hosts?
  I'd like to monitor not only OpenStack VMs but all so OpenStack Hosts(=Computer Nodes&Controller Nodes).
  At this case, should we install OneAgent into OpenStack Hosts?
  Is it possible to capture OpenStack Hosts performance without each Agent ?

I don't understand "OpenStack" itself well, so please tell me if you find some mistakes in this article.

Kohei

[Bartek_Gatz · Accepted solution]

Hi Kohei.

These are really good questions. Thank you for asking them.

Answering them one by one:

• You are correct. In order to monitor the VMs managed by OpenStack, you need to deploy OneAgents into them. The VMs will then be monitored as if they were regular hosts, so all the detection and instrumentation will happen automatically.
• In this question you are referring to the OpenStack infrastructure monitoring feature set, which is currently available as EAP to selected customers. In case other readers wondered: this is not a functionality which is publicly available today. On a side note: we are approaching the beta release and it will soon be announced on our blog post site. Answering your question: the Compute and Controller nodes are best monitored by a combination of Dynatrace OneAgents and Security Gateway. Security Gateway accesses the OpenStack remote APIs, while OneAgents provide additional monitoring of host's metrics and OpenStack behavior. In order to gain the full visibility into the OpenStack's infrastructure it is important to instrument it both with Security Gateway and OneAgents. The configuration steps require adding respective permission roles on OpenStack side, so that Dynatrace could talk to it (or alternatively - it is also possible to use full-access OpenStack admin account). Once OpenStack is configured properly, one needs to configure Security Gateway to communicate with the OpenStack API and deploy OneAgents on Controller and Compute nodes. It is also recommended to configure monitoring of RabbitMQ, MySQL and HAProxy - the services used under the hood by OpenStack.
• If you decided to not instrument Compute and Controller nodes with OneAgents, the visibility into OpenStack health will not be complete. As part of OpenStack monitoring support we (will) provide a set of monitoring dashboards and screens: listing OpenStack services, and their resource utilization metrics, dynamics of VirtualMachines, OpenStack project utilization, but also insight into Storage and Network services of OpenStack and more. Without the OneAgents on Compute and Controller nodes those screens will be missing most of the data. Also we will not be able to generate the events for Glance, Keystone and Neutron services.

To sum up, it is recommended to use Security Gateway, as well as deploy OneAgents on both the VMs and OpenStack hosts.

Regards,

Bartek.

[kohei-saito]

@Bartosz G.

Hi Bartek,

thank you very much for your detailed answer!

OK, I understand the overview of necessary installation as below.

• To monitor VMs managed by OpenStack:
  We need to install OneAgent into each VM. And then VMs will be monitored automatically and application should be restarted to be monitored if needed.
• To monitor fully OpenStack Infrastructure(Compute nodes&Controller nodes):
  In fact, it is not always necessary to install OneAgents into OpenStack Infrastructure but it is better to do so with the configuration of Security Gateway for using OpenStack APIs.
• About dynatrace data we will miss If we DON'T install OneAgent into OpenStack hosts:
  e.g.) especially Glance, Keystone and Neutron services, for more detailed, at least as belows.
  1. listing OpenStack services
  2. their resource utilization metrics
  3. dynamics of VirtualMachines
  4. OpenStack project utilization
  5. Storage and Network services of OpenStack
  and more.
• For now, the OpenStack monitoring function of dynatrace is opened to only selected customers as EAP.
  (Thank you for following up with my question and environment!)

In addition, I'd like to ask more things about licencing and consumption.

1. Is it necessary to purchase both of licences for VMs and OpenStack hosts respectively when we monitor both of them?

dynatrace automatically recognize and start to monitor each hosts that are installed OneAgent into and need some licences per host.

Of course, we can choose to enable/disable monitoring on specific hosts/application/service, but I assume we basically need licence per host, so it is necessary to purchase licences for VMs and OpenStack hosts respectively.

2. If we install OneAgent into not only each VM but also Compute nodes and Controller nodes, how do we consume licences?

I think that dynatrace have two types of licences;
One of them is "Full-stack" monitoring and the other is "Cloud-infrastructure" monitoring.

I assume that we need "Cloud-infrastructure" licence when we monitor Compute nodes and Controller nodes, and on the other hand, we need "Full-Stack" licence for monitoring of VMs managed by OpenStack.

Regards,

Kohei Saito

[Bartek_Gatz]

Hi Kohei.

Answering your further questions:

1. You are correct. There are no exceptions to the licensing rules in case of OpenStack. So in this scenario you need to consider both the host units from VMs as well as host units from Compute and Controller nodes.
2. For Compute and Controller nodes it is enough to use the OneAgent in "Cloud Infrastructure Only" mode. It will then be able to monitor and report on OpenStack infrastructure fully. Having said that, there is a theoretical scenario possible, where in addition to running the OpenStack services and VMs you might want to have additional processes deployed directly onto the Compute nodes. In this case, if you wanted to monitor them with deep code monitoring modules of OneAgent (for Java, Apache, etc.) you will need to have a full-stack OneAgent deployed on those Compute nodes. I have deliberately used the term "theoretical scenario", because I believe such a setup will be breaking the good practices of OpenStack deployment itself (meshed environments where the Compute nodes' resources are used by non-OpenStack-controlled processes are introducing unwanted non-determinism in resource management). In case of VMs you will probably prefer to use OneAgent operating in full-stack mode, because of the fact that the VMs will in fact be hosting all the services which should be monitored with the deep code injection.

Please let me know if you have other questions or comments.

Kind regards,

Bartek.

[kohei-saito]

Hi Bartek,

Thank you for your very kind answers!

Thanks to you, I now have a deeper understanding of OpenStack monitoring!
I'm going to make a trial on OpenStack monitoring in a large environment with dynatrace in the near future.

I care about additional processes for OpenStack on Compute and Controller nodes, too.

Thanks!

Best Regards,

Kohei

[Babar_Qayyum]

Hello @Bartek_Gatz & @kohei-saito 

The OpenStack/OpenShift administrators facing issues with the podman commands when the container monitoring enabled for the CRI-O containers.

If we do not enable CRI-O containers then there is no code-level information for the OpenShift/Kubernetes containers.

What could be the reason behind this issue?

Regards,

Babar

[Julius_Loman]

Hello @Babar_Qayyum ,
as far as I know podman in general is not supported (in terms of monitoring podman containers). But I think you mention your admins have issues with the podman commands itself. I believe in this case you may want to try disabling monitoring of podman processes by defining your own process monitoring rules.

Julius

[Babar_Qayyum]

Hello @Julius_Loman 

There are the following built-in container monitoring rules already applied. 

Also, the following rules applied process group monitoring.

One more rule in the container monitoring rules.

Still the same issue.

Regards,

Babar

[Julius_Loman]

@Babar_Qayyum  I don't know your situation, but I think you have your podman rules wrong. AFAIK the podman is a command, not a kubernetes container name and based on your description you have problem with the podman command itself. Thus you should have the rule not to monitor processes where EXE name equals podman.

Maybe you should consider opening a support ticket.

[Babar_Qayyum]

Hello @Julius_Loman 

I included the following rule as well now. Will check with the admins and update you.

Regards,

Babar

[Babar_Qayyum]

Hello @Julius_Loman 

After implementing all these rules, the admins are still not able to execute the podman commands when the CRI-O container is enalbed.

Regards,

Babar