Steps Required To Support Metrics Ingestion From Prometheus Server Residing in K8s Cluster

[ankurs]

I wanted to check the feasibility and possible way to fulfill this requirement.

What is the possible solution for ingesting metrics to Dynatrace from prometheus server residing in K8s OCP cluster? I am aware of Extension 2.0 (Prometheus Datasource).  can Extension 2.0 be used for this purpose? And while writing an extension is there any architectural point I should keep in mind? 

[MichalOlszewski · Accepted solution]

Hi @ankurs 

The only thing You need to do is expose prometheus endpoints to active gate You will use to monitor it with mentioned by You Extension 2.0. Prometheus can be configured to provide data in prometheus format thus Extensions 2.0 can be used to query its data just like prometheus datasource - we had a chance to do it for Spark data in prometheus  in the past.

Bests 
Michał

[ankurs]

Hello @MichalOlszewski ,

Thank you for the confirmation. One follow-up question, Is this ActiveGate located inside the K8s cluster or outside?

Thanks!

-Ankur

[MichalOlszewski]

Hi @ankurs ,

officially is better to have it outside, but You can modify config for containerized ActiveGate and it works as well, but some automation things doesn't work thus requiring manual steps.
Regards
Michał  

[ankurs]

ok. could you please confirm which module name I should choose exactly for Extension 2.0?  I couldn't find this in the official documentation. Even this sample dynakube yaml mentions only 4 modules.

[MichalOlszewski]

Hi,
It's not exactly that easy, cause like I said it is recommended to have it outside.
Officially having extensions on Containerized AG is not supported - please look https://community.dynatrace.com/t5/Extensions/Support-for-Extension-2-0-in-containerized-ActiveGate/...
You can force it and it works for most cases. I would have to find and revive my " playing with AG env" to check what parameter was allowing to do so - but it was something obvious like extensionmodule or alike. Still there were things like pushing Extension config which wasn't detecting ContainerizedAG as valid, so I had to push configs for that manually. Making it a little bit messy. Thus recommend using external AG and Commenting/Adding votes for DT support of EXT2.0 on Containerized ActiveGates rather that maintaining this half manual approach (that have to be redone every time AG updates) 

[ankurs]

Thanks, @MichalOlszewski for your detailed explanation.  

[Enrico_F · Accepted solution]

Actually, if you've already enabled monitoring of the Kubernetes cluster via the operator (see Connect your Kubernetes/OpenShift clusters to Dynatrace ) it's very easy to ingest gauge, counter and summary metrics from Prometheus exporters in a cloud-native way by simply annotating exporter pods and/or services:

Monitor Prometheus metrics | Dynatrace Docs

[mreider]

I'd like to refer people interested in Prometheus in Kubernetes clusters to this feedback thread for more input:

https://community.dynatrace.com/t5/Feedback-channel/Feedback-Channel-for-Prometheus-Monitoring-in-Ku...