cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Pushing custom metrics to Dynatrace using One agent extension

Mayank_Sh
Participant

In my use case, I am retrieving metrics from endpoint and trying to push to Dynatrace SaaS using One agent SDK (ie.. plugin.py and plugin.json).

 

1. Plugin.json:

- As endpoint contains varied kind of metrics. So, I considered hardcoding each of the metrics under metrics sections along with their respective dimensions.

- I have not used config-ui and ui section.

- For all other fields, kindly have a look at attached screenshot.

 

Plugin_json.png

Similarly, there are 40 more metrics.

 

 

2. Plugin.py:

I am retrieving metrics from endpoint using prometheus_client.parser :

from prometheus_client.parser import text_string_to_metric_families

 

For transformation of metrics in Dynatrace-supported format , I am using this part of code:

Metric_insertion.png

- "metric-server" I have kept as Process group for now. But, not sure whether it is right or wrong ?

 

For pushing metrics to Dynatrace , this is used:

 

plugin_dimension.png

Issues faced by :

- On uploading of SDK in Dynatrace SaaS, though dimensions are being created on Dynatrace with respective metric name but, data is not coming.

- Is there any alternative way from hardcoding metrics in plugin.json and keep it more dynamic ?

 

5 REPLIES 5

Mike_L
Dynatrace Champion
Dynatrace Champion

Hi,

 

if I were you I’d definitely use extension framework 2.0 instead. It comes with a declarative framework for Prometheus so you don’t have to write any code.

 

You can find more details here:

https://www.dynatrace.com/support/help/extend-dynatrace/extensions20/data-sources/prometheus-extensi...

Mike

Thanks for reply Mike,

 

 

We are looking into the extension framework 2.0. But, meanwhile, can you tell me any remedy associated with the above followed procedure to get metrics in Dynatrace ?

 

Also, to avoid hardcoding of plugin.json file.

 

Thanks.

@Mayank_Sh  Extensions 1.0 require metrics specified in the plugin.json. The only way to get around it is to send the metrics using by calling the Metric API from the code instead of the SDK methods. The ingest data point Metric API
You can use the Python API client for that. So you will use the extension code just to fetch and process the metric, but the sending will be using the API. Not ideal, but it will work.

If it's only prometheus as the source, definitely use Extension 2.0 for that.

Certified Dynatrace Master | Alanata a.s., Slovakia, Dynatrace Master Partner

Mayank_Sh
Participant

Hi @Mike_L  and @Julius_Loman  , Thanks for reply.

 

We are trying Prometheus extension 2.0 now.

 

We are monitoring an application which has been deployed on Kubernetes cluster(GKE). We have oneagent running as a container inside the pods. As a part of extension 2.0 process , we are required to upload root certificate(ca.pem) in each Oneagent host. 

 

 

Challenge which are being faced:

 

  1. We are not able to copy root certificate file inside the pod.
    • Command used by us iskubectl cp ca.pem dynatrace-oneagent-bk6ns:/var/lib/dynatrace/oneagent/agent/config/certificates -n dynatrace
    • Error observed is :error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "25efd997d58778142223f246292fd80171dfc99aea1a3531c9ded878ddba9380": OCI runtime exec failed: exec failed: unable to start container process: exec: "tar": executable file not found in $PATH: unknown
  2. Even though we resolve the first issue, if pod is getting restarted then, we will lose root certificate again because pod is not persisting data. What is the remedy for it ?

Also, we are not allowed to create custom OneAgent image.

 

 

Mike_L
Dynatrace Champion
Dynatrace Champion

The remedy from our services team would be to sign it with the Dynatrace cert, but that's not possible for you of course. Looking at the error there are some solutions described here: https://stackoverflow.com/questions/67087858/why-does-kubectl-cp-command-terminates-with-exit-code-1...