How to use OneAgent Metrics API with Dynatrace Operator

[alefcarlos]

Hi,

I have installed the operator(https://github.com/Dynatrace/dynatrace-operator) and then added metrics-ingest to the dynakube, but now, how can I use the metrics ingestion endpoint ?

I'd like to create custom metrics from my deployments, so from my pods I will call the OneAgent metrics api.

Should I call the ActiveGate service ? Should the http://localhost:14499/metrics/ingest be available on my pods ?

I'm stuck on what do now.

[dannemca]

The 14499 port service is located at the node level, not at pod.

Did you tried to point to the node IP instead localhost and test?

[alefcarlos]

Hi, I'm gonna explain my use case:

I'm gonna use OpentelemetryCollector with dynatraceexporter(https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/dynatraceexport...)

So I need to configure the url when it is not possible to use the default value (localhost:14499)

I'm gonna talk to our devops to test that connection using nodeIp:14499.

[alefcarlos]

@dannemca 

I'm testing locally:

kubectl get node -o wide

NAME                               INTERNAL-IP
docker-desktop               192.168.65.4

Then I tried that inside a pod:

curl --data "alef.oneagent,cpu=1 55" http://192.168.65.4:14499/metrics/ingest \
     -H "Content-Type: text/plain; charset=utf-8"

The response was Connection Refused.

I also tried that curl inside oneagent pod, and it worked!

[dannemca]

What about to call the AG using the Ingest Metric Datapoint API?

[alefcarlos]

I tried to call the SaaS and ActiveGate endpoints, but I needed to send the auth header and add the dynatrace attributes(dt.host, dt.entity, etc). manually 😞

Calling the oneagent endpoint all metrics are already tagged with those default attributes.

For example there are receivers on OpenTelemetry Collector(https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kafkametricsrec...) that I cant modify to add those attributes.

So, my first try was to use OneAgent metrics to centralize all metrics ingestion...

[dannemca]

Since you are using OTel, did you check for this already?
https://www.dynatrace.com/support/help/shortlink/opentelemetry-metrics#send-metrics-directly-to-onea...

[alefcarlos]

Yes, i'm using that "Zero configuration" strategy. But it sends the metrics to http://localhost:14499/metrics/ingest(OneAgent metrics api)

But the pod(otel collector) cant call the localhost:14499, so I was searching for a solution that I could call OneAgent APIs. I was planning to create a sidecard to redirect the http://localhost:14499/metrics/ingest  to a that new endpoint.

But I dont know how to call the OneAgent Metrics api outside the oneagent pod.

[Ana_Kuzmenchuk]

Hi, @alefcarlos! Did you manage to solve the issue after all? We'd love to hear about your approach if you did 😉

[sdannewitz]

Hi @alefcarlos where you able to solve it? I'm also stuck here. 

We are pretty Python heavy and therefore need to use OTEL. I did activate the "metric-ingest" in the dynakube crd. The documentation here states: "opens the metrics ingest endpoint on the DynaKube ActiveGate and redirects all pods to it". It doesn't work. 

Trying to call "curl -k http://localhost:14499/otlp/v1/traces" from inside of a pod will end with a connection refused. I mean from the pods perspective, localhost is its own one, not the active gate. I can send traces to the Dynatrace API directly but in this case we'd need to pay for DDUs on top of running OneAgent Operator in Full Stack Mode. 

The documentation is unclear and spread all over the place for this topic. 

Can someone please clarify: How I'm supposed to use OTEL in a Kubernetes environment with OneAgent Operator running in Full Stack Mode without wasting additional DDUs?

Many thanks in advance 🙂 

[alefcarlos]

@sdannewitz / @Ana_Kuzmenchuk  We changed our strategy and now we are using prometheus to ingest our metrics.

https://www.dynatrace.com/support/help/platform-modules/infrastructure-monitoring/container-platform...