11 Jul 2022
11:03 PM
- last edited on
12 Jul 2022
08:30 AM
by
MaciejNeumann
Hi,
I have installed the operator(https://github.com/Dynatrace/dynatrace-operator) and then added metrics-ingest to the dynakube, but now, how can I use the metrics ingestion endpoint ?
I'd like to create custom metrics from my deployments, so from my pods I will call the OneAgent metrics api.
Should I call the ActiveGate service ? Should the http://localhost:14499/metrics/ingest be available on my pods ?
I'm stuck on what do now.
Hi, I'm gonna explain my use case:
I'm gonna use OpentelemetryCollector with dynatraceexporter(https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/dynatraceexport...)
So I need to configure the url when it is not possible to use the default value (localhost:14499)
I'm gonna talk to our devops to test that connection using nodeIp:14499.
I'm testing locally:
kubectl get node -o wide
NAME INTERNAL-IP
docker-desktop 192.168.65.4
Then I tried that inside a pod:
curl --data "alef.oneagent,cpu=1 55" http://192.168.65.4:14499/metrics/ingest \
-H "Content-Type: text/plain; charset=utf-8"
The response was Connection Refused.
I also tried that curl inside oneagent pod, and it worked!
What about to call the AG using the Ingest Metric Datapoint API?
I tried to call the SaaS and ActiveGate endpoints, but I needed to send the auth header and add the dynatrace attributes(dt.host, dt.entity, etc). manually 😞
Calling the oneagent endpoint all metrics are already tagged with those default attributes.
For example there are receivers on OpenTelemetry Collector(https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kafkametricsrec...) that I cant modify to add those attributes.
So, my first try was to use OneAgent metrics to centralize all metrics ingestion...
Since you are using OTel, did you check for this already?
https://www.dynatrace.com/support/help/shortlink/opentelemetry-metrics#send-metrics-directly-to-onea...
Yes, i'm using that "Zero configuration" strategy. But it sends the metrics to http://localhost:14499/metrics/ingest(OneAgent metrics api)
But the pod(otel collector) cant call the localhost:14499, so I was searching for a solution that I could call OneAgent APIs. I was planning to create a sidecard to redirect the http://localhost:14499/metrics/ingest to a that new endpoint.
But I dont know how to call the OneAgent Metrics api outside the oneagent pod.
Hi, @alefcarlos! Did you manage to solve the issue after all? We'd love to hear about your approach if you did 😉
Hi @alefcarlos where you able to solve it? I'm also stuck here.
We are pretty Python heavy and therefore need to use OTEL. I did activate the "metric-ingest" in the dynakube crd. The documentation here states: "opens the metrics ingest endpoint on the DynaKube ActiveGate and redirects all pods to it". It doesn't work.
Trying to call "curl -k http://localhost:14499/otlp/v1/traces" from inside of a pod will end with a connection refused. I mean from the pods perspective, localhost is its own one, not the active gate. I can send traces to the Dynatrace API directly but in this case we'd need to pay for DDUs on top of running OneAgent Operator in Full Stack Mode.
The documentation is unclear and spread all over the place for this topic.
Can someone please clarify: How I'm supposed to use OTEL in a Kubernetes environment with OneAgent Operator running in Full Stack Mode without wasting additional DDUs?
Many thanks in advance 🙂
@sdannewitz / @Ana_Kuzmenchuk We changed our strategy and now we are using prometheus to ingest our metrics.