Dynatrace support for prometheus histograms

[janne_olkoniemi]

We are currently trying to enrich our Kubernetes visibility with OTEL collector since the default Dynatrace visibility lacks many important aspects how the Kubernetes itself is behaving. 

We are currently setting up the kube-apiserver, kubelet and kube-proxy monitoring via otel collector as example. 
The challenge we are facing is that many of the metrics we would be interested about are discarded by the /api/v2/otlp interface with this kind of error.

warn otlphttpexporter@v0.113.0/otlp.go:382 Partial success response {"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "message": "The following issues were encountered while ingesting OTLP metrics:\nErrors:\nUnsupported metric: 'kubelet_pod_start_duration_seconds' - Reason: UNSUPPORTED_METRIC_TYPE_CUMULATIVE_HISTOGRAM

I already tried to enable this feature also on our environment but seems that it does not make any difference on this case. 

https://docs.dynatrace.com/docs/shortlink/release-notes-saas-sprint-300#opentelemetry-metrics-ingest...

Since there is no direct support for these histograms currently I have tried also to utilize this compute histogram summaries to get at least some visibility out but for some reason I get sum metrics out, but not item count metric. Still trying to figure out the reason for this but of course the best case scenario would be if we could send these histograms directly on dynatrace. 

https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector/use-cases/histograms#compute-his...

[gopher · Accepted solution]

Hi @janne_olkoniemi 
As of a couple of versions ago - the OTLP API should have full support for Histograms. 
Dynatrace SaaS release notes version 1.300 — Dynatrace Docs

"Starting with Dynatrace version 1.300, histogram ingestion is supported directly via OTLP ingest API. Consequently, it's not required to use Collector as a workaround. Dynatrace recommends ingesting histograms directly without additional data manipulation."

I've tested on the otel collector metrics & I'm no longer getting these errors - I was previously excluding collecting them.  Only thing I can think of is that Dynatrace can only accept cumulativetodelta format - your error is mentioning cumulative.
 
Assuming you tried setting your processor to  

processors:
cumulativetodelta:

Otherwise, you might need to raise a case to Dyna on this on your specific scenario. Despite the challenges of getting support on the Otel Collector & config - they do support it as long as you are using the dynatrace collector. 

[janne_olkoniemi]

Thanks for your reply and the solution was eventually pretty easy in this case since I forgot to add the cumulativetodelta processor on my collector configuration 🙂