Why are my OpenTelemetry metrics not ingested

[joaograssi]

Problem

When sending OpenTelemetry metrics to Dynatrace several validations are performed on the data which may result in it not being ingested. Here are the main causes for metric rejection:

• Metrics with cumulative aggregation temporality
• Too many attributes on a metric
• Too many metrics in the payload
• Unsupported metric types
• Invalid characters in metric name
• Invalid metric values
• Wrong export URL
• Missing or malformed API token
• Too large OTLP message

 

Troubleshooting Steps

You can use a combination of the HTTP response body and Dynatrace self-monitoring metrics to aid you in finding out what is wrong and how you can address it. In the response body, look out for messages such this:

The following issues were encountered while ingesting OTLP metrics:

Errors:

Unsupported metric: 'my.cumulative.counter' - Reason: NSUPPORTED_METRIC_TYPE_MONOTONIC_CUMULATIVE_SUM

Unsupported metric: 'my.summary' - Reason: UNSUPPORTED_METRIC_TYPE_SUMMARY

Histogram metric dropped. key: 'my.histogram' details: 'Histogram boundaries are not sorted' - Reason: HISTOGRAM_BUCKET_INVALID

Metric dropped: 'my.metric.many.attributes' - Reason: DIMENSION_COUNT_EXCEEDED

For an overview of what is being rejected and why, you can use the OTLP datapoints rejected self-monitoring metric: (dsfm.active_gate.metrics.ingest.otlp.datapoints.rejected ) and split by reason.This will show you all the reasons why metrics are being rejected.

Note that in this view it is not possible to see the metric names.

Rejected metrics, split by reason

For API related errors, such as API token issues or too large requests, you can use the following self-monitoring metrics, both filtered by operation=POST /otlp/v1/metrics and split by response_codes:

• dsfm.active_gate.rest.request_count
• dsfm.active_gate.rest.request_size

Count of OTLP metric requests, split by response code

Size of OTLP metric requests, split by response code

Tip: You can also additionally split by user_agent to help in finding out from which type of client the requests are coming from (e.g., OTel collector, OTel Java SDK).

 

Resolution

The combination of error messages included in the HTTP response and the self-monitoring metrics should give you useful information to help find out what is wrong and how to fix it.

For example, for errors mentioning “Cumulative”, you need to change the aggregation temporality to “delta”. For errors such as DIMENSION_COUNT_EXCEEDED you need to adapt the metric generation to make sure the number of recorded attributes stay under the limits.

For request too large errors, one good strategy is to use the batch processor in the OpenTelemetry collector, to split metrics into smaller batches.

Further Support

If you have any further questions or encounter any issues not listed above, please feel free to contact our support team.