Prometheus Troubleshooting Guide

maciej-tokarz — Thu, 07 Aug 2025 09:16:19 GMT

Introduction

This troubleshooting article will help you resolve some common issues faced while installing and setting up Prometheus-based extensions.

Things to know

Where to look for symptoms?

Specific symptoms mentioned in this article can be seen in the Extensions app / Extension page in the Dynatrace Hub and extension log files. Here is how you can find them.

SFM logs

Self-monitoring logs are available in the health tab on the Extension page in the Extensions app or on Extension page in the Dynatrace Hub.

Log files can be found in the paths:

Linux ActiveGate: /var/lib/dynatrace/remotepluginmodule/log/extensions/datasources/<directory_corresponding_to_the_used_extension>
Windows ActiveGate: %PROGRAMDATA%/dynatrace/remotepluginmodule/log/extensions/datasources/<directory_corresponding_to_the_used_extension>
Linux OneAgent: /var/lib/dynatrace/oneagent/log/extensions/datasources/<directory_corresponding_to_the_used_extension>
Windows OneAgent: %PROGRAMDATA%/dynatrace/oneagent/log/extensions/datasources/<directory_corresponding_to_the_used_extension>
Generated support archive from the ActiveGate or OneAgent (How to collect a Support Archive)

Common issues

1.Bad endpoint type

Symptoms

The Prometheus extensions support two kinds of endpoint types:

/metrics – returns metrics in plain text Prometheus format.

/api/v1/ – API path that could be followed directly by a query or metadata endpoint.

https://docs.dynatrace.com/docs/ingest-from/extensions20/data-sources/prometheus-extensions/prometheus-schema-reference#url

The first type requires HTTP response code from 200 to 399 and meets one of the conditions:

Content-Type="text/plain"

Content-Type="application/openmetrics-text".

Often, these requirements are not met, and they are a cause of the issue.

Related logs:

[GENERIC_ERROR] Invalid content type (http://localhost:9090/metrics//query) Notice: The / can be added to the URL, in the second type of endpoint.

[DEVICE_CONNECTION_ERROR] Endpoint="http://localhost:9090/metrics/" Message="DEVICE_CONNECTION_ERROR Unknown error while checking HTTP Response (Query) code; Server returned response code 404 from Prometheus endpoint"

[DEVICE_CONNECTION_ERROR]:Failed to query 'http://localhost:9090/metrics/' for metrics

Troubleshooting

This can be verified by making a manual request to the scrape endpoint.

Copy the Prometheus scrape endpoint URL from the monitoring configuration.
Curl the Prometheus scrape endpoint and save the output to a file. curl -v -i <scrape-endpoint-url> -o prometheus-output.txt
Verify content-type header.

2.No metrics reported

Symptoms

A manual request returns data related to metrics from the monitored extension, but it can not be viewed in the dashboard. Metric dsfm:datasource.prometheus.metric.line.count does not indicate that the data was collected.

Troubleshooting

Try to take some steps, as described below. It can help diagnose potential blockers.

Verify firewall rules. It can block network traffic for the monitored endpoint.
Verify the result of metric dsfm:server.metrics.rejections. Common cause of issue are exhausted DDUs.
Verify Application Security rules.

3.Metric key missing

Symptoms

Due to the simplicity of Prometheus-based extensions, missing metric data tends to be related to a metric key missing from the Prometheus response

Related logs:

Could not find the following metrics in Prometheus response: confluent_kafka_connect_dead_letter_queue_records, confluent_kafka_server_cluster_link_count, kafka_partition_earliest_offset, confluent_kafka_connect_received_bytes, confluent_kafka_ksql_query_saturation
Endpoint http://localhost:9090/metrics has collected 227 metrics.

Notice: This log means that after parsing the endpoint's answer returned 227 metrics, but not all metrics have to be matched with extension metrics.

Troubleshooting

This can be verified by making a manual request to the scrape endpoint, saving the output to a file, and checking the relevant metric key using the steps below.

Copy the Prometheus scrape endpoint URL from the monitoring configuration.
Curl the Prometheus scrape endpoint and save the output to a file. curl -k <scrape-endpoint-url> -o prometheus-output.txt
Download the extension package from the Dynatrace Hub.
Notice: The information about metrics collected by extension is often described in the "Product information" section.
Extract the extension package.
Inspect the extension.yaml file for the missing metric and note the entry for the value field. - key: bdb_avg_read_latency_max value: metric:bdb_avg_read_latency_max type: gauge
Inspect the prometheus-output.txt file for the metric key from the previous step.
If prometheus-output.txt file contains the metric that is missing in the extension.yaml file. It is possible to create a custom extension to add this metric. More information is available in the links: https://docs.dynatrace.com/docs/ingest-from/extensions20/extensions-concepts and https://www.dynatrace.com/hub/detail/custom-extensions-creator/#overview.

4.Feature set is disabled

Symptoms

The extension and response of the source contain the metric, but the data is not visible.

Troubleshooting

Check if the feature set that contains the metric is enabled.

5.Timeout

Symptoms

Gaps in data or complete lack of metrics.

Achieve rate limits:

https://api.telemetry.confluent.cloud/docs#tag/Version-2/paths/~1v2~1metrics~1%7Bdataset%7D~1export/get

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429

Related logs:

[DEVICE_CONNECTION_ERROR] Failed to send request to Prometheus target (http://localhost:9090/metrics): Get "http://localhost:9090/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[DEVICE_CONNECTION_ERROR] Server returned response code 429 from Prometheus endpoint (http://localhost:9090/metrics)

Troubleshooting

This can be verified by some steps described below.

Curl the Prometheus scrape endpoint and verify that the answer is correct (check especially: HTTP code, content-type and body). curl -i <scrape-endpoint-url>
During the curl request, it is worth collecting network packets. This can be helpful in investigating the issue.
tcpdump -i eth0 -w output.pcap
Increase timeout/retries in the extension monitoring configuration (section advanced).
Notice: Retries * timeout should not be greater than min interval in the extension.
Check if you are using the same credentials for both Dynatrace and some other tool, or if multiple monitoring configs/endpoints in Dynatrace are using the same credentials. If yes, try to create separate service accounts.
Verify firewall rules. It can block network traffic for the monitored endpoint.

6.Invalid content of response of the source

Symptoms

Failing to parse the Prometheus response according to the Prometheus format: https://prometheus.io/docs/instrumenting/exposition_formats/

Popular cases:

Duplicated TYPE# TYPE bdb_up gauge bdb_up{bdb="19",status="active"} 1.0 ... # TYPE bdb_up gauge bdb_up{bdb="19",status="active"} 1.0
Invalid escape sequence \"
Unsupported metric type

Related logs:

[GENERIC_ERROR] Failed to scrape 'http://localhost:9090/metrics'. Reason: Failed to parse Prometheus metrics response (http://localhost:9090/metrics): text format parsing error in line 1224: second TYPE line for metric name "bdb_up", or TYPE reported after samples
[GENERIC_ERROR] Failed to scrape 'http://localhost:9090/metrics'. Reason: Failed to parse Prometheus metrics response (http://localhost:9090/metrics): text format parsing error in line 291: invalid escape sequence '\"'

Notice: Prometheus data source was updated with latest prometheus/common library that covers parsing of \". The fix is available in agent version 305.

[GENERIC_ERROR] Failed to scrape 'http://localhost:9090/metrics'. Reason: Failed to parse Prometheus metrics response (http://localhost:9090/metrics): text format parsing error in line 1: unknown metric type "info"

Troubleshooting

In many situations, it demands changes in the format of Prometheus's answer by the provider. This can be verified by making a manual request to the scrape endpoint.

Curl the Prometheus scrape endpoint. curl -i <scrape-endpoint-url> -o prometheus-output.txt
Verify that the answer is correct with the official Prometheus format mentioned above.

7.Invalid extension YAML

Symptoms

Failing to parse the extension YAML file, assign configuration, and fastcheck.

Related logs:

[INVALID_CONFIG_ERROR] Error in extension configuration: Unmarshalling of extension config failed.
[INVALID_CONFIG_ERROR] Invalid metric type (key 'prometheus.queue'): Invalid or unsupported MINT type (counter)

Troubleshooting

Verify the correctness of the extension YAML file using the documentation https://docs.dynatrace.com/docs/ingest-from/extensions20/data-sources/prometheus-extensions/prometheus-schema-reference#prometheus-data-source-reference.

8.Authentication error

Symptoms

Failed fastchecks and unassigned configuration.

Related logs:

[GENERIC_ERROR] Prometheus (url type check) response for endpoint (http://localhost:9090/metrics) is empty
Fast check status: 27. Details: There is no endpoint that could have connected.
[AUTHENTICATION_ERROR] Server returned response code 401 from Prometheus endpoint (http://localhost:9090/metrics)

Troubleshooting

Check the authentication type and the correctness of the auth parameters.

Curl the Prometheus scrape endpoint from the same host where extension monitoring configuration is running. Use the same authentication parameters. curl -i <scrape-endpoint-url>Notice: The curl command can require authorization option for example curl -i -u <username> http://localhost:9090/metrics.
Try to manually enter the authentication parameters in the extension monitoring configuration.

9. Generic Entity Type pool exhaustion

Symptoms

An extension can create the generic type entry in Dynatrace environment. There is default limit for such generic entities which is 100k. When one of the generic types of your Prometheus extension reach the limit, the sudden lack of fresh data might occur.

In Dynatrace HUB you can check what Generic Type entities are created by your extension :

Troubleshooting:

To troubleshoot this issue you can follow the steps :

Go to Dynatrace HUB and click on your Extension
Go to Extension Content -> Generic Type
Open each Generic Type and look for warning message like this
Open a new ticket in Dynatrace Support and share details and the name of the problematic Generic Type.

10.Prometheus DataSource related knowledge

Useful links

Create Prometheus extension - official documentation how to create Prometheus extensions

Dynatrace HUB - all released extensions, where you can download and gain more information about them

Prometheus official documentation

Extensions 2.0 - what are extensions 2.0

Custom Extensions Creator - create custom extension directly on the Dynatrace web UI

Signing extensions - how to sing an extension

Dt-CLI usage

Metrics ingestion - documentation

Dashboards - documentation

What's next

If none of the solutions above has worked for you and you are still experiencing issues with the Prometheus extension you are trying to configure, there's always a chance that there is a problem with the extension itself or the underlying data source.

In that case feel free to open a support ticket specifying:

A description of the problem

A list of troubleshooting steps that have been performed

Logs from the ActiveGate/OneAgent - support_archive.zip (How to collect a Support Archive)

Curl command output or output file (from the host where the extension monitoring configuration is assigned)

Working link to the environment

Dynatrace version

ActiveGate/OneAgent version

Extensions Module version

Monitoring configuration ID/name

Screens from the UI where the problem is visible (e.g., monitoring configuration error, errors in log monitoring, data gaps, or lack of data on the dashboard/data explorer)

If it's a custom extension, please provide the extension YAML or ZIP file

TCP dump (recorded communication between ActiveGate/OneAgent and source/endpoint)

If this article has helped you and provided you with valuable insight, please give it a thumbs-up (kudos).

Re: Prometheus Troubleshooting Guide

AntonPineiro — Wed, 30 Jul 2025 14:09:16 GMT

Thanks!

Re: Prometheus Troubleshooting Guide

calfanonerey — Wed, 06 Aug 2025 20:33:05 GMT

Amazing! Thanks for sharing

article Prometheus Troubleshooting Guide in Troubleshooting

Prometheus Troubleshooting Guide

Introduction

Things to know

Where to look for symptoms?

Common issues

1.Bad endpoint type

Symptoms

Troubleshooting

2.No metrics reported

Symptoms

Troubleshooting

3.Metric key missing

Symptoms

Troubleshooting

4.Feature set is disabled

Symptoms

Troubleshooting

5.Timeout

Symptoms

Troubleshooting

6.Invalid content of response of the source

Symptoms

Troubleshooting

7.Invalid extension YAML

Symptoms

Troubleshooting

8.Authentication error

Symptoms

Troubleshooting

9. Generic Entity Type pool exhaustion

Symptoms

Troubleshooting:

10.Prometheus DataSource related knowledge

Useful links

What's next

Re: Prometheus Troubleshooting Guide

Re: Prometheus Troubleshooting Guide