on - edited on by
jonghpark
- Introduction
- Things to know
- Where to look for symptoms?
- Common issues
- 1.Bad endpoint type
- Symptoms
- Troubleshooting
- 2.No metrics reported
- Symptoms
- Troubleshooting
- 3.Metric key missing
- Symptoms
- Troubleshooting
- 4.Feature set is disabled
- Symptoms
- Troubleshooting
- 5.Timeout
- Symptoms
- Troubleshooting
- 6.Invalid content of response of the source
- Symptoms
- Troubleshooting
- 7.Invalid extension YAML
- Symptoms
- Troubleshooting
- 8.Authentication error
- Symptoms
- Troubleshooting
- 9. Generic Entity Type pool exhaustion
- Symptoms
- Troubleshooting:
- 10.Prometheus DataSource related knowledge
- Useful links
- What's next
Introduction
This troubleshooting article will help you resolve some common issues faced while installing and setting up Prometheus-based extensions.
Things to know
Where to look for symptoms?
Specific symptoms mentioned in this article can be seen in the Extensions app / Extension page in the Dynatrace Hub and extension log files. Here is how you can find them.
SFM logs
Self-monitoring logs are available in the health tab on the Extension page in the Extensions app or on Extension page in the Dynatrace Hub.
Log files can be found in the paths:
- Linux ActiveGate: /var/lib/dynatrace/remotepluginmodule/log/extensions/datasources/<directory_corresponding_to_the_used_extension>
- Windows ActiveGate: %PROGRAMDATA%/dynatrace/remotepluginmodule/log/extensions/datasources/<directory_corresponding_to_the_used_extension>
- Linux OneAgent: /var/lib/dynatrace/oneagent/log/extensions/datasources/<directory_corresponding_to_the_used_extension>
- Windows OneAgent: %PROGRAMDATA%/dynatrace/oneagent/log/extensions/datasources/<directory_corresponding_to_the_used_extension>
- Generated support archive from the ActiveGate or OneAgent (How to collect a Support Archive)
Common issues
1.Bad endpoint type
Symptoms
The Prometheus extensions support two kinds of endpoint types:
- /metrics – returns metrics in plain text Prometheus format.
- /api/v1/ – API path that could be followed directly by a query or metadata endpoint.
The first type requires HTTP response code from 200 to 399 and meets one of the conditions:
Content-Type="text/plain"
or
Content-Type="application/openmetrics-text".
Often, these requirements are not met, and they are a cause of the issue.
Related logs:
Notice: The / can be added to the URL, in the second type of endpoint.[GENERIC_ERROR] Invalid content type (http://localhost:9090/metrics//query)
[DEVICE_CONNECTION_ERROR] Endpoint="http://localhost:9090/metrics/" Message="DEVICE_CONNECTION_ERROR Unknown error while checking HTTP Response (Query) code; Server returned response code 404 from Prometheus endpoint"
[DEVICE_CONNECTION_ERROR]:Failed to query 'http://localhost:9090/metrics/' for metrics
Troubleshooting
This can be verified by making a manual request to the scrape endpoint.
- Copy the Prometheus scrape endpoint URL from the monitoring configuration.
- Curl the Prometheus scrape endpoint and save the output to a file.
curl -v -i <scrape-endpoint-url> -o prometheus-output.txt - Verify content-type header.
2.No metrics reported
Symptoms
A manual request returns data related to metrics from the monitored extension, but it can not be viewed in the dashboard. Metric dsfm:datasource.prometheus.metric.line.count does not indicate that the data was collected.
Troubleshooting
Try to take some steps, as described below. It can help diagnose potential blockers.
- Verify firewall rules. It can block network traffic for the monitored endpoint.
- Verify the result of metric dsfm:server.metrics.rejections. Common cause of issue are exhausted DDUs.
- Verify Application Security rules.
3.Metric key missing
Symptoms
Due to the simplicity of Prometheus-based extensions, missing metric data tends to be related to a metric key missing from the Prometheus response
Related logs:
Could not find the following metrics in Prometheus response: confluent_kafka_connect_dead_letter_queue_records, confluent_kafka_server_cluster_link_count, kafka_partition_earliest_offset, confluent_kafka_connect_received_bytes, confluent_kafka_ksql_query_saturationEndpoint http://localhost:9090/metrics has collected 227 metrics.
Notice: This log means that after parsing the endpoint's answer returned 227 metrics, but not all metrics have to be matched with extension metrics.
Troubleshooting
This can be verified by making a manual request to the scrape endpoint, saving the output to a file, and checking the relevant metric key using the steps below.
- Copy the Prometheus scrape endpoint URL from the monitoring configuration.
- Curl the Prometheus scrape endpoint and save the output to a file.
curl -k <scrape-endpoint-url> -o prometheus-output.txt - Download the extension package from the Dynatrace Hub.
Notice: The information about metrics collected by extension is often described in the "Product information" section.
-
Extract the extension package.
- Inspect the extension.yaml file for the missing metric and note the entry for the value field.
- key: bdb_avg_read_latency_max value: metric:bdb_avg_read_latency_max type: gauge - Inspect the prometheus-output.txt file for the metric key from the previous step.
- If prometheus-output.txt file contains the metric that is missing in the extension.yaml file. It is possible to create a custom extension to add this metric. More information is available in the links: https://docs.dynatrace.com/docs/ingest-from/extensions20/extensions-concepts and https://www.dynatrace.com/hub/detail/custom-extensions-creator/#overview.
4.Feature set is disabled
Symptoms
The extension and response of the source contain the metric, but the data is not visible.
Troubleshooting
Check if the feature set that contains the metric is enabled.
5.Timeout
Symptoms
Gaps in data or complete lack of metrics.
Achieve rate limits:
Related logs:
[DEVICE_CONNECTION_ERROR] Failed to send request to Prometheus target (http://localhost:9090/metrics): Get "http://localhost:9090/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)[DEVICE_CONNECTION_ERROR] Server returned response code 429 from Prometheus endpoint (http://localhost:9090/metrics)
Troubleshooting
This can be verified by some steps described below.
- Curl the Prometheus scrape endpoint and verify that the answer is correct (check especially: HTTP code, content-type and body).
curl -i <scrape-endpoint-url>During the curl request, it is worth collecting network packets. This can be helpful in investigating the issue.
tcpdump -i eth0 -w output.pcap - Increase timeout/retries in the extension monitoring configuration (section advanced).
Notice: Retries * timeout should not be greater than min interval in the extension.
- Check if you are using the same credentials for both Dynatrace and some other tool, or if multiple monitoring configs/endpoints in Dynatrace are using the same credentials. If yes, try to create separate service accounts.
- Verify firewall rules. It can block network traffic for the monitored endpoint.
6.Invalid content of response of the source
Symptoms
Failing to parse the Prometheus response according to the Prometheus format: https://prometheus.io/docs/instrumenting/exposition_formats/
Popular cases:
- Duplicated TYPE
# TYPE bdb_up gauge bdb_up{bdb="19",status="active"} 1.0 ... # TYPE bdb_up gauge bdb_up{bdb="19",status="active"} 1.0 - Invalid escape sequence \"
- Unsupported metric type
Related logs:
[GENERIC_ERROR] Failed to scrape 'http://localhost:9090/metrics'. Reason: Failed to parse Prometheus metrics response (http://localhost:9090/metrics): text format parsing error in line 1224: second TYPE line for metric name "bdb_up", or TYPE reported after samples[GENERIC_ERROR] Failed to scrape 'http://localhost:9090/metrics'. Reason: Failed to parse Prometheus metrics response (http://localhost:9090/metrics): text format parsing error in line 291: invalid escape sequence '\"'
Notice: Prometheus data source was updated with latest prometheus/common library that covers parsing of \". The fix is available in agent version 305.
[GENERIC_ERROR] Failed to scrape 'http://localhost:9090/metrics'. Reason: Failed to parse Prometheus metrics response (http://localhost:9090/metrics): text format parsing error in line 1: unknown metric type "info"
Troubleshooting
In many situations, it demands changes in the format of Prometheus's answer by the provider. This can be verified by making a manual request to the scrape endpoint.
- Curl the Prometheus scrape endpoint.
curl -i <scrape-endpoint-url> -o prometheus-output.txt - Verify that the answer is correct with the official Prometheus format mentioned above.
7.Invalid extension YAML
Symptoms
Failing to parse the extension YAML file, assign configuration, and fastcheck.
Related logs:
[INVALID_CONFIG_ERROR] Error in extension configuration: Unmarshalling of extension config failed.[INVALID_CONFIG_ERROR] Invalid metric type (key 'prometheus.queue'): Invalid or unsupported MINT type (counter)
Troubleshooting
Verify the correctness of the extension YAML file using the documentation https://docs.dynatrace.com/docs/ingest-from/extensions20/data-sources/prometheus-extensions/promethe....
8.Authentication error
Symptoms
Failed fastchecks and unassigned configuration.
Related logs:
[GENERIC_ERROR] Prometheus (url type check) response for endpoint (http://localhost:9090/metrics) is emptyFast check status: 27. Details: There is no endpoint that could have connected.[AUTHENTICATION_ERROR] Server returned response code 401 from Prometheus endpoint (http://localhost:9090/metrics)
Troubleshooting
Check the authentication type and the correctness of the auth parameters.
- Curl the Prometheus scrape endpoint from the same host where extension monitoring configuration is running. Use the same authentication parameters.
Notice: The curl command can require authorization option for example curl -i -u <username> http://localhost:9090/metrics.curl -i <scrape-endpoint-url> - Try to manually enter the authentication parameters in the extension monitoring configuration.
9. Generic Entity Type pool exhaustion
Symptoms
An extension can create the generic type entry in Dynatrace environment. There is default limit for such generic entities which is 100k. When one of the generic types of your Prometheus extension reach the limit, the sudden lack of fresh data might occur.
In Dynatrace HUB you can check what Generic Type entities are created by your extension :
Troubleshooting:
To troubleshoot this issue you can follow the steps :
- Go to Dynatrace HUB and click on your Extension
- Go to Extension Content -> Generic Type
- Open each Generic Type and look for warning message like this
- Open a new ticket in Dynatrace Support and share details and the name of the problematic Generic Type.
10.Prometheus DataSource related knowledge
Useful links
- Create Prometheus extension - official documentation how to create Prometheus extensions
- Dynatrace HUB - all released extensions, where you can download and gain more information about them
- Extensions 2.0 - what are extensions 2.0
- Custom Extensions Creator - create custom extension directly on the Dynatrace web UI
- Signing extensions - how to sing an extension
What's next
If none of the solutions above has worked for you and you are still experiencing issues with the Prometheus extension you are trying to configure, there's always a chance that there is a problem with the extension itself or the underlying data source.
In that case feel free to open a support ticket specifying:
- A description of the problem
- A list of troubleshooting steps that have been performed
- Logs from the ActiveGate/OneAgent - support_archive.zip (How to collect a Support Archive)
- Curl command output or output file (from the host where the extension monitoring configuration is assigned)
- Working link to the environment
- Dynatrace version
- ActiveGate/OneAgent version
- Extensions Module version
- Monitoring configuration ID/name
- Screens from the UI where the problem is visible (e.g., monitoring configuration error, errors in log monitoring, data gaps, or lack of data on the dashboard/data explorer)
- If it's a custom extension, please provide the extension YAML or ZIP file
- TCP dump (recorded communication between ActiveGate/OneAgent and source/endpoint)
If this article has helped you and provided you with valuable insight, please give it a thumbs-up (kudos).
