How to troubleshoot missing Prometheus metrics

[shahinm]

You can integrate your Prometheus exporter to Dynatrace and make them available for charting, alerting, and analysis. 

If you completed the integration but still cannot find the relevant metrics on the Dynatrace metrics UI, then run the following steps for troubleshooting: 

1. Make sure the Pod template annotations are properly set (link) 
2. “Monitor annotated Prometheus exporters” toggle is enabled from the Kubernetes integration settings
   
3. Metric output format (text/plain) and exposed metric types are valid.  
4. Install “Kubernetes Monitoring Statistics” extension from the Dynatrace Hub. This extension provides you with self-monitoring statistics for all the Kubernetes clusters you already monitor within your Dynatrace environment and the root cause (response code) for failing API requests. 
5. Run one of the following curl commands valid for your deployment:
   
   a) ActiveGate is running outside (non-containerized) the Kubernetes/OpenShift cluster
   

   API=<enter external k8s api>
   NAMESPACE=<enter pod namespace here e.g. openshift-node>
   POD=<enter pod name here e.g. node-exporter-1234>
   PORT=<enter port here e.g. 9100>
   PATH=<enter path here e.g. /metrics>
   TOKEN=<enter bearer token here>
   
   curl https://$API/api/v1/namespaces/$NAMESPACE/pods/$POD:$PORT/proxy$PATH \
        --connect-timeout 20 -v -k -H "Authorization: Bearer $TOKEN"

   
   b) ActiveGate is running inside (containerized) the Kubernetes/OpenShift cluster 
   

   POD_OR_NODE_IP=<enter pod or node ip here>
   METRICS_PORT=<enter port here e.g. 9100>
   METRICS_PATH=<enter path here e.g. /metrics>
   
   kubectl run -n dynatrace curl --image=curlimages/curl --restart=Never -it --rm \
           -- http://$POD_OR_NODE_IP:$METRICS_PORT/$METRICS_PATH \
           --connect-timeout 20 -k -v

   
   c) ActiveGate is running inside (containerized) the Kubernetes/OpenShift cluster and metrics endpoint requires RBAC authentication

   POD_OR_NODE_IP=<enter pod or node ip here>
   METRICS_PORT=<enter port here e.g. 9100>
   METRICS_PATH=<enter path here e.g. /metrics>
   TOKEN=<enter bearer token here>
   
   kubectl run -n dynatrace curl --image=curlimages/curl --restart=Never -it --rm \
       -- http://$POD_OR_NODE_IP:$METRICS_PORT/$METRICS_PATH \
       --connect-timeout 20 -k -v -H "Authorization: Bearer $TOKEN"

6. Use the latest available AG version since it is highly recommended.
7. Collect AG logs and results from all the above steps and create a ticket for Dynatrace Technical Support for further investigation if the metrics are still missing.  
   

[ChadTurner]

Thank you for sharing this! I'm sure it will help out many community members. Always check your AG Version if you have issues lol