Setting frequency for Prometheus metrics polling

[n002213f]

We are able to send custom Prometheus metrics to Dynatrace using podAnnotations as details here 

From the documentation, polling is at 1 minute intervals.

Pulling these metrics makes use of complex queries on the database which when run every minute is placing strain on our database.

Is there a way to configure the polling interval to allow for polling every 5 minutes?

[jaume_reverte · Accepted solution]

Hello @n002213f! 

For now, as far as I know, it is not possible to configure the K8s exporter to not pull the metrics every minute. 

You can try creating your own Prometheus extension (link). This way, you can select the polling interval used for each metric. I have never tested in a K8s environment. 

Hope you a good monitoring! 

[gopher]

Hi @n002213f ,  The newly recommended way (also fixes issues on resource consumption of the AG) is to use the Dynatrace OTEL Collector : Collector deployment - Dynatrace Docs.

This gives you the same capability as a standard Prometheus server and you don't need to worry about the Dynatrace scrape settings.   You can set the prom standard scrape intervals in this and customise it per scrape setting.  

e.g

          scrape_configs:
            - job_name: 'istiod'
              scrape_interval: 5m
              static_configs:
                - targets: ['istiod.istio-internal.svc.cluster.local:15014']

There are a couple of limitations around the metrics Ingest OpenTelemetry metrics - Dynatrace Docs but nothing you that is different to what the AG collects.

Hope this helps

Thanks