Excluded Prometheus metrics are still fetched by ActiveGate

Platform Troubleshooting
, ,
1

If you are monitoring Prometheus metrics you may want to exclude some noise metrics or include only selected metrics that provide more business content.

Symptoms

You have excluded some noise Prometheus metrics from ingestion by using a filtering option, but on the Dynatrace UI (data explorer) the excluded metrics are still visible and AG continues to fetch them.

For example, you have configured the exlusion rule like this

annotations:
        metrics.dynatrace.com/filter: |
          {
              "mode": "exclude",
              "names":
              [
                  "sample_request_client_request_bytes_count",
                  "sample_request_client_request_bytes_max",
                  "sample_request_client_request_bytes_sum",
                  "sample_request_client_response_bytes_count",
                  "sample_request_client_response_bytes_max",
                  "sample_request_client_response_bytes_sum",
                  ...
              ]
          }

Resolution

Looking at the Prometheus metrics from the exposer instance we get this response

...
# TYPE sample_request_client_request_bytes summary
# HELP sample_request_client_request_bytes Size of requests in bytes
sample_request_client_request_bytes_count{method="POST",route="/api/v1/app/versions/validate"} 1480.0
sample_request_client_request_bytes_sum{method="POST",route="/api/v1/app/versions/validate"} 345163.0
sample_request_client_request_bytes_count{method="POST",route="/api/v1/labels/sample/get"} 115.0
sample_request_client_request_bytes_sum{method="POST",route="/api/v1/labels/sample/get"} 28352.0
...

The excluded metrics have an actual name as defined in the “# TYPE sample_request_client_request_bytes summary” line in Prometheus (here: sample_request_client_request_bytes), and may contain additional sum and count datapoints with different metric keys (here: sample_request_client_request_bytes_sum and sample_request_client_request_bytes_count).

When applying the configured operations, the filter groups all data points within a metric together and does not consider the other metric keys contained within this metric. So, in this case the filter will only evaluate the actual metric sample_request_client_request_bytes but not the _sum and _count data points included inside.
If sample_request_client_request_bytes is excluded by the filter, the _sum and _count data points will also be excluded.

Example of how the correct filtering should look like:

metrics.dynatrace.com/filter: |
          {
              "mode": "exclude",
              "names":
              [
                  ...
                  "sample_request_client_request_bytes",
                  ...
             ]
          }
3 Likes