Metrics discrepancy

Gibilisco — Thu, 19 Sep 2024 09:39:11 GMT

Hi,

I've noticed that the following two metrics have some discrepancies. The first one is the CPU usage of all the containers `builtin:containers.cpu.usageMilliCores` of a workload summed up. It's the default query used in the workload dashboard.

The latter one is the cpu_used metric `builtin:kubernetes.workload.cpu_usage` directly defined on the workload.

There are two strange behaviors:
- The time series are almost equal (but almost)

- The latter one seems shifted 1 minutes ahead.

What might be the cause?
This example is taken from a deployment with 4 pods, there might be more complex cases in which the difference is more significant.

Re: Metrics discrepancy

p_devulapalli — Fri, 20 Sep 2024 00:30:31 GMT

I think the difference between the two metrics is to do with the scope containers.cpu.usageMilliCores is specific to individual container CPU usage and kubernetes.workload.cpu_usage provides aggregate of CPU usage of all containers/pods within a specific Kubernetes workload.

A workload might contain a single container or multiple containers/pods, in your case how many containers do you have part of the workload you are trying to compare the CPU util?

topic Re: Metrics discrepancy in Container platforms

Metrics discrepancy

Re: Metrics discrepancy