19 Sep 2024 10:39 AM
Hi,
I've noticed that the following two metrics have some discrepancies. The first one is the CPU usage of all the containers `builtin:containers.cpu.usageMilliCores` of a workload summed up. It's the default query used in the workload dashboard.
The latter one is the cpu_used metric `builtin:kubernetes.workload.cpu_usage` directly defined on the workload.
There are two strange behaviors:
- The time series are almost equal (but almost)
- The latter one seems shifted 1 minutes ahead.
What might be the cause?
This example is taken from a deployment with 4 pods, there might be more complex cases in which the difference is more significant.
20 Sep 2024 01:30 AM
I think the difference between the two metrics is to do with the scope containers.cpu.usageMilliCores is specific to individual container CPU usage and kubernetes.workload.cpu_usage provides aggregate of CPU usage of all containers/pods within a specific Kubernetes workload.
A workload might contain a single container or multiple containers/pods, in your case how many containers do you have part of the workload you are trying to compare the CPU util?