Metrics discrepancy

Platform Container platforms
1

Hi,

I’ve noticed that the following two metrics have some discrepancies. The first one is the CPU usage of all the containers `builtin:containers.cpu.usageMilliCores` of a workload summed up. It’s the default query used in the workload dashboard.

The latter one is the cpu_used metric `builtin:kubernetes.workload.cpu_usage` directly defined on the workload.

There are two strange behaviors:
 - The time series are almost equal (but almost)

- The latter one seems shifted 1 minutes ahead.

Gibilisco_0-1726738542739.png
Gibilisco_0-1726738542739.png2560×1882 197 KB

What might be the cause? 
This example is taken from a deployment with 4 pods, there might be more complex cases in which the difference is more significant.

2

I think the difference between the two metrics is to do with the scope containers.cpu.usageMilliCores is specific to individual container CPU usage and kubernetes.workload.cpu_usage provides aggregate of CPU usage of all containers/pods within a specific Kubernetes workload.

A workload might contain a single container or multiple containers/pods, in your case how many containers do you have part of the workload you are trying to compare the CPU util?