Metrics discrepancy

[Gibilisco]

Hi, 

I've noticed that the following two metrics have some discrepancies. The first one is the CPU usage of all the containers `builtin:containers.cpu.usageMilliCores` of a workload summed up. It's the default query used in the workload dashboard. 

The latter one is the cpu_used metric `builtin:kubernetes.workload.cpu_usage` directly defined on the workload. 

There are two strange behaviors:
 - The time series are almost equal (but almost)

 - The latter one seems shifted 1 minutes ahead. 

What might be the cause? 
This example is taken from a deployment with 4 pods, there might be more complex cases in which the difference is more significant. 

[p_devulapalli · Accepted solution]

I think the difference between the two metrics is to do with the scope containers.cpu.usageMilliCores is specific to individual container CPU usage and kubernetes.workload.cpu_usage provides aggregate of CPU usage of all containers/pods within a specific Kubernetes workload.

A workload might contain a single container or multiple containers/pods, in your case how many containers do you have part of the workload you are trying to compare the CPU util?