Proper kube-state-metrics associations in Dynatrace

mreider — Mon, 07 Aug 2023 06:57:46 GMT

TL;DR

The Dynatrace kubernetes engineering team, based in Klagenfurt, Austria ("Klagifornia") delivered proper kube-state-metrics associations. This will be available in mid-August (SaaS 1.274).

More details

Dynatrace automatically associates metrics, traces, and logs with proper Kubernetes entities to present signals in the right context. Dynatrace makes this association by using Kubernetes API's, along with OneAgent Code Modules, to stitch together a map of each signal, and where it comes from. This is referred to as "topology."

These associations work perfectly well for 99% of the workloads on Kubernetes. A metric, trace, or log is emitted, and Dynatrace associates that signal with a certain process, container, and pod. But there's an important exception we've been wanting to address. The kube-state-metrics pod.

The kube-state-metrics pod is special. In addition to emitting metrics about itself, it also exports metrics about the state of other objects in Kubernetes, including nodes, pods, and deployments. All of these metrics are exposed via Prometheus and automatically available for dashboards, charts and alerts in Dynatrace.

Today - Dynatrace assigns kube-state metrics to the kube-state-metrics pod itself, rather than to the objects it describes. From an engineering perspective this makes sense. From a user perspective, however, these metrics should be re-assigned. This is the issue our team addressed in Klagifornia last month. 😎

Below you can see this list of kube-state metrics in Dynatrace. All of these metrics start with the prefix "kube_" (the underscore is important).

Looking more closely at an example, like kube_deployment_created, we see the workload, namespace, and cluster values are automatically linked to the proper Kubernetes analysis screens in Dynatrace. This illustrates the value of proper association. These links would not be available otherwise.

Licensing improvement

In addition to the functional improvements there are changes to how kube-state-metrics consume DDUs. With correct associations in place, these (free) DDU's are now consumed by related hosts, rather than the host where a kube-state-metrics pod runs.

Hats off to the engineers who made this possible. Let us know if you have questions or feedback. Again, this feature should be ready for production sometime in mid August (SaaS cluster 1.274)

Re: Proper kube-state-metrics associations in Dynatrace

florian_g — Mon, 07 Aug 2023 06:25:42 GMT

fyi @g_diener 🙂

topic Proper kube-state-metrics associations in Dynatrace in Container platforms

Proper kube-state-metrics associations in Dynatrace

Re: Proper kube-state-metrics associations in Dynatrace