Dynatrace Operator - Having a different number of OA/CSI pods than actual K8S nodes

Filip-Rus — Wed, 13 Nov 2024 09:23:04 GMT

Self Service Summary

After deploying the Dynatrace Operator, the OneAgent pod and CSI driver pod are not running on all the nodes.

Issue	Solution	Tasks	Alternative(s)
Not all the nodes are having the OneAgent pod/CSI pod running on them	Check with your K8S administrator the taint(s) on the nodes	Update manifest files with corresponding toleration(s)	Contact Dynatrace Support for further assistance

This is a common scenario for customers that use taints on the Kubernetes nodes in order to restrain what workload are deployed on them. If it is required to have the OneAgent + CSI driver pods running on those nodes, these taints need to be added as a toleration in the dynakube and the csi driver manifest files. Modified manifests need to be re-applied.

In order to check if a node has any taints, execute following command:

kubectl describe node <nodeName>

Identify the following part:

Taints: key1=value1:NoSchedule

It means that the node has a taint and the same value has to be set as a toleration in the manifest files. Nodes can have multiple taints.

The toleration(s) would need to be added in the dynakube and the csi driver manifest files.

tolerations: - key: "key1" operator: "Equal" value: "value1" effect: "NoSchedule"

After the manifest files have been edited, execute following commands:

kubectl apply -f <yourDynakube>.yaml kubectl apply -f <yourCSI>.yaml

The same applies if the used platform is Openshift.

The official documentation from Kubernetes that explains in detail how taint and toleration work can be found here: Taint-and-toleration

article Dynatrace Operator - Having a different number of OA/CSI pods than actual K8S nodes in Troubleshooting

Dynatrace Operator - Having a different number of OA/CSI pods than actual K8S nodes

Self Service Summary