Dynatrace Operator - Having a different number of OA/CSI pods than actual K8S nodes

Self Service Summary

After deploying the Dynatrace Operator, the OneAgent pod and CSI driver pod are not running on all the nodes.

Issue Solution Tasks Alternative(s)
Not all the nodes are having the OneAgent pod/CSI pod running on them Check with your K8S administrator the taint(s) on the nodes Update manifest files with corresponding toleration(s) Contact Dynatrace Support for further assistance

This is a common scenario for customers that use taints on the Kubernetes nodes in order to restrain what workload are deployed on them. If it is required to have the OneAgent + CSI driver pods running on those nodes, these taints need to be added as a toleration in the dynakube and the csi driver manifest files. Modified manifests need to be re-applied.

In order to check if a node has any taints, execute following command:

kubectl describe node <nodeName>

Identify the following part:

Taints:             key1=value1:NoSchedule

It means that the node has a taint and the same value has to be set as a toleration in the manifest files. Nodes can have multiple taints.

The toleration(s) would need to be added in the dynakube and the csi driver manifest files.

tolerations:
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoSchedule"

After the manifest files have been edited,  execute following commands:

kubectl apply -f <yourDynakube>.yaml

kubectl apply -f <yourCSI>.yaml

The same applies if the used platform is Openshift.

The official documentation from Kubernetes that explains in detail how taint and toleration work can be found here: Taint-and-toleration

2 Likes

What is the solution if using the Dyntrace Azure Marketplace extension?