Tagging Kubernetes Clusters, Workloads, and Pods

[shakib]

I’m needing to tag all entities (workloads, pods, & nodes) in a Kubernetes cluster via auto-tagging rules.

There is no way for me to do this with the rule type as “Monitored Entity” as a Kubernetes entity type (workloads, nodes, or anything K8 related) are not available to select. So, I assume the only way I can do this is by choosing the Entity Selector option when creating a tagging rule. But I need a way to tag 100+ workloads and 200+ pods without creating 300+ rules.

I was hoping I could tag all entities under a Management Zone but Dynatrace doesn’t allow that as specifying a Management Zone yields an error. I then thought I could do this at the cluster level but adding a tag to the cluster doesn’t cause it to pass down <type(KUBERNETES_CLUSTER),entityName("myK8ClusterName")>. According to Dynatrace entity types here are the seemingly relevant properties I need:

• Kubernetes Pods = type(CLOUD_APPLICATION_INSTANCE)
• Kubernete workload = type(CLOUD_APPLICATION)
• Kubernete Cluster = type(KUBERNETES_CLUSTER).

It looks like I need to string together these values in order to be able to tag what I need but I am running into issues getting it to work. I talked to Dynatrace support via chat and I was given an example of what worked for them before (but doesn’t work for me because it relies in a pre-existing tag).

• type(CONTAINER_GROUP_INSTANCE),toRelationships.isPgiOfCgi(types(PROCESS_GROUP_INSTANCE),tag([Kubernetes]app.kubernetes.io/instance:dev-atlas))

That support agent couldn’t get me a working syntax for what I need for the tagging rule despite the above example. But I’m looking do something like:

“Tag all pods where type is KUBERNETES_CLUSTER and cluster name is ShakibsClusterName"

Is this even possible via automated tagging rules? 

[ChadTurner · Accepted solution]

From an automated dynamic tagging stand point it isn't. But if you are willing to do a 1 to 1 it can be done via relations. For example. lets say I have a Namespace that resides under a cluster and I want to tag it with my own value. You can use the entity selector and leverage relationship tagging. via this string: 

Apply '<EASYTRAVELCLUSTER>' to Entities matching type("Cloud_Application_Namespace"), toRelationships.isClusterOfNamespace(type("KUBERNETES_CLUSTER"), entityName("<CLUSTERNAME/ID>"))

Now it will apply that "Easytravel" as my custom tag value to the namespace if and when that namespace is related to the defined cluster via Dynatrace AI. You can then do this for the other aspects such as Pods. 

It requires some thinking out of the box and isn't truly scalable for large organizations as you have to define them one by one. 

[ChadTurner]

@shakib I also have created a RFE that is requesting the ability to leverage pod labels as tags and such. this would help reduce the usage of the entity selector. Dynatrace needs to expand the dynamic tagging capabilities to all the auto discovered properties. Feel free to give it a like 🙂 

https://community.dynatrace.com/t5/Dynatrace-product-ideas/Expand-Dynamic-Auto-Tag-constructs-for-Ku... 

[shakib]

Thank you, Chad, that worked perfectly for the name spaces.  

If I have 10 name spaces then it seems I'll have to create 10 individual rules to cover each workload inside each individual namespace. 

I was able to use the same logic for Workloads and the working syntax below shows all workloads being tagged for namespace XYZ which belongs to cluster ABC.

• type(CLOUD_APPLICATION),toRelationships.isNamespaceOfCa(type("CLOUD_APPLICATION_NAMESPACE"), entityName("XYZ"), toRelationships.isClusterOfNamespace(type("KUBERNETES_CLUSTER"), entityName("ABC")))

I'm going to try the same approach for pods next and see how that works out. 

It's far from ideal to have to create this many rules. I really hope Dynatrace addresses this for future users so they don't have to suffer. 

[BaliBan]

hey @shakib ,

can you provide logical syntax  for pods and nodes also like you provide for workload

[DanielS · Accepted solution]

I share my collection of rules for Kubernetes Cluster:

• type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")
• type(CLOUD_APPLICATION_NAMESPACE),toRelationships.isClusterOfNamespace(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(CLOUD_APPLICATION),toRelationships.isClusterOfCa(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(KUBERNETES_SERVICE),toRelationships.isClusterOfKubernetesSvc(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(CONTAINER_GROUP),toRelationships.isInstanceOf(type(CONTAINER_GROUP_INSTANCE),fromRelationships.isCgiOfCluster(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")))
• type(CONTAINER_GROUP_INSTANCE),fromRelationships.isCgiOfCluster(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(HOST),toRelationships.isClusterOfHost(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(PROCESS_GROUP),fromRelationships.runsOn(type(HOST),toRelationships.isClusterOfHost(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")))
• type(PROCESS_GROUP_INSTANCE),fromRelationships.isProcessOf(type(HOST),toRelationships.isClusterOfHost(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")))
• type(SERVICE),fromRelationships.runsOnHost(type(HOST),toRelationships.isClusterOfHost(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")))

Only Manual Tags:

• type(CLOUD_APPLICATION_INSTANCE),toRelationships.isClusterOfCai(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))

[BaliBan]

I have tried your first logic , but it does't showing any output even through i got entity matching ?

• type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")
• type(KUBERNETES_CLUSTER),entityName.in("minikube")

hope can you answer my query

[DanielS]

Hi @BaliBan if you mean through the UI there is no place where you can see the output, but for example if you add to a MZ or reference the Kubernetes Cluster the tags are there. If you use the API you can see the tag reference:

[Brijesh_Saw]

Hi DanialS,

Your Post really helped me a lot. Also, can you please add little detailed step for tagging of Kubernetes Pods(CLOUD_APPLICATION_INSTANCE).

Thanks and Regards,

Brijesh Saw

[Brijesh_Saw · Accepted solution]

Hi DanialS,

I got to know about below option in URL for tagging Kubernetes Pods, Where we can configure custom tags using "Environment API v2" ->Monitored entities - Custom tags ->POST method. All we need is Below mention EntitySelector and tag value

EntitySelector:- "type(CLOUD_APPLICATION_INSTANCE),toRelationships.isClusterOfCai(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))"

URL:- https://www.dynatrace.com/support/help/dynatrace-api/environment-api/custom-tags/post-tags

Thanks and Regards,

Brijesh Saw

[Alejandro_hdez]

@DanielS Do you have on your collection rules for nodes and pods?

[DanielS]

Hello @Alejandro_hdez , yes:

• type(KUBERNETES_NODE),toRelationships.isClusterOfNode(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(CLOUD_APPLICATION_INSTANCE),toRelationships.isClusterOfCai(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))

[Alejandro_hdez]

Do you also know a way to easily tag GCP integration (custom devices)? I've noticed that all gcp entities from integration are identified as custom devices. 

[vinnu_arige_moe]

how to we get workloads label values as tags so we can filter by those labels in data explorer?

[DanielS]

Hi @vinnu_arige_moe I leave you this link  but have in mind this observation. As a result, Kubernetes processes monitored in your Dynatrace environment will have Kubernetes labels attached as Kubernetes tags. For namespaces, pods, and workloads, Kubernetes tags are not evaluated.

[Jennnyhz · Accepted solution]

Everyone ok? Thank you very much for the topic, it helped me a lot here in my environment.

I was facing the same problem and I noticed that after entering the cluster through the Kubernetes menu, in settings (...) the option to create a management zone for the cluster is presented, within which there are the following rules:

 

- Dimensional rule for METRIC whre DIMESION k8s.cluster.name equals YOUR CLUSTER NAME
- Kubernetes namespace on Hosts where Kubernetes cluster name equals "CLUSTER NAME"
- Kubernetes services on Hosts where Kubernetes cluster name equals "CLUSTER NAME"
- Kubernetes workloads on Hosts where Kubernetes cluster name equals "CLUSTER NAME"

I was having trouble inserting the Nodes and pods into the Management zone and the first rule did that, I just needed to insert it into my existing mz

 

I hope this helps in some way, in my case it worked.

thanks.