Tagging Kubernetes Clusters, Workloads, and Pods

[shakib]

I’m needing to tag all entities (workloads, pods, & nodes) in a Kubernetes cluster via auto-tagging rules.

There is no way for me to do this with the rule type as “Monitored Entity” as a Kubernetes entity type (workloads, nodes, or anything K8 related) are not available to select. So, I assume the only way I can do this is by choosing the Entity Selector option when creating a tagging rule. But I need a way to tag 100+ workloads and 200+ pods without creating 300+ rules.

I was hoping I could tag all entities under a Management Zone but Dynatrace doesn’t allow that as specifying a Management Zone yields an error. I then thought I could do this at the cluster level but adding a tag to the cluster doesn’t cause it to pass down <type(KUBERNETES_CLUSTER),entityName("myK8ClusterName")>. According to Dynatrace entity types here are the seemingly relevant properties I need:

• Kubernetes Pods = type(CLOUD_APPLICATION_INSTANCE)
• Kubernete workload = type(CLOUD_APPLICATION)
• Kubernete Cluster = type(KUBERNETES_CLUSTER).

It looks like I need to string together these values in order to be able to tag what I need but I am running into issues getting it to work. I talked to Dynatrace support via chat and I was given an example of what worked for them before (but doesn’t work for me because it relies in a pre-existing tag).

• type(CONTAINER_GROUP_INSTANCE),toRelationships.isPgiOfCgi(types(PROCESS_GROUP_INSTANCE),tag([Kubernetes]app.kubernetes.io/instance:dev-atlas))

That support agent couldn’t get me a working syntax for what I need for the tagging rule despite the above example. But I’m looking do something like:

“Tag all pods where type is KUBERNETES_CLUSTER and cluster name is ShakibsClusterName"

Is this even possible via automated tagging rules? 

[ChadTurner · Accepted solution]

From an automated dynamic tagging stand point it isn't. But if you are willing to do a 1 to 1 it can be done via relations. For example. lets say I have a Namespace that resides under a cluster and I want to tag it with my own value. You can use the entity selector and leverage relationship tagging. via this string: 

Apply '<EASYTRAVELCLUSTER>' to Entities matching type("Cloud_Application_Namespace"), toRelationships.isClusterOfNamespace(type("KUBERNETES_CLUSTER"), entityName("<CLUSTERNAME/ID>"))

Now it will apply that "Easytravel" as my custom tag value to the namespace if and when that namespace is related to the defined cluster via Dynatrace AI. You can then do this for the other aspects such as Pods. 

It requires some thinking out of the box and isn't truly scalable for large organizations as you have to define them one by one. 

[ChadTurner]

@shakib I also have created a RFE that is requesting the ability to leverage pod labels as tags and such. this would help reduce the usage of the entity selector. Dynatrace needs to expand the dynamic tagging capabilities to all the auto discovered properties. Feel free to give it a like 🙂 

https://community.dynatrace.com/t5/Dynatrace-product-ideas/Expand-Dynamic-Auto-Tag-constructs-for-Ku... 

[shakib]

Thank you, Chad, that worked perfectly for the name spaces.  

If I have 10 name spaces then it seems I'll have to create 10 individual rules to cover each workload inside each individual namespace. 

I was able to use the same logic for Workloads and the working syntax below shows all workloads being tagged for namespace XYZ which belongs to cluster ABC.

• type(CLOUD_APPLICATION),toRelationships.isNamespaceOfCa(type("CLOUD_APPLICATION_NAMESPACE"), entityName("XYZ"), toRelationships.isClusterOfNamespace(type("KUBERNETES_CLUSTER"), entityName("ABC")))

I'm going to try the same approach for pods next and see how that works out. 

It's far from ideal to have to create this many rules. I really hope Dynatrace addresses this for future users so they don't have to suffer. 

[BaliBan]

hey @shakib ,

can you provide logical syntax  for pods and nodes also like you provide for workload

[DanielS · Accepted solution]

I share my collection of rules for Kubernetes Cluster:

• type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")
• type(CLOUD_APPLICATION_NAMESPACE),toRelationships.isClusterOfNamespace(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(CLOUD_APPLICATION),toRelationships.isClusterOfCa(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(KUBERNETES_SERVICE),toRelationships.isClusterOfKubernetesSvc(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(CONTAINER_GROUP),toRelationships.isInstanceOf(type(CONTAINER_GROUP_INSTANCE),fromRelationships.isCgiOfCluster(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")))
• type(CONTAINER_GROUP_INSTANCE),fromRelationships.isCgiOfCluster(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(HOST),toRelationships.isClusterOfHost(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(PROCESS_GROUP),fromRelationships.runsOn(type(HOST),toRelationships.isClusterOfHost(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")))
• type(PROCESS_GROUP_INSTANCE),fromRelationships.isProcessOf(type(HOST),toRelationships.isClusterOfHost(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")))
• type(SERVICE),fromRelationships.runsOnHost(type(HOST),toRelationships.isClusterOfHost(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")))

Only Manual Tags:

• type(CLOUD_APPLICATION_INSTANCE),toRelationships.isClusterOfCai(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))

[BaliBan]

I have tried your first logic , but it does't showing any output even through i got entity matching ?

• type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>")
• type(KUBERNETES_CLUSTER),entityName.in("minikube")

hope can you answer my query

[DanielS]

Hi @BaliBan if you mean through the UI there is no place where you can see the output, but for example if you add to a MZ or reference the Kubernetes Cluster the tags are there. If you use the API you can see the tag reference:

[Brijesh_Saw · Accepted solution]

Hi DanialS,

Your Post really helped me a lot. Also, can you please add little detailed step for tagging of Kubernetes Pods(CLOUD_APPLICATION_INSTANCE).

Thanks and Regards,

Brijesh Saw

[Brijesh_Saw · Accepted solution]

Hi DanialS,

I got to know about below option in URL for tagging Kubernetes Pods, Where we can configure custom tags using "Environment API v2" ->Monitored entities - Custom tags ->POST method. All we need is Below mention EntitySelector and tag value

EntitySelector:- "type(CLOUD_APPLICATION_INSTANCE),toRelationships.isClusterOfCai(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))"

URL:- https://www.dynatrace.com/support/help/dynatrace-api/environment-api/custom-tags/post-tags

Thanks and Regards,

Brijesh Saw

[Alejandro_hdez]

@DanielS Do you have on your collection rules for nodes and pods?

[DanielS]

Hello @Alejandro_hdez , yes:

• type(KUBERNETES_NODE),toRelationships.isClusterOfNode(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• type(CLOUD_APPLICATION_INSTANCE),toRelationships.isClusterOfCai(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))

[Alejandro_hdez]

Do you also know a way to easily tag GCP integration (custom devices)? I've noticed that all gcp entities from integration are identified as custom devices. 

[Veleda]

@DanielS 

Good afternoon,

Sorry to bother you.

Where did you get the tags to make those tags and how do you know the relationship between them?
I check the documentation and I can't find anything.

Thanks 

Best Regard

[DanielS]

Hi @Veleda I'm glad to help.

All the information needed is under the entity types API in the Environment v2. The request is Gets a list of properties for all entity types. I leave you the link to docs but feel free to contact me for further details.

[vinnu_arige_moe]

how to we get workloads label values as tags so we can filter by those labels in data explorer?

[DanielS]

Hi @vinnu_arige_moe I leave you this link  but have in mind this observation. As a result, Kubernetes processes monitored in your Dynatrace environment will have Kubernetes labels attached as Kubernetes tags. For namespaces, pods, and workloads, Kubernetes tags are not evaluated.

[Jennnyhz · Accepted solution]

Everyone ok? Thank you very much for the topic, it helped me a lot here in my environment.

I was facing the same problem and I noticed that after entering the cluster through the Kubernetes menu, in settings (...) the option to create a management zone for the cluster is presented, within which there are the following rules:

 

- Dimensional rule for METRIC whre DIMESION k8s.cluster.name equals YOUR CLUSTER NAME
- Kubernetes namespace on Hosts where Kubernetes cluster name equals "CLUSTER NAME"
- Kubernetes services on Hosts where Kubernetes cluster name equals "CLUSTER NAME"
- Kubernetes workloads on Hosts where Kubernetes cluster name equals "CLUSTER NAME"

I was having trouble inserting the Nodes and pods into the Management zone and the first rule did that, I just needed to insert it into my existing mz

 

I hope this helps in some way, in my case it worked.

thanks.

[Babar_Qayyum]

Hello @DanielS 

What is the right way to organize a management zone for a namespace where the user can view the infrastructure-related issue (CPU-limit quota saturation)?

Also, what are the rules to apply the tag on the following for a specific namespace?

• Pod
• Container
• Kubernetes service

Regards,

Babar

[DanielS]

Hi @Babar_Qayyum I've made for you the rules based on Namespace Entity ID as you requested because is the most restrictive approach, have in mind that for access you can only navigate via Kubernetes Workloads UI Menu (NOTE: Host information for the Kubernetes Cluster have been omitted because I understand that you only need from Namespace to below):

* type(CLOUD_APPLICATION_INSTANCE),toRelationships.isNamespaceOfCai(type(CLOUD_APPLICATION_NAMESPACE),entityld(CLOUD_APPLICATION_NAMESPACE-XXXXXXXXXXXXXXX))

* type(CLOUD_APPLICATION_NAMESPACE),entityld(CLOUD_APPLICATION_NAMESPACE-XXXXXXXXXXXXXXX)

* type(CLOUD_APPLICATION),toRelationships.isNamespaceOfCa(type(CLOUD_APPLICATION_NAMESPACE),entityld(CLOUD_APPLICATION_NAMESPACE-XXXXXXXXXXXXXXX))

* type(CONTAINER_GROUP_INSTANCE),fromRelationships.isCgiOfNamespace(type(CLOUD_APPLICATION_NAMESPACE),entityld(CLOUD_APPLICATION_NAMESPACE-XXXXXXXXXXXXXXX))

* type(CONTAINER_GROUP),torelationships.isInstanceOf(type(CONTAINER_GROUP_INSTANCE),fromRelationships.isCgiOfNamespace(type(CLOUD_APPLICATION_NAMESPACE),entityld(CLOUD_APPLICATION_NAMESPACE-XXXXXXXXXXXXXXX)))

* type(KUBERNETES_SERVICE),toRelationships.isNamespaceOfKubernetesSvc(type(CLOUD_APPLICATION_NAMESPACE),entityld(CLOUD_APPLICATION_NAMESPACE-XXXXXXXXXXXXXXX))

* type(PROCESS_GROUP_INSTANCE),fromRelationships.isMainPgiOfCgi(type(CONTAINER_GROUP_INSTANCE),fromRelationships.isCgiOfNamespace(type(CLOUD_APPLICATION_NAMESPACE),entityId(CLOUD_APPLICATION_NAMESPACE-XXXXXXXXXXXXXXX)))

* type(SERVICE),fromRelationships.isServiceOf(type(CLOUD_APPLICATION),toRelationships.isNamespaceOfCa(type(CLOUD_APPLICATION_NAMESPACE),entityld(CLOUD_APPLICATION_NAMESPACE-XXXXXXXXXXXXXXX)))

[DanielS]

Look here @Babar_Qayyum, if you try to access it through the Kubernetes menu, you won't see anything, but in Kubernetes Workloads menu you will only see the namespace with the entity ID that you use in the rules.

[DanielS]

please let me know if they are working ok of is something missing. 

[Babar_Qayyum]

Hello @DanielS 

First of all, thank you for all your support.

Do we need to configure the following rules to make a management zone for a particular namespace?

• type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>") type(CLOUD_APPLICATION_NAMESPACE),toRelationships.isClusterOfNamespace(type(KUBERNETES_CLUSTER),entityName.in("<YOUR KUBERNETES CLUSTER NAME>"))
• Dimensional rule for METRIC where DIMESION k8s.cluster.name equals YOUR CLUSTER NAME 

The following rules have an issue with the relationship.

• type(CONTAINER_GROUP_INSTANCE),toRelationships.isCgiOfNamespace(type(CLOUD_APPLICATION_NAMESPACE),entityName.in("XYZ"))
  entitySelector The entitySelector has an invalid relationship defined: CLOUD_APPLICATION_NAMESPACE IS_CGI_OF_NAMESPACE CONTAINER_GROUP_INSTANCE
• type(PROCESS_GROUP_INSTANCE),fromRelationships.isMainPgiOfCgi(type(CONTAINER_GROUP_INSTANCE),toRelationships.isCgiOfNamespace(type(CLOUD_APPLICATION_NAMESPACE),entityName.in("XYZ")))
  entitySelector The entitySelector has an invalid relationship defined: CLOUD_APPLICATION_NAMESPACE IS_CGI_OF_NAMESPACE CONTAINER_GROUP_INSTANCE

Also, the user has a 403 permission issue while accessing the CPU-limit quota saturation of the namespace. Will the above/or missing rules cover this as well?

Regards,

Babar

[DanielS]

Hi @Babar_Qayyum let me replicate the rules in my environment and I reach you.

[DanielS]

Sorry I didn't have time to test it yesterday, but today I tested everything in my environment. Do not hesitate to contact me. Please let me know if the rules worked for you. I edited my previous answer to prevent incorrect rules from confusing anyone.

[DanielS]

Now, for the user to enter to the section of Anomaly detection of the namespace, it must have the following policy:

ALLOW settings:objects:read, settings:objects:write, settings:schemas:read
WHERE settings:schemaId = "builtin:anomaly-detection.kubernetes.namespace";

[Babar_Qayyum]

Hello @DanielS 

Once again, thank  you for all your great support. Mostly, things worked as expected. I faced a challenge with the following rule while applying the automatic tagging.

type(CLOUD_APPLICATION_INSTANCE),toRelationships.isNamespaceOfCai(type(CLOUD_APPLICATION_NAMESPACE),entityId("CLOUD_APPLICATION_NAMESPACE-XXXXXXXXXXXXX"))
entitySelector Entity selector contains type which does not support automatically applied tags

What about the Dimensional rule for METRIC? Do we need to include it in the management zone? What should be the dimension keys in case they are required?

Regards,

Babar

[DanielS]

Hello, your welcome @Babar_Qayyum  you are not doing anything wrong and is not an error, as you can see in this post  auto-tags for type CLOUD_APPLICATION_INSTANCE is not supported due to performance implications. 

From the test in my environment I didn't require the DIMENSIONAL Metric, but that depends on what kind of permissions you want to give to the user.

[Babar_Qayyum]

Hello @DanielS 

Thank you for sharing the post. In the post, it is mentioned that we can apply the manual tag on the Pods level, whereas, I do not see this option in the settings except for the Container.

Regards,

Babar

[Babar_Qayyum]

Hello @DanielS 

Will the policy apply on the environmental/or management zones level? How can it be assigned only to an individual management zone?

Regards,

Babar

[DanielS]

You apply the policy under the environment but you give the group a Permission under the MZ only.