How to ingest additional Kubernetes Custom Resources (CRs) from a monitored cluster as a metric to be able to configure alerting based on that metric

[sheriflashin]

Table of Contents

• Summary
• Problem
• Workaround
• What's next

Summary

Kubernetes Platform Monitoring is ingesting a set of Kubernetes resources from the Kube-APIserver on the monitored Kubernetes cluster and present those resources as metrics in Dynatrace, e.g(workload metrics (CPU, Memory), namespace resource quotas,......,etc)

In some scenarios, those Kubernetes resources are not enough and customers might need to ingest additional Kubernetes resources from a monitored cluster, and setup an alert based on the ingested resources metric value.

 

Problem

Ingesting additional Kubernetes Custom Resources (CRs) from a monitored Kubernetes cluster that is not ingested into Dynatrace out of the box and present this Custom Resources as a metric in Dynatrace to be able to setup an alerting if the value of this metric breached a configured threshold value.

 

Workaround

Ingesting additional Custom Resources from Kubernetes cluster as Smartscape nodes:

Prerequisites

• Dynatrace SaaS environment powered by Grail and AppEngine
• DPS license that includes the Kubernetes Platform Monitoring capability
• Sufficient permissions to use the New Kubernetes app within your Dynatrace environment
• ActiveGate version 1.335+
• Enabling this feature will increase the ActiveGate's resource consumption, so make sure to watch for this and adapt resource's requests/limits accordingly.

For the below steps, we will Ingest additional Custom Resource (ClusterResourceQuota) from a monitored OpenShift Cluster as an example:

Step 1: Add the new permissions to the ClusterRole:

Please refer to this link to extend the permissions for Kubernetes Platform monitoring, by creating a ClusterRole with the required permissions and bind it to the dynatrace-activegate service account using a ClusterRoleBinding, below are the needed permissions to ingest ClusterResourceQuota CR from a monitored OpenShift Cluster:

- verbs:
  - list
  - watch
  - get
  apiGroups:
  - quota.openshift.io
  resources:
  - clusterresourcequotas

Step 2: Modify the ActiveGate's customProperties to include kubernetes_smartscape_monitored_custom_resources parameter for Kubernetes monitoring in the DynaKube YAML file:

Please refer to this link to extend monitoring of custom resources (CRs) using kubernetes_smartscape_monitored_custom_resources parameter on the ActiveGate's customProperties, below are the parameter's values needed to ingest ClusterResourceQuota CR from a monitored OpenShift Cluster:

spec:
  activeGate:
    capabilities:
    - kubernetes-monitoring
    customProperties:
      value: |
        [kubernetes_monitoring]
        kubernetes_smartscape_monitored_custom_resources = [{ "resource": "clusterresourcequotas"\, "kind": "ClusterResourceQuota"\, "apiGroup": "quota.openshift.io"\, "version": "v1"\, "namespaced": false\, "usePreferredApiVersion": false }]

Note: The resource value must be the lowercase plural name of the resource, and kind must match the Kubernetes kind exactly (PascalCase). Optionally, set usePreferredApiVersion for ActiveGate to use the preferred API version (if the preferred version can’t be determined, the version specified in version is used as a fallback).

Once the new permissions has been added to the ClusterRole and the ActiveGate's customProperties modified to include the kubernetes_smartscape_monitored_custom_resources parameter successfully, you can query the data via the below DQL query:

smartscapeNodes K8S_CLUSTERRESOURCEQUOTA

Step 3:  After successfully querying the K8S_CLUSTERRESOURCEQUOTA via the provided DQL query, you can create a workflow to read the smartscapeNode 'K8S_CLUSTERRESOURCEQUOTA' every 1 minute or so, extract the data and ingest it as custom metric:

Please refer to this link to create a Workflow with DQL query and HTTP request actions, attached to the article an example Workflow JSON file 'clusterresourcequota-pods-used-metric.workflow.json' that can be uploaded to a Dynatrace environment on the Workflow app to perform the below:

1) The Workflow gets triggered every 1 minute.
2) Reads ClusterResourceQuota Smartscape nodes and calculates used pods in percent.
3) Ingest the result as metric via REST API.

Notes:

• Please replace the below placeholders with the actual values on the Workflow:
  1. <DT-SaaS-FQDN> with the FQDN of your SaaS environment. 
  2. <OCP-CLUSTER-NAME> with the name of the OpenShift cluster where you modified the ActiveGate's customProperties for Kubernetes monitoring in the DynaKube YAML to include kubernetes_smartscape_monitored_custom_resources parameter and added the new permissions to the ClusterRole,
  3. <token-value> with an access token with metrics.ingest scope
• This Workflow is just a starting point and must be adapted based on your desired outcome.
• The Workflow can only handle one ClusterResourceQuota object.
• You might need to adjust the max result limits, and other parameters on the Workflow.

 

What's next

• Up to 5 extra Custom Resources types can be ingested as Smartscape nodes.
• We are working on a more easier way to extend monitoring of custom resources (CRs) that can be done via the Dynatrace UI, However, there is still no ETA yet on when this feature will be available.

• References:

https://docs.dynatrace.com/docs/shortlink/operator-security-rbac#activegate 

https://docs.dynatrace.com/docs/shortlink/custom-properties-file 

https://docs.dynatrace.com/docs/shortlink/semantic-dictionary-smartscape-kubernetes 

https://docs.dynatrace.com/docs/shortlink/workflows 

https://docs.dynatrace.com/docs/manage/identity-access-management/access-tokens-and-oauth-clients/ac... 

https://docs.dynatrace.com/docs/observe/infrastructure-observability/kubernetes-app/enable-k8s-exper...