Disclaimer: Some changes will be introduced with SaaS 1.294 and 1.296, you can read more about it in Introducing Intergrated Persistent Volume Claim Monitoring for Kubernetes.
- new introduced PVC metrics, current ones will not be supported anymore with Saas 1.296 (writing stops)
- new metrics are written by default
- new metrics are going to be supported out of the box, so there is no need for an extension anymore
However, the steps mentioned for analysis are still valid.
Please note that there are a few prerequisites that you must meet in order for your PVC data to be reflected in Dynatrace.
- ensure that the get rule and the nodes/metrics resources are enabled in the Kubernetes ClusterRole.
- enable Monitor persistent volume claims in the Settings->Monitoring section of your Kubernetes connection. (will be deprecated with SaaS 1.296)
- add the "Kubernetes persistent volume claims" from the Dynatrace Hub to your environment. (will be deprecated with SaaS 1.294)
- Activegate 1.289+ will be needed to fetch the new metrics
Please note that there is a limitation:
This feature is only available if your Kubernetes cluster is connected to a local Kubernetes API endpoint.If, however, you still do not see any data on the dashboard, please check the following points:
Please execute the following command on the Kubernetes cluster and substitute the {NODE-NAME} with your node name, which corresponds to the PVC.
kubectl get --raw /api/v1/nodes/{NODE-NAME}/proxy/metrics | grep kubelet_volume_stats_
This query will retrieve all volume stats metrics exposed by the kubelet. To locate a specific pod, use the pod, namespace, and persistentvolumeclaim labels. If the CSI driver mounting a volume does not expose volume statistics, this is indicated by missing PVC metrics for a pod claiming that volume.
For further information, please consult the documentation of a specific CSI driver to see if it exposes volume statistics.
If, however, you have results based on the previous kubectl command, it may be a connection issue with the metrics endpoint.
To check the status of connectivity, please execute the following query in Dynatrace Data Explorer. Please ensure that you replace the {K8S_CLUSTER_NAME} with your actual Kubernetes cluster name.
dsfm:active_gate.kubernetes.api.query_count:filter(and(or(in("dt.entity.kubernetes_cluster",entitySelector("type(kubernetes_cluster),entityName.equals(~"{K8S_CLUSTER_NAME}~")"))),or(eq(path,"/metrics")))):splitBy(status_code,status_reason):sum:sort(value(sum,descending)):limit(20)
The connection status can also be checked from the logs of the ActiveGate container. To do so, open the Properties and tags of your Kubernetes cluster connection, and click on the hyperlink of the ActiveGate ID. This will redirect you to the deployment status, filtering directly for the ActiveGate ID in question.
Please run the ActiveGate diagnostics, save the results locally, and unzip them. In the log folder, open the dynatracegateway.0.0.log file and look for log lines similar to the following:
- Query failed for endpoint /metrics on DirectIp with statusReason: ConnectionTimeout.
- Query failed for endpoint /metrics/cadvisor on DirectIp with statusReason: ConnectionTimeout.
The IP address displayed indicates the location where the ActiveGate pod attempts to connect. It is recommended that you consult with your Kubernetes administrator to verify the connectivity.
If you do not see a timeout message in Data Explorer or in the ActiveGate logs, please submit a support ticket. Please include a detailed description of the steps you have taken to check the issue, as this will help us to investigate it more efficiently.
