topic Re: Introducing built-in Persistent Volume Claim Monitoring for Kubernetes in Dynatrace tips

Introducing built-in Persistent Volume Claim Monitoring for Kubernetes

Alisopp — Wed, 16 Oct 2024 14:28:41 GMT

We are happy to announce, that PVC (Persistent Volume Claim) monitoring is becoming a built-in feature of our product. Previously, PVC monitoring was based on custom metric, and hence subject to additional licensing. With this change, PVC monitoring becomes part of your existing license, meaning there won't be any additional charges for using it. Going forward, we will also enable this feature for everyone per default.

I'm already using PVC monitoring today. Is there anything I need to do?

The integrated PVC monitoring will replace the current set of custom metrics with new built-in metrics. The assets provided by Dynatrace for PVC monitoring will be automatically updated by us. Specifically, the built-in anomaly detection for PVCs will automatically utilize the new metrics with version 1.294. For the PVC dashboard, we will deprecate the existing extensions, and ship an updated dashboard, based on the new metrics, as a built-in dashboard of Dynatrace.

If you are using today`s PVC metrics in any custom dashboard, metric event, or SLO, you need to manually migrate these assets by following the guide below.

Prerequisites

Kubernetes monitoring
ActiveGate version 1.289+
Managed/Saas classic the metrics are available with the existing license (e.g. Data Explorer, Dashboard Classic)
Saas platform (e.g. Notebooks app, Dashboards app) requires the New Kubernetes experience

Which metrics are affected

The affected metrics are the ones that are written upon enabling the PVC toggle in the monitoring settings of a Kubernetes cluster. Following table shows which metrics are affected and by which metric they are replaced:

Deprecated metric key (Decommission: Dynatrace 1.296)	1. Dynatrace Classic - New metric key (Kubernetes classic, Dashboards Classic, Data explorer, Metrics, ...) 2. Dynatrace Platform/Grail - New metric key* (Kubernetes, Dashboards, Notebooks, ...)	New metric name
kubelet_volume_stats_available_bytes	builtin:kubernetes.persistentvolumeclaim.available dt.kubernetes.persistentvolumeclaim.available	Kubernetes: Volume available bytes
kubelet_volume_stats_capacity_bytes	builtin:kubernetes.persistentvolumeclaim.capacity dt.kubernetes.persistentvolumeclaim.capacity	Kubernetes: Volume capacity
kubelet_volume_stats_used_bytes	builtin:kubernetes.persistentvolumeclaim.used dt.kubernetes.persistentvolumeclaim.used	Kubernetes: Volume used bytes

* Using the new PVC metrics in the platform requires the new Kubernetes experience.

How to get the new metrics

The new metrics will be written with updating your ActiveGates to version 1.289+. They are written per default, and are independent of the PVC toggle. The PVC toggle only switches writing of the deprecated PVC metrics.

How to successfully migrate to the new metrics

As there is no straight forward migration from an old metric to a new metric, we can't automatically migrate your custom dashboards, custom events for alerting, SLOs, or any other tool that utilizes our API to access these metrics.

To minimize the effort on your side, we offer a web application, called Metric Audit Report, guiding you through this process. So please make sure to update your ActiveGates to 1.289+, when this version becomes available for you and migrate your assets until Dynatrace 1.296 is released.

Deprecation: What happens to the old metrics?

They will still be written if the PVC toggle is enabled until Dynatrace 1.296. With Dynatrace 1.296 their decommission is planned. The PVC toggle will be no longer available.

For troubleshooting steps, see the article: Troubleshooting PVC not showing data.

Re: Introducing Integrated Persistent Volume Claim Monitoring for Kubernetes

sivart_89 — Fri, 31 May 2024 18:17:37 GMT

Will the method in which the metrics are being captured, remain the same? In other words, I believe the below is how dynatrace get some of the metrics, will the same endpoint be used?

If there are different endpoints that will be used can you provide one of them please? We have some clusters that are not exposing the pvc metrics in their current state of k8 version, I'm wondering if I will be able to start seeing them come 1.294 cluster version & 1.289 AG

/api/v1/nodes/{k8-node-name}/proxy/metrics

Re: Introducing Integrated Persistent Volume Claim Monitoring for Kubernetes

Alisopp — Wed, 05 Jun 2024 09:07:58 GMT

How we collect the data from Kubernetes remains the same, yes we are still using the node metrics endpoint for pvc.

With not exposing, you mean that the Kubernetes clusters do not provide any kubelet_volume_stats_ (e.g. kubelet_volume_stats_used_bytes) ,metric on these endpoints? Otherwise, please follow this Troubleshooting guide for PVC.

Also, the new metrics are already available with cluster version 1.290 and AG 1.289. In cluster version 1.294 we adapt our builtin features to use the new metrics instead of the old ones.

Re: Introducing Integrated Persistent Volume Claim Monitoring for Kubernetes

sivart_89 — Tue, 04 Jun 2024 17:35:55 GMT

That is correct, the diamanti clusters we have are on an older version of diamanti which puts us on k8 version 1.21. Not entirely sure if it is the k8 version itself being and older version that doesn't expose the metrics or if its entirely due to the diamanti version we are using.

Being the method in collecting the metrics is the same, we won't be able to get these in even in the new way. Nothing at fault from dynatrace, entirely on us as we just haven't moved very quickly with the k8 updates here 😀

Re: Introducing built-in Persistent Volume Claim Monitoring for Kubernetes

DavidFernandez — Thu, 04 Jul 2024 08:19:33 GMT

Hi team,

Should we expect any difference between the old and new metrics? For example, if we check the following metrics in Data Explorer, we don't see the same value (they are close, though, but we are expecting to see the exact same values if the way how metrics are gathered has not changed). How can we explain this? Adding a couple of examples here:

Thanks in advance!

Re: Introducing built-in Persistent Volume Claim Monitoring for Kubernetes

Alisopp — Mon, 08 Jul 2024 07:27:06 GMT

Hello David,

no the old and the new metrics have the same metric values, different is how they are displayed and the condition when they are written.

Both metrics are stored in Bytes. The old metrics were using the decimal prefix for display (KB, MB, GB, e.g. 1 GB = 1000 MB), the new one are using the binary prefix for display (KiB, MiB, GiB, 1 GiB = 1024MiB). In this case, check the unit of the displayed metrics.

The old metrics are written when you activate the PVC toggle in the monitoring settings of a Kubernetes cluster. The new metrics are written by default (independent of the PVC toggle state). In this case compare the metrics by splitting by Kubernetes cluster.

Re: Introducing built-in Persistent Volume Claim Monitoring for Kubernetes

sivart_89 — Mon, 06 Jan 2025 15:51:42 GMT

@Alisopp the metrics audit report link provided in the original post is not valid, is there an updated link?

Re: Introducing built-in Persistent Volume Claim Monitoring for Kubernetes

Peter_Jelitsch1 — Tue, 07 Jan 2025 11:23:31 GMT

I'm sorry, @sivart_89, but the metric audit tool has been discontinued in this form. If you found it helpful and you think, that it should be part of our product (either on platform or classic or both) in whatsoever form, feel free to create an RFE (request for enhancement). BR, PJ.

Re: Introducing built-in Persistent Volume Claim Monitoring for Kubernetes

AurelienGravier — Mon, 20 Jan 2025 10:55:57 GMT

Hello,

I understand that we're discussing metrics.

I would like to create advanced Davis anomaly detection alerts with thresholds based on the values of PVC labels.

In fact, I would like to set a threshold at 5% for the majority of PVCs, but have the option to lower this threshold to 1% for certain PVCs (by using tag or label), particularly those with large capacities of several terabytes.

For example:

timeseries {avg(dt.kubernetes.persistentvolumeclaim.available)}, by: { k8s.cluster.name, dt.entity.kubernetes_cluster, k8s.namespace.name,k8s.persistentvolumeclaim.name} | lookup [ fetch dt.entity.cloud_application_namespace | parse persistentVolumeClaimLabels[ACCEPT_LOW_DISK_SPACE], "DATA:ACCEPT_LOW_DISK_SPACE" ] | filter ACCEPT_LOW_DISK_SPACE == "true"

However, currently, there is no equivalent to the cloudApplicationLabels command for PVCs, and Dynatrace does not currently provide a dedicated entity for Persistent Volume Claims (PVCs) to set a tag.

Do you see another way to achieve this?

Thank you.
Regards