Summary
This article walks you through validating a known situation of high disk usage on Kubernetes Nodes where the OA CSI Driver is running and provides a workaround via a Dynakube configuration change.
Please note that these observations and provided numbers are all experience-based and subject to environmental differences.
Problem
The problem presents itself as high disk usage due to OneAgent code module libraries being duplicated into each container's specific paths instead of being downloaded once and shared between all deeply monitored containers.
Troubleshooting steps
Confirm disk usage by logging into your Kubernetes nodes and validating the following path using the du tool:
cd /var/lib/kubelet/plugins/csi.oneagent.dynatrace.com/data
du -xh -d2
1.6G ./_dynakubes/<Dynakube CRD Name>
1.6G ./_dynakubes
1004M ./codemodules/<Currently deployed OA Code Module version>
2.0G ./codemodules
44K ./appmounts/csi-596216c7ef94c7c5de0ad9bac898a9c4790698fa698c64b4f7c16ff65f8b0ab2
44K ./appmounts/csi-b061260662a103676a7e8f63e567c9fd4f118cbff61e5b70d97bcf1db34954f1
44K ./appmounts/csi-3a0ea8f3b5e1f921c5f0d617bb6d86e0326869397c420bc922832e81349d3f43
136K ./appmounts
3.6G .
In the above example the container specific , ie ./appmounts/* , folders and their children are only using 44K each and the larger data usage is centralized under ./_dynakubes and ./codemodules as we would normally expect to see.
However, it is possible to see usage closer to 1000M or more per folder if the codemodules are being fully copied into each directory instead of being referenced from the base ./codemodules/* folders due to unexpected filesystem operations such as file metadata alterations etc.
Please note that all of these paths and values can change based on deployment details (i.e. the name of your dynakube custom resource and the version of the Dynatrace Operator used). Depending on your Operator version you may see ./pods/* instead of ./appmounts/* within the above du output.
Resolution
Currently there are two options available to workaround this situation.
Option 1
1. convert the CSI driver mounts to read only mode using the available dynakube flag:
apiVersion: dynatrace.com/v1beta2
kind: DynaKube
metadata:
name: dynakube
namespace: dynatrace
annotations:
feature.dynatrace.com/injection-readonly-volume: "true"
https://docs.dynatrace.com/docs/ingest-from/setup-on-k8s/reference/dynakube-feature-flags
2. After making this annotation change and reapplying the dynakube.yaml file, recycle the pods to have the config change take effect and see the disk usage decrease.
Option 2
Upgrade to Dynatrace Operator version 1.5.0+ so you can use Cloud Native Full Stack without the CSI Driver: https://docs.dynatrace.com/docs/shortlink/release-notes-dto-fix-1-5-0#new-features-and-enhancements
*** Usage of DT Operator 1.5.1 or above is recommended over 1.5.0 due to critical patches
What's next
If this article didn't help or you would like to investigate further, please open a support ticket, mention that this article was used and provide the following in the ticket:
