Hi,
We've been using Dynatrace on Kubernetes for some time and over last few months we've updated versions and ended up with new cloud native full stack installation.
# previous step
kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v1.0.1/kubernetes.yaml
# updated version
kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v1.5.0/kubernetes-csi.yaml
Due to the fact that there is a dependency between dynakube-oneagent and dynatrace-oneagent-csi-driver pods it's already adding extra 1 minute before dynakube-oneagent can start. Then because of init container dependencies on application pods if dynakube-oneagent is taking long time to start, the application pods are affected. Timing wise it's sometimes between 10-15 minutes before applications pods are ready to serve a traffic (in k8s world I would say it's terrible, in case of autoscaling and other events like node crashes, etc.).
As we cannot do too much with csi driver pods in above steps, I started to look at what exactly it's happening with the containers in dynakube-oneagent. Base on the logs I could see e.g. extracting/unpacking took 2minutes, however this was triggered due to the version differs and upgrade requirements.
08:27:31 Bootstrapping read-only deployment
08:27:31 AGENT_CONTAINER_IMAGE_VERSION=1.321.24.20250820-221352
08:28:46 Finished setting up /mnt/volume_storage_mount/host_root
...
08:28:49 Moving downloaded installer to /mnt/volume_storage_mount/host_root/opt/dynatrace/oneagent/Dynatrace-OneAgent-Linux.sh
08:28:52 Verifying agent installer signature
08:30:48 CMS Verification successful
08:30:48 Creating /mnt/volume_storage_mount/host_root/opt/dynatrace/oneagent/agent/conf/dockerdeployment.conf
08:30:48 Downloaded version differs from installed version, will re-run installer
...
08:31:00 Installation started, version 1.321.51.20250905-075429, build date: 05.09.2025, PID 19017.
08:31:01 Detected platform: LINUX arch: X86_64
08:31:01 Checking /opt/dynatrace/oneagent for prohibited mount flags
08:31:02 Checking free space in /opt/dynatrace/oneagent
08:31:03 Extracting...
08:31:10 Unpacking. This may take a while...
08:33:32 Unpacking complete.
...
08:35:37 Starting Dynatrace OneAgent from within the container.
So here is a problem or at least one of them. I've tried to stop that check version to update during container startup (tbh, I don't get it why you would update version on ephemeral container) and I couldn't figured out why it's not working. Here is a k8s manifest which is being applied.
apiVersion: dynatrace.com/v1beta5
kind: DynaKube
metadata:
name: dynakube
namespace: dynatrace
spec:
apiUrl: <redacted>
oneAgent:
# https://docs.dynatrace.com/docs/ingest-from/setup-on-k8s/reference/dynakube-parameters
cloudNativeFullStack:
# autoUpdate: false # Deprecated field to be removed in a future release.
version: "1.321.24.20250820-221352"
So main questions from me are:
- is it something I'm missing in the installation steps to stop upgrading app everytime oneagent is starting up.
- ideally application should start without waiting for oneagent, however base on what I can see, don't think so it's possible as injection is part of init container for each application pod?
Thanks
