Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

ryan_korteway — Wed, 15 Oct 2025 14:22:50 GMT

Summary

This article is to help identify potential blockers to deep monitoring pod-contained apps within Kubernetes clusters.

Before we get started, we expect there to already be a Dynatrace Operator deployment in your cluster. If needed, see our documentation for deployment details .
You can learn more about controlling Pod Monitoring while using Dynatrace Operator Cloud Native Full Stack from our docs here: Configure monitoring for namespaces and pods

Troubleshooting Pod Injections

Step 1 - Validate Namespace

The first step of Pod injection is adding labels to application namespaces. Namespace labels can be listed using:

kubectl describe namespace <application namespace>

Look for the label: dynakube.internal.dynatrace.com/instance

C:\Users\ryan.korteway>kubectl describe ns default Name: default Labels: dynakube.internal.dynatrace.com/instance=sup-enablement-k8s-cnfs Annotations: <none> Status: Active

If you DO see this label but no deep monitoring data, move on to step 2.

If you do NOT see this label, then,

validate that communication between the Dynatrace namespace / pods and the kube-system namespace is functional
- See this page of Dynatrace Namespace Networking docs
Collect a Dynatrace Operator support archive before reaching out to Dynatrace Support.
- How to collect a Dynatrace Operator Support Archive.
- It is essential to send the whole archive, and not to just send the logs. Configuration and diagnostic runtime information is also collected, and extremely valuable to speed up the investigation of your reported issue.

Additionally, you can validate Namespace Labels within containerized process properties and tags dropdowns / pop-out menus within the Dynatrace UI:

Step 2 - Validate Pods

The next step is to describe the pod and look for signs of pod-level injection from the Dynatrace Webhook pods.

kubectl describe pod tomcat-10-deployment-5498c479d4-5hhjf -n default

Look for the dynatrace-operator init container^[1] being present, and if it finished successfully:

Controlled By: ReplicaSet/tomcat-10-deployment-5498c479d4 Init Containers: dynatrace-operator: Container ID: containerd://0ba7ca10b2d0554566c60611da43d6feb310aff829e6313c76226b658bfda093 Image: public.ecr.aws/dynatrace/dynatrace-operator:v1.5.1 Image ID: public.ecr.aws/dynatrace/dynatrace-operator@sha256:4b0994d8bd1d4723760f793421959377de391600de74104913ef1b1905d986ae Port: <none> Host Port: <none> SeccompProfile: RuntimeDefault Args: init State: Terminated Reason: Completed Exit Code: 0

With the right configuration, such as having metadataEnrichment enabled but a namespaceSelector at the oneAgent cloudNativeFullStack level, its possible to have an install-oneagent (operator version < 1.4.0) or dynatrace-operator (operator version > 1.4.0) init container but no deep monitoring for the pods.

As such, other key items to watch for are the LD_PRELOAD environment variable and the volume mounts for /opt/dynatrace/oneagent-paas volumes:

Containers: tomcat10-container: Container ID: containerd://409633c542b9d1fa6b3e93231cc61f17a1ce4e12dd49a5c4156936338ec07de6 Image: tomcat:10 Image ID: docker.io/library/tomcat@sha256:0e763cfc66a42679d63132ceacfbf7082a219c0385ca75f85baf3294e76bacf7 Port: 8080/TCP Host Port: 0/TCP Command: /bin/sh -c echo 'Hello from Kubernetes!' & catalina.sh run State: Running Started: Fri, 06 Dec 2024 12:53:02 -0500 Ready: True Restart Count: 0 Environment: DT_CUSTOM_PROP: cnfs_cluster location_us_east DT_RELEASE_STAGE: cloudNativeFullStack DT_DEPLOYMENT_METADATA: orchestration_tech=Operator-cloud_native_fullstack;script_version=v1.3.2;orchestrator_id=f27c4518-9ead-4488-bab0-b2cde9649669 LD_PRELOAD: /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so DT_NETWORK_ZONE: azure-us-east Mounts: /etc/ld.so.preload from oneagent-share (rw,path="ld.so.preload") /opt/dynatrace/oneagent-paas from oneagent-bin (rw) /var/lib/dynatrace/enrichment/dt_metadata.json from metadata-enrichment (rw,path="dt_metadata_tomcat10-container.json") /var/lib/dynatrace/enrichment/dt_metadata.properties from metadata-enrichment (rw,path="dt_metadata_tomcat10-container.properties") /var/lib/dynatrace/enrichment/endpoint from metadata-enrichment-endpoint (rw) /var/lib/dynatrace/oneagent/agent/config/container.conf from oneagent-share (rw,path="container_tomcat10-container.conf") /var/lib/dynatrace/oneagent/agent/customkeys/curl_options.conf from oneagent-share (rw,path="curl_options.conf") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lzdk6 (ro)

If you DO see the init container and mounts, etc. but still no deep monitoring data, move on to step 3.

Now would also be a good time to double-check for possible metadataEnrichment injection, but not Cloud Native Full Stack injection due to having a namespaceSelector defined at that level.

If you do NOT see the init container or any of the mounts/environment variables needed, then,

validate that communication between the Dynatrace namespace/ pods and the kube-system namespace is functional. It's the Dynatrace Webhook that provides the mutating webhook that adds these items to the pods (see ingress docs above if necessary)
and collect a Dynatrace Operator support archive before reaching out to Dynatrace Support:
- How to collect a Dynatrace Operator Support Archive.
- It is essential to send the whole archive, and not to just send the logs. Configuration and diagnostic runtime information is also collected, and extremely valuable to speed up the investigation of your reported issue.
- Depending on the health/status of the dynatrace-operator^[1] init container, we may also require the logs from the relevant container:
- ```
kubectl logs <App pod name> -n <App Pod Namespace> -c dynatrace-operator
ex.
kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c dynatrace-operator
```

Step 3 - Validate Processes within the Pod

The first step of process validation within the pod is to exec into the pod and look for logs within /opt/dynatrace/oneagent-paas/log/<tech type, java, nginx, go, etc.>

kubectl exec -it tomcat-10-deployment-5498c479d4-5hhjf -n default -- /bin/sh # bash root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# cd /opt/dynatrace/oneagent-paas/log/java/ root@tomcat-10-deployment-5498c479d4-5hhjf:/opt/dynatrace/oneagent-paas/log/java# ls ruxitagent_tomcat_tomcat-_-deployment-__cloudNativeFullStack_8.0.log root@tomcat-10-deployment-5498c479d4-5hhjf:/opt/dynatrace/oneagent-paas/log/java#

If there are logs, we most likely have a networking or config blocker issue and can learn about what is going on by reading the log files and reaching out to Dynatrace Support for assistance if needed.

If there are no logs, then this usually means that there is no OneAgent library loading taking place. Still, we can double-check on that possibility using the "env" command to validate that LD_PRELOAD is set within the Pod's environment and not just being within the pod "describe":

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# env | grep LD_PRELOAD LD_PRELOAD=/opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so

Note: It is easier to do this phase of the investigation with ps aux available within the pod to find the process IDs but I know some images do not include the procps package so I will also provide a pure bash alternative.

Now look for the above-mentioned library within your processes' loaded libraries memory addresses via the /proc/<PID>/maps file within the Linux /proc files.

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 6060 2156 ? Ss Dec06 0:00 /bin/sh -c echo 'Hello from Kubernetes!' & catalina.s root 8 0.2 1.6 7969628 273124 ? Sl Dec06 22:38 /opt/java/openjdk/bin/java -Djava.util.logging.config // OR use the pure bash command: root@tomcat-10-deployment-5498c479d4-5hhjf:/ find /proc/*[0-9]/cmdline -type f -print -exec cat {} \; -exec printf '\n\n' \; /proc/1/cmdline /bin/sh-cecho 'Hello from Kubernetes!' & catalina.sh run .......... /proc/8/cmdline /opt/java/openjdk/bin/java-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager-Djdk.tls.ephemeralDHKeySize=2048-Djava.protocol.handler.pkgs=org.apache.catalina.webresources-Dorg.apache.catalina.security.SecurityListener.UMASK=0027--add-opens=java.base/java.lang=ALL-UNNAMED--add-opens=java.base/java.io=ALL-UNNAMED--add-opens=java.base/java.util=ALL-UNNAMED--add-opens=java.base/java.util.concurrent=ALL-UNNAMED--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED-classpath/usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar-Dcatalina.base=/usr/local/tomcat-Dcatalina.home=/usr/local/tomcat-Djava.io.tmpdir=/usr/local/tomcat/temporg.apache.catalina.startup.Bootstrapstart

Once the app PID is identified, in this case the app is tomcat/ Java with a pid of 8, then we can use /proc/8/maps to learn about what is loaded into the app:

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# cat /proc/8/maps | grep oneagent 7f3b5781e000-7f3b5785a000 r--s 003af000 00:2e5 3893709 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.livedebugger.jar 7f3b5785a000-7f3b5789d000 r--s 0030d000 00:2e5 3893712 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.jar 7f3b64023000-7f3b64027000 r--s 00019000 00:2e5 3893716 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.8.jar 7f3b64027000-7f3b64029000 r--s 0000c000 00:2e5 3893719 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.11.jar 7f3b660f8000-7f3b6625e000 r--p 00000000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so 7f3b6625e000-7f3b675b8000 r-xp 00166000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so 7f3b675b8000-7f3b67af1000 r--p 014c0000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so 7f3b67af1000-7f3b67bd3000 r--p 019f8000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so 7f3b67bd3000-7f3b67bdd000 rw-p 01ada000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so 7f3b69718000-7f3b69719000 r--s 00005000 00:2e5 3893718 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.17.jar 7f3b69736000-7f3b69741000 r--p 00000000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so 7f3b69741000-7f3b697de000 r-xp 0000b000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so 7f3b697de000-7f3b69821000 r--p 000a8000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so 7f3b69821000-7f3b69827000 r--p 000ea000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so 7f3b69827000-7f3b69828000 rw-p 000f0000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so

Above, we can see both liboneagentproc within the file, which means we had OneAgent process injection/ file loading AND liboneagentjava, which is for OneAgent Java code module Injection/ loading.

If you can see both types of OneAgent libraries loaded but still no logs visible in the expected location, then you can also use /proc/PID/fd to find open file handles for the app process:

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# ls -l /proc/8/fd | grep log lrwx------ 1 root root 64 Dec 13 18:17 4 -> /opt/dynatrace/oneagent-paas/log/java/ruxitagent_tomcat_tomcat-_-deployment-__cloudNativeFullStack_8.0.log l-wx------ 1 root root 64 Dec 13 18:17 65 -> /usr/local/tomcat/logs/catalina.2024-12-06.log l-wx------ 1 root root 64 Dec 13 18:17 68 -> /usr/local/tomcat/logs/localhost_access_log.2024-12-06.txt

In this case, the OneAgent Java code module is file handle 4, and we can now see the path we would need to take to read the log if it was in a "non-standard location."

Alternatively, you can print the OneAgent logs to stdout, aka kubectl log <pod name>, and prepare a Dynatrace support ticket by adding the following environment variables within your pods deployment yaml file or the container env spec section of any deployment template you're using before restarting the deployment:

Environment variable name	Environment variable value
DT_LOGLEVEL_PROC	config=debug,pgid=debug
DT_LOGCON_PROC	stdout
DT_LOGLEVELCON	info

What's Next?

If you've not been able to find the OneAgent logs in the container or find the OneAgent libraries within your processes, etc., then open a Dynatrace Support ticket with the following information collected:

A Dynatrace Operator Support archive.
- How to collect a Dynatrace Operator Support Archive.
- It is essential to send the whole archive, and not to just send the logs. Configuration and diagnostic runtime information is also collected, and extremely valuable to speed up the investigation of your reported issue.
The logs from the pod itself, preferably with the above environment variables in place:
- ```
kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default
```

And the dynatrace-operator^1] init container logs:

kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c dynatrace-operator

[1] Starting with Dynatrace Operator 1.4.0, the install-oneagent initcontainer was renamed the dynatrace-operator initcontainer. Please substitute commands as appropriate. IE.

kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c dynatrace-operator

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

sivart_89 — Thu, 20 Feb 2025 12:11:00 GMT

@ryan_korteway thank you for this right up. The 2nd link is invalid (DT Documentation about pod and namespace annotations). I'm currently looking through this because I've noticed that 1 cluster of ours has label dynakube.internal.dynatrace.com/instance in our namespaces while others do not. If the label does not exist in a namespace does that mean no dynatrace agent is being installed (as an init container) into their pod? I'm trying to understand what we are missing by not having this label in our namespaces.

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

ryan_korteway — Tue, 25 Feb 2025 16:26:09 GMT

Hello Sivart,

Not a problem at all on the write up and thank you for the heads up on the broken doc link which I will be fixing right after this comment.

So my understanding is if your namespace is missing that annotation, then the namespace and thus all of the pods within it are not being watched for pod starts and therefore will not have any pod injections.

Sounds to me like you may have some namespaceSelectors set up within the dynakube yaml files for some of your clusters / DT operator deployments.

It may also warrant checking the dynakube yaml file for any dynatrace operator config annotations that disable automatic monitoring for all namespaces within the cluster etc.

Please let me know if you have any further questions or concerns.

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

steven_v — Sat, 25 Oct 2025 15:44:39 GMT

@ryan_korteway Thank you for this informative and detailed write up! I was struggling to understand why my pods were not getting the injection and instrumentation. It turns out we had a namespaceSelector value set in our Dynakube yaml to disable injection on our default/system labeled namespaces 🙂

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

StrangerThing — Sat, 25 Oct 2025 15:49:40 GMT

Have you tried to automate this process with something in the pipeline or on the DT side with a Workflow? Seems like a great opportunity to automate this validation so we don't always have to check every time there's a deployment to k8s.

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

steven_v — Sat, 25 Oct 2025 17:43:46 GMT

I have not personally but that's a fantastic idea to explore. The k8s connector sounds like something cool we can look at:

https://docs.dynatrace.com/docs/analyze-explore-automate/workflows/actions/kubernetes-automation/get-started-with-kubernetes-workflows

article Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting in Troubleshooting

Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

Summary

Troubleshooting Pod Injections

Step 1 - Validate Namespace

Step 2 - Validate Pods

Step 3 - Validate Processes within the Pod

What's Next?

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

Re: Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting