Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

[ryan_korteway]

• Summary
• Troubleshooting Pod Injections
• Step 1 - Validate Namespace
• Step 2 - Validate Pods
• Step 3 - Validate Processes within the Pod
• What's Next?

 

Summary

This article is to help identify potential blockers to deep monitoring pod-contained apps within Kubernetes clusters.

• Before we get started, we expect there to already be a Dynatrace Operator deployment in your cluster. If needed, see our documentation for deployment details.
• You can learn more about controlling Pod Monitoring while using Dynatrace Operator Cloud Native Full Stack from our docs here: Configure monitoring for namespaces and pods 

Troubleshooting Pod Injections

Step 1 - Validate Namespace

The first step of Pod injection is adding labels to application namespaces. Namespace labels can be listed using:

kubectl describe namespace <application namespace>

Look for the label: dynakube.internal.dynatrace.com/instance

C:\Users\ryan.korteway>kubectl describe ns default
Name:         default
Labels:       dynakube.internal.dynatrace.com/instance=sup-enablement-k8s-cnfs
Annotations:  <none>
Status:       Active

If you DO see this label but no deep monitoring data, move on to step 2. 

If you do NOT see this label, then,

• validate that communication between the Dynatrace namespace / pods and the kube-system namespace is functional
  • See this page of Dynatrace Namespace Networking docs 
• Collect a Dynatrace Operator support archive before reaching out to Dynatrace Support.
  • How to collect a Dynatrace Operator Support Archive. 
  • It is essential to send the whole archive, and not to just send the logs. Configuration and diagnostic runtime information is also collected, and extremely valuable to speed up the investigation of your reported issue. 
    
    

Additionally, you can validate Namespace Labels within containerized process properties and tags dropdowns / pop-out menus within the Dynatrace UI: 

 

Step 2 - Validate Pods

The next step is to describe the pod and look for signs of pod-level injection from the Dynatrace Webhook pods.

kubectl describe pod tomcat-10-deployment-5498c479d4-5hhjf -n default

Look for the dynatrace-operator init container^[1] being present, and if it finished successfully:

Controlled By:  ReplicaSet/tomcat-10-deployment-5498c479d4
Init Containers:
   dynatrace-operator:
    Container ID:  containerd://0ba7ca10b2d0554566c60611da43d6feb310aff829e6313c76226b658bfda093
    Image:         public.ecr.aws/dynatrace/dynatrace-operator:v1.5.1
    Image ID:      public.ecr.aws/dynatrace/dynatrace-operator@sha256:4b0994d8bd1d4723760f793421959377de391600de74104913ef1b1905d986ae
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Args:
      init
    State:          Terminated
      Reason:       Completed
      Exit Code:    0​

With the right configuration, such as having metadataEnrichment enabled but a namespaceSelector at the oneAgent cloudNativeFullStack level, its possible to have an install-oneagent (operator version < 1.4.0) or dynatrace-operator (operator version > 1.4.0) init container but no deep monitoring for the pods.

As such, other key items to watch for are the LD_PRELOAD environment variable and the volume mounts for /opt/dynatrace/oneagent-paas volumes: 

Containers:
  tomcat10-container:
    Container ID:  containerd://409633c542b9d1fa6b3e93231cc61f17a1ce4e12dd49a5c4156936338ec07de6
    Image:         tomcat:10
    Image ID:      docker.io/library/tomcat@sha256:0e763cfc66a42679d63132ceacfbf7082a219c0385ca75f85baf3294e76bacf7
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
      -c
      echo 'Hello from Kubernetes!' &
      catalina.sh run

    State:          Running
      Started:      Fri, 06 Dec 2024 12:53:02 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      DT_CUSTOM_PROP:          cnfs_cluster location_us_east
      DT_RELEASE_STAGE:        cloudNativeFullStack
      DT_DEPLOYMENT_METADATA:  orchestration_tech=Operator-cloud_native_fullstack;script_version=v1.3.2;orchestrator_id=f27c4518-9ead-4488-bab0-b2cde9649669
      LD_PRELOAD:              /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
      DT_NETWORK_ZONE:         azure-us-east
    Mounts:
      /etc/ld.so.preload from oneagent-share (rw,path="ld.so.preload")
      /opt/dynatrace/oneagent-paas from oneagent-bin (rw)
      /var/lib/dynatrace/enrichment/dt_metadata.json from metadata-enrichment (rw,path="dt_metadata_tomcat10-container.json")
      /var/lib/dynatrace/enrichment/dt_metadata.properties from metadata-enrichment (rw,path="dt_metadata_tomcat10-container.properties")
      /var/lib/dynatrace/enrichment/endpoint from metadata-enrichment-endpoint (rw)
      /var/lib/dynatrace/oneagent/agent/config/container.conf from oneagent-share (rw,path="container_tomcat10-container.conf")
      /var/lib/dynatrace/oneagent/agent/customkeys/curl_options.conf from oneagent-share (rw,path="curl_options.conf")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lzdk6 (ro)​

If you DO see the init container and mounts, etc. but still no deep monitoring data, move on to step 3.

• Now would also be a good time to double-check for possible metadataEnrichment injection, but not Cloud Native Full Stack injection due to having a namespaceSelector defined at that level.

If you do NOT see the init container or any of the mounts/environment variables needed, then,

• validate that communication between the Dynatrace namespace/ pods and the kube-system namespace is functional. It's the Dynatrace Webhook that provides the mutating webhook that adds these items to the pods (see ingress docs above if necessary)
• and collect a Dynatrace Operator support archive before reaching out to Dynatrace Support: 
  • How to collect a Dynatrace Operator Support Archive. 
  • It is essential to send the whole archive, and not to just send the logs. Configuration and diagnostic runtime information is also collected, and extremely valuable to speed up the investigation of your reported issue. 
  • Depending on the health/status of the dynatrace-operator^[1] init container, we may also require the logs from the relevant container:

  • kubectl logs <App pod name> -n <App Pod Namespace> -c dynatrace-operator
    ex.
    kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c dynatrace-operator

 

Step 3 - Validate Processes within the Pod

The first step of process validation within the pod is to exec into the pod and look for logs within /opt/dynatrace/oneagent-paas/log/<tech type, java, nginx, go, etc.>

kubectl exec -it tomcat-10-deployment-5498c479d4-5hhjf -n default -- /bin/sh
# bash

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# cd /opt/dynatrace/oneagent-paas/log/java/

root@tomcat-10-deployment-5498c479d4-5hhjf:/opt/dynatrace/oneagent-paas/log/java# ls
ruxitagent_tomcat_tomcat-_-deployment-__cloudNativeFullStack_8.0.log

root@tomcat-10-deployment-5498c479d4-5hhjf:/opt/dynatrace/oneagent-paas/log/java#​

If there are logs, we most likely have a networking or config blocker issue and can learn about what is going on by reading the log files and reaching out to Dynatrace Support for assistance if needed.

If there are no logs, then this usually means that there is no OneAgent library loading taking place. Still, we can double-check on that possibility using the "env" command to validate that LD_PRELOAD is set within the Pod's environment and not just being within the pod "describe": 

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# env | grep LD_PRELOAD
LD_PRELOAD=/opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so​

Note: It is easier to do this phase of the investigation with ps aux available within the pod to find the process IDs but I know some images do not include the procps package so I will also provide a pure bash alternative.

Now look for the above-mentioned library within your processes' loaded libraries memory addresses via the /proc/<PID>/maps file within the Linux /proc files.

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   6060  2156 ?        Ss   Dec06   0:00 /bin/sh -c echo 'Hello from Kubernetes!' & catalina.s
root           8  0.2  1.6 7969628 273124 ?      Sl   Dec06  22:38 /opt/java/openjdk/bin/java -Djava.util.logging.config

// OR use the pure bash command:
root@tomcat-10-deployment-5498c479d4-5hhjf:/ find /proc/*[0-9]/cmdline -type f -print -exec cat {} \; -exec printf '\n\n' \;

/proc/1/cmdline
/bin/sh-cecho 'Hello from Kubernetes!' &
catalina.sh run

..........

/proc/8/cmdline
/opt/java/openjdk/bin/java-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager-Djdk.tls.ephemeralDHKeySize=2048-Djava.protocol.handler.pkgs=org.apache.catalina.webresources-Dorg.apache.catalina.security.SecurityListener.UMASK=0027--add-opens=java.base/java.lang=ALL-UNNAMED--add-opens=java.base/java.io=ALL-UNNAMED--add-opens=java.base/java.util=ALL-UNNAMED--add-opens=java.base/java.util.concurrent=ALL-UNNAMED--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED-classpath/usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar-Dcatalina.base=/usr/local/tomcat-Dcatalina.home=/usr/local/tomcat-Djava.io.tmpdir=/usr/local/tomcat/temporg.apache.catalina.startup.Bootstrapstart​

Once the app PID is identified, in this case the app is tomcat/ Java with a pid of 8, then we can use /proc/8/maps to learn about what is loaded into the app:

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# cat /proc/8/maps | grep oneagent
7f3b5781e000-7f3b5785a000 r--s 003af000 00:2e5 3893709                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.livedebugger.jar
7f3b5785a000-7f3b5789d000 r--s 0030d000 00:2e5 3893712                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.jar
7f3b64023000-7f3b64027000 r--s 00019000 00:2e5 3893716                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.8.jar
7f3b64027000-7f3b64029000 r--s 0000c000 00:2e5 3893719                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.11.jar
7f3b660f8000-7f3b6625e000 r--p 00000000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b6625e000-7f3b675b8000 r-xp 00166000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b675b8000-7f3b67af1000 r--p 014c0000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b67af1000-7f3b67bd3000 r--p 019f8000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b67bd3000-7f3b67bdd000 rw-p 01ada000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b69718000-7f3b69719000 r--s 00005000 00:2e5 3893718                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.17.jar
7f3b69736000-7f3b69741000 r--p 00000000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b69741000-7f3b697de000 r-xp 0000b000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b697de000-7f3b69821000 r--p 000a8000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b69821000-7f3b69827000 r--p 000ea000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b69827000-7f3b69828000 rw-p 000f0000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so

Above, we can see both liboneagentproc within the file, which means we had OneAgent process injection/ file loading AND liboneagentjava, which is for OneAgent Java code module Injection/ loading.

If you can see both types of OneAgent libraries loaded but still no logs visible in the expected location, then you can also use /proc/PID/fd to find open file handles for the app process:

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# ls -l /proc/8/fd | grep log
lrwx------ 1 root root 64 Dec 13 18:17 4 -> /opt/dynatrace/oneagent-paas/log/java/ruxitagent_tomcat_tomcat-_-deployment-__cloudNativeFullStack_8.0.log
l-wx------ 1 root root 64 Dec 13 18:17 65 -> /usr/local/tomcat/logs/catalina.2024-12-06.log
l-wx------ 1 root root 64 Dec 13 18:17 68 -> /usr/local/tomcat/logs/localhost_access_log.2024-12-06.txt​

In this case, the OneAgent Java code module is file handle 4, and we can now see the path we would need to take to read the log if it was in a "non-standard location."

Alternatively, you can print the OneAgent logs to stdout, aka kubectl log <pod name>, and prepare a Dynatrace support ticket by adding the following environment variables within your pods deployment yaml file or the container env spec section of any deployment template you're using before restarting the deployment:

Environment variable name	Environment variable value
DT_LOGLEVEL_PROC	config=debug,pgid=debug
DT_LOGCON_PROC	stdout
DT_LOGLEVELCON	info

What's Next?

If you've not been able to find the OneAgent logs in the container or find the OneAgent libraries within your processes, etc., then open a Dynatrace Support ticket with the following information collected:

• A Dynatrace Operator Support archive.
  • How to collect a Dynatrace Operator Support Archive. 
  • It is essential to send the whole archive, and not to just send the logs. Configuration and diagnostic runtime information is also collected, and extremely valuable to speed up the investigation of your reported issue. 
• The logs from the pod itself, preferably with the above environment variables in place:

  • kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default

• And the dynatrace-operator^1]  init container logs: 
  

  • kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c dynatrace-operator

[1] Starting with Dynatrace Operator 1.4.0, the install-oneagent initcontainer was renamed the dynatrace-operator initcontainer. Please substitute commands as appropriate. IE.

• kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c dynatrace-operator

[sivart_89]

@ryan_korteway thank you for this right up. The 2nd link is invalid (DT Documentation about pod and namespace annotations). I'm currently looking through this because I've noticed that 1 cluster of ours has label dynakube.internal.dynatrace.com/instance in our namespaces while others do not. If the label does not exist in a namespace does that mean no dynatrace agent is being installed (as an init container) into their pod? I'm trying to understand what we are missing by not having this label in our namespaces.

[ryan_korteway]

Hello Sivart,

Not a problem at all on the write up and thank you for the heads up on the broken doc link which I will be fixing right after this comment. 

So my understanding is if your namespace is missing that annotation, then the namespace and thus all of the pods within it are not being watched for pod starts and therefore will not have any pod injections.

Sounds to me like you may have some namespaceSelectors set up within the dynakube yaml files for some of your clusters / DT operator deployments.

It may also warrant checking the dynakube yaml file for any dynatrace operator config annotations that disable automatic monitoring for all namespaces within the cluster etc.

Please let me know if you have any further questions or concerns.

[steven_v]

@ryan_korteway Thank you for this informative and detailed write up! I was struggling to understand why my pods were not getting the injection and instrumentation. It turns out we had a namespaceSelector value set in our Dynakube yaml to disable injection on our default/system labeled namespaces 🙂 

[StrangerThing]

Have you tried to automate this process with something in the pipeline or on the DT side with a Workflow? Seems like a great opportunity to automate this validation so we don't always have to check every time there's a deployment to k8s. 

[steven_v]

I have not personally but that's a fantastic idea to explore. The k8s connector sounds like something cool we can look at:

https://docs.dynatrace.com/docs/analyze-explore-automate/workflows/actions/kubernetes-automation/get...