Dynatrace Operator - Cloud Native Full Stack - Pod Injection Validation and Troubleshooting

ryan_korteway · ‎23 Dec 2024

Getting started

This Partial Self-Service guide is to help identify potential blockers to deeply monitoring pod contained apps within kubernetes clusters.

Before we get started, the expectation is that there will already be a Dynatrace Operator deployment in your cluster. See our documentation for deployment details if needed.
You can learn more about controlling Pod Monitoring while using Dynatrace Operator Cloud Native Full Stack from our docs here: Configure monitoring for namespaces and pods

Troubleshooting Pod Injections

Step 1 - Validate Namespace

The first step of Pod injection is Labels being added to application namespaces and so Troubleshooting Step 1 - is validating namespace Labels:

kubectl describe namespace <application namespace>

look for the label: dynakube.internal.dynatrace.com/instance

C:\Users\ryan.korteway>kubectl describe ns default
Name:         default
Labels:       dynakube.internal.dynatrace.com/instance=sup-enablement-k8s-cnfs
Annotations:  <none>
Status:       Active

If you DO see this label but no deep monitoring data, move on to step 2.

If you do NOT see this label then,

validate that communication between the dynatrace namespace / pods and the kube-system namespace is functional
- For more details see this page of Dynatrace Namespace Networking docs
collect a Dynatrace Operator support archive before reaching out to Dynatrace Support.
- Please follow these steps in our documentation to collect said Dynatrace Operator Support Archive.
- Please note that it is important not to just send the logs from the support archive. Configuration Information/Diagnostic runtime information is also collected and extremely valuable to speed up the investigation of your reported issue.

Additionally you can validate Namespace Labels within containerized process properties and tags dropdowns / pop out menus within the Dynatrace UI:

alternative sensoring.png

Step 2 - Validate Pods

Next step is to describe the pod and look for signs of pod level injection from the Dynatrace Webhook pods.

kubectl describe pod tomcat-10-deployment-5498c479d4-5hhjf -n default

Look for the install-oneagent initcontainer^[1] being present and if it finished successfully:

Controlled By:  ReplicaSet/tomcat-10-deployment-5498c479d4
Init Containers:
  install-oneagent:
    Container ID:    containerd://35a91cf04a8b266be5a2648540d52e5c1247b60ef22abf6d71b5dc1bd0180093
    Image:           public.ecr.aws/dynatrace/dynatrace-operator:v1.3.2
    Image ID:        public.ecr.aws/dynatrace/dynatrace-operator@sha256:f8ecdcd87d7d84b87e645074084dd7f57dd62c76e120bb21e5abde158755be56
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Args:
      init
    State:          Terminated
      Reason:       Completed
      Exit Code:    0

Other key items to watch for are the LD_PRELOAD environment variable and the key volume mounts for /opt/dynatrace/oneagent-paas volumes:

Containers:
  tomcat10-container:
    Container ID:  containerd://409633c542b9d1fa6b3e93231cc61f17a1ce4e12dd49a5c4156936338ec07de6
    Image:         tomcat:10
    Image ID:      docker.io/library/tomcat@sha256:0e763cfc66a42679d63132ceacfbf7082a219c0385ca75f85baf3294e76bacf7
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
      -c
      echo 'Hello from Kubernetes!' &
      catalina.sh run

    State:          Running
      Started:      Fri, 06 Dec 2024 12:53:02 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      DT_CUSTOM_PROP:          cnfs_cluster location_us_east
      DT_RELEASE_STAGE:        cloudNativeFullStack
      DT_DEPLOYMENT_METADATA:  orchestration_tech=Operator-cloud_native_fullstack;script_version=v1.3.2;orchestrator_id=f27c4518-9ead-4488-bab0-b2cde9649669
      LD_PRELOAD:              /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
      DT_NETWORK_ZONE:         azure-us-east
    Mounts:
      /etc/ld.so.preload from oneagent-share (rw,path="ld.so.preload")
      /opt/dynatrace/oneagent-paas from oneagent-bin (rw)
      /var/lib/dynatrace/enrichment/dt_metadata.json from metadata-enrichment (rw,path="dt_metadata_tomcat10-container.json")
      /var/lib/dynatrace/enrichment/dt_metadata.properties from metadata-enrichment (rw,path="dt_metadata_tomcat10-container.properties")
      /var/lib/dynatrace/enrichment/endpoint from metadata-enrichment-endpoint (rw)
      /var/lib/dynatrace/oneagent/agent/config/container.conf from oneagent-share (rw,path="container_tomcat10-container.conf")
      /var/lib/dynatrace/oneagent/agent/customkeys/curl_options.conf from oneagent-share (rw,path="curl_options.conf")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lzdk6 (ro)

If you DO see the init container and mounts etc. but still no deep monitoring data, move on to step 3.

If you do NOT see the initcontainer or any of the mounts / environment variables needed then,

validate that communication between the dynatrace namespace / pods and the kube-system namespace is functional - as it is the Dynatrace Webhook that provides the mutating webhook that adds these items to the pods (see ingress docs above if necessary)
and collect a Dynatrace Operator support archive before reaching out to Dynatrace Support:
- Please follow these steps in our documentation to collect said Dynatrace Operator Support Archive.
  - Please note that it is important not to just send the logs from the support archive. Configuration Information/Diagnostic runtime information is also collected and extremely valuable to speed up the investigation of your reported issue.
- Depending on the health/status of the install-oneagent^[1] init container, we may also require the logs from said container:
- ```
kubectl logs <App pod name> -n <App Pod Namespace> -c install-oneagent
ex.
kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c install-oneagent
```

Step 3 - Validate Processes within the Pod

First step of process validation within the pod is to exec into the pod and look for logs within /opt/dynatrace/oneagent-paas/log/<tech type, java, nginx, go, etc.>

kubectl exec -it tomcat-10-deployment-5498c479d4-5hhjf -n default -- /bin/sh
# bash

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# cd /opt/dynatrace/oneagent-paas/log/java/

root@tomcat-10-deployment-5498c479d4-5hhjf:/opt/dynatrace/oneagent-paas/log/java# ls
ruxitagent_tomcat_tomcat-_-deployment-__cloudNativeFullStack_8.0.log

root@tomcat-10-deployment-5498c479d4-5hhjf:/opt/dynatrace/oneagent-paas/log/java#

If there Are Logs, then most likely we have a networking or config blocker issue and we can learn about which is going on by reading the log files and reaching out to Dynatrace Support for assistance if needed.

If there Are NO Logs, then this usually means that there is no OA library loading taking place but we can double check on that possibility using the "env" command to validate that LD_PRELOAD is set within the Pod's environment and not just being within the pod "describe":

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# env | grep LD_PRELOAD
LD_PRELOAD=/opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so

Note: It is easier to do this phase of the investigation with ps aux available within the pod to find the process ID's but I know some images do not include the procps package so I will also provide a pure bash alternative.

Now look for the above mentioned library within your processes loaded libraries memory addresses via the /proc/<PID>/maps file within the Linux /proc files.

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   6060  2156 ?        Ss   Dec06   0:00 /bin/sh -c echo 'Hello from Kubernetes!' & catalina.s
root           8  0.2  1.6 7969628 273124 ?      Sl   Dec06  22:38 /opt/java/openjdk/bin/java -Djava.util.logging.config

// OR use the pure bash command:
root@tomcat-10-deployment-5498c479d4-5hhjf:/ find /proc/*[0-9]/cmdline -type f -print -exec cat {} \; -exec printf '\n\n' \;

/proc/1/cmdline
/bin/sh-cecho 'Hello from Kubernetes!' &
catalina.sh run

..........

/proc/8/cmdline
/opt/java/openjdk/bin/java-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager-Djdk.tls.ephemeralDHKeySize=2048-Djava.protocol.handler.pkgs=org.apache.catalina.webresources-Dorg.apache.catalina.security.SecurityListener.UMASK=0027--add-opens=java.base/java.lang=ALL-UNNAMED--add-opens=java.base/java.io=ALL-UNNAMED--add-opens=java.base/java.util=ALL-UNNAMED--add-opens=java.base/java.util.concurrent=ALL-UNNAMED--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED-classpath/usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar-Dcatalina.base=/usr/local/tomcat-Dcatalina.home=/usr/local/tomcat-Djava.io.tmpdir=/usr/local/tomcat/temporg.apache.catalina.startup.Bootstrapstart

Once the App PID is identified, in this case the app is tomcat/java with a pid of 8, then we can use /proc/8/maps to learn about what is loaded into the app:

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# cat /proc/8/maps | grep oneagent
7f3b5781e000-7f3b5785a000 r--s 003af000 00:2e5 3893709                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.livedebugger.jar
7f3b5785a000-7f3b5789d000 r--s 0030d000 00:2e5 3893712                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.jar
7f3b64023000-7f3b64027000 r--s 00019000 00:2e5 3893716                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.8.jar
7f3b64027000-7f3b64029000 r--s 0000c000 00:2e5 3893719                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.11.jar
7f3b660f8000-7f3b6625e000 r--p 00000000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b6625e000-7f3b675b8000 r-xp 00166000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b675b8000-7f3b67af1000 r--p 014c0000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b67af1000-7f3b67bd3000 r--p 019f8000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b67bd3000-7f3b67bdd000 rw-p 01ada000 00:2e5 3897447                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b69718000-7f3b69719000 r--s 00005000 00:2e5 3893718                   /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.17.jar
7f3b69736000-7f3b69741000 r--p 00000000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b69741000-7f3b697de000 r-xp 0000b000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b697de000-7f3b69821000 r--p 000a8000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b69821000-7f3b69827000 r--p 000ea000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b69827000-7f3b69828000 rw-p 000f0000 00:2e5 3900038                   /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so

Above we can see both liboneagentproc.so, within the file which means we had OA Process Injection/file loading, AND liboneagentjava.so for OA Java Code Module Injection / loading.

If you can see both types of OneAgent libraries loaded but still no logs visible in the expected location, then you can also use /proc/PID/fd to find open file handles for the app process:

root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# ls -l /proc/8/fd | grep log
lrwx------ 1 root root 64 Dec 13 18:17 4 -> /opt/dynatrace/oneagent-paas/log/java/ruxitagent_tomcat_tomcat-_-deployment-__cloudNativeFullStack_8.0.log
l-wx------ 1 root root 64 Dec 13 18:17 65 -> /usr/local/tomcat/logs/catalina.2024-12-06.log
l-wx------ 1 root root 64 Dec 13 18:17 68 -> /usr/local/tomcat/logs/localhost_access_log.2024-12-06.txt

In this case the oneagent java code module is file handle 4, and we can now see the path we would need to take to read the log if it was in a "non standard location".

Alternatively you can print the oneagent logs to stdout aka kubectl log <pod name> and prepare for a DT support ticket by adding the following environment variables within your pods deployment yaml file or the container env spec section of any deployment template you are using before restarting the deployment:

Environment variable name	Environment variable value
DT_LOGLEVEL_PROC	config=debug,pgid=debug
DT_LOGCON_PROC	stdout
DT_LOGLEVELCON	info

Need further help?

If you are at this point of the guide and you still have not been able to find the OA logs in the container or find the OA libraries within your processes etc. then I would recommend reaching out to Dynatrace Support with the following information collected and we will be happy to help you from there:

A Dynatrace Operator Support archive as mentioned above.
- Please follow these steps in our documentation to collect said Dynatrace Operator Support Archive.
The logs from the pod itself, preferably with the above environment variables in place:
- ```
kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default
```

And the install-oneagent^[1] initcontainer logs:

kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c install-oneagent

[1] Starting with Dynatrace Operator 1.4.0, the install-oneagent initcontainer was renamed to the dynatrace-operator initcontainer. Please substitute commands as appropriate. IE.

kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c dynatrace-operator

sivart_89 · ‎20 Feb 2025

@ryan_korteway thank you for this right up. The 2nd link is invalid (DT Documentation about pod and namespace annotations). I'm currently looking through this because I've noticed that 1 cluster of ours has label dynakube.internal.dynatrace.com/instance in our namespaces while others do not. If the label does not exist in a namespace does that mean no dynatrace agent is being installed (as an init container) into their pod? I'm trying to understand what we are missing by not having this label in our namespaces.

ryan_korteway · ‎25 Feb 2025

Hello Sivart,

Not a problem at all on the write up and thank you for the heads up on the broken doc link which I will be fixing right after this comment.

So my understanding is if your namespace is missing that annotation, then the namespace and thus all of the pods within it are not being watched for pod starts and therefore will not have any pod injections.

Sounds to me like you may have some namespaceSelectors set up within the dynakube yaml files for some of your clusters / DT operator deployments.

It may also warrant checking the dynakube yaml file for any dynatrace operator config annotations that disable automatic monitoring for all namespaces within the cluster etc.

Please let me know if you have any further questions or concerns.