on 23 Dec 2024 03:04 PM
This Partial Self-Service guide is to help identify potential blockers to deeply monitoring pod contained apps within kubernetes clusters.
The first step of Pod injection is Labels being added to application namespaces and so Troubleshooting Step 1 - is validating namespace Labels:
kubectl describe namespace <application namespace>
look for the label: dynakube.internal.dynatrace.com/instance
C:\Users\ryan.korteway>kubectl describe ns default
Name: default
Labels: dynakube.internal.dynatrace.com/instance=sup-enablement-k8s-cnfs
Annotations: <none>
Status: Active
If you DO see this label but no deep monitoring data, move on to step 2.
If you do NOT see this label then,
Additionally you can validate Namespace Labels within containerized process properties and tags dropdowns / pop out menus within the Dynatrace UI:
kubectl describe pod tomcat-10-deployment-5498c479d4-5hhjf -n default
install-oneagent initcontainer
[1] being present and if it finished successfully:Controlled By: ReplicaSet/tomcat-10-deployment-5498c479d4
Init Containers:
install-oneagent:
Container ID: containerd://35a91cf04a8b266be5a2648540d52e5c1247b60ef22abf6d71b5dc1bd0180093
Image: public.ecr.aws/dynatrace/dynatrace-operator:v1.3.2
Image ID: public.ecr.aws/dynatrace/dynatrace-operator@sha256:f8ecdcd87d7d84b87e645074084dd7f57dd62c76e120bb21e5abde158755be56
Port: <none>
Host Port: <none>
SeccompProfile: RuntimeDefault
Args:
init
State: Terminated
Reason: Completed
Exit Code: 0
LD_PRELOAD
environment variable and the key volume mounts for /opt/dynatrace/oneagent-paas
volumes: Containers:
tomcat10-container:
Container ID: containerd://409633c542b9d1fa6b3e93231cc61f17a1ce4e12dd49a5c4156936338ec07de6
Image: tomcat:10
Image ID: docker.io/library/tomcat@sha256:0e763cfc66a42679d63132ceacfbf7082a219c0385ca75f85baf3294e76bacf7
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/sh
-c
echo 'Hello from Kubernetes!' &
catalina.sh run
State: Running
Started: Fri, 06 Dec 2024 12:53:02 -0500
Ready: True
Restart Count: 0
Environment:
DT_CUSTOM_PROP: cnfs_cluster location_us_east
DT_RELEASE_STAGE: cloudNativeFullStack
DT_DEPLOYMENT_METADATA: orchestration_tech=Operator-cloud_native_fullstack;script_version=v1.3.2;orchestrator_id=f27c4518-9ead-4488-bab0-b2cde9649669
LD_PRELOAD: /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
DT_NETWORK_ZONE: azure-us-east
Mounts:
/etc/ld.so.preload from oneagent-share (rw,path="ld.so.preload")
/opt/dynatrace/oneagent-paas from oneagent-bin (rw)
/var/lib/dynatrace/enrichment/dt_metadata.json from metadata-enrichment (rw,path="dt_metadata_tomcat10-container.json")
/var/lib/dynatrace/enrichment/dt_metadata.properties from metadata-enrichment (rw,path="dt_metadata_tomcat10-container.properties")
/var/lib/dynatrace/enrichment/endpoint from metadata-enrichment-endpoint (rw)
/var/lib/dynatrace/oneagent/agent/config/container.conf from oneagent-share (rw,path="container_tomcat10-container.conf")
/var/lib/dynatrace/oneagent/agent/customkeys/curl_options.conf from oneagent-share (rw,path="curl_options.conf")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lzdk6 (ro)
If you DO see the init container and mounts etc. but still no deep monitoring data, move on to step 3.
If you do NOT see the initcontainer or any of the mounts / environment variables needed then,
kubectl logs <App pod name> -n <App Pod Namespace> -c install-oneagent
ex.
kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c install-oneagent
/opt/dynatrace/oneagent-paas/log/<tech type, java, nginx, go, etc.>
kubectl exec -it tomcat-10-deployment-5498c479d4-5hhjf -n default -- /bin/sh
# bash
root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# cd /opt/dynatrace/oneagent-paas/log/java/
root@tomcat-10-deployment-5498c479d4-5hhjf:/opt/dynatrace/oneagent-paas/log/java# ls
ruxitagent_tomcat_tomcat-_-deployment-__cloudNativeFullStack_8.0.log
root@tomcat-10-deployment-5498c479d4-5hhjf:/opt/dynatrace/oneagent-paas/log/java#
root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# env | grep LD_PRELOAD
LD_PRELOAD=/opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
ps aux
available within the pod to find the process ID's but I know some images do not include the procps package so I will also provide a pure bash alternative./proc/<PID>/maps
file within the Linux /proc files.root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 6060 2156 ? Ss Dec06 0:00 /bin/sh -c echo 'Hello from Kubernetes!' & catalina.s
root 8 0.2 1.6 7969628 273124 ? Sl Dec06 22:38 /opt/java/openjdk/bin/java -Djava.util.logging.config
// OR use the pure bash command:
root@tomcat-10-deployment-5498c479d4-5hhjf:/ find /proc/*[0-9]/cmdline -type f -print -exec cat {} \; -exec printf '\n\n' \;
/proc/1/cmdline
/bin/sh-cecho 'Hello from Kubernetes!' &
catalina.sh run
..........
/proc/8/cmdline
/opt/java/openjdk/bin/java-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager-Djdk.tls.ephemeralDHKeySize=2048-Djava.protocol.handler.pkgs=org.apache.catalina.webresources-Dorg.apache.catalina.security.SecurityListener.UMASK=0027--add-opens=java.base/java.lang=ALL-UNNAMED--add-opens=java.base/java.io=ALL-UNNAMED--add-opens=java.base/java.util=ALL-UNNAMED--add-opens=java.base/java.util.concurrent=ALL-UNNAMED--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED-classpath/usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar-Dcatalina.base=/usr/local/tomcat-Dcatalina.home=/usr/local/tomcat-Djava.io.tmpdir=/usr/local/tomcat/temporg.apache.catalina.startup.Bootstrapstart
Once the App PID is identified, in this case the app is tomcat/java with a pid of 8, then we can use /proc/8/maps
to learn about what is loaded into the app:
root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# cat /proc/8/maps | grep oneagent
7f3b5781e000-7f3b5785a000 r--s 003af000 00:2e5 3893709 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.livedebugger.jar
7f3b5785a000-7f3b5789d000 r--s 0030d000 00:2e5 3893712 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.jar
7f3b64023000-7f3b64027000 r--s 00019000 00:2e5 3893716 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.8.jar
7f3b64027000-7f3b64029000 r--s 0000c000 00:2e5 3893719 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.11.jar
7f3b660f8000-7f3b6625e000 r--p 00000000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b6625e000-7f3b675b8000 r-xp 00166000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b675b8000-7f3b67af1000 r--p 014c0000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b67af1000-7f3b67bd3000 r--p 019f8000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b67bd3000-7f3b67bdd000 rw-p 01ada000 00:2e5 3897447 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/linux-x86-64/liboneagentjava.so
7f3b69718000-7f3b69719000 r--s 00005000 00:2e5 3893718 /opt/dynatrace/oneagent-paas/agent/bin/1.306.0.20241128-144422/any/java/oneagentjava.17.jar
7f3b69736000-7f3b69741000 r--p 00000000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b69741000-7f3b697de000 r-xp 0000b000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b697de000-7f3b69821000 r--p 000a8000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b69821000-7f3b69827000 r--p 000ea000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7f3b69827000-7f3b69828000 rw-p 000f0000 00:2e5 3900038 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
/proc/PID/fd
to find open file handles for the app process:root@tomcat-10-deployment-5498c479d4-5hhjf:/usr/local/tomcat# ls -l /proc/8/fd | grep log
lrwx------ 1 root root 64 Dec 13 18:17 4 -> /opt/dynatrace/oneagent-paas/log/java/ruxitagent_tomcat_tomcat-_-deployment-__cloudNativeFullStack_8.0.log
l-wx------ 1 root root 64 Dec 13 18:17 65 -> /usr/local/tomcat/logs/catalina.2024-12-06.log
l-wx------ 1 root root 64 Dec 13 18:17 68 -> /usr/local/tomcat/logs/localhost_access_log.2024-12-06.txt
Environment variable name | Environment variable value |
DT_LOGLEVEL_PROC | config=debug,pgid=debug |
DT_LOGCON_PROC | stdout |
DT_LOGLEVELCON | info |
kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default
kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c install-oneagent
[1] Starting with Dynatrace Operator 1.4.0, the install-oneagent initcontainer was renamed to the dynatrace-operator initcontainer. Please substitute commands as appropriate. IE.
kubectl logs tomcat-10-deployment-5498c479d4-5hhjf -n default -c dynatrace-operator