Scenario
You're running cloudNativeFullStack, app only kubernetes injection or have a Dynatrace integration in any other PaaS platform. One of your pods are in crashloopbackoff and consistently crashing when it attempts to start.
Ask
How do I get logs from this failing container?
Option 1:
Set a debug flag to cause the oneagent logs to print to standard out. This can be set at the deployment or pod level so that the agent can read this on startup. This can be useful for any PaaS type setup: google cloud run, cloud foundry, app only kubernetes, etc.
You can set this via an environment variable
env DT_LOGLEVELCON="info"
or use a more kubernetes friendly version in the pod spec/deployment
- env:
- name: DT_LOGLEVELCON
value: "info"
This will make agent logs print to stdout of the pod. So you are able to view the container's log stream to see why it might be crashing.
It's also a good idea to enable process agent standard out logs as well using the below environment variable:
DT_LOGCON_PROC=stdout
Option 2:
Inspect the pod and find the underlying container information to access the pod volume on the underlying node.
We're looking for the main container its container id
$kubectl describe pod frontend-4jzzx
Relevant section:
Containers:
php-redis:
Container ID: containerd://01ac620d1bc9e1ce308d4a69625f8d2e271ec41388003c394af54e0526541a77
now we want to inspect the container and find the pod path, you can either just look through the results of the inspect or pipe grep for mounts and include some of the lines after it
$sudo crictl inspect 01ac620d1bc9e1ce308d4a69625f8d2e271ec41388003c394af54e0526541a77
$sudo crictl inspect 01ac620d1bc9e1ce308d4a69625f8d2e271ec41388003c394af54e0526541a77 | grep -i -b10 mounts
And from that I'm looking for the oneagent-paas directory:
{
"destination": "/opt/dynatrace/oneagent-paas",
"type": "bind",
"source": "/var/lib/kubelet/pods/9d2d4699-8f84-496f-ba0b-f57929d803db/volumes/kubernetes.io~csi/oneagent-bin/mount",
"options": [
"rbind",
"rprivate",
"rw"
]
},
So now that we know the relevant path /var/lib/kubelet/pods/9d2d4699-8f84-496f-ba0b-f57929d803db/ we can cd there and run a tree
$sudo su
$cd /var/lib/kubelet/pods/9d2d4699-8f84-496f-ba0b-f57929d803db/
$tree
....
├── volumes
│ ├── kubernetes.io~csi
│ │ └── oneagent-bin
│ │ ├── mount
│ │ │ ├── log
│ │ │ │ ├── apache
│ │ │ │ │ ├── ruxitagent_Apache_Web_Server_apache__frontend_1.0.log
│ │ │ │ │ ├── ruxitagent_Apache_Web_Server_apache__frontend_bootstrap_1.0.log
│ │ │ │ │ └── ruxitagent_Apache_Web_Server_apache__frontend_startup_1.0.log
│ │ │ │ ├── ignore_me.txt
│ │ │ │ └── php
│ │ │ │ └── ruxitagent_Apache_Web_Server_apache__frontend_bootstrap_1.0.log
so cd to the subpath to find the logs:
cd /var/lib/kubelet/pods/9d2d4699-8f84-496f-ba0b-f57929d803db/volumes/kubernetes.io~csi/oneagent-bin/mount/log/apache
Support ticket content
Upload these logs to your support ticket to provide Dynatrace OneAgent context about why the pod is crashing.
