cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
danmichael
Dynatrace Guide
Dynatrace Guide

Scenario

You're running cloudNativeFullStack, app only kubernetes injection or have a Dynatrace integration in any other PaaS platform. One of your pods are in crashloopbackoff and consistently crashing when it attempts to start.

Ask

How do I get logs from this failing container?

Option 1:

Set a debug flag to cause the oneagent logs to print to standard out. This can be set at the deployment or pod level so that the agent can read this on startup. This can be useful for any PaaS type setup: google cloud run, cloud foundry, app only kubernetes, etc.

You can set this via an environment variable

env DT_LOGLEVELCON="info"

or use a more kubernetes friendly version in the pod spec/deployment

- env:
    - name: DT_LOGLEVELCON
      value: "info"

This will make agent logs print to stdout of the pod. So you are able to view the container's log stream to see why it might be crashing.

It's also a good idea to enable process agent standard out logs as well using the below environment variable:

DT_LOGCON_PROC=stdout

Option 2:

Inspect the pod and find the underlying container information to access the pod volume on the underlying node.

We're looking for the main container its container id

$kubectl describe pod frontend-4jzzx

Relevant section:

Containers:
  php-redis:
    Container ID:   containerd://01ac620d1bc9e1ce308d4a69625f8d2e271ec41388003c394af54e0526541a77

now we want to inspect the container and find the pod path, you can either just look through the results of the inspect or pipe grep for mounts and include some of the lines after it

$sudo crictl inspect 01ac620d1bc9e1ce308d4a69625f8d2e271ec41388003c394af54e0526541a77
$sudo crictl inspect 01ac620d1bc9e1ce308d4a69625f8d2e271ec41388003c394af54e0526541a77 | grep -i -b10 mounts

And from that I'm looking for the oneagent-paas directory:

        {
          "destination": "/opt/dynatrace/oneagent-paas",
          "type": "bind",
          "source": "/var/lib/kubelet/pods/9d2d4699-8f84-496f-ba0b-f57929d803db/volumes/kubernetes.io~csi/oneagent-bin/mount",
          "options": [
            "rbind",
            "rprivate",
            "rw"
          ]
        },

So now that we know the relevant path /var/lib/kubelet/pods/9d2d4699-8f84-496f-ba0b-f57929d803db/ we can cd there and run a tree

$sudo su
$cd /var/lib/kubelet/pods/9d2d4699-8f84-496f-ba0b-f57929d803db/
$tree
....
├── volumes
│   ├── kubernetes.io~csi
│   │   └── oneagent-bin
│   │       ├── mount
│   │       │   ├── log
│   │       │   │   ├── apache
│   │       │   │   │   ├── ruxitagent_Apache_Web_Server_apache__frontend_1.0.log
│   │       │   │   │   ├── ruxitagent_Apache_Web_Server_apache__frontend_bootstrap_1.0.log
│   │       │   │   │   └── ruxitagent_Apache_Web_Server_apache__frontend_startup_1.0.log
│   │       │   │   ├── ignore_me.txt
│   │       │   │   └── php
│   │       │   │       └── ruxitagent_Apache_Web_Server_apache__frontend_bootstrap_1.0.log

so cd to the subpath to find the logs:

cd /var/lib/kubelet/pods/9d2d4699-8f84-496f-ba0b-f57929d803db/volumes/kubernetes.io~csi/oneagent-bin/mount/log/apache

Support ticket content

Upload these logs to your support ticket to provide Dynatrace OneAgent context about why the pod is crashing.

Version history
Last update:
‎14 Sep 2023 08:54 AM
Updated by: