Container issues and initial information gathering

danmichael — Mon, 20 Mar 2023 14:51:35 GMT

Initial information gathering commands

These will help describe Dynatrace entities and their status, also can help gather the Dynakube Custom Resource if the location of the yaml used to apply it isn't known. Also including some basic exec commands to get into pods

Kubernetes commands:

kubectl get all -n dynatrace
kubectl describe all -n dynatrace
kubectl get events -n dynatrace --sort-by=.metadata.creationTimestamp
kubectl describe nodes
kubectl get dynakube -n dynatrace -o yaml

init container logs (app only/cloudnative):
kubectl logs frontend-79db8fc578-48qxh -c install-oneagent

OpenShift Commands:

oc get all -n dynatrace
oc describe all -n dynatrace
oc get events -n dynatrace --sort-by=.metadata.creationTimestamp
oc describe nodes
oc get dynakube -n dynatrace -o yaml

Generic 1st response commands

kubectl get all -n dynatrace
kubectl describe all -n dynatrace
kubectl get events -n dynatrace --sort-by=.metadata.creationTimestamp
kubectl logs -n dynatrace nameofdynatracepods (this is good for oneagent pods and csi pods)
kubectl describe nodes
kubectl get dynakube -n dynatrace -o yaml

Exec into pods

kubectl logs oneagent-pod
kubectl exec oneagentPodname -n dynatrace -it – sh
kubectl exec oneagentPodname -n dynatrace -it –- /bin/bash
chroot /mnt/host_root /bin/bash

cloudNativeFullStack injection issues

These labels are subject to change and this information might not be the most up to date source for this.

Quick rundown of how the Webhook pod does the container injections while we have the Dynatrace Operator running in CloudNativeFullStack:

The install of the webhook should lead to all namespaces (with dynatrace operator deployed NS along with anything kube- or openshift- starting) to be applied with the label:
dynakube.internal.dynatrace.com/instance

As long as that label is on the namespace that the app pods are starting in, check it with kubectl describe appNamespace , then you should be seeing pod injections on that kind of container.

You can see if a pod was seen by the webhook for injection or not consideration by checking its logs with these two steps:

Gather pod UID for next command:
kubectl get pods -n <namespace> <pod-name> -o jsonpath='{.metadata.uid}'

Use Pod UID to search all webhook pod logs:
kubectl logs <webhook pod name> -n dynatrace | grep <UID of the pod that was launched>

And Further validation on if a pod was injected into by the webhook or not can be seen by inspecting the pods labels for the label: dynakube.dynatrace.com/injected
kubectl describe pod podName -n PodNamespace | grep dynakube.dynatrace.com/injected

cloudNative logs

Also.. if you're working on cloudNative issues and wondering where the application/pod logs are.. they're located on the instrumented/injected application pods, you can find the log locations if you exec in and run something like the below:

sudo ls -lash /proc/*/fd | grep oneagent

Copying logs out of container
If tar binary is on the app pod you can use kubectl cp
kubectl cp namespace/podname:/opt/dynatrace/oneagent-paas/log /tmp/logsIf tar is unavailable on the app pod you can try something like the below:
kubectl exec <pod-name> – find <directory-path> -type f | xargs -I {} sh -c 'kubectl exec <pod-name> – cat {} > <local-path>/${1#<directory-path>}' _ {}
or
kubectl exec myapp-123 – find /data/logs -type f | xargs -I {} sh -c 'kubectl exec myapp-123 – cat {} > /tmp/${1#/data/logs}' _ {}

Re: Container issues and initial information gathering

ChadTurner — Fri, 12 May 2023 18:53:45 GMT

Thank you for sharing this! I'm sure it will help out many community members.

article Container issues and initial information gathering in Troubleshooting

Container issues and initial information gathering

Initial information gathering commands

cloudNativeFullStack injection issues

cloudNative logs

Re: Container issues and initial information gathering