Initial information gathering commands
These will help describe Dynatrace entities and their status, also can help gather the Dynakube Custom Resource if the location of the yaml used to apply it isn't known. Also including some basic exec commands to get into pods
Kubernetes commands:
kubectl get all -n dynatrace
kubectl describe all -n dynatrace
kubectl get events -n dynatrace --sort-by=.metadata.creationTimestamp
kubectl describe nodes
kubectl get dynakube -n dynatrace -o yaml
init container logs (app only/cloudnative):
kubectl logs frontend-79db8fc578-48qxh -c install-oneagent
OpenShift Commands:
oc get all -n dynatrace
oc describe all -n dynatrace
oc get events -n dynatrace --sort-by=.metadata.creationTimestamp
oc describe nodes
oc get dynakube -n dynatrace -o yaml
Generic 1st response commands
kubectl get all -n dynatrace
kubectl describe all -n dynatrace
kubectl get events -n dynatrace --sort-by=.metadata.creationTimestamp
kubectl logs -n dynatrace nameofdynatracepods (this is good for oneagent pods and csi pods)
kubectl describe nodes
kubectl get dynakube -n dynatrace -o yaml
Exec into pods
kubectl logs oneagent-pod
kubectl exec oneagentPodname -n dynatrace -it – sh
kubectl exec oneagentPodname -n dynatrace -it –- /bin/bash
chroot /mnt/host_root /bin/bash
cloudNativeFullStack injection issues
These labels are subject to change and this information might not be the most up to date source for this.
Quick rundown of how the Webhook pod does the container injections while we have the Dynatrace Operator running in CloudNativeFullStack:
The install of the webhook should lead to all namespaces (with dynatrace operator deployed NS along with anything kube- or openshift- starting) to be applied with the label:
dynakube.internal.dynatrace.com/instance
As long as that label is on the namespace that the app pods are starting in, check it with kubectl describe appNamespace , then you should be seeing pod injections on that kind of container.
You can see if a pod was seen by the webhook for injection or not consideration by checking its logs with these two steps:
Gather pod UID for next command:
kubectl get pods -n <namespace> <pod-name> -o jsonpath='{.metadata.uid}'
Use Pod UID to search all webhook pod logs:
kubectl logs <webhook pod name> -n dynatrace | grep <UID of the pod that was launched>
And Further validation on if a pod was injected into by the webhook or not can be seen by inspecting the pods labels for the label: dynakube.dynatrace.com/injected
kubectl describe pod podName -n PodNamespace | grep dynakube.dynatrace.com/injected
cloudNative logs
Also.. if you're working on cloudNative issues and wondering where the application/pod logs are.. they're located on the instrumented/injected application pods, you can find the log locations if you exec in and run something like the below:
sudo ls -lash /proc/*/fd | grep oneagent
Copying logs out of container
If tar binary is on the app pod you can use kubectl cp
kubectl cp namespace/podname:/opt/dynatrace/oneagent-paas/log /tmp/logsIf tar is unavailable on the app pod you can try something like the below:
kubectl exec <pod-name> – find <directory-path> -type f | xargs -I {} sh -c 'kubectl exec <pod-name> – cat {} > <local-path>/${1#<directory-path>}' _ {}
or
kubectl exec myapp-123 – find /data/logs -type f | xargs -I {} sh -c 'kubectl exec myapp-123 – cat {} > /tmp/${1#/data/logs}' _ {}
