Dynatrace OneAgent Pod Failing with CrashLoopBackOff in Kubernetes Cluster on Docker Desktop

[Marwa]

I am encountering an issue with the Dynatrace OneAgent pod in my Kubernetes cluster, which is provided by Docker Desktop. The pod dynatrace-montor-1-oneagent-jdpfs is in a CrashLoopBackOff state, while the dynatrace-oneagent-csi-driver-dx5pv pod is running without issues. The error logs indicate:

Error: Cannot determine volume host path from /proc/self/mountinfo: /data/kubelet/plugins/csi.oneagent.dynatrace.com/data/cfq65015/osagent does not exist on the host

[NavenduGupta]

I have the exact same issue on windows 10. 

Using minikube and docker on windows 10

19:10:24 Error: Cannot determine volume host path from /proc/self/mountinfo: /docker/volumes/minikube/_data/lib/kubelet/plugins/csi.oneagent.dynatrace.com/data/1cadc1b7-cc94-46a7-a5b8-47ade37d437f/osagent does not exist on the host

I notice that there is no response to this thread... any guidance?

Thanks in advance,

Navendu

[p_devulapalli]

@NavenduGupta , are you using Helm or the manifest to setup? I've recently tried setting it up on minikube and I remember getting this error when using Helm . I would suggest using the manifest option as it worked for me and was bit easy to troubleshoot  

[NavenduGupta]

@p_devulapalli I'm using helm... will try that and get back.
I think the issue is with cloudNativeFullStack vs classicFullStack mode. When I downloaded the dynakuke.yaml, it default mode was set to cloudNativeFullStack and it was looking for the some image in  /proc/self/mountinfo file which is not mounted/doesn't exist. In cloudNativeFullStack mode, probably we have to provide the image path in dynakube.yaml. May be minikube needs to be started with different options.
In  classicFullStack mode, it worked.

[p_devulapalli]

@Marwa @NavenduGupta , can you please check if you have enough space on the partition ?

also, do you see any error in events?

kubectl describe pod dynatrace-montor-1-oneagent-jdpfs -n dynatrace

[NavenduGupta]

@p_devulapalli yes, there is enough space on the partition. As mentioned above, the issue is with cloudNativeFullStack vs classicFullStack mode. it works in classicFullStack mode and looks for an image at a specific location in cloudNativeFullStack mode.

[Peter_Youssef]

Hello @Marwa , @NavenduGupta 

Regarding the highlighted error simply increase or re-adjust the allocated oneagent resources.

By default, it's presented in Dynakube.yaml file 

After increasing the allocated resources for example to 2x or 3x it should work as expected without issues.

KR,

Peter.

[NavenduGupta]

@Peter_Youssef i noticed that the issue was because of classicFullStack vs cloudNativeFullStack mode. By default, in the dynakube.yaml file, OA was getting installed in cloudNativeFullStack mode. I changed it to classicFullStack model and it started working...

I'll try your solution in the cloudNativeFullStack mode and get back.

Regards,

Navendu

[Peter_Youssef]

Thanks, keep us posted.
Hopefully it will work in cloudNativeFullStack mode.

• It's recommended to proceed with K8s monitoring in cloudNativeFullStack whenever possible.

Best of Luck @NavenduGupta 

[NavenduGupta]

@Peter_Youssef @p_devulapalli slightly off topic here... 

I'm running easytravel in my K8s cluster(minikube) and in classicFullstack mode, 

1) when I connect it with SaaS, then I see all the kubernetes cluster/workloads/pods etc under the namespace I created in dynakube.yaml. I also see minikube top container (only one node for minikube with 8GB mem) as "host" under infrastructure. Here I have provided the "SaaS env url/api" for apiURL field
2) when I do the same with "managed", then I don't see top minikube under "host" under infrastructure. For managed, the apiUrl is "CAG url followed by env id /api". Here I get the kubernetes workload. but not the kubernetes top container under "host" under infrastructure... Also I'm not getting any easytravel application traffic under RUM. Even if my application detection may be wrong (i don't think so) but I should see the application traffic under "My web application" catch all application.

Only thing I see the difference between the AG in SaaS and Managed is that cluster AG doesn't have kubernetes module switched on/enabled. Could that be issue? but that doesn't make sense as the kubernetes AG is sending the data to cluster AG and it is only doing routing....

Any ideas?

[NavenduGupta]

Anybody... any ideas?

[p_devulapalli]

@NavenduGupta , can you please check if the below is enabled on your side?

Please ping me directly when you are free, we can take a look at it together if possible .

[p_devulapalli]

@NavenduGupta  other options you can try is to look at the operator logs

kubectl logs dynatrace-operator-xxxx -n dynatrace

Also try the troubleshoot command and see if you can find something 

kubectl exec deploy/dynatrace-operator -n dynatrace -- dynatrace-operator troubleshoot

OneAgent logs

kubectl logs dynakube-oneagent-<id> -n dynatrace

[NavenduGupta]

To All,

Any ideas?

Regards,

Navendu