cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Oneagent Container Deployment to GKE Fails (Container CrashLoop)

traiano
Newcomer

Hi,

We're doing a PoC to determine if dynatrace can be used to monitor our containerized applications running on Google's GKE (kubernetes) platform.


However, your process described below for getting dynatrace oneagent to work on GKE does not work:
https://help.dynatrace.com/infrastructure-monitori...

Following this procedure, the container fails to deploy with this error in the console:

      "Error syncing pod, skipping: failed to "StartContainer" for "dynatrace-oneagent" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=dynatrace-oneagent pod=dynatrace-oneagent-nqf6q_default(500a76c4-6f83-11e7-902a-42010a840026)"

Investigating further, the container logs have just this line in before crashing:

10:29:53 This script can be only executed when Dynatrace OneAgent is deployed as docker container

(kubectl logs dynatrace-oneagent-mjr9v --previous)

Is this a defect in the oneagent container image itself, installation process or something unrelated?

Please help me troubleshoot this container failure further?

Many thanks in advance!

Traiano

14 REPLIES 14

daniela_rabiser
Dynatrace Helper
Dynatrace Helper

Hi Traiano,

since your goal is to deploy Dynatrace OneAgent on Google's GKE platform, you may want to follow the instructions described here: https://help.dynatrace.com/infrastructure-monitoring/containers/how-do-i-deploy-oneagent-in-gke-base...

Hope this helps.

Best,
Daniela

Hi Daniela

Firstly, thanks for your response. However, the link you posted is of no use because:

1) This link only describes how to set up the GKE container cluster
infrastructure - it actually tells you nothing about how to deploy the
the dynatrace oneagent container itself. For that, you have to refer to
this link, which I had to find by myself:

https://help.dynatrace.com/infrastructure-monitori...

2) I have already set up a GKE container cluster - this is my existing system. I have also followed this link as a separate test and it is fine, but:

The problem is that even once I've deployed the oneagent container to GKE, the oneagent container crashes with the error I described in my original post.

So the solution I need is:

a) To fix the root cause of why the oneagent container is crashing on GKE.

b) optionally: how to troubleshoot further to discover the reason why the oneagent container is crashing on GKE.

I'd be really thankful if you could help me with this. So far it appears that Dynatrace offers a solution to APM on GKE , but in practice we are not able to get this to work.

Hi Traiano,

sorry that the help topic around deploying OneAgent in GKE-based clusters is not helpful to you. I've asked one of our Docker experts to have a look at your problem description. Please stay tuned for further answers.

Best,
Daniela

Thanks, Daniela! I await in anticipation ...

Tomasz_Gajger
Dynatrace Pro
Dynatrace Pro

Hi Traiano,

I assume that you have tried to deploy the latest image from Docker Hub, which is currently version 1.119. We had a problem with this version that causes it to not be able to recognize docker used by kubernetes correctly, the problem was fixed in version 1.121, that should be made available on Docker Hub in a matter of few days.
Furthermore, please be aware that currently we only support container-vm type of node image and deployment will not work with cos type.

Regards,
Tomek

Hi Tomasz

I've converted the GKE cluster to use container-vm.

We are using the "latest" oneagent image available. So if the current image is not 1.121, then I assume it's one of the defective images?

So, to conclude, would this mean we have to wait some days to complete our evaluation of Dynatrace as an APM for GKE? Any date for when we can evaluate 1.121?

Thanks in advance!

Traiano

Hi Traiano,

yes, the previous latest image (version 1.119) was defective.
However, version 1.121 was just released to Docker Hub and tagged as latest, this one should work with your GKE setup.

Regards,
Tomek

Hi Tomasz, thanks for your response.

However, I see the container (latest) still appears to have issues, the containers fail to startup due to a permissions issue in mounting the installation directory. Here is the container logs from kubernetes when deploying with "latest":

bash-3.2$ kubectl logs dynatrace-oneagent-2q2jv

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent directory.
chmod: /mnt/root//opt/dynatrace/oneagent: No such file or directory
07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent directory to 755.
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent/log directory.
chmod: /mnt/root//opt/dynatrace/oneagent/log: No such file or directory
07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent/log directory to 777.
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent/log/installer directory.
chmod: /mnt/root//opt/dynatrace/oneagent/log/installer: No such file or directory
07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent/log/installer directory to 755.
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent/log/network directory.
chmod: /mnt/root//opt/dynatrace/oneagent/log/network: No such file or directory
07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent/log/network directory to 755.
07:07:47 Started agent deployment as docker image, PID 26567.
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
07:07:47 Container version: 1.121.196.20170720-130455
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
07:07:48 Deploying to: /mnt/root
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:48 Cannot create /mnt/root//opt/dynatrace/oneagent directory.
chmod: /mnt/root//opt/dynatrace/oneagent: No such file or directory
07:07:48 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent directory to 755.
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
cp: can't create '/mnt/root//opt/dynatrace/oneagent/Dynatrace-OneAgent-Linux.sh': No such file or directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

Here is my deployment .yaml:

apiVersion: extensions/v1beta1kind: DaemonSetmetadata:  name: dynatrace-oneagentspec:  template:  metadata:  labels:  run: dynatrace-oneagent  spec:  hostPID: true  hostIPC: true  hostNetwork: true  volumes:  - name: host-root  hostPath:  path: /  containers:  - name: dynatrace-oneagent  image: dynatrace/oneagent:latest  args:  - "TENANT=lfn20468"  - "TENANT_TOKEN=IRV1c15bIcRWq5KQ"  - "SERVER=https://lfn20468.live.dynatrace.com"  volumeMounts:  - mountPath: /mnt/root  name: host-root  securityContext:  privileged: true

Snapshot of status of the daemonset:

bash-3.2$ kubectl get daemonsets
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE
dynatrace-oneagent 3 3 0 3 0 <none> 5m
bash-3.2$
bash-3.2$


bash-3.2$
bash-3.2$ kubectl get pods
NAME READY STATUS RESTARTS AGE
dynatrace-oneagent-2q2jv 0/1 CrashLoopBackOff 5 5m
dynatrace-oneagent-5mqv1 0/1 CrashLoopBackOff 5 5m
dynatrace-oneagent-6vvtf 0/1 CrashLoopBackOff 5 5m

Hi Traiano,

are you sure that this node uses container-vm image?
I recall seeing similar errors when cos was used.

Hi Tomasz! Thanks, that was it, it looks like my cos->container-vm conversion didn't complete successfully last time, so I did it again. We have success 🙂

scott_brenner
Newcomer

Having the same issue, what was the resolution?

Tomasz_Gajger
Dynatrace Pro
Dynatrace Pro

Hi Scott,

such a problem can occur as a result of different root causes.
Please make sure that you followed the instructions from

https://help.dynatrace.com/infrastructure/containers/how-do-i-deploy-oneagent-in-gke-based-clusters/ (this page was recently updated).

If the problem still persists, please kindly open a support ticket on

https://www.dynatrace.com/support/

Regards,
Tomek

Answer by Tomasz G. Jul 24, 2017 at 09:14 AM

Hi Traiano,

I assume that you have tried to deploy the latest image from Docker Hub, which is currently version 1.119. We had a problem with this version that causes it to not be able to recognize docker used by kubernetes correctly, the problem was fixed in version 1.121, that should be made available on Docker Hub in a matter of few days.
Furthermore, please be aware that currently we only support container-vm type of node image and deployment will not work with cos type.

Regards,
Tomek

Where is Dynatrace now with supporting the cos type? We are doing some evaluations and this is a requirement. Thanks!

We're actively working on COS support, it should soon be released as an EAP feature.