topic Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop) in Container platforms

Oneagent Container Deployment to GKE Fails (Container CrashLoop)

traiano — Sun, 23 Jul 2017 10:41:07 GMT

Hi,

We're doing a PoC to determine if dynatrace can be used to monitor our containerized applications running on Google's GKE (kubernetes) platform.

However, your process described below for getting dynatrace oneagent to work on GKE does not work:
https://help.dynatrace.com/infrastructure-monitori...

Following this procedure, the container fails to deploy with this error in the console:

      "Error syncing pod, skipping: failed to "StartContainer" for "dynatrace-oneagent" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=dynatrace-oneagent pod=dynatrace-oneagent-nqf6q_default(500a76c4-6f83-11e7-902a-42010a840026)"

Investigating further, the container logs have just this line in before crashing:

10:29:53 This script can be only executed when Dynatrace OneAgent is deployed as docker container

(kubectl logs dynatrace-oneagent-mjr9v --previous)

Is this a defect in the oneagent container image itself, installation process or something unrelated?

Please help me troubleshoot this container failure further?

Many thanks in advance!

Traiano

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

daniela_rabiser — Mon, 24 Jul 2017 07:16:33 GMT

Hi Traiano,

since your goal is to deploy Dynatrace OneAgent on Google's GKE platform, you may want to follow the instructions described here: https://help.dynatrace.com/infrastructure-monitoring/containers/how-do-i-deploy-oneagent-in-gke-based-clusters/

Hope this helps.

Best,
Daniela

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

traiano — Mon, 24 Jul 2017 07:31:57 GMT

Hi Daniela

Firstly, thanks for your response. However, the link you posted is of no use because:

1) This link only describes how to set up the GKE container cluster
infrastructure - it actually tells you nothing about how to deploy the
the dynatrace oneagent container itself. For that, you have to refer to
this link, which I had to find by myself:

https://help.dynatrace.com/infrastructure-monitori...

2) I have already set up a GKE container cluster - this is my existing system. I have also followed this link as a separate test and it is fine, but:

The problem is that even once I've deployed the oneagent container to GKE, the oneagent container crashes with the error I described in my original post.

So the solution I need is:

a) To fix the root cause of why the oneagent container is crashing on GKE.

b) optionally: how to troubleshoot further to discover the reason why the oneagent container is crashing on GKE.

I'd be really thankful if you could help me with this. So far it appears that Dynatrace offers a solution to APM on GKE , but in practice we are not able to get this to work.

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

daniela_rabiser — Mon, 24 Jul 2017 07:44:11 GMT

Hi Traiano,

sorry that the help topic around deploying OneAgent in GKE-based clusters is not helpful to you. I've asked one of our Docker experts to have a look at your problem description. Please stay tuned for further answers.

Best,
Daniela

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

traiano — Mon, 24 Jul 2017 07:47:40 GMT

Thanks, Daniela! I await in anticipation ...

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

Tomasz_Gajger — Mon, 24 Jul 2017 09:14:06 GMT

Hi Traiano,

I assume that you have tried to deploy the latest image from Docker Hub, which is currently version 1.119. We had a problem with this version that causes it to not be able to recognize docker used by kubernetes correctly, the problem was fixed in version 1.121, that should be made available on Docker Hub in a matter of few days.
Furthermore, please be aware that currently we only support container-vm type of node image and deployment will not work with cos type.

Regards,
Tomek

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

traiano — Mon, 24 Jul 2017 09:30:16 GMT

Hi Tomasz

I've converted the GKE cluster to use container-vm.

We are using the "latest" oneagent image available. So if the current image is not 1.121, then I assume it's one of the defective images?

So, to conclude, would this mean we have to wait some days to complete our evaluation of Dynatrace as an APM for GKE? Any date for when we can evaluate 1.121?

Thanks in advance!

Traiano

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

Tomasz_Gajger — Tue, 25 Jul 2017 05:29:06 GMT

Hi Traiano,

yes, the previous latest image (version 1.119) was defective.
However, version 1.121 was just released to Docker Hub and tagged as latest, this one should work with your GKE setup.

Regards,
Tomek

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

traiano — Tue, 25 Jul 2017 07:13:05 GMT

Hi Tomasz, thanks for your response.

However, I see the container (latest) still appears to have issues, the containers fail to startup due to a permissions issue in mounting the installation directory. Here is the container logs from kubernetes when deploying with "latest":

bash-3.2$ kubectl logs dynatrace-oneagent-2q2jv

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system

07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent directory.

chmod: /mnt/root//opt/dynatrace/oneagent: No such file or directory

07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent directory to 755.

mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system

07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent/log directory.

chmod: /mnt/root//opt/dynatrace/oneagent/log: No such file or directory

07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent/log directory to 777.

mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system

07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent/log/installer directory.

chmod: /mnt/root//opt/dynatrace/oneagent/log/installer: No such file or directory

07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent/log/installer directory to 755.

mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system

07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent/log/network directory.

chmod: /mnt/root//opt/dynatrace/oneagent/log/network: No such file or directory

07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent/log/network directory to 755.

07:07:47 Started agent deployment as docker image, PID 26567.

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

07:07:47 Container version: 1.121.196.20170720-130455

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

07:07:48 Deploying to: /mnt/root

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system

07:07:48 Cannot create /mnt/root//opt/dynatrace/oneagent directory.

chmod: /mnt/root//opt/dynatrace/oneagent: No such file or directory

07:07:48 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent directory to 755.

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

cp: can't create '/mnt/root//opt/dynatrace/oneagent/Dynatrace-OneAgent-Linux.sh': No such file or directory

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory

Here is my deployment .yaml:

apiVersion: extensions/v1beta1kind: DaemonSetmetadata:  name: dynatrace-oneagentspec:  template:  metadata:  labels:  run: dynatrace-oneagent  spec:  hostPID: true  hostIPC: true  hostNetwork: true  volumes:  - name: host-root  hostPath:  path: /  containers:  - name: dynatrace-oneagent  image: dynatrace/oneagent:latest  args:  - "TENANT=lfn20468"  - "TENANT_TOKEN=IRV1c15bIcRWq5KQ"  - "SERVER=https://lfn20468.live.dynatrace.com"  volumeMounts:  - mountPath: /mnt/root  name: host-root  securityContext:  privileged: true

Snapshot of status of the daemonset:

bash-3.2$ kubectl get daemonsets
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE
dynatrace-oneagent 3 3 0 3 0 <none> 5m
bash-3.2$
bash-3.2$

bash-3.2$
bash-3.2$ kubectl get pods
NAME READY STATUS RESTARTS AGE
dynatrace-oneagent-2q2jv 0/1 CrashLoopBackOff 5 5m
dynatrace-oneagent-5mqv1 0/1 CrashLoopBackOff 5 5m
dynatrace-oneagent-6vvtf 0/1 CrashLoopBackOff 5 5m

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

Tomasz_Gajger — Tue, 25 Jul 2017 11:34:13 GMT

Hi Traiano,

are you sure that this node uses container-vm image?
I recall seeing similar errors when cos was used.

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

traiano — Tue, 25 Jul 2017 12:35:50 GMT

Hi Tomasz! Thanks, that was it, it looks like my cos->container-vm conversion didn't complete successfully last time, so I did it again. We have success 🙂

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

scott_brenner — Tue, 10 Oct 2017 18:12:44 GMT

Having the same issue, what was the resolution?

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

Tomasz_Gajger — Thu, 12 Oct 2017 08:44:30 GMT

Hi Scott,

such a problem can occur as a result of different root causes.
Please make sure that you followed the instructions from

https://help.dynatrace.com/infrastructure/containers/how-do-i-deploy-oneagent-in-gke-based-clusters/ (this page was recently updated).

If the problem still persists, please kindly open a support ticket on

https://www.dynatrace.com/support/

Regards,
Tomek

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

larry_roberts — Tue, 01 May 2018 20:38:38 GMT

Answer by Tomasz G. Jul 24, 2017 at 09:14 AM
Hi Traiano,

I assume that you have tried to deploy the latest image from Docker Hub, which is currently version 1.119. We had a problem with this version that causes it to not be able to recognize docker used by kubernetes correctly, the problem was fixed in version 1.121, that should be made available on Docker Hub in a matter of few days.
Furthermore, please be aware that currently we only support container-vm type of node image and deployment will not work with cos type.

Regards,
Tomek

Where is Dynatrace now with supporting the cos type? We are doing some evaluations and this is a requirement. Thanks!

Re: Oneagent Container Deployment to GKE Fails (Container CrashLoop)

Tomasz_Gajger — Fri, 01 Jun 2018 04:38:03 GMT

We're actively working on COS support, it should soon be released as an EAP feature.