23 Jul 2017 11:41 AM
Hi,
We're doing a PoC to determine if dynatrace can be used to monitor our containerized applications running on Google's GKE (kubernetes) platform.
However, your process described below for getting dynatrace oneagent to work on GKE does not work:
https://help.dynatrace.com/infrastructure-monitori...
Following this procedure, the container fails to deploy with this error in the console:
"Error syncing pod, skipping: failed to "StartContainer" for "dynatrace-oneagent" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=dynatrace-oneagent pod=dynatrace-oneagent-nqf6q_default(500a76c4-6f83-11e7-902a-42010a840026)"
Investigating further, the container logs have just this line in before crashing:
10:29:53 This script can be only executed when Dynatrace OneAgent is deployed as docker container
(kubectl logs dynatrace-oneagent-mjr9v --previous)
Is this a defect in the oneagent container image itself, installation process or something unrelated?
Please help me troubleshoot this container failure further?
Many thanks in advance!
Traiano
Solved! Go to Solution.
24 Jul 2017 08:16 AM
Hi Traiano,
since your goal is to deploy Dynatrace OneAgent on Google's GKE platform, you may want to follow the instructions described here: https://help.dynatrace.com/infrastructure-monitoring/containers/how-do-i-deploy-oneagent-in-gke-base...
Hope this helps.
Best,
Daniela
24 Jul 2017 08:31 AM
Hi Daniela
Firstly, thanks for your response. However, the link you posted is of no use because:
1) This link only describes how to set up the GKE container cluster
infrastructure - it actually tells you nothing about how to deploy the
the dynatrace oneagent container itself. For that, you have to refer to
this link, which I had to find by myself:
https://help.dynatrace.com/infrastructure-monitori...
2) I have already set up a GKE container cluster - this is my existing system. I have also followed this link as a separate test and it is fine, but:
The problem is that even once I've deployed the oneagent container to GKE, the oneagent container crashes with the error I described in my original post.So the solution I need is:
a) To fix the root cause of why the oneagent container is crashing on GKE.
b) optionally: how to troubleshoot further to discover the reason why the oneagent container is crashing on GKE.
I'd be really thankful if you could help me with this. So far it appears that Dynatrace offers a solution to APM on GKE , but in practice we are not able to get this to work.
24 Jul 2017 08:44 AM
Hi Traiano,
sorry that the help topic around deploying OneAgent in GKE-based clusters is not helpful to you. I've asked one of our Docker experts to have a look at your problem description. Please stay tuned for further answers.
Best,
Daniela
24 Jul 2017 08:47 AM
Thanks, Daniela! I await in anticipation ...
24 Jul 2017 10:14 AM
Hi Traiano,
I assume that you have tried to deploy the latest image from Docker Hub, which is currently version 1.119. We had a problem with this version that causes it to not be able to recognize docker used by kubernetes correctly, the problem was fixed in version 1.121, that should be made available on Docker Hub in a matter of few days.
Furthermore, please be aware that currently we only support container-vm type of node image and deployment will not work with cos type.
Regards,
Tomek
24 Jul 2017 10:30 AM
Hi Tomasz
I've converted the GKE cluster to use container-vm.
We are using the "latest" oneagent image available. So if the current image is not 1.121, then I assume it's one of the defective images?
So, to conclude, would this mean we have to wait some days to complete our evaluation of Dynatrace as an APM for GKE? Any date for when we can evaluate 1.121?
Thanks in advance!
Traiano
25 Jul 2017 06:29 AM
Hi Traiano,
yes, the previous latest image (version 1.119) was defective.
However, version 1.121 was just released to Docker Hub and tagged as latest, this one should work with your GKE setup.
Regards,
Tomek
25 Jul 2017 08:13 AM
Hi Tomasz, thanks for your response.
However, I see the container (latest) still appears to have issues, the containers fail to startup due to a permissions issue in mounting the installation directory. Here is the container logs from kubernetes when deploying with "latest":
bash-3.2$ kubectl logs dynatrace-oneagent-2q2jv
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent directory.
chmod: /mnt/root//opt/dynatrace/oneagent: No such file or directory
07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent directory to 755.
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent/log directory.
chmod: /mnt/root//opt/dynatrace/oneagent/log: No such file or directory
07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent/log directory to 777.
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent/log/installer directory.
chmod: /mnt/root//opt/dynatrace/oneagent/log/installer: No such file or directory
07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent/log/installer directory to 755.
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:47 Cannot create /mnt/root//opt/dynatrace/oneagent/log/network directory.
chmod: /mnt/root//opt/dynatrace/oneagent/log/network: No such file or directory
07:07:47 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent/log/network directory to 755.
07:07:47 Started agent deployment as docker image, PID 26567.
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
07:07:47 Container version: 1.121.196.20170720-130455
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
07:07:48 Deploying to: /mnt/root
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
mkdir: can't create directory '/mnt/root//opt/dynatrace/': Read-only file system
07:07:48 Cannot create /mnt/root//opt/dynatrace/oneagent directory.
chmod: /mnt/root//opt/dynatrace/oneagent: No such file or directory
07:07:48 Cannot change permisions of /mnt/root//opt/dynatrace/oneagent directory to 755.
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
cp: can't create '/mnt/root//opt/dynatrace/oneagent/Dynatrace-OneAgent-Linux.sh': No such file or directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
/tmp/entrypoint.sh: line 352: can't create /mnt/root//opt/dynatrace/oneagent/log/installer/installation_docker_26567.log: nonexistent directory
Here is my deployment .yaml:
apiVersion: extensions/v1beta1kind: DaemonSetmetadata: name: dynatrace-oneagentspec: template: metadata: labels: run: dynatrace-oneagent spec: hostPID: true hostIPC: true hostNetwork: true volumes: - name: host-root hostPath: path: / containers: - name: dynatrace-oneagent image: dynatrace/oneagent:latest args: - "TENANT=lfn20468" - "TENANT_TOKEN=IRV1c15bIcRWq5KQ" - "SERVER=https://lfn20468.live.dynatrace.com" volumeMounts: - mountPath: /mnt/root name: host-root securityContext: privileged: true
Snapshot of status of the daemonset:
bash-3.2$ kubectl get daemonsets
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE
dynatrace-oneagent 3 3 0 3 0 <none> 5m
bash-3.2$
bash-3.2$
bash-3.2$
bash-3.2$ kubectl get pods
NAME READY STATUS RESTARTS AGE
dynatrace-oneagent-2q2jv 0/1 CrashLoopBackOff 5 5m
dynatrace-oneagent-5mqv1 0/1 CrashLoopBackOff 5 5m
dynatrace-oneagent-6vvtf 0/1 CrashLoopBackOff 5 5m
25 Jul 2017 12:34 PM
Hi Traiano,
are you sure that this node uses container-vm image?
I recall seeing similar errors when cos was used.
25 Jul 2017 01:35 PM
Hi Tomasz! Thanks, that was it, it looks like my cos->container-vm conversion didn't complete successfully last time, so I did it again. We have success 🙂
10 Oct 2017 07:12 PM
Having the same issue, what was the resolution?
12 Oct 2017 09:44 AM
Hi Scott,
such a problem can occur as a result of different root causes.
Please make sure that you followed the instructions from
https://help.dynatrace.com/infrastructure/containers/how-do-i-deploy-oneagent-in-gke-based-clusters/ (this page was recently updated).
If the problem still persists, please kindly open a support ticket on
https://www.dynatrace.com/support/
Regards,
Tomek
01 May 2018 09:38 PM
Answer by Tomasz G. Jul 24, 2017 at 09:14 AM
Hi Traiano,
I assume that you have tried to deploy the latest image from Docker Hub, which is currently version 1.119. We had a problem with this version that causes it to not be able to recognize docker used by kubernetes correctly, the problem was fixed in version 1.121, that should be made available on Docker Hub in a matter of few days.
Furthermore, please be aware that currently we only support container-vm type of node image and deployment will not work with cos type.
Regards,
Tomek
Where is Dynatrace now with supporting the cos type? We are doing some evaluations and this is a requirement. Thanks!
01 Jun 2018 05:38 AM
We're actively working on COS support, it should soon be released as an EAP feature.