topic Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke) in Container platforms

Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

matthias_dillie — Thu, 26 May 2022 07:36:35 GMT

Hi all

We have a kubernetes cluster with Rancher v1.21.10+rke2r2 and a local installation of dynatrace managed. We tried to install dynatrace full stack monitoring with the dynatrace operator as this is the installation which is recommended by dynatrace. Monitoring of kubernetes worked but not the monitoring of the applications in the cluster (Java, ga, webserver etc.).

The support answer is now, that dynatrace does not support kubernetes with rancher and rke, but it my work with other installation options.

Did someone succeed installing dynatrace operator and monitoring kubernetes with rancher and rke (monitoring kubernetes and the applications running in the pods)? Any hints, which konfigurations may work?
Thank you for any hints.

Kind regards

Matthias

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

techean — Thu, 26 May 2022 04:14:56 GMT

its solved already right in below post.

Solved: Re: Is there any way to monitor Rancher with Dynatrace? - Dynatrace Community

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

matthias_dillie — Fri, 27 May 2022 08:04:47 GMT

As I understand the post, the problem there is how to monitor rancher itself. We like to monitor the whole kubernetes cluster, which is installed with rancher and the rancher kubernetes engine. We want to monitor the applications running in pods in the cluster and the cluster itself. So I would say, it is not solved.

The dynatrace support answered, that rancher and rke is a not supported installation ("unsupported k8s distribution"), but 'there might be a workaround' using other installation options of dynatrace than using classic full stack with the dynatrace operator.

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

Julius_Loman — Fri, 27 May 2022 09:14:56 GMT

Hello @matthias_dillie ,
we have a customer running RKE2 and the Dynatrace Operator installation worked. There were only some issues with pulling the images, as RKE2 used proxy to pull them and avoided connecting to container image repository located on the Dynatrace Managed. There are only some issues with connecting the platform monitoring and the data from OneAgents as the OneAgent is not able to fetch some metadata from the containerd due to its location. However deep monitoring works without issues. Dynatrace Operator 0.5.0 and OA 1.237.

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

matthias_dillie — Fri, 27 May 2022 09:21:30 GMT

Hi Julius
Thanks for your reply. Can you give us some more details on the installation? We tried classic full stack injection (https://www.dynatrace.com/support/help/shortlink/dto-deploy-options-k8s#classic) which is the recommended option, but this did not work.

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

oliver_arnold — Fri, 27 May 2022 09:38:10 GMT

What you mean by this answer? We connected to a local Kubernetes API endpoint automatically as documented. Do we have to use the public Kubernetes API option for Ranger? See documentation
https://www.dynatrace.com/support/help/setup-and-configuration/setup-on-container-platforms/kubernetes/enable-kubernetes-api-monitoring/connect-your-k8s-clusters#local

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

Julius_Loman — Fri, 27 May 2022 09:42:53 GMT

@matthias_dillie can you please share what went wrong? Maybe share logs of the dynatrace operator and errors you are encountering. There are many points where a deployment into k8s may fail depending on your environment.

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

techean — Fri, 27 May 2022 09:49:33 GMT

@matthias_dillie did this help? just following up.

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

matthias_dillie — Fri, 27 May 2022 10:25:07 GMT

Thank you but no, not really, see my post below.

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

matthias_dillie — Fri, 27 May 2022 12:04:44 GMT

@Julius_Loman we first tried with classic full stack option. Monitoring Kubernetes itself worked good. The problem was, that we did not see the deep monitoring part of the applications running in the pods eg. java (OpenLiberty), go (Prometheus), (ingress-)nginx and so on. We saw the processes and cpu, memory and so they use but we did not see service calls, methods and all this details. With the dynatrace support, we looked at the system and found no errors in logfiles or other hints. The answer then was: not supported. See https://one.dynatrace.com/hc/en-us/requests/39972

Now we try with 'Automatic application-only injection'. We see now the OpenLiberty applications. This is ok now. But ingress-nginx, go and so on does not work. We no longer see this processes in Dynatrace. Maybe we have a ressource problem as we see a tmp.xxxx-file in the agent directory in the pods with go applications and ngnix instances?

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

Julius_Loman — Sun, 29 May 2022 20:08:29 GMT

@matthias_dillie I don't have access to your support tickets. Did support reveal any details on where is the injection failing with classic fullstack?

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

matthias_dillie — Mon, 30 May 2022 08:31:06 GMT

No. We had a skype session to solve the problem but we did not find any errors or wrong configurations. Finally the answer was, that this combination of Rancher with rke is 'not supported' by dynatrace. No more explanation. Their hint was to try application only injection, this 'might' work.

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

Prakash — Thu, 13 Oct 2022 04:09:27 GMT

@Julius_Loman We aretrying to setup Dynatrace oneagent on RKE2 cluster ( Backend Vmware ).

After installation , Dynatrace oneagent pod keeps crashing with below error

03:47:18 Bootstrapping read-only deployment Executing /tmp/bootstrap.sh --set-proxy=http://vzen01.internal.bunnings.com.au:80 --set-host-property=OperatorVersion=v0.9.0 --set-deployment-metadata=orchestration_tech=Operator-cloud_native_fullstack --set-deployment-metadata=script_version=v0.9.0 --set-deployment-metadata=orchestrator_id=96654e16-1513-4dd0-84b2-5360a3f563e5 --set-host-id-source=k8s-node-name with ambient capabilities set 03:47:18 Started agent deployment as a container, PID 128145. 03:47:18 System version: Linux snd-dc2-rke-emp-001-wrksnd-4fe631fb-9r6d6 5.4.17-2136.307.3.5.el7uek.x86_64 #2 SMP Thu May 26 19:06:29 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux 03:47:18 Command line: --set-proxy=*** --set-host-property=OperatorVersion=v0.9.0 --set-deployment-metadata=orchestration_tech=Operator-cloud_native_fullstack --set-deployment-metadata=script_version=v0.9.0 --set-deployment-metadata=orchestrator_id=96654e16-1513-4dd0-84b2-5360a3f563e5 --set-host-id-source=k8s-node-name 03:47:18 Installed version: 03:47:18 Using pre-packed installer, no download from external location will be performed. 03:47:19 ONEAGENT_INSTALLER_SCRIPT_URL= 03:47:19 ONEAGENT_INSTALLER_DOWNLOAD_TOKEN= 03:47:19 ONEAGENT_INSTALLER_DOWNLOAD_VERBOSE= 03:47:19 ONEAGENT_INSTALLER_SKIP_CERT_CHECK= 03:47:19 ONEAGENT_ENABLE_VOLUME_STORAGE= 03:47:19 ONEAGENT_CONTAINER_STORAGE_PATH= 03:47:19 ONEAGENT_NO_REMOUNT_ROOT= 03:47:19 ONEAGENT_ADDITIONAL_UNMOUNT_PATTERN= 03:47:19 ONEAGENT_DISABLE_CONTAINER_INJECTION=true 03:47:19 ONEAGENT_READ_ONLY_MODE=true 03:47:19 AGENT_CONTAINER_IMAGE_VERSION=1.249.2.20220812-040632 03:47:19 Path: /usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 03:47:19 Started with capabilities: self: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_sys_chroot,cap_sys_ptrace,cap_sys_admin,cap_setfcap=eip 03:47:19 Started with user: uid=1000 gid=1000 groups=1000 03:47:19 Using volume-based storage 03:47:19 Volume mount detected under /mnt/volume_storage_mount 03:47:19 Error: /mnt/volume_storage_mount exists but sanity check for path on host (/mnt/root/lib/kubelet/plugins/csi.oneagent.dynatrace.com/data/uun69539/osagent) failed 03:47:21 Error: Initialization procedure failed

Have you seen this in your setup ?

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

Julius_Loman — Thu, 13 Oct 2022 07:48:35 GMT

@Prakash no. What deployment mode of Dynatrace operator are you using? I was trying that only with classic full stack. Based on your error, Dynatrace likely tries to use the CSI driver.

Re: Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

andre_vdveen — Fri, 04 Aug 2023 13:38:16 GMT

I assume you're aware that at the time of writing this, RKE2 is not supported, only RKE1.

https://www.dynatrace.com/support/help/shortlink/section-technology-support#kubernetes