cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Monitor Kubernetes with rancher and the rancher kubernetes engine (rke)

matthias_dillie
Advisor

Hi all

We have a kubernetes cluster with Rancher v1.21.10+rke2r2 and a local installation of dynatrace managed. We tried to install dynatrace full stack monitoring with the dynatrace operator as this is the installation which is recommended by dynatrace. Monitoring of kubernetes worked but not the monitoring of the applications in the cluster (Java, ga, webserver etc.).

The support answer is now, that dynatrace does not support kubernetes with rancher and rke, but it my work with other installation options.

Did someone succeed installing dynatrace operator and monitoring kubernetes with rancher and rke (monitoring kubernetes and the applications running in the pods)? Any hints, which konfigurations may work?
Thank you for any hints.

Kind regards

Matthias

14 REPLIES 14

techean
Dynatrace Champion
Dynatrace Champion

As I understand the post, the problem there is how to monitor rancher itself. We like to monitor the whole kubernetes cluster, which is installed with rancher and the rancher kubernetes engine. We want to monitor the applications running in pods in the cluster and the cluster itself. So I would say, it is not solved.

The dynatrace support answered, that rancher and rke is a not supported installation ("unsupported k8s distribution"), but 'there might be a workaround' using other installation options of dynatrace than using classic full stack with the dynatrace operator.

What you mean by this answer? We connected to a local Kubernetes API endpoint automatically as documented. Do we have to use the public Kubernetes API option for Ranger? See documentation
https://www.dynatrace.com/support/help/setup-and-configuration/setup-on-container-platforms/kubernet... 

 

techean
Dynatrace Champion
Dynatrace Champion

@matthias_dillie  did this help? just following up.

KG

Thank you but no, not really, see my post below.

Julius_Loman
DynaMight Legend
DynaMight Legend

Hello @matthias_dillie ,
we have a customer running RKE2 and the Dynatrace Operator installation worked. There were only some issues with pulling the images, as RKE2 used proxy to pull them and avoided connecting to container image repository located on the Dynatrace Managed. There are only some issues with connecting the platform monitoring and the data from OneAgents as the OneAgent is not able to fetch some metadata from the containerd due to its location. However deep monitoring works without issues. Dynatrace Operator 0.5.0 and OA 1.237.

Certified Dynatrace Master | Alanata a.s., Slovakia, Dynatrace Master Partner

Hi Julius
Thanks for your reply. Can you give us some more details on the installation? We tried classic full stack injection (https://www.dynatrace.com/support/help/shortlink/dto-deploy-options-k8s#classic) which is the recommended option, but this did not work.

@matthias_dillie can you please share what went wrong? Maybe share logs of the dynatrace operator and errors you are encountering. There are many points where a deployment into k8s may fail depending on your environment.

Certified Dynatrace Master | Alanata a.s., Slovakia, Dynatrace Master Partner

@Julius_Loman we first tried with classic full stack option. Monitoring Kubernetes itself worked good. The problem was, that we did not see the deep monitoring part of the applications running in the pods eg. java (OpenLiberty), go (Prometheus), (ingress-)nginx and so on. We saw the processes and cpu, memory and so they use but we did not see service calls, methods and all this details. With the dynatrace support, we looked at the system and found no errors in logfiles or other hints. The answer then was: not supported. See https://one.dynatrace.com/hc/en-us/requests/39972

Now we try with 'Automatic application-only injection'. We see now the OpenLiberty applications. This is ok now. But ingress-nginx, go and so on does not work. We no longer see this processes in Dynatrace. Maybe we have a ressource problem as we see a tmp.xxxx-file in the agent directory in the pods with go applications and ngnix instances?

@matthias_dillie I don't have access to your support tickets. Did support reveal any details on where is the injection failing with classic fullstack? 

Certified Dynatrace Master | Alanata a.s., Slovakia, Dynatrace Master Partner

No. We had a skype session to solve the problem but we did not find any errors or wrong configurations. Finally the answer was, that this combination of Rancher with rke is 'not supported' by dynatrace. No more explanation. Their hint was to try application only injection, this 'might' work.

Prakash
Visitor

@Julius_Loman  We aretrying to setup Dynatrace oneagent on RKE2 cluster ( Backend Vmware ). 

After installation , Dynatrace oneagent pod keeps crashing with below error 

 

03:47:18 Bootstrapping read-only deployment
Executing /tmp/bootstrap.sh --set-proxy=http://vzen01.internal.bunnings.com.au:80 --set-host-property=OperatorVersion=v0.9.0 --set-deployment-metadata=orchestration_tech=Operator-cloud_native_fullstack --set-deployment-metadata=script_version=v0.9.0 --set-deployment-metadata=orchestrator_id=96654e16-1513-4dd0-84b2-5360a3f563e5 --set-host-id-source=k8s-node-name with ambient capabilities set
03:47:18 Started agent deployment as a container, PID 128145.
03:47:18 System version: Linux snd-dc2-rke-emp-001-wrksnd-4fe631fb-9r6d6 5.4.17-2136.307.3.5.el7uek.x86_64 #2 SMP Thu May 26 19:06:29 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
03:47:18 Command line: --set-proxy=*** --set-host-property=OperatorVersion=v0.9.0 --set-deployment-metadata=orchestration_tech=Operator-cloud_native_fullstack --set-deployment-metadata=script_version=v0.9.0 --set-deployment-metadata=orchestrator_id=96654e16-1513-4dd0-84b2-5360a3f563e5 --set-host-id-source=k8s-node-name
03:47:18 Installed version:
03:47:18 Using pre-packed installer, no download from external location will be performed.
03:47:19 ONEAGENT_INSTALLER_SCRIPT_URL=
03:47:19 ONEAGENT_INSTALLER_DOWNLOAD_TOKEN=
03:47:19 ONEAGENT_INSTALLER_DOWNLOAD_VERBOSE=
03:47:19 ONEAGENT_INSTALLER_SKIP_CERT_CHECK=
03:47:19 ONEAGENT_ENABLE_VOLUME_STORAGE=
03:47:19 ONEAGENT_CONTAINER_STORAGE_PATH=
03:47:19 ONEAGENT_NO_REMOUNT_ROOT=
03:47:19 ONEAGENT_ADDITIONAL_UNMOUNT_PATTERN=
03:47:19 ONEAGENT_DISABLE_CONTAINER_INJECTION=true
03:47:19 ONEAGENT_READ_ONLY_MODE=true
03:47:19 AGENT_CONTAINER_IMAGE_VERSION=1.249.2.20220812-040632
03:47:19 Path: /usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
03:47:19 Started with capabilities: self: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_sys_chroot,cap_sys_ptrace,cap_sys_admin,cap_setfcap=eip
03:47:19 Started with user: uid=1000 gid=1000 groups=1000
03:47:19 Using volume-based storage
03:47:19 Volume mount detected under /mnt/volume_storage_mount
03:47:19 Error: /mnt/volume_storage_mount exists but sanity check for path on host (/mnt/root/lib/kubelet/plugins/csi.oneagent.dynatrace.com/data/uun69539/osagent) failed

03:47:21 Error: Initialization procedure failed

Have you seen this in your setup ?

 

@Prakash no. What  deployment mode of Dynatrace operator are you using? I was trying that only with classic full stack. Based on your error, Dynatrace likely tries to use the CSI driver.

Certified Dynatrace Master | Alanata a.s., Slovakia, Dynatrace Master Partner

andre_vdveen
DynaMight Champion
DynaMight Champion

I assume you're aware that at the time of writing this, RKE2 is not supported, only RKE1.

https://www.dynatrace.com/support/help/shortlink/section-technology-support#kubernetes

andre_vdveen_0-1691156287630.png

 

Featured Posts