cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Watchdog causing issues with Oneagent in cloudNativeFullStack deployment.

chahat_bhatia
Newcomer

I recently installed the Cloud Native Full Stack  for dynatrace. 
All the components seem to come up fine except `eks-sit1b-oneagent`.

 

NAME                                                         READY  STATUS 
dynatrace-oneagent-csi-driver-gwphr       4/4 Running
dynatrace-oneagent-csi-driver-r2bk9         4/4 Running 
dynatrace-operator-6c44b8fcff-2zhv6       1/1 Running 
dynatrace-webhook-5cb47fd4cd-pdx85     1/1 Running
dynatrace-webhook-5cb47fd4cd-qlhnj       1/1 Running
eks-sit1b-activegate-0                                  1/1 Running
eks-sit1b-oneagent-bhhqt                            0/1 Running 
eks-sit1b-oneagent-hpnv7                            0/1 Running 


The oneagent pod won't come up; logs in the pod:
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied) 
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied) 
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied) 
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied) 
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied)


The readiness probe failing with:

[2024-06-04 23:43:54.085] [error] Watchdog pid file not found
[2024-06-04 23:43:54.085] [info] Watchdog healthcheck returned FAILURE


Also some other pods in the cluster, erroring out with logs:

>>  MountVolume.SetUp failed for volume "oneagent-bin" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name csi.oneagent.dynatrace.com not found in the list of registered CSI drivers

But the CSI Driver is very much present on the cluster 😕


Any ideas on how to fix these issues? Even the slighest help would be appreciated. Thanks.

4 REPLIES 4

Esam_Eid
Pro

Have you installed it by root user?

Dynatrace Certified Professional

PacoPorro
Dynatrace Leader
Dynatrace Leader

Can you check if something is blocking the watchdog ports?
On startup, OneAgent watchdog tries to open the first available port between ports 50000 and 50100.

chahat_bhatia
Newcomer

I recently installed the Cloud Native Full Stack  for dynatrace. 
All the components seem to come up fine except `eks-sit1b-oneagent`.

 

NAME                                                         READY  STATUS 
dynatrace-oneagent-csi-driver-gwphr       4/4 Running
dynatrace-oneagent-csi-driver-r2bk9         4/4 Running 
dynatrace-operator-6c44b8fcff-2zhv6       1/1 Running 
dynatrace-webhook-5cb47fd4cd-pdx85     1/1 Running
dynatrace-webhook-5cb47fd4cd-qlhnj       1/1 Running
eks-sit1b-activegate-0                                  1/1 Running
eks-sit1b-oneagent-bhhqt                            0/1 Running 
eks-sit1b-oneagent-hpnv7                            0/1 Running 


The oneagent pod won't come up; logs in the pod:
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied) 
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied) 
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied) 
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied) 
dynatrace-oneagent Cannot open log file "/var/log/dynatrace/oneagent/watchdog/oneagentwatchdog_218530.log": error code: 13 (Permission denied)

The readiness probe failing with:

[2024-06-04 23:43:54.085] [error] Watchdog pid file not found
[2024-06-04 23:43:54.085] [info] Watchdog healthcheck returned FAILURE

Also some other pods in the cluster, erroring out with logs:
>>  MountVolume.SetUp failed for volume "oneagent-bin" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name csi.oneagent.dynatrace.com not found in the list of registered CSI drivers

But the CSI Driver is very much present on the cluster 😕


Any ideas on how to fix these issues? Even the slighest help would be appreciated. Thanks.

Featured Posts