Error while running dynakube.yaml

[kduebc]

Hi,

We have k8s behind proxy and are trying to install full-stack observe. kubernetes-csi.yaml was successful but when I run dynakube.yaml I get the following error:

# kubectl apply -f dynakube.yaml
Error from server (InternalError): error when creating "dynakube.yaml": Internal error occurred: failed calling webhook "v1beta3.dynakube.webhook.dynatrace.com": failed to call webhook: the server could not find the requested resource

Could this be due to network connectivity? pods are:

# kubectl get pods -n dynatrace
NAME READY STATUS RESTARTS AGE
dynatrace-oneagent-csi-driver-2zprh 3/4 CrashLoopBackOff 18 (5m3s ago) 26m
dynatrace-oneagent-csi-driver-8zjjt 3/4 CrashLoopBackOff 18 (5m6s ago) 26m
dynatrace-oneagent-csi-driver-dp5vb 3/4 CrashLoopBackOff 18 (4m55s ago) 26m
dynatrace-oneagent-csi-driver-pp2nj 3/4 CrashLoopBackOff 18 (4m50s ago) 26m
dynatrace-oneagent-csi-driver-rsvdm 4/4 CrashLoopBackOff 19 (5m2s ago) 26m
dynatrace-oneagent-csi-driver-slpzs 3/4 CrashLoopBackOff 18 (4m54s ago) 26m
dynatrace-oneagent-csi-driver-zxx8v 3/4 CrashLoopBackOff 19 (13s ago) 26m
dynatrace-operator-587886577d-wzr4k 1/1 Running 0 23h
dynatrace-webhook-6d878bcc77-82l5n 1/1 Running 0 23h
dynatrace-webhook-6d878bcc77-ttgpr 1/1 Running 0 23h


describe warning:

Warning BackOff 2m27s (x131 over 27m) kubelet Back-off restarting failed container server in pod dynatrace-oneagent-csi-driver-rsvdm_dynatrace

[gopher]

Hi @kduebc ,

I had this issue, it was resolved by updating the active-gate service name in the no-proxy annotation. 
If you already have it from a previous release, doublecheck it as the service names change with the 1.4+ operator version. 

 annotations:
    feature.dynatrace.com/no-proxy: ".dynatrace.svc,.artifactory.gcp.xxx,.amazonaws.com" #private repo
  

[pawel_harbuz]

It looks like Dynatrace Operator can't communicate with webhook. Do you see that all of your CSI driver pods are in back-off state? Without running them you can't deploy an operator. Could you describe one pod of CSI driver and provide output? In example: kubectl describe pod dynatrace-oneagent-csi-driver-zxx8v -n=dynatrace

[kduebc]

Following is the output

Name: dynatrace-oneagent-csi-driver-q4tlg
Namespace: dynatrace
Priority: 1000000
Priority Class Name: dynatrace-high-priority
Service Account: dynatrace-oneagent-csi-driver
Node: s-k8s-w21/10.1.90.224
Start Time: Mon, 10 Mar 2025 10:45:50 +1000
Labels: app.kubernetes.io/component=csi-driver
app.kubernetes.io/name=dynatrace-operator
app.kubernetes.io/version=1.4.1
controller-revision-hash=67d78b6557
internal.oneagent.dynatrace.com/app=csi-driver
internal.oneagent.dynatrace.com/component=csi-driver
pod-template-generation=1
Annotations: cluster-autoscaler.kubernetes.io/enable-ds-eviction: false
cni.projectcalico.org/containerID: f9df8604943a7764319945db68c9f5cbf8f30fc0e12e259709b4f5035d6906f4
cni.projectcalico.org/podIP: 10.244.187.34/32
cni.projectcalico.org/podIPs: 10.244.187.34/32
dynatrace.com/inject: false
kubectl.kubernetes.io/default-container: provisioner
Status: Running
IP: 10.244.187.34
IPs:
IP: 10.244.187.34
Controlled By: DaemonSet/dynatrace-oneagent-csi-driver
Init Containers:
csi-init:
Container ID: cri-o://d4306d2b7a31717ca0226288e0cd7502948560cfdcc2769518c20fdf15757bd5
Image: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Image ID: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Port: <none>
Host Port: <none>
Args:
csi-init
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 10 Mar 2025 10:45:55 +1000
Finished: Mon, 10 Mar 2025 10:45:56 +1000
Ready: True
Restart Count: 0
Limits:
cpu: 50m
memory: 100Mi
Requests:
cpu: 50m
memory: 100Mi
Environment: <none>
Mounts:
/data from data-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pzp4g (ro)
Containers:
server:
Container ID: cri-o://63084fda638277f4acdd49ac990b990957160b38ee680fb8aa8c44d5c0be7937
Image: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Image ID: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Ports: 9808/TCP, 8080/TCP
Host Ports: 0/TCP, 0/TCP
Args:
csi-server
--endpoint=unix:/csi/csi.sock
--node-id=$(KUBE_NODE_NAME)
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 10 Mar 2025 10:56:42 +1000
Finished: Mon, 10 Mar 2025 10:56:43 +1000
Ready: False
Restart Count: 7
Limits:
cpu: 50m
memory: 100Mi
Requests:
cpu: 50m
memory: 100Mi
Liveness: http-get http://:healthz/healthz delay=15s timeout=10s period=15s #success=1 #failure=5
Environment:
POD_NAMESPACE: dynatrace (v1:metadata.namespace)
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from plugin-dir (rw)
/data from data-dir (rw)
/tmp from tmp-dir (rw)
/var/lib/kubelet/pods/ from mountpoint-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pzp4g (ro)
provisioner:
Container ID: cri-o://01c97c0ddd72ee46093c3306616cbbbeaa19d23d065a37df1393a1a5cdfa466d
Image: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Image ID: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Ports: 10090/TCP, 8090/TCP
Host Ports: 0/TCP, 0/TCP
Args:
csi-provisioner
--health-probe-bind-address=:10090
State: Running
Started: Mon, 10 Mar 2025 10:45:58 +1000
Ready: True
Restart Count: 0
Requests:
cpu: 300m
memory: 100Mi
Liveness: http-get http://:livez/livez delay=5s timeout=1s period=5s #success=1 #failure=3
Startup: exec [/usr/local/bin/dynatrace-operator startup-probe] delay=0s timeout=5s period=10s #success=1 #failure=1
Environment:
POD_NAMESPACE: dynatrace (v1:metadata.namespace)
modules.json: {
"csiDriver": true,
"activeGate": true,
"oneAgent": true,
"extensions": true,
"logMonitoring": true,
"edgeConnect": true,
"supportability": true,
"kspm": true
}

Mounts:
/data from data-dir (rw)
/tmp from tmp-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pzp4g (ro)
registrar:
Container ID: cri-o://6f7b9c0421b521383147bd98a678af140752599501a2f3c63102a1f3b66f9b29
Image: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Image ID: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Port: <none>
Host Port: <none>
Command:
csi-node-driver-registrar
Args:
--csi-address=/csi/csi.sock
--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 10 Mar 2025 10:54:43 +1000
Finished: Mon, 10 Mar 2025 10:55:14 +1000
Ready: False
Restart Count: 6
Limits:
cpu: 20m
memory: 30Mi
Requests:
cpu: 20m
memory: 30Mi
Environment:
DRIVER_REG_SOCK_PATH: /var/lib/kubelet/plugins/csi.oneagent.dynatrace.com/csi.sock
Mounts:
/csi from plugin-dir (rw)
/registration from registration-dir (rw)
/var/lib/kubelet/plugins/csi.oneagent.dynatrace.com/ from lockfile-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pzp4g (ro)
liveness-probe:
Container ID: cri-o://8c27d4c70ca8c61ed796bc4cded1833acf8ab60f03d3cb971d6e10a1d76a24f8
Image: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Image ID: s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a
Port: <none>
Host Port: <none>
Command:
livenessprobe
Args:
--csi-address=/csi/csi.sock
--health-port=9808
--probe-timeout=9s
State: Running
Started: Mon, 10 Mar 2025 10:46:02 +1000
Ready: True
Restart Count: 0
Limits:
cpu: 20m
memory: 30Mi
Requests:
cpu: 20m
memory: 30Mi
Environment: <none>
Mounts:
/csi from plugin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pzp4g (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry/
HostPathType: Directory
plugin-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/csi.oneagent.dynatrace.com/
HostPathType: DirectoryOrCreate
data-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/csi.oneagent.dynatrace.com/data
HostPathType: DirectoryOrCreate
mountpoint-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/pods/
HostPathType: DirectoryOrCreate
lockfile-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tmp-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-pzp4g:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: ToBeDeletedByClusterAutoscaler:NoSchedule op=Exists
kubernetes.io/arch=amd64:NoSchedule
kubernetes.io/arch=arm64:NoSchedule
kubernetes.io/arch=ppc64le:NoSchedule
kubernetes.io/arch=s390x:NoSchedule
node-role.kubernetes.io/control-plane:NoSchedule op=Exists
node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned dynatrace/dynatrace-oneagent-csi-driver-q4tlg to s-k8s-w21
Normal Pulling 13m kubelet Pulling image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a"
Normal Pulled 13m kubelet Successfully pulled image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a" in 81.178869ms (81.190076ms including waiting)
Normal Created 13m kubelet Created container csi-init
Normal Started 13m kubelet Started container csi-init
Normal Pulled 13m kubelet Successfully pulled image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a" in 60.464681ms (60.476568ms including waiting)
Normal Pulling 13m kubelet Pulling image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a"
Normal Started 13m kubelet Started container provisioner
Normal Pulled 13m kubelet Successfully pulled image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a" in 59.984636ms (59.994808ms including waiting)
Normal Pulling 13m kubelet Pulling image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a"
Normal Pulled 13m kubelet Successfully pulled image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a" in 61.40022ms (61.411261ms including waiting)
Normal Created 13m kubelet Created container provisioner
Normal Created 13m kubelet Created container registrar
Normal Started 13m kubelet Started container registrar
Normal Pulling 13m kubelet Pulling image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a"
Normal Pulled 13m kubelet Successfully pulled image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a" in 70.71851ms (70.723591ms including waiting)
Normal Pulling 13m (x2 over 13m) kubelet Pulling image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a"
Normal Created 13m kubelet Created container liveness-probe
Normal Started 13m kubelet Started container liveness-probe
Normal Pulled 13m kubelet Successfully pulled image "s-harbor.example.com/atis/dynatrace/dynatrace-operator@sha256:123275d7ff395a29bf5524fd5d69573e593d80e7597a40fde9953c0dd78ffc5a" in 60.778914ms (60.785462ms including waiting)
Normal Started 12m (x2 over 13m) kubelet Started container server
Normal Created 12m (x2 over 13m) kubelet Created container server
Warning BackOff 2m57s (x60 over 12m) kubelet Back-off restarting failed container server in pod dynatrace-oneagent-csi-driver-q4tlg_dynatrace(1160d563-a955-40a4-a462-c404a9a9953b)