Hi everyone,
Question:
We are experiencing high memory consumption of the liveness-probe container in the CSI driver Pod when using Dynatrace Operator.
Has anyone experienced the similar issues?
Or, does anyone have insights or best practices for addressing high memory usage by the liveness-probe?
I noticed that Dynatrace Operator v1.7.0 introduced Helm switches for the livenessProbe parameter:
https://docs.dynatrace.com/docs/shortlink/release-notes-dto-fix-1-7-0#new-features-and-enhancements
Would tuning the livenessProbe parameters help reduce resource consumption?
Background:
We previously encountered a CrashLoopBackOff issue with the CSI driver caused by liveness-probe timeouts. Despite tuning various parameters based on guidance from the support team and the Community article, the issue persisted.
Eventually, increasing the memory limit for liveness-probe from the default 30 MB to 600 MB resolved the CrashLoopBackOff.
In a test environment, we intentionally set the memory limit to 6 MB, which reproduced the timeout issue. This suggests that memory exhaustion in liveness-probe leads to timeouts.
However, the root cause of the high memory consumption remains unclear. We are investigating ways to reduce memory usage to avoid CrashLoopBackOff.
Environment:
Dynatrace Operator v1.4.1
OpenShift 4.14
Kubernetes 1.27
Current livenessProbe settings:
livenessProbe:
failureThreshold: 5 #default 3
httpGet:
path: /healthz
port: healthz
scheme: HTTP
initialDelaySeconds: 15 #default 5
periodSeconds: 15 #default 5
successThreshold: 1
timeoutSeconds: 10 #default 1
name: server
ports:
- containerPort: 9808
name: healthz
protocol: TCP
- containerPort: 8080
name: metrics
protocol: TCP
resources:
limits:
cpu: "1" #default 20m
memory: 600Mi #default 30Mi
requests:
cpu: 300m #default 20m
memory: 600Mi #default 30Mi
