CPU Throttling on CSI Driver Pods

[ntanaka]

Question

We are experiencing frequent CPU throttling on CSI driver Pods of Dynatrace Operator.
The CPU throttling rate is 600-700%.

Has anyone experienced similar issues in their environment?

If yes, what actions did you take to resolve them? How much should we increase the CPU limits?

We have already increased the CPU limits compared to the defaults, but throttling still occurs.
In your experience, how much did you increase the CPU limits to mitigate this?

Current Situation

CPU throttling is happening on CSI driver Pods, but there is no immediate impact on workloads.
However, in the past, we had a CrashLoopBackOff issue with the CSI driver, which prevented business-critical Pods from starting. So, we concerned about potential CPU shortages.

Environment

• OpenShift 4.14
• Kubernetes 1.27
• Dynatrace Operator v1.4.1

Resource Configuration

name: server
resources:
  limits:
    cpu: "1"       # default: 50m
    memory: 600Mi  # default: 100Mi
  requests:
    cpu: 300m      # default: 50m
    memory: 600Mi  # default: 100Mi

name: provisioner
resources:
  requests:
    cpu: 300m
    memory: 100Mi

name: registrar
resources:
  limits:
    cpu: 20m
    memory: 30Mi
  requests:
    cpu: 20m
    memory: 30Mi

name: liveness-probe
resources:
  limits:
    cpu: 200m      # default: 20m
    memory: 300Mi  # default: 30Mi
  requests:
    cpu: 20m
    memory: 30Mi

[Michal_Gebacki · Accepted solution]

Hello, @ntanaka!

Is this Dynatrace Documentation page helpful in any way? Please let us know, any feedback is greatly appreciated!

-> Set resource limits for Dynatrace Operator components

[Julius_Loman · Accepted solution]

Hello @ntanaka 

I highly recommend upgrading your Dynatrace Operator first to 1.7.1 or 1.7.2 at least as there have been some issues resolved which might cause symptoms you see, especially the CrashLoopBackOff.

[ntanaka]

Hi @Michal_Gebacki and @Julius_Loman ,

Thank you very much for your reply, and sorry for the delay.

After my previous post, we discovered that CPU throttling occurred in the registrar container.
We did not increase the CPU limit for the registrar container, which remains at 20mCore, the same as the default.

However, we cannot determine why the registrar container is consuming so much CPU.
I checked the registrar log, but it only showed activity during installation.

log:

I0404 04:17:44.467787       1 main.go:150] "Version" version="v2.13.0"
I0404 04:17:44.467870       1 main.go:151] "Running node-driver-registrar" mode=""
I0404 04:17:44.867955       1 node_register.go:56] "Starting Registration Server" socketPath="/registration/csi.oneagent.dynatrace.com-reg.sock"
I0404 04:17:44.967883       1 node_register.go:66] "Registration Server started" socketPath="/registration/csi.oneagent.dynatrace.com-reg.sock"
I0404 04:17:44.967960       1 node_register.go:96] "Skipping HTTP server"
I0404 04:17:45.720760       1 main.go:96] "Received GetInfo call" request="&InfoRequest{}"
I0404 04:17:45.728137       1 main.go:108] "Received NotifyRegistrationStatus call" status="&RegistrationStatus{PluginRegistered:true,Error:,}"

Question:

@Michal_Gebacki,
I reviewed the documentation of resource limits and found that we should increase the limit when there are many pods.

However, our cluster has only 16 nodes and 587 pods.
The documentation mentions that a baseline environment has 25 nodes and 5,000 pods, while large environments have more than 50 nodes and 10,000 pods.
I don’t think our cluster qualifies as large.
Should we still increase the CPU limit?

@Julius_Loman ,
Do you have any information on whether Dynatrace Operator 1.7.1 or 1.7.2 changed the registrar container of the CSI driver?