GKE Autopilot DTO Cloud-native installation

[dcinciripini]

Hi everyone 🙂
we encountered a issue installing DTO 1.1 on Cloud-Native mode on GKE w autopilot when tryied to install CSI-driver:

Error from server (GKE Warden constraints violations): error when creating "https://github.com/Dynatrace/dynatrace-operator/releases/download/v1.1.0/kubernetes-csi.yaml": admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints. Violations details: {"[denied by autogke-disallow-privilege]":["container server is privileged; not allowed in Autopilot","container provisioner is privileged; not allowed in Autopilot"],"[denied by autogke-no-write-mode-hostpath]":["hostPath volume plugin-dir in container server is accessed in write mode; disallowed in Autopilot.","hostPath volume mountpoint-dir in container server is accessed in write mode; disallowed in Autopilot.","hostPath volume data-dir in container server is accessed in write mode; disallowed in Autopilot.","hostPath volume data-dir in container provisioner is accessed in write mode; disallowed in Autopilot.","hostPath volume plugin-dir in container registrar is accessed in write mode; disallowed in Autopilot.","hostPath volume registration-dir in container registrar is accessed in write mode; disallowed in Autopilot.","hostPath volume plugin-dir in container liveness-probe is accessed in write mode; disallowed in Autopilot.","hostPath volume data-dir in container csi-init is accessed in write mode; disallowed in Autopilot."]}

reading Dynatrace and GKE documentation about Autopilot Pod Security I found this links:
https://docs.dynatrace.com/docs/shortlink/pod-security-standards#configure-pod-security-for-the-name...

https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-security?hl=it#autopilot-pss

where seems to be impossible installing pods with privileged permission: autopilot GKE doesn't allow setting privileged criterion on namespace.

This means that we can't install OA and CSI-driver (pod w privileged permission required) on GKE autopilot except using this workaround (v0.12)?
https://community.dynatrace.com/t5/Heads-up-from-Dynatrace/CSI-Driver-pods-with-Dynatrace-Operator-0...

Thanks everyone,
Davide

[dcinciripini]

Searching again we found this article where:
- is explained that GKE autopilot installation is possible using Helm

- is linked Dynatrace documentation about application observability (by Helm)

So I suppose that, for the installation:
- We need to use "Installation with additional configuration of the Helm chart" using values.yaml where we can specify platform: gke-autopilot

- We can install only in application observability mode

[dannemca]

Hi @dcinciripini where you able to resolve this?

What have you changed on the values.yaml template file?

Thanks.

[dcinciripini · Accepted solution]

Hi dannemca,
we managed to solve it by installing the Kubernetes.yaml via Manifest in Application-observability classic mode, the only way that GKE-autopilot offers for installation because OA is not a privileged pod allowed, without CSI-driver. Theorically it's possible installing CSI-driver because of Google-Dynatrace partnership but we encountered again the same problem trying to install it, so we preferred to install without it to accelarate the operations. By the way we have been informed that other Dynatrace customers succeded with CSI-driver.

I hope that this answer help you a little bit! I would be curious to know if you can install with CSI-driver 🙂

[ian_cervantez · Accepted solution]

In Dynatrace for GKE Autopilot, the Google-Dynatrace partnership "Allowlisted Autopilot partner workloads" only pertains to containers hosted in the following repositories:

gcr.io/dynatrace-marketplace-prod/dynatrace-operator:<version>
docker.io/dynatrace/dynatrace-operator:<version>

So, when applying with helm you will need to specify imageRef.repository (like so):

helm upgrade --install dynatrace-operator oci://public.ecr.aws/dynatrace/dynatrace-operator \
    --create-namespace \
    --namespace dynatrace \
    --atomic \
    --set "platform=gke-autopilot" \
    --set "imageRef.repository=docker.io/dynatrace/dynatrace-operator" \
    --set "csidriver.enabled=true"

Or, with a values.yaml file override.

[dannemca]

Thank you @dcinciripini and @ian_cervantez for your updates... indeed, I could deploy the operator with CIS enabled.

But I have one question... were you able to deploy the oneagents in your nodes? Or the dynakube was the application only?

I tried to use the full one, and got this error

Message:               A problem occurred when using the Kubernetes API: admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by autogke-default-linux-capabilities]":["linux capability 'DAC_READ_SEARCH,NET_ADMIN,SYS_ADMIN,SYS_RESOURCE' on container 'dynatrace-oneagent' not allowed; Autopilot only allows the capabilities: 'AUDIT_WRITE,CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,MKNOD,NET_BIND_SERVICE,NET_RAW,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT,SYS_PTRACE'."],"[denied by autogke-disallow-hostnamespaces]":["enabling hostPID is not allowed in Autopilot.","enabling hostNetwork is not allowed in Autopilot."],"[denied by autogke-no-write-mode-hostpath]":["hostPath volume host-root used in container dynatrace-oneagent uses path / which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/]."]}

Thanks

[ian_cervantez]

Yeah, Dynakube can only be configured for application only monitoring mode.  And that's all you need with autopilot, since you can't control the worker nodes anyway.

I was able to configure dynakube for classic monitoring mode, but I didn't run that for very long so I don't know if that's 100% stable.

apiVersion: dynatrace.com/v1beta2
kind: DynaKube
spec:
  activeGate:
    capabilities:
    - routing
    - kubernetes-monitoring
    - dynatrace-api
    group: <GROUP_NAME_HERE>
    replicas: 1
    resources:
      limits:
        cpu: 1000m
        ephemeral-storage: 1Gi
        memory: 1.5Gi
      requests:
        cpu: 1000m
        ephemeral-storage: 1Gi
        memory: 1.5Gi
  apiUrl: https://{environmentid}.live.dynatrace.com/api
  dynatraceApiRequestThreshold: 15
  metadataEnrichment:
    enabled: true
  networkZone: <NETWORK_ZONE_NAME_HERE>
  oneAgent:
    applicationMonitoring:
      namespaceSelector: {}
      useCSIDriver: true
  skipCertCheck: false

[dannemca]

Thanks, @ian_cervantez