OpenShift Dynatrace operator 1.5.1: app-mon pod fail to start for runAsUser 1001 not allowed

[gilles_tabary]

Hi all.

We are on Managed.

We are testing app-monitoring in OpenShift with Dynatrace OpenShift operator 1.5.1 : https://docs.dynatrace.com/docs/whats-new/dynatrace-operator/dto-fix-1-5-1 (the requirements are satisfied: Cluster v1.312.51 + AG v1.311 + OA  v1.311.72).

When a deployment try to start replicat we get this error : runAsUser 1001 not allowed, must be in the range [1001060000, 1001069999].

30s Warning FailedCreate replicaset/x0m-56854c4ffd Error creating: pods "x0m-56854c4ffd-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "trident-controller": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .initContainers[1].runAsUser: Invalid value: 1001: must be in the ranges: [1001060000, 1001069999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "logging-scc": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "trident-node-linux": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

AFAIU Dynatrace Operator should not try to assign a User ID for it's OneAgent agent / code module, because OpenShift handles automatically the assignment. OpenShift doc @ security-context-constraints-example.

Any feed back from someone on that ?

Regards.

[yanezza]

Hi,
this should be investigated in a support ticket. Attach also support archive and information regarding the Openshift version.

[gilles_tabary]

Could it be that setting a runAsUser value at https://github.com/Dynatrace/dynatrace-operator/blob/main/pkg/webhook/mutation/pod/v2/init.go#L75 is interfering with default OpenShift automatic uid assigment ? Doc 15.2. About pre-allocated security context constraints values.

(Yes we have a ticket open :-))

[shahinm]

Hi there,

This is a known issue, and a fix is in progress.

While we are working on a fix, apply one of the following changes to work around the problem:

• add spec.securityContext.RunAsUser(Group) on the POD level valid ID value can be found in the app namespace’s annotations

kind: Namespace
metadata:
  annotations:
    openshift.io/sa.scc.supplemental-groups: 1000660000/10000
    openshift.io/sa.scc.uid-range: 1000660000/10000

• add nonroot-v2 SCC to the ServiceAccount

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <name>
  namespace: <app namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: systemscc:nonroot-v2
subjects:
- kind: ServiceAccount
  name: <app ServiceAccount>
  namespace: <app namespace>

 Regards

[BertEvo]

Hi there,

Any update on this issue? Will it be fixed in the next release and if so ..any idea when it will be released?

From what I read in the work around, both seem actions that need to be performed per namespace?

So a customer with +500 namespaces need to implement this +500 times and any future deployment

[shahinm]

The Operator release 1.6.0 includes the fix. You can already try a release candidate in any affected non-prod environment.

https://github.com/Dynatrace/dynatrace-operator/tree/v1.6.0-rc.2

[gilles_tabary]

AFAIU this is (for now) available only throuh helm (requiering more hacky work for integration), not through OpenShift operator hub and standard operator integration.

[MichaelL]

I have used the YAML files of https://github.com/Dynatrace/dynatrace-operator/releases/tag/v1.6.0-rc.2 for OpenShift (4.13):

• openshift.yaml
• openshift-csi.yaml
• dynatrace-operator-crd.yaml

In the first two files I replaced the redhat image link with the one of Docker "dynatrace/dynatrace-operator:v1.6.0-rc.2". Since redhat one does not exist.

I had to add the CRD via OpenShift gui or I got an error.

I then Added the DynaKube config for beta5 and everything is running. I can see my cluster in Dynatrace.

However, we still have the issue our GitLab runners and other deployments are getting:

ERROR: Error cleaning up pod: resource name may not be empty
ERROR: Job failed (system failure): prepare environment: setting up build pod: pods "runner-t1qz363-project-22774-concurrent-0-bzq4fg5s" is forbidden: unable to validate against any security context constraint: [spec.volumes[5]: Invalid value: "csi": csi volumes are not allowed to be used, provider restricted-v2: .initContainers[1].runAsUser: Invalid value: 1001: must be in the ranges: [1006100000, 1006109999], provider restricted: .initContainers[1].runAsUser: Invalid value: 1001: must be in the ranges: [1006100000, 1006109999], provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "rsync-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "elasticsearch-scc": Forbidden: not usable by user or serviceaccount, provider "logging-scc": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider restricted-csi: .initContainers[1].runAsUser: Invalid value: 1001: must be in the ranges: [1006100000, 1006109999], provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "velero-privileged": Forbidden: not usable by user or serviceaccount]. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

The GitLab runner pods were restarted.

As soon as we uninstall DT, it works again.

Am I doing something wrong?