oneAgent pod in crashLoopBackOff

[skatshell]

I upgraded the version of oneagent to latest from 1.277.196-raw. Since then one agent is crashing with these lines of log. What can I do to resolve the issue?

│ 15:58:21 Command line: --set-host-id-source=k8s-node-name --set-host-property=OperatorVersion=v1.2.2 --set-network-zone=aws.ap-southeast-1.sky --set-server={https://172.20.142.222:443/communication;https://prod-apac-eks-activegate.dynatrace:443/communicatio │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Installed version: │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Using pre-packed installer, no download from external location will be performed. │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_INSTALLER_SCRIPT_URL= │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_INSTALLER_DOWNLOAD_TOKEN= │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_INSTALLER_DOWNLOAD_VERBOSE= │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_INSTALLER_SKIP_CERT_CHECK= │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_ENABLE_VOLUME_STORAGE= │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_CONTAINER_STORAGE_PATH= │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_NO_REMOUNT_ROOT= │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_ADDITIONAL_UNMOUNT_PATTERN= │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_DISABLE_CONTAINER_INJECTION=true │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 ONEAGENT_READ_ONLY_MODE=true │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 AGENT_CONTAINER_IMAGE_VERSION=1.277.159.20231019-200414 │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Path: /usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Started with capabilities: self: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_sys_chroot,cap_sys_ptrace,cap_sys_admin,cap_setfcap=eip │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Started with user: uid=1000 gid=1000 groups=1000 │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Using volume-based storage │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Volume mount detected under /mnt/volume_storage_mount │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Warning: Command 'mkdir -p /mnt/host_root' failed, return code: 1, message: mkdir: cannot create directory '/mnt/host_root': Read-only file system │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Warning: Cannot create /mnt/host_root directory. │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ 15:58:21 Error: Initialization procedure failed │
│ /tmp/bootstrap_common.sh: line 42: /tmp/container_init.log: Read-only file system │
│ Stream closed EOF for dynatrace/prod-apac-eks-oneagent-qc7pl (dynatrace-oneagent)

[MostafaHussein]

Hi @skatshell ,

I think this post is related to your issue

https://community.dynatrace.com/t5/Troubleshooting/CrashLoopBackOff-Downgrading-OneAgent-is-not-supp...

hope this help.

BR,
Mostafa Hussein.

[gopher]

@skatshell  which OS are you using on EKS?  if I had to guess 'bottlerocket'? 
Also are you deploying as full stack or App Only? 
Implementation on a read only system will vary, however you would typically set the 'readonly' feature flag to true where a read only environment is present to allow for the install (below)

apiVersion: dynatrace.com/v1beta2
kind: DynaKube
metadata:
  name: dynakube
  namespace: dynatrace
  annotations:
    feature.dynatrace.com/automatic-kubernetes-api-monitoring-cluster-name: "${TF_VAR_cluster_name}"  
    feature.dynatrace.com/no-proxy: "dynakube-activegate.dynatrace,.amazonaws.com" #private repo
    feature.dynatrace.com/injection-readonly-volume: "true"
    feature.dynatrace.com/oneagent-initial-connect-retry-ms: "10000"
    feature.dynatrace.com/init-container-seccomp-profile: "true"