OneAgent in Minikube failing to deploy

[TomLichti]

I am trying to work through the DTU Kubernetes lab, and I am stuck on what feels like the last step. The OneAgent pod will not start, the log is ending with the following:

00:49:55 Started with user: uid=0(root) gid=0(root) groups=0(root)
00:49:55 Error: Container was not launched in host's cgroup namespace
00:49:55 Error: Dynatrace OneAgent requires parameters: --pid, --net and --cgroupns set to 'host'.
00:49:55 If you are not sure how to launch the container please visit: https://www.dynatrace.com/support/help/shortlink/oneagent-docker
00:49:55 Error: Initialization procedure failed

I have googled every error message, combed through yaml files, and I have not been able to figure out if this is the real issue, or something else? I've tried adding the 'useUnprivilegedMode: true' setting in the dynakube.yaml, but it didn't make a difference. I'm fairly new to deploying Dynatrace in K8S, not new to Dynatrace though, this one has me stumped!

Thanks for any help.

Tom

[andre_f]

Hi @TomLichti,

Can you give some more information?
E.g. link to the lab or did you use a Dynatrace guide for deployment? If yes, which one? For example, have you used this guide for Kubernetes classicFullStack? Did you use manual or automatic or helm for deployment?

[TomLichti]

Hi @andre_f,

I used the automated Dynatrace Operator guide here: Get started with Kubernetes/OpenShift monitoring | Dynatrace Docs

The lab I am working on is this one: https://university.dynatrace.com/assets/23274/LabKubernetes.pdf

I created another full Kubernetes cluster on another completely separate server, and had the exact same problem and error. On the new cluster, before I installed K8S, I had a docker instance, oneagent and activegate installed outside docker and it worked perfectly. I removed oneagent, activegate and docker completely, and started with K8S from scratch, and had the same problem.

As a test I installed the onegent again outside the cluster and that seems to work.

[andre_f · Accepted solution]

Hi @TomLichti,
I tried to reproduce it, but didn't have these problems. The OneAgent pod was able to start.
My setup looks like this:

1. created VM with multipass
2. minikube installed
3. docker installed
4. started Minikube
5. created yaml for the application and run apply command
6. in the Dynatrace menu in the area Manage
   Deploy Dynatrace > Start Installation > click here Kubernetes
7. enter a name, create Dynatrace operator token (see screenshot - (1) )
8. download dynakube.yaml and copy file to minikube VM or copy content to a file dynakube.yaml on the vm
9. I copied the commands in the window (see screenshot - (2) ) individually and executed them one after the other
   
10. Wait until pods Operator, ActiveGate and OneAgent(s) are running
    "kubectl get all -n dynatrace" or "minikube kubectl -- get all -n dynatrace" should show this
    

It was a new VM only minikube and docker were installed according to instructions.

Is it possible that you have also executed steps from this documentation?

[TomLichti]

I followed the exact same steps that you did. The only difference, and this may be the root cause, is that on the new server I had previously installed Docker (not Kubernetes), and standalone (not containerized) ActiveGate and OneAgent. I thought I had removed all of those components and started fresh, but it was still on the same server. Having said that, my first attempt was on a VirtualBox VM that I don't think had anything other than a K8S deployment, but I will start that from scratch with a new VM and see how it goes.

Thank you for investigating!

[techean]

Tom just to confirm and also @andre_f ,

What are the underline OS being used in your VMs? is it open source Linux or red hat linux

[TomLichti]

Both of my tests were Ubuntu, the first 20.04 LTS and the second 22.04 LTS.

[andre_f]

@TomLichti Yes, you are probably right and it is related to one of the previously installed components that some of it was not completely uninstalled. Difficult to say. Hope the attempt with a new VM will work.

@techean I used ubuntu 20.04

[techean]

Should work !😉

[TomLichti]

I just re-did the whole thing from scratch and I get the same error again:

The only difference I can see is that I used Ubuntu 22.04. I guess the last thing to try would be to use 20.04 as that is literally the only thing that is different. I would appreciate if anyone else who has used Ubuntu 22.04 and Dynatrace K8S via the Operator would chime in to see if it's just me.

[TomLichti]

I built a new VM on Ubunto server 20.04 with microk8s and it works. So it seems like there is something either with 22.04 or 22.04 Desktop that it didn't like.

Thanks everyone!

[techean]

Yes @TomLichti .

I believe we can report this to support to fix it. Not working for my Vm too on 22.04.

[andre_f]

Hi @TomLichti, @techean 

I also tried it with Ubuntu 22.04. It works. My ActiveGate doesn't run because I do not have enough CPU 😇. But the agent works. However, I am surprised that the agent shows me Ubuntu 20.04 as OS, but it was set up in VM 22.04.

I don't know what the difference is to your attempts. But as I said, the detection of the OS also makes me wonder 🤔.

[TomLichti]

That is very strange. Did you use Ubuntu Desktop or Server? I was using 22.04 Desktop and it failed every time. I used 20.04 Server and it worked. If I get time I'll try 22.04 Server.

[andre_f]

I used the Desktop version

[andre_f]

@TomLichti After restarting the VM, I have the error as well.

14:11:41 Started with capabilities: self: cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_net_admin,cap_net_raw,cap_sys_chroot,cap_sys_ptrace,cap_sys_admin,cap_sys_resource,cap_setfcap=ep
14:11:41 Started with user: uid=0(root) gid=0(root) groups=0(root)
14:11:41 Error: Container was not launched in host's cgroup namespace
14:11:41 Error: Dynatrace OneAgent requires parameters: --pid, --net and --cgroupns set to 'host'.
14:11:41 If you are not sure how to launch the container please visit: https://www.dynatrace.com/support/help/shortlink/oneagent-docker
14:11:41 Error: Initialization procedure failed

 Until now I had always started with a new VM and reinstalled everything. The error did not occur. After a reboot now it does.

[TomLichti]

Ah, my VM would have restarted a few times before I installed DT, as I installed VM Guest additions. Glad someone reproduced it, I was beginning to doubt my sanity!

[andre_f]

Got this info, but have not been able to test it myself yet.

"We support only minikube started with --vm-driver=none parameter.
It's recommended to use only docker runtime (it's controlled by --container-runtime minikube's parameter)."

[andre_vdveen]

@andre_f I followed the exact same steps, but for some reason, my AG pod doesn't get created/doesn't start up - any idea why that would be? I've tried it using EC2 instances as well as local VMs. Do I need to change anything in the dynakube.yaml file, or use it as is from the deployment screen?

[andre_f]

I used the default dynakube.yaml file and didn't customize anything. Did you try a "describe" to see what the problem might be?

kubectl describe pod/<AG-POD-NAME> -n dynatrace

[andre_vdveen]

If I run "kubectl get all -n dynatrace" the AG pod doesn't show up at all, so "describe" won't work, as I need the pod name, right?

[andre_f]

Ah okay. Sorry about that. I thought that the pod was there but would not come into the status "running".
If you download the dynakube.yaml here (see screenshot), you should not need to adjust anything.

This part is already included there (I have removed some comments here):

  # Configuration for ActiveGate instances.
  activeGate:
    # Enables listed ActiveGate capabilities
    capabilities:
      - routing
      - kubernetes-monitoring

    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 1.5Gi

[andre_vdveen]

I did exactly what you did, used the dynakube.yaml as is, no modifications...strange. I'll give it another go, thanks @andre_f 

[andre_f]

I usually use k3s instead of minikube. This works without problems. Perhaps you can give it a try.