Two AKS environment and switching between them

[Peter_Lepsis]

Dear all,

I have a customer with two AKS environment(PROD and PROD2). From time to time they are switching these in a way that they delete one and deploy the other included the dynatrace operator in classic full stack mode- The dynatrace cluster remains the same(they are using managed DT running in Azure) . Once they are done with the switch they experienced a strange thing. Sometimes it takes days sometime only hours but there is always a lack between the operator is up and running and the AKS services are starting to be monitored and showing in the services. Is it normal behaviour that it takes different amount of time while the services are shown?

Thanks a lot!

Kind regards

Peter

[Julius_Loman]

No, definitely the AKS cluster should be operational immediately when OneAgents connect. There can be number of issues which may cause the behaviour you are encountering. Most likely, in my experience, it's a lack of network connectivity. 
You metion your customer users Managed cluster. One of the caveats (not sure if this still applies with classicFullStack, I try to avoid it as it's a deprecated method) is that in recent Dynatrace Operator versions, the operator creates a custom certificate for the ActiveGate and add it to OneAgents. If OneAgents start before the ActiveGate, they will not know the AG address (as it's embedded AG). Since Operator adds CA certificate for the OneAgent trust store, OneAgents verify the certificate on connection. Since the trust store includes only a certificate for the embedded activegate, your OneAgents will not connect unless they are started after embedded AG comes online or they are restarted after the embedded AG comes online - even if they have connectivity to other environment ActiveGates in the environment.

It does not happen only if you have other environments ActiveGates with your issued custom certificates and you provide the CA cert in the trusted CA map for the Operator. 
Also, this connectivity issue cannot be seen in the console log of the OneAgent pods 

When such issues happen, investigate OneAgent logs, including the ruxitagent_os log for any connectivity errors.

[Peter_Lepsis]

Hello Julius,

yes, these were my ideas at first also, but we also tried to delete the oneagent pods after the activegate has started. I also requested the ruxitagent logs from oneagent pod but the folder is not existing in the pod:
/mnt/volume_storage_mount/var_log/os
So it's strange. Also it seems that the deep monitoring is working however in the describe of the monitored pod there is no init container.

Thanks

Kind regards

Peter

[Julius_Loman]

Hello Peter,

You use classicFullStack, which works differently. First of all, there is no init container for injecting OneAgent code module. Secondly, the OS agent logs are written to the node file system directly in /var/log/dynatrace/oneagent . 

Is there any specific reason for using classicFullStack instead of cloudNativeFullStack? 

[Peter_Lepsis]

Hello Julius,

to be honest I have no idea why are they using classic fullstack. I suggested to them to change it to cloud native I'm waiting for the answers. Thanks a lot about the information of the init container and the log location I asked them also to send the log files .

Thanks

Kind regards

Peter