Memory dump download for Kubernetes installations

[hari-m94]

Hi , 

We run all our workloads in AKS Clusters and we have Dynatrace integrated via the Kubernetes Classic full-stack injection or Cloud-native full-stack injection. With my current setup, I am able to trigger memory dump but I am not able to download the triggered dump directly from Dynatrace. 

Upon some reading, I understand that the active gates have to be configured to enable such direct download, but I am having a hard time figuring out how I can accomplish that configuration in the Dynakube-based Kubernetes integration. I assume some configurations needed to be updated in the Dynakube to reflect the configurations outlines in https://www.dynatrace.com/support/help/how-to-use-dynatrace/diagnostics/memory-dump-analysis/configu...

Need some help to get this set on my environment, thanks in advance.

[The_AM · Accepted solution]

Hi @hari-m94 ,

Please note that as of now, I do not believe memory dumps are tested by Dynatrace in Kubernetes environments. Therefore, not officially supported.

However, I believe you could add custom properties to the activegate section of the Dynakube CR, similar to:

customProperties:
    value: |-
        [collector]
        DumpSupported = true
        [dump]
        dumpDir = dump
        downloadUrl = FQDN_OF_YOUR_INGRESS:PORT

You would also need an accessible downloadUrl to retrieve the memory dump files. I believe this requires an ingress (such as a gateway with Istio) to be able to retrieve the dump from outside of the cluster:

See further Kubernetes Ingress documentation - https://kubernetes.io/docs/concepts/services-networking/ingress/

I've heard of reported issues, such as the following:

• dump path not matching (eg: volume storage is/not used, so the relative path doesn't match what ActiveGate is expecting)
• Memory dumps are a "stop-the-world" event, so they could cause liveness probes to fail, therefore K8s could kill and restart your container. This would interrupt the memory dump process.
• Memory dumps could cause additional memory load on your container, hitting limits. Again, K8s could kill and restart your container. This would interrupt the memory dump process.

[edward_ma]

How to create the ingress to retrieve the dump from Activegate? Are there any Activegate APIs that can be used for this purpose?

[The_AM]

The custom properties section also allows you to set a custom DNS load-balancer address for the ActiveGate like so:

    customProperties:
      value: |
        [connectivity]
        dnsEntryPoint=https://dynakube-custom-activegate.dynatrace.svc.cluster.local:443/communication 

I haven't tried to modify that to the ingress address, it might work. It's in the realm of untested & unsupported & undocumented.

It's also going to be creating a scenario where OA will use the ingress address for connectivity. That may be undesirable.

[edward_ma]

I have tried this configuration and unfortunately, failed in getting it work.

[paum]

I was advised by Dynatrace to use an external ActiveGate(external to K8s). That works as intended, however I just find this whole setup a little weird due to quite a few of these interdependencies.

[np9]

Hello @paum , 

Can you please share how you setup this for Kubernetes environment? We have similar requirement of generating a heap dump for a process running on the container of kubernetes cluster. 

[anthony_sciarin]

Hey I hope you don't mind chiming in here,

I have reached out to our product team and confirmed that this is not supported but can be suggested as a future enhancement via a product idea. I have asked that one of my clients who have also inquired about having memory dumps for container ActiveGates file and product idea. As soon as it is posted I highly recommend voting for this feature to warrant testing and investigation from the Dynatrace product management team.

Best regards,
-Anthony (Dynatrace CSE)