cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Memory dump download for Kubernetes installations

hari-m94
Participant

Hi , 

 

We run all our workloads in AKS Clusters and we have Dynatrace integrated via the Kubernetes Classic full-stack injection or Cloud-native full-stack injection. With my current setup, I am able to trigger memory dump but I am not able to download the triggered dump directly from Dynatrace. 

 

Upon some reading, I understand that the active gates have to be configured to enable such direct download, but I am having a hard time figuring out how I can accomplish that configuration in the Dynakube-based Kubernetes integration. I assume some configurations needed to be updated in the Dynakube to reflect the configurations outlines in https://www.dynatrace.com/support/help/how-to-use-dynatrace/diagnostics/memory-dump-analysis/configu...

 

Need some help to get this set on my environment, thanks in advance.

6 REPLIES 6

The_AM
Dynatrace Champion
Dynatrace Champion

Hi @hari-m94 ,

 

Please note that as of now, I do not believe memory dumps are tested by Dynatrace in Kubernetes environments. Therefore, not officially supported.

 

However, I believe you could add custom properties to the activegate section of the Dynakube CR, similar to:

 

customProperties:
    value: |-
[collector] DumpSupported = true
[dump] dumpDir = dump
        downloadUrl = FQDN_OF_YOUR_INGRESS:PORT

 

You would also need an accessible downloadUrl to retrieve the memory dump files. I believe this requires an ingress (such as a gateway with Istio) to be able to retrieve the dump from outside of the cluster:

See further Kubernetes Ingress documentation - https://kubernetes.io/docs/concepts/services-networking/ingress/

 

I've heard of reported issues, such as the following:

  • dump path not matching (eg: volume storage is/not used, so the relative path doesn't match what ActiveGate is expecting)
  • Memory dumps are a "stop-the-world" event, so they could cause liveness probes to fail, therefore K8s could kill and restart your container. This would interrupt the memory dump process.
  • Memory dumps could cause additional memory load on your container, hitting limits. Again, K8s could kill and restart your container. This would interrupt the memory dump process.
Regards,
Andrew M.

How to create the ingress to retrieve the dump from Activegate? Are there any Activegate APIs that can be used for this purpose?

The_AM
Dynatrace Champion
Dynatrace Champion

The custom properties section also allows you to set a custom DNS load-balancer address for the ActiveGate like so:

    customProperties:
      value: |
        [connectivity]
        dnsEntryPoint=https://dynakube-custom-activegate.dynatrace.svc.cluster.local:443/communication 


I haven't tried to modify that to the ingress address, it might work. It's in the realm of untested & unsupported & undocumented.

It's also going to be creating a scenario where OA will use the ingress address for connectivity. That may be undesirable.

Regards,
Andrew M.

I have tried this configuration and unfortunately, failed in getting it work.

paum
Participant

I was advised by Dynatrace to use an external ActiveGate(external to K8s). That works as intended, however I just find this whole setup a little weird due to quite a few of these interdependencies.

np9
Participant

Hello @paum , 

Can you please share how you setup this for Kubernetes environment? We have similar requirement of generating a heap dump for a process running on the container of kubernetes cluster. 

 

Featured Posts