Containerized AG change memory limits

[Michael_M]

Is it possible to change the memory limits of the AG in a containerized environment like it is descriped here for a non-containerized AG: https://docs.dynatrace.com/docs/shortlink/sgw-configure#memory-limits

The reason for my question is simpl: In the container the AG is alone and it still only uses 55% of max memory.
From my point of view, this is not really a efficient use of the memory in the Pod.

I deploy the AG with the help of the Dynatrace Operator in our OpenShift environment.

[DanielS]

Yes, you can do it in the section activegate (Resource settings for ActiveGate container) of your Dynakube YAML :

apiVersion: dynatrace.com/v1beta4
kind: DynaKube
metadata:
  name: dynakube
  namespace: dynatrace
spec:
  # Link to api reference for further information: https://docs.dynatrace.com/docs/ingest-from/setup-on-k8s/reference/dynakube-parameters

  # Required: Dynatrace apiUrl including the `/api` path at the end.
  apiUrl: https://ENVIRONMENTID.live.dynatrace.com/api

  # Optional: Name of the secret holding the tokens used for connecting to Dynatrace.
  #
  # tokens: ""

  # Optional: Defines a custom pull secret in case you use a private registry when pulling images from the Dynatrace environment
  #
  # customPullSecret: "custom-pull-secret"

  # Optional: Disable certificate check for the connection between Dynatrace Operator and the Dynatrace Cluster.
  #
  # skipCertCheck: false

  # Optional: Set custom proxy settings either directly or from a secret with the field 'proxy'
  #
  # proxy:
  #   value: my-proxy-url.com
  #   valueFrom: name-of-my-proxy-secret

  # Optional: Add custom RootCAs from a configmap.
  #
  # trustedCAs: name-of-my-ca-configmap

  # Optional: Set a network zone for the OneAgent and ActiveGate pods.
  #
  # networkZone: name-of-my-network-zone

  # Optional: Configure istio to allow access to the Dynatrace Cluster from the OneAgent or ActiveGate pods.
  #
  # enableIstio: true

  # Optional: Minimum minutes between Dynatrace API requests.
  #
  # dynatraceApiRequestThreshold: 15

  # Configuration for Metadata Enrichment.
  #
  # metadataEnrichment:
  #   enabled: true

    # Optional: The namespaces where you want Dynatrace Operator to inject
    #
    # namespaceSelector:
    #   matchLabels:
    #     app: my-app
    #   matchExpressions:
    #   - key: app
    #     operator: In
    #     values: [my-frontend, my-backend, my-database]

  # Configuration for Log monitoring.
  #
  # logMonitoring: {}
  
    # Optional: Specifies the rules and conditions for matching ingest attributes.
    #
    # ingestRuleMatchers:
    #   - attribute: "k8s.namespace.name"
    #     values:
    #       - "kube-system"
    #       - "dynatrace"
    #       - "default"
    #   - attribute: "k8s.pod.annotation",
    #     values:
    #       - "logs.dynatrace.com/ingest=true"
    #       - "category=security"

  # Configuration for OneAgent
  #
  oneAgent:
    # Optional: Set a host group for OneAgent
    #
    # hostGroup: ""

    classicFullStack:
      # Optional: Specify the node selector that controls on which nodes OneAgent will be deployed.
      #
      # nodeSelector: {}

      # Optional: Assign a priority class to the OneAgent pods.
      #
      # priorityClassName: priority-class

      # Optional: Tolerations to include with the OneAgent DaemonSet.
      #
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists

      # Optional: Resource settings for OneAgent container.
      #
      # oneAgentResources:
      #   requests:
      #     cpu: 100m
      #     memory: 512Mi
      #   limits:
      #     cpu: 300m
      #     memory: 1.5Gi

      # Optional: Set additional arguments to the OneAgent installer.
      #
      # args: []

      # Optional: Set additional environment variables for the OneAgent pods.
      #
      # env: []

      # Optional: Disable automatic restarts of OneAgent pods in case a new version is available
      #
      # autoUpdate: true

      # Optional: Set the DNS Policy for OneAgent pods.
      #
      # dnsPolicy: "ClusterFirstWithHostNet"

      # Optional: Add custom OneAgent annotations.
      #
      # annotations:
      #   custom: annotation

      # Optional: Add custom labels to OneAgent pods
      #
      # labels:
      #   custom: label

      # Optional: Use a custom OneAgent image.
      #
      # image: ""

      # Optional: The OneAgent version to be used.
      # Example: <major>.<minor>.<release>.<timestamp>, e.g. 1.200.0.20240101-000000
      #
      # version: ""

      # Optional: The SecComp Profile that will be configured in order to run in secure computing mode.
      #
      # secCompProfile: ""

  # Configuration for ActiveGate instances.
  #
  activeGate:
    # Defines the ActiveGate capabilities
    #
    capabilities:
      - routing
      - kubernetes-monitoring
      - dynatrace-api

    # Optional: Amount of replicas of ActiveGate pods.
    #
    # replicas: 1

    # Optional: Use a custom ActiveGate image
    #
    # image: ""

    # Optional: Set the ActiveGate group
    #
    # group: ""

    # Optional: Add a custom properties file by providing it as a value or by referencing it from a secret.
    #
    # customProperties:
    #   value: |
    #     [connectivity]
    #     networkZone=
    #   valueFrom: myCustomPropertiesSecret

    # Optional: Resource settings for ActiveGate container.
    #
    resources:
      requests:
        cpu: 500m
        memory: 1.5Gi
      limits:
        cpu: 1000m
        memory: 1.5Gi

    # Optional: Specify the node selector that controls on which nodes ActiveGate will be deployed.
    #
    # nodeSelector: {}

    # Optional: Set tolerations for the ActiveGate pods.
    #
    # tolerations:
    # - effect: NoSchedule
    #   key: node-role.kubernetes.io/master
    #   operator: Exists

    # Optional: Add custom labels to ActiveGate pods
    #
    # labels:
    #   custom: label

    # Optional: Add custom environment variables to ActiveGate pods
    #
    # env: []

    # Optional: Name of a secret containing ActiveGate TLS certificate, key, and password.
    #
    # tlsSecretName: "my-tls-secret"

    # Optional: Set the DNS policy for ActiveGate pods.
    #
    # dnsPolicy: "Default"

    # Optional: Assign a priority class to the ActiveGate pods.
    #
    # priorityClassName: priority-class

    # Optional: Add custom annotations to ActiveGate pods
    #
    # annotations:
    #   custom: annotation

    # Optional: Add TopologySpreadConstraints to the ActiveGate pods
    #
    # topologySpreadConstraints: []

[Michael_M]

Hi Daniel,

thanks for your suggestion but this is not exactly what I meant.

I was talking about how much memory of the assigned memory of the Pod the AG will use.
For example: The pod has a limit and request of 3Gi but the AG will only use 50% of it.
Log output of my AG
Memory [maxMemory=1536M, initHeap=768M, maxHeap=1536M, usedMeta=7M, committedMeta=7M, totalPhysicalMemory=3072M, freePhysicalMemory=2988M]
50% (-XX:MaxRAMPercentage=50) is the default for the AG and can be changed with the -java.xmx.relative_part switch in the launcheruserconfig.conf.
I understand that this is necessary for a non-containerized AG but in the container world I can spin up a container for each use case I have.
The current use case is to monitor the OpenShift cluster with the kubernetes_monitoring feature and my AG needs some memory for this. But unfortuneally because of this 50% default I have to define a relative high limit only to have use 50 % of the memory.

That why I would like to change this but I wasn't able to finde a way with the operator. Not one I have found in the docs.

Regards

Michael

[Romanenkov_Al3x · Accepted solution]

Hello @Michael_M 

This thread for me is very interesting, bcs Dynatrace  + kubernetes  ❤️ I’ll do everything I can, whatever it takes. 

There is two ways to do this:

First - If you prefer Kubernetes Hard Way (Joking 😃) Second - GitOps is everything.  (So true 😋)

Results: Memory [maxMemory=5521M, initHeap=3072M, maxHeap=5521M, usedMeta=14M, committedMeta=15M, totalPhysicalMemory=6144M, freePhysicalMemory=6031M] from your values  [maxMemory=1536M ...

results

1)  Only If you’re into challenges and spending fun evenings alone with the terminal.

If you have chance to update the StatefulSet and create several ConfigMaps, it’s not a big deal to get what you want.

Ideal if you don't want change dynakube for some reasons.  

You can create configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: activegate-memory-config
  namespace: dynatrace
data:
  DT_INIT_HEAP_PERCENT: '50'
  DT_MAX_HEAP_PERCENT: '90'

Modify statefulSet + boost req/lim

...
      containers:
        - name: activegate
...
          env:
...            
           - name: DT_INIT_HEAP_PERCENT
              valueFrom:
                configMapKeyRef:
                  name: activegate-memory-config
                  key: DT_INIT_HEAP_PERCENT
            - name: DT_MAX_HEAP_PERCENT
              valueFrom:
                configMapKeyRef:
                  name: activegate-memory-config
                  key: DT_MAX_HEAP_PERCENT
...
            limits:
              cpu: '1'
              memory: 6Gi
            requests:
              cpu: 500m
              memory: 6Gi

2) As mentioned @DanielS you can update dynakube.

You need to provide same env variables to container with name "activeagate"

You can define them here:

    # Optional: Add custom environment variables to ActiveGate pods
    #
    # env: []

You need all the same environment variables from the first point.

DT_INIT_HEAP_PERCENT: 50

DT_MAX_HEAP_PERCENT: 90

Boost request/limits for example to 6Gi from 1.5Gi

Then the operator will work for you.

Mischief managed.

--

After this, you’ll probably want to tune something else — take a look at custom properties:

    # customProperties:
    #   value: |
    #     [connectivity]
    #     networkZone=

cat /var/lib/dynatrace/gateway/config/config.properties

Have fun! 

Regards,

Aleksandr Romanenkov

[Michael_M]

Hi Aleksandr,

that was exactly what I was looking for.

I set the environment variables in my dynakube an the activegate use the memory how I want it to use.

One question: I couldn't find this env variables in the documentation. Where did you get them? 

Regards

Michael

[Romanenkov_Al3x]

Hey Michael, great to hear it worked out.

Honestly, it’s not that hard — you can get that info too. This is not a secret in Vault (haha😄).

All you have to do is know the container’s entrypoint, check out the script (the sh that launches the gateway), and the parameters are right there.

Regards,

Aleksandr Romanenkov

[Michael_M]

THX for the info, I will do that 😉