# \[Kubernetes\] Node memory usage is greater than memory limits

**URL:** <https://community.dynatrace.com/t/kubernetes-node-memory-usage-is-greater-than-memory-limits/45010>\
**Category:** Troubleshooting\
**Tags:** kubernetes\
**Created:** [April 25, 2024, 7:59am UTC](https://community.dynatrace.com/t/kubernetes-node-memory-usage-is-greater-than-memory-limits/45010 "2024-04-25T07:59:20Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alisopp](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Alisopp](https://community.dynatrace.com/u/Alisopp)\
**Post date:** [April 25, 2024, 7:59am UTC](https://community.dynatrace.com/t/kubernetes-node-memory-usage-is-greater-than-memory-limits/45010/1 "2024-04-25T07:59:20Z")

</div>

# Symptom/Question

When looking at the memory chart of my node,&nbsp;I noticed that the memory usage is bigger than the memory limits.&nbsp;How can this happen?

| Kubernetes app | Kubernetes classic |
| --- | --- |
| ![kubernetes_app_node_screen.png](https://global.discourse-cdn.com/dynatrace/original/2X/b/b1c183e933a0d4c256f2853bbbf95262a7ca97fa.png) | ![kubernetes_classic_node_screen.png](https://global.discourse-cdn.com/dynatrace/original/3X/8/2/82bc94a043f019073475cebf1133c51cd1948696.png) |

# Answer

The reason for this is that every container has a usage,&nbsp;but not every container has to have a limit set.&nbsp;On node screens,&nbsp;memory limits/request/usage are calculated by summing up the respective value from the application containers of a pod.&nbsp;Furthermore,&nbsp;it is not required to set any requests/limit for a container.&nbsp;Therefore,&nbsp;when&nbsp;summing up usage and limits on node scope,&nbsp;it can happen that you get memory usage for all application containers,&nbsp;but limits are only set for a few application containers.

Let us inspect this scenario with an example:

Let us assume,&nbsp;Pod1,&nbsp;Pod2 and Pod3 are running on our node.&nbsp;Each pod has one application container and the&nbsp;following table shows the memory limits and memory usage defined for each container.

| **Pod** | **Memory limits** | **Memory usage** |
| --- | --- | --- |
| Pod1 | 1GiB | 500MiB |
| Pod2 | - | 1GiB |
| Pod3 | 500MiB | 250MiB |

Now if we sum up this value to the node scope,&nbsp;we end up with the following total values for memory limits and usage:

Memory limits total&nbsp;=&nbsp;1 GiB&nbsp;+&nbsp;0&nbsp;+&nbsp;500 MiB&nbsp;=&nbsp;1.5 GiB

Memory usage total&nbsp;=&nbsp;500 MiB&nbsp;+&nbsp;1 GiB&nbsp;+&nbsp;250 MiB&nbsp;=&nbsp;1,75 GiB

So we can easily see how this scenario can happen.&nbsp;As due to that for the container of Pod2,&nbsp;no limit was defined,&nbsp;the sum of&nbsp;memory usage can be bigger than the limits.

Now,&nbsp;to verify this behavior on a real node,&nbsp;this can be easily achieved by using a DQL query&nbsp;(Note:&nbsp;in order to successfully&nbsp;execute this query,&nbsp;the&nbsp;[new Kubernetes experience](https://community.dynatrace.com/topic/234188)&nbsp;is required).

```java
fetch dt.entity.container_group_instance 
| fields container.name = entity.name, container.id = id, cluster.id = belongs_to[dt.entity.kubernetes_cluster], node.name = nodeName
| filter cluster.id in [fetch dt.entity.kubernetes_cluster | filter entity.name == "{CLUSTER_NAME}" | fields id] 
and matchesPhrase(node.name, "{NODE_NAME}")
| lookup [timeseries {
limits_memory=sum(dt.kubernetes.container.limits_memory)}, by:{k8s.container.name, dt.entity.container_group_instance}
| fields k8s.container.name, limits_memory=arrayLast(limits_memory), dt.entity.container_group_instance]
, sourceField:container.id, lookupField: dt.entity.container_group_instance, fields:{limits_memory}
| lookup [timeseries {
memory_usage=sum(dt.kubernetes.container.memory_working_set)}, by:{k8s.container.name, dt.entity.container_group_instance}
| fields k8s.container.name, memory_usage = arrayLast(memory_usage), dt.entity.container_group_instance]
, sourceField:container.id, lookupField: dt.entity.container_group_instance, fields:{memory_usage}
| fields container.name, node.name, limits_memory, memory_usage
| sort limits_memory

```

Replace **{CLUSTER\_NAME}** and **{NODE\_NAME}** with the name of your Kubernetes cluster and the node for which you want to execute this query.

 ![note_book_container_list.png](https://global.discourse-cdn.com/dynatrace/original/3X/4/c/4c2e5f9af0bfa07d4b83398de68b8c013c8784c2.png)

Here we see that for our node&nbsp;“ip-10-0-130-92.ec2.internal”,&nbsp;only four containers have even set memory limits.

By adding summarize to the end of the query,&nbsp;we can also see the values which are displayed in the node screen.

```java
fetch dt.entiy.container_group_instance
....
| sort limits_memory
| summarize by:{}, {limits_memory=sum(limits_memory), memory_usage=sum(memory_usage)}

```

 ![note_book_memory_summary.png](https://global.discourse-cdn.com/dynatrace/original/3X/d/d/dd001c66267cdb6b17b2ce8f61f9b56b25584a7e.png)