NUMA-Clustered-Mode

Information:

Environment

AppMon: all versions

Symptoms

The AppMon environment is reporting the incorrect number of cores for a given known quantity of physical processors.

Solution

In the system BIOS, the NUMA mode needs to be changed from “Clustered” to
“Flat.” This will allow all processes access to all physical processor sockets and will alleviate issues with incorrect reporting of CPU cores.

Root Cause

N.U.M.A. (Non-Uniform Memory Access) is a system by which memory is accessed in a Symmetric Multi-Processing environment.

  • • In SMP all CPU (cores) have equal access to the system memory.
  • • In NUMA (Clustered), groups of processors (cores) are separated into nodes. This makes for faster access, but limits total memory available to each node.
  • • In NUMA (Flat), all CPU (cores) have equal access to the system memory.


The trend in hardware has been towards more than one system bus, each serving a small set of processors. Each group of processors has its own memory and possibly its own I/O channels. However, each CPU can access memory associated with the other groups in a coherent way. Each group is called a NUMA node. The number of CPUs within a NUMA node depends on the hardware vendor. It is faster to access local memory than the memory associated with other NUMA nodes. This is the reason for the name, non-uniform memory access architecture.
On NUMA hardware, some regions of memory are on physically different buses from other regions. Because NUMA uses local and foreign memory, it will take longer to access some regions of memory than others. Local memory and foreign memory are typically used in reference to a currently running thread. Local memory is the memory that is on the same node as the CPU currently running the thread. Any memory that does not belong to the node on which the thread is currently running is foreign. Foreign memory is also known as remote memory. The ratio of the cost to access foreign memory over that for local memory is called the NUMA ratio. If the NUMA ratio is 1, it is symmetric multiprocessing (SMP).

 

An example of the IBM Numa-Q system can be seen in the following image: