On a server with two CPUs, using memory attached to the other CPU slows access down.
Why Threads and their memory sit on different CPU sockets → Effect Memory access slows down (1.5–2× depending on hardware) → On screen Same specs, but performance differs from process to process
Pin processes and their memory to one socket (numactl), on a two-socket machine run separate game server processes per socket.
On the graph
Outliers only · Tick time per process, memory per node
Where to look
Use numastat -p PID to see which NUMA node holds the game server process’s memory, check whether numa_miss and other_node in numastat are rising, and compare with the node of the CPU the process runs on
Confirmed if
Only the slow processes have most of their memory on a different node from the CPU they run on, and the gap disappears after restarting them with CPU and memory pinned to one node via numactl
Ruled out if
Slow even with the same node placement as the fast processes: another cause such as a noisy neighbor, CPU throttling, or that process’s load
Check with
Infra tools (no game code needed)
Sources
What is NUMA?Linux kernel Memory in the same cell is faster and has higher bandwidth; memory in another (remote) cell is slower to access
numastat(8) — Linux manual pagenumactl numa_miss (allocated on a node other than the intended one) and other_node (allocated on this node by a process running on another node) counters; -p shows a process’s memory per node