Game Lag White Paper › L10 Memory

Cache miss CPU cache misses

Cause ID mem-cache-miss · Primary owner Game team (Server development)

Open the interactive card with figures and simulations →

When data is scattered all over memory, the CPU has to go all the way out to slow RAM and wait every time.

Why Objects scattered behind pointers and accessed in no particular order → Effect Data isn’t in the CPU cache, so every read goes to RAM (roughly 100 times slower) → On screen The same work costs several times more tick time; in bad cases, slow motion

Symptoms

Slow motion

Factors

Stall

Who’s affected

Whole server

When

Always, When crowds gather

Owner

Primary owner Game team (Server development)

Game team action items

Lay out data that is used together contiguously (data-oriented design).

On the graph

Always high · Tick time, CPU utilization

Where to look

Attach perf stat -d -p PID to the game server process to measure instructions per cycle (insn per cycle) and L1/LLC cache misses, and view them alongside tick time and CPU utilization

Confirmed if

CPU stays busy, but insn per cycle is low and LLC misses are high. Confirmed if a build with a changed data layout cuts tick time sharply at the same player count

Ruled out if

Low CPU utilization but slow ticks: a cause that waits outside the CPU, such as locks or I/O waits

Check with

Infra tools (no game code needed)

Sources

1. Designs, Lessons and Advice from Building Large Distributed Systems (LADIS 2009 keynote) Google
   L1 cache 0.5 ns, L2 cache 7 ns, main memory 100 ns (as of 2009): going out to RAM is one to two orders of magnitude slower than cache
2. perf-stat(1) — Linux manual page perf
   -p counts hardware events for a running process and shows insn per cycle; -d adds L1 and LLC data cache events

See also

Same layer: L10 Memory

• Server GC stop-the-world pause
• Script engine GC pause
• Allocation surge
• Memory leak
• GC thrashing (too little heap headroom)
• Swap
• Memory fragmentation
• Remote NUMA memory

Same symptom (Slow motion), other layers

• Fixed-timestep catch-up spiral L1 Client game process
• Load balancer skew and misjudged health checks L5 Data center network equipment
• Too many threads and context switching L7 Server OS (kernel)
• Container CPU throttling (CFS quota) L7 Server OS (kernel)
• Latency spikes from server power management (C-states, frequency scaling) L7 Server OS (kernel)
• Scheduled jobs L7 Server OS (kernel)
• Performance changes after OS, kernel, driver, or firmware updates L7 Server OS (kernel)
• Blocking sends caused by slow clients L8 Sockets and protocols

View the interactive card with figures and simulations