한국어English日本語简体中文繁體中文DeutschไทยTiếng ViệtРусскийPortuguês (Brasil)EspañolBahasa Indonesia

Game Lag White Paper › L9 Server game process

Single-threaded zone overload (hotspot) Single-threaded hot zone

Cause ID sp-hotzone · Primary owner Game team (Server development) · Also Infra team (Server infrastructure)

Open the interactive card with figures and simulations →

When each area runs on a single thread and players crowd into one place, only that one core hits 100%.

Why One thread runs each area (channel) → Effect When players crowd into one place, only that core saturates while the other cores have room to spare → On screen Only that area lags; other areas are fine

Symptoms
Slow motion, Input lag
Factors
Stall
Who’s affected
Specific zone/channel
When
When crowds gather
Owner
Primary owner Game team (Server development) · Also Infra team (Server infrastructure)
Game team action items
Spread players across channels, parallelize work within an area, cap player count.
Infra team action items
Add per-core CPU utilization to monitoring and alerts (a single saturated core gets buried in the server-wide average).
Ballpark numbers
On a 16-core server, one core at 100% shows up as only about 6% total CPU utilization. You have to look at per-core utilization to find it.
On the graph
Rises with load · Per-core CPU utilization, per-thread CPU
Where to look
Per-core utilization from mpstat -P ALL 1 and per-thread CPU of the game process from pidstat -t 1, compared with the player count in the zone the busiest thread runs
Confirmed if
Total server CPU is low, but one thread (one core) sits near 100%, and at that moment players are crowded into the zone that thread runs
Ruled out if
Several cores evenly high: overall server overload. Only one core’s %soft (receive processing) high: points to NIC interrupts concentrated on one core
Check with
Infra tools (no game code needed)
Learn more
Other areas stay fine only when each area runs its own tick independently. If the area threads wait for each other every tick and move on to the next tick together, the busiest area slows the tick for the whole server.
Real incidents
CCP Games 2014: Server overload in EVE Online’s massive HED-GP fleet battle

Sources

  1. Time Dilation – How’s That Going? CCP Games
    EVE Online’s Time Dilation works per node, so even distant star systems on the same node slow down; big battles run on reinforced nodes that host only 4 star systems
  2. mpstat(1) — Linux manual page sysstat
    Shows per-processor utilization separately from the overall average (-P ALL); %soft is the share of time spent handling software interrupts
  3. pidstat(1) — Linux manual page sysstat
    -t shows per-thread statistics (CPU utilization and so on) for the threads in a process

See also

Same layer: L9 Server game process

Same symptom (Slow motion), other layers

View the interactive card with figures and simulations