한국어English日本語简体中文繁體中文DeutschไทยTiếng ViệtРусскийPortuguês (Brasil)EspañolBahasa Indonesia

Game Lag White Paper › L7 Server OS (kernel)

CPU steal (virtual machines) CPU steal time

Cause ID so-steal · Primary owner Infra team (Server infrastructure) · Also External (External)

Open the interactive card with figures and simulations →

While the physical server (hypervisor) briefly gives a virtual machine’s CPU time to another VM (CPU steal), the game server stalls.

Why Other VMs on the same host use a lot of CPU → Effect The game server’s VM loses its turn on the CPU for a few ms to tens of ms at a time → On screen Unexplained tick-time spikes: stutter, freeze

Symptoms
Stutter, Freeze
Factors
Stall
Who’s affected
Whole server
When
Randomly
Owner
Primary owner Infra team (Server infrastructure) · Also External (External)
Infra team action items
Monitor steal (st in top and vmstat), use dedicated cores or hosts, avoid burstable instances that slow down when CPU credits run out, stop and start instances with persistently high steal to move them to another host.
External action items
Report hosts with persistently high steal to the cloud provider.
On the graph
Random spikes · %steal, server tick time
Where to look
%steal from mpstat -P ALL 1 on the same time axis as server tick time
Confirmed if
%steal spikes whenever the tick spikes, and drops after a stop and start moves the instance to another host
Ruled out if
%steal near 0 while ticks still spike: a cause inside the game server (“Server GC stop-the-world pause,” “Lock contention”). In a container: “Container CPU throttling (CFS quota)”
Check with
Infra tools (no game code needed)

Sources

  1. proc_stat(5) — Linux manual page Linux man-pages
    steal: time lost while other operating systems ran in a virtualized environment
  2. Standard mode for burstable performance instances AWS
    Burstable instances spend credits to run above baseline performance, and when the credits run out, CPU utilization drops to the baseline level
  3. How EC2 instance stop and start works AWS
    Stopping and starting an instance usually moves it to a new host
  4. mpstat(1) — Linux manual page sysstat
    %steal: share of time this virtual CPU was forced to wait while the hypervisor ran other virtual CPUs

See also

Same layer: L7 Server OS (kernel)

Same symptom (Stutter), other layers

View the interactive card with figures and simulations