한국어English日本語简体中文繁體中文DeutschไทยTiếng ViệtРусскийPortuguês (Brasil)EspañolBahasa Indonesia

Game Lag White Paper › L9 Server game process

Lock contention Lock contention

Cause ID sp-lock · Primary owner Game team (Server development)

Open the interactive card with figures and simulations →

When several threads wait on one lock to write the same data, they run one at a time no matter how many threads you add.

Why Several threads use shared data at once, such as the auction house or guild storage → Effect The others wait until the thread holding the lock finishes → On screen Only certain features are slow; in bad cases, the whole tick is delayed

Symptoms
Input lag, Freeze
Factors
Stall
Who’s affected
One feature only, Whole server
When
When crowds gather
Owner
Primary owner Game team (Server development)
Game team action items
Split locks into finer-grained ones, do less work inside locks, move to a message-based design (give each piece of data an owning thread, and have other threads only send it requests as messages).
Ballpark numbers
If 20% of the work happens inside the lock, throughput tops out at 5 times that of one thread no matter how many threads you add; at 40%, it stops at 2.5 times.
On the graph
Rises with load · Request processing time, per-thread CPU and context switches
Where to look
Per-thread voluntary context switches (cswch/s, times a thread stopped to wait for a resource) from pidstat -w -t 1; where threads wait after leaving the CPU (wait time per call stack) from bcc offcputime -p. For .NET, the lock contention count in dotnet-counters (dotnet.monitor.lock_contentions on .NET 9 and later, Monitor Lock Contention Count on 8 and earlier)
Confirmed if
Processing time grows as load rises while CPU utilization stays low, most of the wait time is concentrated in call stacks trying to acquire a lock, and the lock contention count rises along with it
Ruled out if
CPU maxed out: a compute problem (tick overrun, single-threaded zone overload). Waiting on DB or file calls: points to blocking calls on the game thread
Check with
Infra tools (no game code needed)
Learn more
This happens in designs where several threads modify game data together. A design where one thread owns each area or feature and threads communicate only through messages has almost no locks, but you have to watch for work piling up on one thread (single-threaded zone overload). If the game thread waits for a lock held by a slow save operation, that whole tick stalls.
Real incidents
Roblox 2021: Roblox 73-hour outage: contention in the service discovery (Consul) cluster

Sources

  1. Amdahl's Law in the Multicore Era IEEE
    IEEE Computer 2008 paper (authors’ copy). If the fraction that can’t be parallelized is 1−f, the speedup can never exceed 1/(1−f) no matter how many cores you add (Amdahl’s law)
  2. Request scheduling Microsoft
    Orleans grains (actors) use a single-threaded execution model that runs each request to completion one at a time, so state is never modified concurrently; grains waiting on each other’s responses can deadlock
  3. pidstat(1) — Linux manual page sysstat
    cswch/s in -w is the number of voluntary context switches from stopping to wait for a resource; -t shows it per thread
  4. Demonstrations of offcputime, the Linux eBPF/bcc version IO Visor
    Sums the time threads spent blocked off the CPU (off-CPU) per call stack; -p selects the process
  5. Well-known EventCounters in .NET Microsoft
    Monitor Lock Contention Count (monitor-lock-contention-count): number of times contention occurred when trying to acquire a monitor lock
  6. .NET runtime metrics .NET
    dotnet.monitor.lock_contentions since .NET 9: number of times contention occurred when trying to acquire a monitor lock since the process started

See also

Same layer: L9 Server game process

Same symptom (Input lag), other layers

View the interactive card with figures and simulations