Game Lag White Paper › L9 Server game process

Infinite loops and runaway logic Infinite loop / runaway logic

Cause ID sp-infinite-loop · Primary owner Game team (Server development)

Open the interactive card with figures and simulations →

When a bug keeps a tick from ever finishing, the server stops, and the watchdog forces a restart.

Why A loop that never ends because of a wrong condition, or runaway recursion → Effect The tick never finishes, and the server stops → On screen A freeze, then everyone disconnects

Symptoms

Freeze, Disconnect

Factors

Stall

Who’s affected

Specific zone/channel, Whole server

When

During specific actions, Randomly

Owner

Primary owner Game team (Server development)

Game team action items

Cap loop iterations, add a watchdog, write tests that reproduce the problem input.

On the graph

Mass disconnect · Connection count, per-thread CPU

Where to look

Per-thread CPU from pidstat -t 1 during the hang, and which function the thread spinning at 100% is in, from perf top -t (thread ID) or gdb. If it has already restarted, the watchdog timeout record (WatchdogSec in systemd, a failed Kubernetes liveness probe)

Confirmed if

While the server is stopped, one game thread sits at 100% CPU, and its stack keeps looping inside the same function or loop

Ruled out if

CPU near 0 during the hang: points to a deadlock or waiting on an external response

Check with

Infra tools (no game code needed)

Sources

1. systemd.service(5) — Linux manual page systemd
   WatchdogSec=: if the service doesn’t send a keep-alive signal (WATCHDOG=1) within the set time, it’s treated as failed and stopped, then restarted automatically depending on the Restart= setting
2. Liveness, Readiness, and Startup Probes Kubernetes
   A liveness probe catches a state where the app is running but can’t make progress and restarts it; by default it checks every 10 seconds and restarts after 3 consecutive failures
3. pidstat(1) — Linux manual page sysstat
   -t shows per-thread statistics (CPU utilization and so on) for the threads in a process
4. perf-top(1) — Linux manual page perf
   Shows live CPU share per function (symbol) for a running thread (-t) or process (-p)

See also

Same layer: L9 Server game process

• Tick overrun
• AOI calculation blowup (N²)
• Broadcast fan-out overload
• Single-threaded zone overload (hotspot)
• Lock contention
• Deadlock
• Blocking calls on the game thread
• Message queue backlog
• Timers firing all at once
• Pathfinding storm
• Serialization and compression cost
• Server crash
• Thread pool exhaustion
• Combat concentrated on one target (world boss)
• Spawn burst when entering a crowded area
• Entity buildup (items and summons never cleaned up)
• Patch changes the traffic pattern

Same symptom (Freeze), other layers

• Frame time spike L1 Client game process
• Client garbage collection L1 Client game process
• Synchronous loading and shader compilation on the main thread L1 Client game process
• Slow storage delays asset streaming L1 Client game process
• Anti-cheat scans L1 Client game process
• Wi-Fi ↔ LTE/5G switching L2 Client OS and device
• Client low on memory and swapping L2 Client OS and device
• Out of graphics memory (VRAM) L2 Client OS and device

View the interactive card with figures and simulations