한국어English日本語简体中文繁體中文DeutschไทยTiếng ViệtРусскийPortuguês (Brasil)EspañolBahasa Indonesia

Game Lag White Paper › L8 Sockets and protocols

Uneven SO_REUSEPORT distribution SO_REUSEPORT imbalance, stuck worker

Cause ID sk-reuseport · Primary owner Game team (Server development) · Also Infra team (Server infrastructure)

Open the interactive card with figures and simulations →

When several processes share one port, the kernel assigns each connection to a process by address hash and never reassigns it. If one of those processes stalls, only the players assigned to it wait.

Why A gateway or login server runs several processes on one port with SO_REUSEPORT → Effect Even when one process stalls from GC or overload, the new connections and UDP packets assigned to it don’t move to another process → On screen Only some players can’t connect or freeze. During a restart that changes the process count, some UDP sessions drop

Symptoms
Can’t connect / infinite loading, Freeze, Disconnect
Factors
Stall, Packet loss
Who’s affected
Just me, Whole server
When
Right after login or maintenance, Randomly
Owner
Primary owner Game team (Server development) · Also Infra team (Server infrastructure)
Game team action items
Never let the receiving thread stall, implement a procedure for handing over sessions on restart.
Infra team action items
Monitor the connection queue per process (Recv-Q in ss), make sure deploys that change the process count follow the session handover procedure.
On the graph
Outliers only · Connection queue (Recv-Q) per listening socket
Where to look
Recv-Q (connections waiting for accept) and the owning process of each listening socket on the same port from ss -ltnp, plus throughput compared per process
Confirmed if
Of the sockets on the same port, only one keeps building up Recv-Q, and its process is stalled or its throughput is near 0
Ruled out if
Recv-Q builds up evenly on all sockets: overall overload (“Connection queue (listen backlog) overflow”)
Check with
Infra tools (no game code needed)

Sources

  1. socket(7) — Linux manual page Linux man-pages
    SO_REUSEPORT lets several sockets bind to the same address and share incoming TCP connections and UDP packets
  2. net/core/sock_reuseport.c (Linux v6.12) Linux kernel
    Without a BPF program, the packet hash is mapped onto the number of sockets in the group to pick the socket
  3. Why does one NGINX worker take all the load? Cloudflare
    SO_REUSEPORT splits queues per worker with a simple hash, so if one worker is blocked, every connection queued to it stalls
  4. net/ipv4/tcp_diag.c (Linux v6.12) Linux kernel
    Recv-Q and Send-Q in ss: for a listening socket, connections waiting for accept and the backlog limit; for a connected socket, bytes the app hasn’t read yet and sent bytes not yet ACKed

See also

Same layer: L8 Sockets and protocols

Same symptom (Can’t connect / infinite loading), other layers

View the interactive card with figures and simulations