한국어English日本語简体中文繁體中文DeutschไทยTiếng ViệtРусскийPortuguês (Brasil)EspañolBahasa Indonesia

Game Lag White Paper › Root causes of TCP retransmission

Zero window (a stall that looks like retransmission) Zero window, often mistaken for retransmission

Cause ID rt-zero-window · Primary owner Game team (Client development) · Also Game team (Server development), Infra team (Server infrastructure)

Open the interactive card with figures and simulations →

When the receiving program doesn’t read its socket in time and the buffer fills up, the sender stops sending and sends only zero window probes. The network itself is fine.

Why A client frame freeze or a blocked server thread keeps the socket from being read → Effect The receive window drops to 0, so the sender stops sending and sends only probes (at growing intervals) → On screen Freeze then fast-forward. A packet capture shows “ZeroWindow” and no loss

Symptoms
Freeze, Fast-forward
Factors
Stall
Who’s affected
Just me, Whole server
When
When crowds gather, Randomly
Owner
Primary owner Game team (Client development) · Also Game team (Server development), Infra team (Server infrastructure)
Game team action items
Start with the side that sent ZeroWindow in the packet capture (the side that can’t read its socket), keep reading network input on a separate thread, size the receive buffer appropriately. Client: fix the causes of frame freezes such as loading and GC. Server: fix whatever blocks the thread that reads the socket.
Infra team action items
Add the server’s nstat TcpExtTCPToZeroWindowAdv (times the server advertised a zero receive window) to monitoring (if it rises, the problem is on the server side, so pass it to server development), provide server-side packet captures.
On the graph
Gap then burst · Per-connection receive volume, zero window count
Where to look
In a packet capture, the side that advertised window 0, found with the Wireshark filter tcp.analysis.zero_window. In the server’s nstat, TcpExtTCPToZeroWindowAdv (the server advertised window 0) and TcpExtTCPWinProbe (probes sent in response to the peer’s window 0) viewed separately, plus Recv-Q on the server socket (bytes in ss the program hasn’t read yet)
Confirmed if
No retransmissions during the freeze, only zero window and probe packets. TcpExtTCPToZeroWindowAdv or server socket Recv-Q rising: the server isn’t reading in time. TcpExtTCPWinProbe rising: the client isn’t reading in time
Ruled out if
No zero window in the capture and the same data being resent: points to loss or spurious retransmission
Check with
Infra tools (no game code needed)
Real incidents
Roblox 2021: Roblox 73-hour outage: contention in the service discovery (Consul) cluster

Sources

  1. RFC 9293: Transmission Control Protocol (TCP) IETF
    When the receive window is 0, the sender sends zero window probes and backs off the probe interval exponentially
  2. 7.5. TCP Analysis Wireshark
    TCP ZeroWindow: a packet in which the receiver advertises window 0, telling the sender to stop sending
  3. Display Filter Reference: Transmission Control Protocol Wireshark
    tcp.analysis.zero_window, tcp.analysis.zero_window_probe display filters
  4. SNMP counter Linux kernel
    TcpExtTCPToZeroWindowAdv: times the receive window was advertised as 0 after being nonzero
  5. net/ipv4/proc.c Linux kernel
    Counter names as shown by nstat: TCPToZeroWindowAdv, TCPWinProbe
  6. net/ipv4/tcp_output.c Linux kernel
    TCPWinProbe: increments for each probe sent while the peer’s receive window is 0 (tcp_send_probe0)
  7. net/ipv4/tcp_diag.c Linux kernel
    For a connected socket, Recv-Q in ss is the number of bytes received that the program hasn’t read yet

See also

Same layer: Root causes of TCP retransmission

Same symptom (Freeze), other layers

View the interactive card with figures and simulations