Game Lag White Paper › L8 Sockets and protocols

Slow start after idle Slow start after idle

Cause ID sk-slowstart · Primary owner Infra team (Server infrastructure) · Also Game team (Server development)

Open the interactive card with figures and simulations →

When a connection has been idle for a while, TCP shrinks the congestion window (how much it can send at once) again, so a sudden large send goes out in several rounds.

Why A large burst of data (on entering a town, for example) goes out over a connection that was idle → Effect The congestion window has shrunk, so the data is spread over several round trips → On screen Right after entering, nearby characters and NPCs appear a few round trips late (more noticeable on distant servers)

Symptoms

Input lag, Invisible / ghost entities

Factors

Latency

Who’s affected

Just me

When

While moving or changing zones, After sitting idle

Owner

Primary owner Infra team (Server infrastructure) · Also Game team (Server development)

Game team action items

Shrink the entry data (send what’s essential first).

Infra team action items

Turn off tcp_slow_start_after_idle (Linux, server-wide setting).

Ballpark numbers

After an idle period longer than the RTO, the congestion window starts shrinking, and after a long idle period it drops to about 14 KB (10 packets). Then 100 KB can’t go out at once and takes 3 round trips.

On the graph

Outliers only · Transfer time right after entering (players with long RTT)

Where to look

The value of sysctl net.ipv4.tcp_slow_start_after_idle, and whether cwnd (congestion window) in ss -ti for that connection has shrunk at the moment a player enters an area after being idle

Confirmed if

The setting is 1 (the default), and on entering after idle, cwnd drops to around 10 and the transfer is split over several round trips. The longer a player’s RTT, the later things appear, and setting it to 0 makes the problem go away

Ruled out if

cwnd stays large, yet things still appear late: server-side entry handling (“Spawn burst when entering a crowded area”)

Check with

Infra tools (no game code needed)

Sources

1. IP Sysctl Linux kernel
   tcp_slow_start_after_idle on by default, the congestion window shrinks after an RTO of idle time (RFC 2861 method)
2. RFC 5681: TCP Congestion Control IETF
   If no data has been sent for longer than the RTO, the congestion window is cut to at most the restart window min(IW, cwnd) and slow start begins again
3. RFC 6928: Increasing TCP's Initial Window IETF
   Initial window 10 segments, up to 14,600 bytes
4. ss(8) — Linux manual page iproute2
   cwnd (congestion window) and ssthresh (slow start threshold) in -i

See also

Same layer: L8 Sockets and protocols

• TCP head-of-line blocking
• TCP RTO and exponential backoff
• Nagle’s algorithm + delayed ACK
• Blocking sends caused by slow clients
• Slow client (slow consumer) handling policy
• Keepalive default of 2 hours
• IP fragmentation of UDP packets
• Reliable UDP retransmission settings
• Sending rate plunges under congestion control
• Last data lost to an abortive close (RST)
• Blocking I/O design
• Uneven SO_REUSEPORT distribution
• WSAECONNRESET errors on Windows UDP sockets

Same symptom (Input lag), other layers

• Rendering load from large crowds L1 Client game process
• Packet processing bottleneck on the main thread L1 Client game process
• V-Sync and the render queue L1 Client game process
• Power saving and thermal throttling L2 Client OS and device
• Other apps on the same device using up bandwidth L2 Client OS and device
• Display, input device, and frame generation latency L2 Client OS and device
• Congested Wi-Fi channel L3 Home network
• Bufferbloat (router queue) L3 Home network

View the interactive card with figures and simulations