Game Lag White Paper › Root causes of TCP retransmission

Packet drops on the receiving host Receiver host drops (ring, softirq, CPU)

Cause ID rt-host-drop · Primary owner Infra team (Server infrastructure)

Open the interactive card with figures and simulations →

Packets reach the server but get dropped, because the NIC’s ring buffer (which briefly holds arriving packets) overflows or the kernel cores that handle receive processing are saturated.

Why A surge of players, interrupts piled on one core, CPU steal on a virtual machine, or an overloaded virtual switch → Effect Drops at the ring buffer (rx_missed_errors and similar; the name varies by driver) or at the kernel receive queue (softnet dropped) → On screen When crowds gather, input registers late and hitches hit the whole server at once

Symptoms

Input lag, Freeze, Fast-forward, Teleporting

Factors

Packet loss, Stall

Who’s affected

Whole server

When

When crowds gather

Owner

Primary owner Infra team (Server infrastructure)

Infra team action items

Add drop counters such as rx_missed_errors in ethtool -S and dropped in /proc/net/softnet_stat to monitoring, enlarge ring buffers (ethtool -G), spread RSS and interrupts across multiple cores, keep the game thread and receive-processing cores separate, keep CPU headroom, and on virtual machines check CPU steal and virtual switch load.

Ballpark numbers

Packets the server drops on receive are resent by the client. So they barely show up in the server’s retransmission metrics and appear first in drop counters such as rx_missed_errors in ethtool -S (the name varies by driver) and in dropped in /proc/net/softnet_stat.

On the graph

Hits a ceiling · softirq utilization per core, NIC drop counters

Where to look

Drop counters in ethtool -S (rx_missed_errors and similar; on mlx5, rx_out_of_buffer and rx_discards_phy), missed in ip -s -s link, and the 2nd column (dropped) and 3rd column (time_squeeze) of /proc/net/softnet_stat, plus %soft (softirq processing) per core from mpstat -P ALL. On virtual machines, %steal too

Confirmed if

Drop counters or softnet dropped rise when crowds gather, and %soft on the receive-processing cores sits near 100% and can’t go higher. Input slows on every connection to that server at the same time

Ruled out if

Server drop counters flat while retransmissions concentrate on connections from a specific region or ISP: loss on the path. When the path loses packets the server sent, the server’s nstat TcpRetransSegs rises while these counters stay flat

Check with

Infra tools (no game code needed)

Sources

1. Interface statistics Linux kernel
   rx_missed_errors counts packets the host missed because no buffer was available (counted into drop in /proc/net/dev); check with ip -s -s link
2. drivers/net/ethernet/intel/igb/igb_ethtool.c Linux kernel
   ethtool -S counter names are set by the driver (e.g., igb’s rx_missed_errors and rx_no_buffer_count)
3. Ethtool counters Linux kernel
   The mlx5 driver’s rx_out_of_buffer (no buffer in the receive queue) and rx_discards_phy (dropped for lack of port buffer)
4. net/core/net-procfs.c Linux kernel
   /proc/net/softnet_stat has one line per CPU, in hex; the 2nd column is dropped and the 3rd is time_squeeze
5. Documentation for /proc/sys/net/ Linux kernel
   netdev_max_backlog: limit of the receive queue that holds packets arriving faster than the kernel can process them
6. Scaling in the Linux Networking Stack Linux kernel
   RSS: the NIC splits traffic across multiple receive queues handled by multiple CPUs
7. ethtool(8) — Linux manual page ethtool
   -G (--set-ring) changes the ring buffer size
8. proc_stat(5) — Linux manual page Linux man-pages
   steal in /proc/stat: time spent in other operating systems when running in a virtualized environment
9. mpstat(1) — Linux manual page sysstat
   mpstat -P ALL shows per-core utilization; %soft is time spent servicing softirqs, and %steal is time spent waiting while the hypervisor serviced another virtual CPU
10. net/ipv4/proc.c Linux kernel
    TcpRetransSegs as shown by nstat (RetransSegs under Tcp)

See also

Same layer: Root causes of TCP retransmission

• Wireless link loss
• Bottleneck queue overflow (congestion loss)
• Send bursts overflow shallow buffers
• Policer drops excess traffic
• Physical errors (bad cable, optics, connectors)
• Duplex mismatch
• Firewall and connection tracking drops
• Middlebox over capacity (firewall, IPS, DDoS protection)
• MTU black hole (only large packets keep getting lost)
• NAT or load balancer mapping expires mid-connection
• Route change / bad ECMP path
• Spurious retransmission from latency spikes
• Spurious fast retransmit from reordering
• Late or lost ACKs (saturated upload)
• RTO settings that don’t fit the environment
• Slow recovery on thin streams
• Middlebox strips TCP options
• Zero window (a stall that looks like retransmission)
• Connection request (SYN) retransmission

Same symptom (Input lag), other layers

• Rendering load from large crowds L1 Client game process
• Packet processing bottleneck on the main thread L1 Client game process
• V-Sync and the render queue L1 Client game process
• Power saving and thermal throttling L2 Client OS and device
• Other apps on the same device using up bandwidth L2 Client OS and device
• Display, input device, and frame generation latency L2 Client OS and device
• Congested Wi-Fi channel L3 Home network
• Bufferbloat (router queue) L3 Home network

View the interactive card with figures and simulations