With small send and receive buffers, a burst of traffic makes the kernel drop packets arriving over UDP, and TCP sends block because the buffer has no room left.
Why SO_SNDBUF and SO_RCVBUF left at their defaults or set too small → Effect During a burst, or while the receiving thread pauses briefly, the UDP receive buffer overflows and drops packets; TCP waits because the send buffer has no room → On screen Teleporting (UDP loss) or fast-forward (TCP waiting)
Primary owner Infra team (Server infrastructure) · Also Game team (Server development)
Game team action items
Set buffer sizes (SO_SNDBUF, SO_RCVBUF) in code to match the traffic, remember that setting a TCP buffer size explicitly turns off Linux autotuning, keep sizes moderate because oversized buffers let stale data pile up and add delay, keep the receiving thread from stalling.
Infra team action items
Tune the kernel caps (rmem_max, wmem_max; buffer sizes set in code can’t exceed them either) and the default (rmem_default), monitor the buffer overflow counter (RcvbufErrors).
Ballpark numbers
The default UDP receive buffer on Linux is about 208 KB. Even a small packet takes up far more kernel memory than its actual size, so tens to hundreds of packets fill it. On a server receiving 100,000 packets per second, it overflows if the receiving thread stalls for just a few ms.
On the graph
Random spikes · UDP receive buffer overflows (UdpRcvbufErrors)
Where to look
Increase in UdpRcvbufErrors from nstat -az, and skmem in ss -uamn (rb is the receive buffer size, d is packets dropped because they couldn’t be queued to the socket); for TCP, whether send-queue memory (w) in the skmem of ss -tm has reached the send buffer size (tb)
Confirmed if
UdpRcvbufErrors (or the socket’s d) rises during bursts or when the receiving thread stalls, with rb near the default (about 208 KB). For TCP, w stays pinned at tb and send blocks
Ruled out if
Counters unchanged but loss present: the NIC stage (“Ring buffer too small”) or the network path
Check with
Infra tools (no game code needed)
Sources
socket(7) — Linux manual pageLinux man-pages SO_RCVBUF and SO_SNDBUF default to rmem_default and wmem_default and are capped at rmem_max and wmem_max; the kernel doubles the value you set
include/net/sock.h (Linux v6.18)Linux kernel The default socket buffer is defined as 256 packets of 256 bytes including sk_buff overhead (SKB_TRUESIZE(256)×256); even a small frame counts as sk_buff + MTU (about 208 KB is the computed value on x86-64)
IP SysctlLinux kernel tcp_rmem, tcp_wmem: setting SO_RCVBUF or SO_SNDBUF directly turns off autotuning for that socket
net/ipv4/udp.c (Linux v6.12)Linux kernel When the UDP receive queue exceeds the socket buffer size, packets are dropped immediately and RcvbufErrors goes up
net/ipv4/proc.c (Linux v6.12)Linux kernel Counter names shown by nstat: RcvbufErrors and SndbufErrors in the Udp group
ss(8) — Linux manual pageiproute2 skmem in -m: rb receive buffer size, tb send buffer size, w send-queue memory, d packets dropped before reaching the socket