When several servers send packets to thousands of players at the same instant, the small buffer on the switch port where that traffic converges overflows in less than 1 ms.
Why A world boss spawn or massive skills, or ticks on several servers lining up so they all send at once → Effect Buffers where several ports feed into one, or where a fast port feeds a slower one (hundreds of KB to a few MB per port), fill up in an instant → On screen Some packets dropped; many players teleport or have skills fail to go off at the same moment
Primary owner Game team (Server development) · Also Infra team (Network infrastructure), Infra team (Server infrastructure)
Game team action items
Spread sends evenly across the tick (pacing), offset each server’s tick start time slightly.
Infra team action items
Network: use switches with bigger buffers, spread traffic (place servers across several switches and ports), watch drop counters per switch port. Servers/OS: cap each server’s total send rate (Linux tc shaper).
Ballpark numbers
A 10 Gbps port can send about 1.25 MB in 1 ms. If traffic from two ports converges on one port at once, 1.25 MB piles up every 1 ms. Even at 10% average utilization over 1 second, the port can overflow at the 1 ms scale.
On the graph
Rises with load · Switch port output drops
Where to look
Collect output drop counters (ifOutDiscards, or output drops depending on the device) at the shortest interval you can on the switch ports the servers connect to and on the ports where their traffic converges, and match them against boss spawns and big battles. 1-second or 1-minute average utilization graphs won’t show it
Confirmed if
Average utilization is low, but output drops rise every time players crowd into one place, and at those moments many players report teleporting and skills not going off
Ruled out if
Drops rise steadily during hours of high average utilization: “Data center link saturation.” Input errors (CRC) rise: “Bad cables and port errors”
Data Center TCP (DCTCP) (SIGCOMM 2010)ACM Commodity switches have shallow buffers (48 ports share 4 MB, and one port can use up to about 700 KB); loss occurs when many flows converge on one port for a brief moment
RFC 2863: The Interfaces Group MIBIETF ifOutDiscards: number of packets discarded even though no error occurred, for reasons such as freeing buffer space