When packets get out of order crossing multiple paths or bundled links, the receiver signals “a packet is missing” with duplicate ACKs, and the sender resends a packet that arrived fine.
Why Devices that split traffic across paths per packet, LAGs (link bundles) that spread traffic per packet, and route changes shuffle packet order → Effect Later packets arrive first and three duplicate ACKs pile up → fast retransmit → On screen Sparse game packets are barely affected. Large updates in crowded areas and patch downloads slow down, with occasional stutter
Primary owner Infra team (Network infrastructure) · Also Infra team (Server infrastructure)
Infra team action items
Network: switch from per-packet to per-connection load balancing (hash ECMP and LAG on address and port). Servers/OS: use RACK (time-based loss detection that tolerates reordering; when DSACK reveals spurious retransmissions, it automatically widens its reordering allowance), check the reordering degree Linux estimates automatically for each connection (the reordering value in ss -ti, starting from tcp_reordering=3).
On the graph
Always high · Reordering detections, DSACKs received
Where to look
nstat TcpExtTCPSACKReorder and TcpExtTCPTSReorder (reordering detections) and TcpExtTCPDSACKRecv; per connection, reordering (shown when not 3) and reord_seen in ss -ti. In a packet capture, the Wireshark filter tcp.analysis.out_of_order
Confirmed if
Reordering counters and DSACK climb steadily at all hours, and connections through a specific path or device show a reordering value above 3. The receiver-side capture shows later packets arriving first, with the earlier ones following shortly
Ruled out if
Reordering counters flat while TcpExtTCPLostRetransmit rises: real loss. DSACK rising only at moments when RTT jumps: “Spurious retransmission from latency spikes”
IP SysctlLinux kernel tcp_reordering starts at 3 (adjusted automatically per connection up to tcp_max_reordering), RACK setting in tcp_recovery
misc/ss.ciproute2 ss -ti shows reordering:value when a connection’s reordering differs from the default of 3, and reord_seen:count once the connection has seen reordering
SNMP counterLinux kernel TcpExtTCPSACKReorder, TcpExtTCPTSReorder (reordering detected), TcpExtTCPDSACKRecv (DSACKs received), TcpExtTCPLostRetransmit (a retransmitted packet lost again)
include/uapi/linux/tcp.hLinux kernel tcpi_reord_seen in tcp_info: how many times the connection has seen reordering