Game Lag White Paper › L6 Server network card

GRO/LRO batching delay GRO/LRO batching

Cause ID nic-offload · Primary owner Infra team (Server infrastructure)

Open the interactive card with figures and simulations →

GRO and LRO bundle several packets into one to reduce CPU load. Depending on settings, a small game packet may wait briefly for the next packet to bundle with.

Why The NIC and kernel bundle arriving packets together for processing → Effect With hardware aggregation (LRO) or a batching wait time setting turned on, packets wait briefly for the next one → On screen A small rise in latency (usually tens of µs or less)

Symptoms

Input lag

Factors

Latency

Who’s affected

Whole server

When

Always

Owner

Primary owner Infra team (Server infrastructure)

Infra team action items

Tune for game traffic (turn off LRO, check the batching wait time setting), check it after other causes since the effect is usually small.

On the graph

Always high · Round-trip time within the same data center

Where to look

Check lro and gro status with ethtool -k and the device’s gro_flush_timeout sysfs setting, and compare round-trip time for small packets within the data center before and after changing them

Confirmed if

LRO is on or gro_flush_timeout is above 0, and turning it off or setting it to 0 reduces round-trip time for small packets

Ruled out if

Difference stays within a few µs after the change: not this cause

Check with

Infra tools (no game code needed)

Sources

1. NAPI Linux kernel
   A large gro_flush_timeout batches more work together but adds latency under low load
2. Linux Base Driver for the Intel(R) Ethernet 10 Gigabit PCI Express Adapters (ixgbe) Linux kernel
   GRO saves CPU by merging incoming traffic into large chunks; it evolved from LRO
3. ethtool(8) — Linux manual page ethtool
   The gro and lro on|off settings of ethtool -K

See also

Same layer: L6 Server network card

• NIC interrupts concentrated on one core
• Ring buffer too small
• Excessive interrupt coalescing
• Cloud PPS limit exceeded
• NIC bandwidth saturation
• Virtualization overhead and noisy neighbors
• Cloud host maintenance and live migration
• NIC driver and firmware problems

Same symptom (Input lag), other layers

• Rendering load from large crowds L1 Client game process
• Packet processing bottleneck on the main thread L1 Client game process
• V-Sync and the render queue L1 Client game process
• Power saving and thermal throttling L2 Client OS and device
• Other apps on the same device using up bandwidth L2 Client OS and device
• Display, input device, and frame generation latency L2 Client OS and device
• Congested Wi-Fi channel L3 Home network
• Bufferbloat (router queue) L3 Home network

View the interactive card with figures and simulations