Game Lag White Paper › L6 Server network card

NIC bandwidth saturation NIC bandwidth saturation

Cause ID nic-saturate · Primary owner Game team (Server development) · Also Infra team (Server infrastructure)

Open the interactive card with figures and simulations →

Running a 1 Gbps or 10 Gbps card at its limit makes the transmit queue grow until packets get dropped.

Why More broadcasts push traffic to the card’s limit → Effect The transmit queue grows, and packets are dropped when it overflows → On screen Latency and loss across the whole server (input lag, teleporting)

Symptoms

Input lag, Teleporting

Factors

Latency, Packet loss

Who’s affected

Whole server

When

When crowds gather

Owner

Primary owner Game team (Server development) · Also Infra team (Server infrastructure)

Game team action items

Reduce traffic (area of interest, compression, send only what changed).

Infra team action items

Upgrade the card (a faster NIC, or a larger instance in the cloud), alert on NIC utilization.

On the graph

Hits a ceiling · NIC transmit volume, transmit drops

Where to look

Compare txkB/s and %ifutil (utilization relative to interface speed) from sar -n DEV 1 with the NIC speed or instance bandwidth, alongside TX dropped from ip -s link

Confirmed if

Transmit volume flattens near the NIC or instance bandwidth, and from then on transmit drops and server-wide latency rise

Ruled out if

Bandwidth has headroom: not this cause. Many small packets and loss: “Cloud PPS limit exceeded”

Check with

Infra tools (no game code needed)

Sources

1. Interface statistics Linux kernel
   tx_dropped: number of packets dropped during transmission for lack of resources
2. Amazon EC2 instance network bandwidth AWS
   The bandwidth an instance can use is set by its vCPU count (instance size)
3. sar(1) — Linux manual page sysstat
   rxkB/s, txkB/s, and %ifutil (utilization relative to interface speed) from -n DEV

See also

Same layer: L6 Server network card

• NIC interrupts concentrated on one core
• Ring buffer too small
• Excessive interrupt coalescing
• Cloud PPS limit exceeded
• Virtualization overhead and noisy neighbors
• Cloud host maintenance and live migration
• NIC driver and firmware problems
• GRO/LRO batching delay

Same symptom (Input lag), other layers

• Rendering load from large crowds L1 Client game process
• Packet processing bottleneck on the main thread L1 Client game process
• V-Sync and the render queue L1 Client game process
• Power saving and thermal throttling L2 Client OS and device
• Other apps on the same device using up bandwidth L2 Client OS and device
• Display, input device, and frame generation latency L2 Client OS and device
• Congested Wi-Fi channel L3 Home network
• Bufferbloat (router queue) L3 Home network

View the interactive card with figures and simulations