Game Lag White Paper › L4 Internet path

Peak-hour congestion at peering links Peak-hour congestion at peering

Cause ID isp-peak · Primary owner Infra team (Network infrastructure) · Also External (External)

Open the interactive card with figures and simulations →

Around 9–11 PM, video traffic surges and the links between ISPs (peering) tend to get congested.

Why Evening streaming and downloads pile up → Effect Queues build and packets drop on peering links → On screen Players on certain ISPs get stutter and teleporting only in the evening

Symptoms

Stutter, Teleporting, Rubber-banding

Factors

Jitter, Packet loss, Latency

Who’s affected

Specific region/ISP

When

Evening peak hours

Owner

Primary owner Infra team (Network infrastructure) · Also External (External)

Infra team action items

Add more direct connections with the affected ISP, route around congested paths, monitor evening loss and ping per ISP.

External action items

Ask the ISP to add capacity on the peering link.

On the graph

High at certain hours · RTT/loss (by ISP)

Where to look

Plot RTT and loss per ISP (ASN) by time of day, and get mtr runs in the evening and during the day from RIPE Atlas probes on that ISP or from players to find the hop where loss starts

Confirmed if

Only one ISP sees RTT and loss rise around 9–11 PM every evening, and in mtr the loss and delay persist from the inter-ISP link all the way to the destination

Ruled out if

All ISPs rise together: points to our own links or servers. Only one household is bad in the evening: “Congested Wi-Fi channel”

Check with

Infra tools (no game code needed)

Sources

1. Inferring Persistent Interdomain Congestion (SIGCOMM 2018) ACM
   Recurring congestion on some inter-ISP links, with latency rising at peak hours every day and loss rates also rising during congested periods
2. Probe Selection (RIPE Atlas REST API) RIPE NCC
   Select RIPE Atlas probes by country, region, ASN, or address prefix and run ping and traceroute from them

See also

Same layer: L4 Internet path

• Propagation delay (physical distance)
• Satellite internet (LEO/GEO)
• Detour routing
• Submarine cable / international link outage
• BGP route changes and convergence
• One faulty ECMP path
• ISP throttling and traffic management
• Country- or ISP-level UDP restrictions and packet inspection
• Poor line quality
• DNS failures and delays
• Shared links saturated by DDoS
• ISP-shared IP addresses (CGNAT)
• Routing through a VPN or game booster

Same symptom (Stutter), other layers

• Frame time spike L1 Client game process
• Client garbage collection L1 Client game process
• Synchronous loading and shader compilation on the main thread L1 Client game process
• Slow storage delays asset streaming L1 Client game process
• Rendering load from large crowds L1 Client game process
• Missing or too-short interpolation buffer L1 Client game process
• Excessive extrapolation (dead reckoning) L1 Client game process
• Fixed-timestep catch-up spiral L1 Client game process

View the interactive card with figures and simulations