Game Lag White Paper › L11 Disk

HDD seek latency HDD seek latency

Cause ID dk-hdd · Primary owner Infra team (Server infrastructure) · Also Infra team (DB infrastructure), Game team (Server development)

Open the interactive card with figures and simulations →

An HDD has to move its head across the platter (a seek), so reading or writing scattered data takes close to 10 ms each time.

Why HDDs in old servers or low-cost storage → Effect About 10 ms for every scattered read or write → On screen Slow saves and loading across the board

Symptoms

Input lag

Factors

Latency

Who’s affected

Whole server

When

Always

Owner

Primary owner Infra team (Server infrastructure) · Also Infra team (DB infrastructure), Game team (Server development)

Game team action items

Design around sequential writes.

Infra team action items

Servers/OS: replace with SSDs (starting with storage disks that see the most scattered reads and writes). DB hosts: replace the DB disks with the most scattered reads and writes with SSDs first.

On the graph

Always high · Disk read/write latency (r_await, w_await)

Where to look

Check whether it’s a rotating disk (HDD) with lsblk -d -o NAME,ROTA, and look at r/s, w/s, r_await, and w_await from iostat -x 1. For virtual servers, check the disk type in the cloud or storage specs

Confirmed if

A rotating disk, with r_await and w_await always at a few ms to tens of ms even at only tens to about a hundred requests per second

Ruled out if

High latency on an SSD: points to queue saturation (dk-iops) or burst credit depletion (dk-burst)

Check with

Infra tools (no game code needed)

Sources

1. Designs, Lessons and Advice from Building Large Distributed Systems (LADIS 2009 keynote) Google
   One disk seek 10 ms, 1 MB sequential read from disk 20 ms
2. Exos X18 Data Sheet Seagate
   7,200 rpm HDD average rotational latency 4.16 ms, 4K random reads 170 IOPS
3. lsblk(8) — Linux manual page util-linux
   -o selects output columns; device topology columns include ROTA (whether the device is rotational)
4. ABI stable symbols Linux kernel
   /sys/block/(disk)/queue/rotational: whether the device is rotational or non-rotational
5. iostat(1) — Linux manual page sysstat
   -x: r/s and w/s, r_await and w_await (average time per request, including time spent waiting in the queue)

See also

Same layer: L11 Disk

• Synchronous log writes
• fsync surge
• Cloud disk out of burst credits
• IOPS limit / queue saturation
• Disk full
• Backup / compression / scan jobs
• Server-side lazy loading
• Writing a core dump

Same symptom (Input lag), other layers

• Rendering load from large crowds L1 Client game process
• Packet processing bottleneck on the main thread L1 Client game process
• V-Sync and the render queue L1 Client game process
• Power saving and thermal throttling L2 Client OS and device
• Other apps on the same device using up bandwidth L2 Client OS and device
• Display, input device, and frame generation latency L2 Client OS and device
• Congested Wi-Fi channel L3 Home network
• Bufferbloat (router queue) L3 Home network

View the interactive card with figures and simulations