If the server reads dungeon or map data from disk the first time it’s requested, everyone freezes for that tick.
Why Someone enters a dungeon or area for the first time → Effect The server reads the data from disk on the game thread → On screen Everyone on that server freezes briefly
Primary owner Game team (Server development) · Also Infra team (Server infrastructure)
Game team action items
Preload at server startup, load asynchronously.
Infra team action items
For servers just created from a snapshot, warm up the disk before putting them into service (read every block once) or use fast snapshot restore.
On the graph
Random spikes · Server tick time, disk reads
Where to look
Match freeze times to first-entry records for dungeons and areas in the game server log, and at those moments check the game server’s disk reads (kB_rd/s from pidstat -d) and slow read and open calls with perf trace --duration. On a newly launched cloud server, compare EBS VolumeAvgReadLatency with older servers
Confirmed if
Freezes only on the first entry, with no freeze on the second entry to the same place. During the freeze, the game thread is waiting on a file read
Ruled out if
Freezes just the same in areas already loaded: another cause such as tick overrun or GC
Check with
Game server or client logs and metrics
Learn more
In the cloud, a server just created from a snapshot (a copy of a disk) fetches every block from remote storage the first time it reads it, so reads are far slower than usual. Suspect this if first entry takes unusually long only on servers newly launched by autoscaling.
Sources
Initialize Amazon EBS volumesAWS Volumes created from snapshots have higher latency and lower performance while blocks are fetched from S3; initialize them in advance by reading every block with dd or fio
Amazon EBS fast snapshot restoreAWS Fast snapshot restore provides volumes that are fully initialized at creation, removing first-access latency