한국어English日本語简体中文繁體中文DeutschไทยTiếng ViệtРусскийPortuguês (Brasil)EspañolBahasa Indonesia

Game Lag White Paper › L11 Disk

Writing a core dump Core dump writing

Cause ID dk-coredump · Primary owner Infra team (Server infrastructure) · Also Game team (Server development)

Open the interactive card with figures and simulations →

When the server crashes, writing several GB of memory to disk can delay the restart by several minutes.

Why A server crash writes all of memory to a file → Effect No restart until several GB have been written → On screen After the server dies and players disconnect, they can’t connect again for a long time

Symptoms
Can’t connect / infinite loading
Factors
Stall
Who’s affected
Whole server
When
Randomly
Owner
Primary owner Infra team (Server infrastructure) · Also Game team (Server development)
Game team action items
Consider small dumps holding only the needed memory (minidumps), fix the cause of the crash.
Infra team action items
Limit dump size (OS core dump settings), use fast disks, decouple the restart from the dump (compress and upload dumps separately after the restart).
On the graph
Mass disconnect · Connection count, server restart times
Where to look
Line up the crash time, core file size (coredumpctl list and info, or the file where core_pattern points), the time the file finished writing, and the time the service came back up, and check wkB/s from iostat -x during that window
Confirmed if
After the crash, disk writes stay near the limit while a multi-GB core file is written, and the restart begins only after the write finishes
Ruled out if
Restart is still slow with core dumps off or finished small: points to the server startup process, such as map loading or a DB cold cache (db-cold-cache)
Check with
Infra tools (no game code needed)

Sources

  1. core(5) — Linux manual page Linux man-pages
    RLIMIT_CORE caps core file size, coredump_filter selects which memory regions to include, and core dumps can be piped to a program for separate handling
  2. Minidump Files Microsoft
    A minidump holds only a useful subset of crash dump information, so it is fast and small
  3. coredumpctl(1) — Linux manual page systemd
    list: core dumps recorded in the journal (TIME is the crash time reported by the kernel); info: details for each dump and the size written to disk
  4. iostat(1) — Linux manual page sysstat
    -x: wkB/s (disk write volume per second)

See also

Same layer: L11 Disk

Same symptom (Can’t connect / infinite loading), other layers

View the interactive card with figures and simulations