한국어English日本語简体中文繁體中文DeutschไทยTiếng ViệtРусскийPortuguês (Brasil)EspañolBahasa Indonesia

Game Lag White Paper › L13 Server architecture and operations

Clock skew between servers Clock skew between servers

Cause ID in-clock-skew · Primary owner Infra team (Server infrastructure) · Also Game team (Server development)

Open the interactive card with figures and simulations →

When each server’s clock is slightly off from the others, cooldown, buff, and event start checks disagree from server to server.

Why A server whose time sync stopped drifts hundreds of ms to several seconds away from the other servers → Effect Passing absolute times, such as when a buff ends, between servers makes their checks disagree → On screen A buff disappears or a cooldown starts over after moving to another server

Symptoms
Dropped action / rollback
Factors
Latency
Who’s affected
Just me
When
While moving or changing zones
Owner
Primary owner Infra team (Server infrastructure) · Also Game team (Server development)
Game team action items
Pass remaining time between servers in place of absolute timestamps.
Infra team action items
Monitor time sync (NTP, chrony), alert on clock differences between servers.
Ballpark numbers
With healthy time sync (NTP, chrony), servers in the same data center usually stay within a few ms of each other. If sync stops, or a virtual machine is paused for a long time and then resumes, the gap grows to hundreds of ms to several seconds.
On the graph
Slow climb · Clock offset per server
Where to look
chronyc tracking on every server, comparing System time (difference between the system clock and NTP time), Last offset, and Ref time (when a measurement from the time source was last applied)
Confirmed if
The problem server’s offset is hundreds of ms or more away from the other servers, or its Ref time stopped long ago, and the mismatched checks happen only on moves to or from that server
Ruled out if
All servers’ offsets are within a few ms: points to the game’s own time calculation or client clock sync error
Check with
Infra tools (no game code needed)
Learn more
A single server’s clock jumping forward or backward all at once is covered in the server OS layer under “System clock jump (NTP step).”

Sources

  1. RFC 5905: Network Time Protocol Version 4: Protocol and Algorithms Specification IETF
    NTP clients on a fast LAN usually stay within a few hundred µs
  2. chrony – Frequently Asked Questions chrony
    Ordinary computer clocks drift less than 100 ppm, but virtual machines can drift more; a VM that was paused and resumed can be far enough off to need a step correction
  3. clock_gettime(2) — Linux manual page Linux man-pages
    CLOCK_REALTIME can jump discontinuously from manual changes or NTP adjustments; CLOCK_MONOTONIC is not affected by such jumps
  4. chronyc(1) chrony
    chronyc tracking fields: System time (difference between NTP time and the system clock), Last offset (offset estimated at the last update), Ref time (when the last measurement from the time source was applied)

See also

Same layer: L13 Server architecture and operations

Same symptom (Dropped action / rollback), other layers

View the interactive card with figures and simulations