한국어English日本語简体中文繁體中文DeutschไทยTiếng ViệtРусскийPortuguês (Brasil)EspañolBahasa Indonesia

Game Lag White Paper › L12 Database

Login storm and N+1 queries Login storm, N+1 queries

Cause ID db-login-storm · Primary owner Game team (Server development) · Also Infra team (DB infrastructure)

Open the interactive card with figures and simulations →

If loading one character takes dozens of separate queries, tens of thousands of simultaneous logins turn into millions of queries.

Why Character loading queries items, skills, and quests one by one → Effect Simultaneous logins right after maintenance make the query count explode → On screen Infinite loading at login, and even saves for players already in the game get held up

Symptoms
Can’t connect / infinite loading, Input lag
Factors
Stall, Latency
Who’s affected
Whole server
When
Right after login or maintenance
Owner
Primary owner Game team (Server development) · Also Infra team (DB infrastructure)
Game team action items
Fetch in batched queries, use a login queue and caching, check how many queries ORM lazy loading generates.
Infra team action items
Pull a ranking of the most frequently called queries and share it, monitor query and connection counts during the login window right after maintenance.
On the graph
Surge after opening · DB queries per second, login count
Where to look
Overlay the login count right after maintenance with the DB’s queries per second (growth of Questions in MySQL) and compute queries per login. Pull the most frequently called queries from COUNT_STAR in MySQL events_statements_summary_by_digest or calls in PostgreSQL pg_stat_statements
Confirmed if
Dozens of queries per login, and the top queries are short queries of the same shape that look up by a single character ID. If queries per login went up after a patch, that patch is the starting point
Ruled out if
Few queries per login but each one is slow: cold cache (db-cold-cache) or indexes (db-no-index)
Check with
Infra tools (no game code needed)
Learn more
Lazy loading in an ORM (a library that builds DB queries for you) generates queries like these without developers even noticing. On a dev server with only a few characters it goes unnoticed, and it first shows up with simultaneous logins on live servers.

Sources

  1. Efficient Querying .NET
    ORM lazy loading creates the N+1 problem, sending one more query per item and badly hurting performance; recommends loading in one batch (eager loading)
  2. pg_stat_statements — track statistics of SQL planning and execution PostgreSQL
    Collects execution count (calls) and total execution time per statement to rank the most frequently called queries
  3. Performance Schema Statement Digests and Sampling MySQL
    events_statements_summary_by_digest groups queries of the same shape and aggregates counts and times
  4. Statement Summary Tables MySQL
    COUNT_STAR (execution count) and SUM_TIMER_WAIT (total time) in summary tables
  5. Server Status Variables MySQL
    Questions: number of statements sent by clients

See also

Same layer: L12 Database

Same symptom (Can’t connect / infinite loading), other layers

View the interactive card with figures and simulations