Firewall and connection tracking drops Stateful firewall / conntrack drops
Cause ID rt-stateful-fw · Primary owner Infra team (Network infrastructure) · Also Infra team (Server infrastructure), Game team (Server development), Game team (Client development)
Firewalls and Linux connection tracking (conntrack, which records passing connections in a table) drop packets when the table is full or when they decide a packet doesn’t match the connection’s state.
Why The connection tracking table is full (table full), or traffic takes a different path each way so only one direction passes through the firewall (asymmetric routing) → Effect The firewall treats the packets as belonging to an “unknown connection” or carrying a “sequence number outside the window” and drops them → On screen A full table blocks new connections; a path mismatch makes only players on that path disconnect after repeated retransmissions
When crowds gather, Right after login or maintenance, Randomly
Owner
Primary owner Infra team (Network infrastructure) · Also Infra team (Server infrastructure), Game team (Server development), Game team (Client development)
Game team action items
Server: in case the table fills up, throttle connection surges with a login queue, reuse connections so you don’t keep opening short-lived ones (including server-to-server calls), proactively close connections whose heartbeats have stopped. Client: when connecting fails or the connection drops, retry at growing, randomized intervals (so players don’t all pile back in at once while the table is full).
Infra team action items
Network: enlarge the firewall’s connection tracking table, exempt game ports from connection tracking, align routing so both directions pass through the same firewall, check the firewall’s TCP window checking settings. Servers/OS: enlarge the Linux table (nf_conntrack_max), exempt game ports from connection tracking (NOTRACK), check the TCP window checking setting (nf_conntrack_tcp_be_liberal), and on AWS also check conntrack_allowance_exceeded.
Ballpark numbers
The default Linux conntrack limit (nf_conntrack_max) ranges from tens of thousands to hundreds of thousands of entries depending on memory. When the current count (nf_conntrack_count) reaches the limit, the log shows “nf_conntrack: table full, dropping packet”.
On the graph
Hits a ceiling · conntrack entry count (nf_conntrack_count), failed new connections
Where to look
On Linux servers, nf_conntrack_count and nf_conntrack_max, “nf_conntrack: table full, dropping packet” in dmesg, and drop and invalid in /proc/net/stat/nf_conntrack (one line per core, in hex). On firewalls, session table usage and drop logs; on AWS, conntrack_allowance_exceeded in ethtool -S
Confirmed if
Entry count flattens at the limit, and table full logs and drop, or conntrack_allowance_exceeded, rise at the same moment. With asymmetric routing, the table has headroom, but invalid and firewall drop logs rise for connections on a specific path
Ruled out if
Entry count far from the limit, with invalid and drop logs flat: a different cause. Table has headroom but the firewall’s CPU or packets per second is maxed out: “Middlebox over capacity (firewall, IPS, DDoS protection)”
Check with
Infra tools (no game code needed)
Sources
Netfilter Conntrack Sysfs variablesLinux kernel nf_conntrack_max defaults to the number of hash buckets (memory ÷ 16384, 1,024–262,144), the current count is nf_conntrack_count, and nf_conntrack_tcp_be_liberal marks only out-of-window RSTs as INVALID
net/netfilter/nf_conntrack_core.cLinux kernel When the table is full, logs “nf_conntrack: table full, dropping packet” and drops the packet (the drop stat increases); packets that don’t match the connection state increase the invalid stat
Amazon EC2 security group connection trackingAWS When an instance exceeds its tracked-connection limit, packets are dropped, visible in conntrack_allowance_exceeded; recommends avoiding asymmetric routing
net/netfilter/nf_conntrack_standalone.cLinux kernel /proc/net/stat/nf_conntrack has one line per core, in hex, with columns such as entries, invalid, insert_failed, drop, and early_drop