Brad Fitzpatrick 2ebb76b1af kernel/linux/src: scale to 50,000 peers, and rename peers atomically
For a 50,000-node tailnet, every peer operation must be O(1), and a peer
that rotates its key must keep its traffic. Measured in a lab VM with
up to 50,000 peers (tailscale.com's TestLeanScale), three things were
off, plus key rotation:

Deleting a device removed its peers one at a time, each waiting for an
RCU grace period (netif_napi_del) and three workqueue flushes: 6ms a
peer, 5 minutes for 50,000, on every tailscaled restart. Now all of a
device's peers wait together (peer_remove_dead_batch): 50,000 in
0.1-0.2s.

Each peer's route cache was per CPU, 40 bytes on each: 5KB a peer on a
128-CPU host, 250MB for 50,000 peers, most of which never send. It's now
allocated on a peer's first send. A peer is 2.1KB, whatever the CPUs.

The pubkey and index hash tables had 2^11 and 2^13 buckets, chains of
25 at 50,000 peers; now 2^16 and 2^17 (1.5MB a device).

And TSWGPEER_A_NEW_PUBLIC_KEY renames a peer: in one step, under
device_update_lock, it moves to its new key in the pubkey table, with
its static-static and cookie keys recomputed and its sessions cleared,
keeping its allowed IPs, endpoint and queues. Removing it and adding
it anew would drop its traffic in between.

At 50,000 peers, adding, removing, renaming and editing one take 26-83
microseconds, as at 500.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I2abc5369ce8321267b39df9d9dc137ef7dc040a7
2026-11-09 03:10:05 +00:00
Languages
Go 96.5%
C 1.2%
TypeScript 0.8%
Shell 0.5%
Swift 0.3%
Other 0.3%