From 2ebb76b1aff6c05eec9590c93dbcbbed83a894dd Mon Sep 17 00:00:00 2001 From: Brad Fitzpatrick Date: Mon, 9 Nov 2026 03:10:05 +0000 Subject: [PATCH] kernel/linux/src: scale to 50,000 peers, and rename peers atomically For a 50,000-node tailnet, every peer operation must be O(1), and a peer that rotates its key must keep its traffic. Measured in a lab VM with up to 50,000 peers (tailscale.com's TestLeanScale), three things were off, plus key rotation: Deleting a device removed its peers one at a time, each waiting for an RCU grace period (netif_napi_del) and three workqueue flushes: 6ms a peer, 5 minutes for 50,000, on every tailscaled restart. Now all of a device's peers wait together (peer_remove_dead_batch): 50,000 in 0.1-0.2s. Each peer's route cache was per CPU, 40 bytes on each: 5KB a peer on a 128-CPU host, 250MB for 50,000 peers, most of which never send. It's now allocated on a peer's first send. A peer is 2.1KB, whatever the CPUs. The pubkey and index hash tables had 2^11 and 2^13 buckets, chains of 25 at 50,000 peers; now 2^16 and 2^17 (1.5MB a device). And TSWGPEER_A_NEW_PUBLIC_KEY renames a peer: in one step, under device_update_lock, it moves to its new key in the pubkey table, with its static-static and cookie keys recomputed and its sessions cleared, keeping its allowed IPs, endpoint and queues. Removing it and adding it anew would drop its traffic in between. At 50,000 peers, adding, removing, renaming and editing one take 26-83 microseconds, as at 500. Updates #426 Signed-off-by: Brad Fitzpatrick Change-Id: I2abc5369ce8321267b39df9d9dc137ef7dc040a7 --- kernel/linux/DESIGN.md | 78 +++++++++++++++++++--- kernel/linux/src/generated/netlink.c | 1 + kernel/linux/src/include/uapi/linux/tswg.h | 9 +++ kernel/linux/src/netlink.c | 72 +++++++++++++++++--- kernel/linux/src/noise.h | 3 + kernel/linux/src/peer.c | 40 +++++++++-- kernel/linux/src/peerlookup.h | 14 ++-- kernel/linux/src/socket.c | 19 ++++++ 8 files changed, 208 insertions(+), 28 deletions(-) diff --git a/kernel/linux/DESIGN.md b/kernel/linux/DESIGN.md index e8c2e5f78..30ffe9117 100644 --- a/kernel/linux/DESIGN.md +++ b/kernel/linux/DESIGN.md @@ -1348,8 +1348,9 @@ Other TODOs: * Only magicsock's first socket per family is shared; `UDPSockets` > 1 isn't supported. -* A full sync (on any config change) still walks every peer; fine for - now, but worth making incremental for huge tailnets. +* Full syncs walk every peer, but now happen only at startup, on a new + private key, and when the config source is (re)installed; each peer + change is synced alone (see "Lazy peer configuration"). ### Hybrid post-quantum handshakes @@ -1403,10 +1404,66 @@ asked for it. tswg does it (2026-11-08): tailscaled only programs recently active peers into wireguard-go, mostly because of per-peer cost in wireguard-go. Kernel peers are cheap and -hashed, so first try programming every peer. If that doesn't scale, add an -unknown-peer upcall from `wg_noise_handshake_consume_initiation` with a -short stash of the initiation for replay (to avoid a 5s `REKEY_TIMEOUT` -stall). +hashed, so the kernel data plane programs every peer, which holds up to +50,000 (2026-11-09; `TestLeanScale`, in a lab VM, adding 50,000 peers +with an endpoint and two allowed IPs each, 5,000 at a time): + +| | 500 peers | 5,000 | 50,000 | +|---|---|---|---| +| kernel memory per peer | | 2.2 KB slab | 2.1 KB slab, no per-CPU (106 MB total) | +| add, per peer in bulk | 48 us | 35 us | 37 us | +| add one / remove one | 48 / 58 us | 59 / 54 us | 39 / 65 us | +| update one's endpoint / replace its allowed IPs | 10 / 13 us | 6 / 15 us | 26 / 80 us | +| rename one (key rotation) | 73 us | | 83 us | +| dump the device | 4 ms | 17 ms | 0.5-0.7 s | +| delete the device | 38 ms | 32 ms | 0.1-0.2 s | + +The single-peer operations' growth from 5,000 to 50,000 is cache misses +in bigger tables and tries, not O(N) work: at 20,000 and 50,000, at 16 +vCPUs, they're the same (22 and 23 us to update an endpoint, 64 and 79 +to replace allowed IPs). Fixed for it: deleting a device +removed its peers one at a time, each waiting for an RCU grace period +(`netif_napi_del`) and three workqueue flushes, 6ms a peer, so 5 minutes +for 50,000; now all of them wait together (`peer_remove_dead_batch`), +1000x faster. Each peer's route cache (`dst_cache`) was per CPU, 40 +bytes on each, so 5KB a peer on a 128-CPU host, 250MB for 50,000; it's +now allocated on the peer's first send. And the pubkey and index hash +tables have 2^16 and 2^17 buckets (1.5MB a device), where upstream's +2^11 and 2^13 had chains of 25 at 50,000 peers. + +In tailscaled (tailscale.com `wgengine/kerneldp_linux.go`), each changed +peer (`Engine.SyncDevicePeer`, called for each peer a netmap delta +changes) used to trigger a full sync, walking every peer through +magicsock and the config source: O(N) per change, so O(N^2) for a +netmap of N changes. Now `syncPeer` syncs just that peer, O(1): its +magicsock path and config decide whether to add, update or remove it. +magicsock's kernel path hook now also fires when it adds or removes a +peer, so a peer magicsock learns of after its `SyncDevicePeer` is still +synced. + +Key rotation is atomic: a peer that rotates its node key comes to +tailscaled as the removal of the old key and the addition of the new, +in either order, perhaps in different syncs, or as a full netmap. Done +as such, the peer's allowed IPs and traffic would be gone in between. +So tailscaled matches the two by the peer's Tailscale IP (`peerAddr`; +by its own map of pushed peers' addresses, and LocalBackend's peer by IP +lookup for the other order) and tells the kernel to rename the peer +(`TSWGPEER_A_NEW_PUBLIC_KEY`, `tswg_peer_rename`): in one step, under +`device_update_lock`, the peer moves in the pubkey table to its new key, +with its precomputed static-static and cookie keys recomputed and its +sessions cleared, keeping its allowed IPs, endpoint and queues; O(1). +`TestTSWGTailscaledKeyRotation` rotates a peer's key (`tailscale up +--force-reauth`) and checks that the other node's kernel always had +exactly one peer, that it renamed it, and that the tunnel works. + +Not done: the device dump is O(N) by nature (0.5-0.7s at 50,000), and +tailscaled does one after each full sync (`refreshListenPort`); the +kernel's peer events, rate limited to one per peer per 2 seconds each +way, could be 50,000 a second with every peer busy; magicsock and +LocalBackend at 50,000 peers are as in TUN mode. If programming every +peer stops scaling, the fallback is an unknown-peer upcall from +`wg_noise_handshake_consume_initiation`, with a short stash of the +initiation for replay (to avoid a 5s `REKEY_TIMEOUT` stall). ### DKMS and fallback @@ -1658,9 +1715,12 @@ Everything else: (2026-11-08; it left a new kernel device without its key until a later reconcile), in 29 rounds of the tests above, each with the tunnel working on the first ping both ways. -- [ ] Scale to a 50,000-node tailnet: memory per peer, in the module and - in tailscaled's kernel data plane, and no O(N) work per peer change - (netlink pushes, reconcile, allowedips). +- [x] Scale to a 50,000-node tailnet (2026-11-09; see "Lazy peer + configuration"): 2.1 KB of kernel memory per peer, whatever the CPU + count; O(1) adds, removals, edits and atomic key rotations, in the + module and in tailscaled. +- [ ] tailscaled: skip the full device dump after each full sync, and + measure the kernel data plane's own memory per peer at 50,000. - [x] Fallback to wireguard-go when tswg is unavailable (module not loaded). - [ ] Fallback on version or feature mismatch between tailscaled and diff --git a/kernel/linux/src/generated/netlink.c b/kernel/linux/src/generated/netlink.c index 6975cbcd3..fc97cf764 100644 --- a/kernel/linux/src/generated/netlink.c +++ b/kernel/linux/src/generated/netlink.c @@ -28,6 +28,7 @@ const struct nla_policy wireguard_wgallowedip_nl_policy[WGALLOWEDIP_A_FLAGS + 1] const struct nla_policy tswg_peer_nl_policy[TSWGPEER_A_MAX + 1] = { [TSWGPEER_A_FLAGS] = NLA_POLICY_MASK(NLA_U32, TSWGPEER_F_NO_ROAM | TSWGPEER_F_HYBRID), [TSWGPEER_A_CLEAR_SESSION] = { .type = NLA_FLAG, }, + [TSWGPEER_A_NEW_PUBLIC_KEY] = NLA_POLICY_EXACT_LEN(WG_KEY_LEN), }; const struct nla_policy wireguard_wgpeer_nl_policy[WGPEER_A_TSWG + 1] = { /* TSWG */ diff --git a/kernel/linux/src/include/uapi/linux/tswg.h b/kernel/linux/src/include/uapi/linux/tswg.h index 11ecea045..951ce820f 100644 --- a/kernel/linux/src/include/uapi/linux/tswg.h +++ b/kernel/linux/src/include/uapi/linux/tswg.h @@ -121,6 +121,15 @@ enum tswgpeer_attribute { * disco key. */ TSWGPEER_A_CLEAR_SESSION, + /* + * NLA_BINARY (32 bytes), set only: change the peer's public key to + * this one, atomically, keeping its allowed IPs, endpoint and other + * settings and dropping its sessions (as TSWGPEER_A_CLEAR_SESSION), + * for when a peer rotates its key. The rest of the peer's attributes + * in the same message apply after. Fails with EEXIST if another peer + * has the key. + */ + TSWGPEER_A_NEW_PUBLIC_KEY, __TSWGPEER_A_LAST }; diff --git a/kernel/linux/src/netlink.c b/kernel/linux/src/netlink.c index 07594d67b..a4e842f35 100644 --- a/kernel/linux/src/netlink.c +++ b/kernel/linux/src/netlink.c @@ -9,6 +9,7 @@ #include "socket.h" #include "queueing.h" #include "messages.h" +#include "cookie.h" /* TSWG */ #include "generated/netlink.h" #include "tswg.h" @@ -643,6 +644,9 @@ int tswg_set_device(struct wg_device *wg, struct nlattr *attr, return 0; } +static void tswg_peer_clear_session(struct wg_peer *peer); +static int tswg_peer_rename(struct wg_peer *peer, const u8 key[NOISE_PUBLIC_KEY_LEN]); + /* TSWG: tswg_set_peer applies the nested WGPEER_A_TSWG attribute to peer. */ int tswg_set_peer(struct wg_peer *peer, struct nlattr *attr) { @@ -653,23 +657,75 @@ int tswg_set_peer(struct wg_peer *peer, struct nlattr *attr) NULL); if (ret < 0) return ret; + if (tb[TSWGPEER_A_NEW_PUBLIC_KEY]) { + ret = tswg_peer_rename(peer, nla_data(tb[TSWGPEER_A_NEW_PUBLIC_KEY])); + if (ret < 0) + return ret; + } if (tb[TSWGPEER_A_FLAGS]) { u32 flags = nla_get_u32(tb[TSWGPEER_A_FLAGS]); WRITE_ONCE(peer->tswg_no_roam, !!(flags & TSWGPEER_F_NO_ROAM)); tswg_noise_set_hybrid(&peer->handshake, flags & TSWGPEER_F_HYBRID); } - if (tb[TSWGPEER_A_CLEAR_SESSION]) { - /* As wg_pm_notification does for every peer before suspend. */ - timer_delete(&peer->timer_zero_key_material); - wg_noise_handshake_clear(&peer->handshake); - wg_noise_keypairs_clear(&peer->keypairs); - /* Don't make the next handshake wait for REKEY_TIMEOUT. */ - wg_noise_reset_last_sent_handshake(&peer->last_sent_handshake); - } + if (tb[TSWGPEER_A_CLEAR_SESSION]) + tswg_peer_clear_session(peer); return 0; } +/* TSWG: tswg_peer_clear_session forgets peer's sessions; see tswg_set_peer. */ +static void tswg_peer_clear_session(struct wg_peer *peer) +{ + /* As wg_pm_notification does for every peer before suspend. */ + timer_delete(&peer->timer_zero_key_material); + wg_noise_handshake_clear(&peer->handshake); + wg_noise_keypairs_clear(&peer->keypairs); + /* Don't make the next handshake wait for REKEY_TIMEOUT. */ + wg_noise_reset_last_sent_handshake(&peer->last_sent_handshake); +} + +/* + * TSWG: tswg_peer_rename changes peer's public key to key, atomically: the + * peer stays in allowedips, with its endpoint, settings and queues, so + * traffic to it isn't dropped, but its sessions, made with the old key, + * are cleared. All of it is O(1). The caller holds device_update_lock. + */ +static int tswg_peer_rename(struct wg_peer *peer, const u8 key[NOISE_PUBLIC_KEY_LEN]) +{ + struct wg_device *wg = peer->device; + struct wg_peer *other; + int ret = 0; + + lockdep_assert_held(&wg->device_update_lock); + if (!crypto_memneq(key, peer->handshake.remote_static, NOISE_PUBLIC_KEY_LEN)) + return 0; + other = wg_pubkey_hashtable_lookup(wg->peer_hashtable, key); + if (other) { + wg_peer_put(other); + return -EEXIST; + } + down_read(&wg->static_identity.lock); + if (wg->static_identity.has_identity && + !memcmp(key, wg->static_identity.static_public, NOISE_PUBLIC_KEY_LEN)) { + ret = -EINVAL; /* our own key */ + goto out; + } + tswg_peer_clear_session(peer); + wg_pubkey_hashtable_remove(wg->peer_hashtable, peer); + down_write(&peer->handshake.lock); + memcpy(peer->handshake.remote_static, key, NOISE_PUBLIC_KEY_LEN); + up_write(&peer->handshake.lock); + wg_noise_precompute_static_static(peer); + wg_cookie_init(&peer->latest_cookie); + wg_cookie_checker_precompute_peer_keys(peer); + wg_pubkey_hashtable_add(wg->peer_hashtable, peer); + /* An in-flight handshake may have raced the clearing above. */ + tswg_peer_clear_session(peer); +out: + up_read(&wg->static_identity.lock); + return ret; +} + /* TSWG: tswg_get_peer adds peer's WGPEER_A_TSWG attribute to skb. */ int tswg_get_peer(struct sk_buff *skb, struct wg_peer *peer) { diff --git a/kernel/linux/src/noise.h b/kernel/linux/src/noise.h index 0b83644a1..6c4d722f4 100644 --- a/kernel/linux/src/noise.h +++ b/kernel/linux/src/noise.h @@ -96,6 +96,9 @@ struct noise_handshake { /* Protects all members except the immutable (after noise_handshake_ * init): remote_static, precomputed_static_static, static_identity. + * TSWG: but tswg_peer_rename changes remote_static (and so + * precomputed_static_static), under this lock and the device's + * device_update_lock, having first cleared the handshake and keypairs. */ struct rw_semaphore lock; }; diff --git a/kernel/linux/src/peer.c b/kernel/linux/src/peer.c index 55764e927..277d0a425 100644 --- a/kernel/linux/src/peer.c +++ b/kernel/linux/src/peer.c @@ -33,8 +33,7 @@ struct wg_peer *wg_peer_create(struct wg_device *wg, peer = kmem_cache_zalloc(peer_cache, GFP_KERNEL); if (unlikely(!peer)) return ERR_PTR(ret); - if (unlikely(dst_cache_init(&peer->endpoint_cache, GFP_KERNEL))) - goto err; + /* TSWG: endpoint_cache is allocated on first send; see socket.c. */ peer->device = wg; wg_noise_handshake_init(&peer->handshake, &wg->static_identity, @@ -65,9 +64,6 @@ struct wg_peer *wg_peer_create(struct wg_device *wg, pr_debug("%s: Peer %llu created\n", wg->dev->name, peer->internal_id); return peer; -err: - kmem_cache_free(peer_cache, peer); - return ERR_PTR(ret); } struct wg_peer *wg_peer_get_maybe_zero(struct wg_peer *peer) @@ -166,6 +162,37 @@ void wg_peer_remove(struct wg_peer *peer) peer_remove_after_dead(peer); } +/* + * TSWG: peer_remove_dead_batch is peer_remove_after_dead for all of the + * dead peers on list, with each of its waits (workqueue flushes and RCU + * grace periods) done once for all of them rather than once per peer. + * Removing a device's 5,000 peers took 30 seconds, 6ms per peer, mostly + * netif_napi_del's synchronize_net, so 50,000 would have taken 5 minutes. + */ +static void peer_remove_dead_batch(struct wg_device *wg, struct list_head *list) +{ + struct wg_peer *peer, *temp; + + list_for_each_entry(peer, list, peer_list) { + WARN_ON(!peer->is_dead); + wg_noise_keypairs_clear(&peer->keypairs); + wg_timers_stop(peer); + } + /* As in peer_remove_after_dead. */ + flush_workqueue(wg->packet_crypt_wq); + flush_workqueue(wg->packet_crypt_wq); + list_for_each_entry(peer, list, peer_list) { + napi_disable(&peer->napi); + __netif_napi_del(&peer->napi); + } + synchronize_net(); /* for all the __netif_napi_dels */ + flush_workqueue(wg->handshake_send_wq); + list_for_each_entry_safe(peer, temp, list, peer_list) { + --wg->num_peers; + wg_peer_put(peer); + } +} + void wg_peer_remove_all(struct wg_device *wg) { struct wg_peer *peer, *temp; @@ -181,8 +208,7 @@ void wg_peer_remove_all(struct wg_device *wg) list_add_tail(&peer->peer_list, &dead_peers); } synchronize_net(); - list_for_each_entry_safe(peer, temp, &dead_peers, peer_list) - peer_remove_after_dead(peer); + peer_remove_dead_batch(wg, &dead_peers); /* TSWG */ } static void rcu_release(struct rcu_head *rcu) diff --git a/kernel/linux/src/peerlookup.h b/kernel/linux/src/peerlookup.h index ced811797..392156ecc 100644 --- a/kernel/linux/src/peerlookup.h +++ b/kernel/linux/src/peerlookup.h @@ -15,8 +15,11 @@ struct wg_peer; struct pubkey_hashtable { - /* TODO: move to rhashtable */ - DECLARE_HASHTABLE(hashtable, 11); + /* + * TSWG: 2^16 buckets (512KB), for a 50,000-node tailnet, rather than + * upstream's 2^11, at which lookups walked chains of 25. + */ + DECLARE_HASHTABLE(hashtable, 16); siphash_key_t key; struct mutex lock; }; @@ -31,8 +34,11 @@ wg_pubkey_hashtable_lookup(struct pubkey_hashtable *table, const u8 pubkey[NOISE_PUBLIC_KEY_LEN]); struct index_hashtable { - /* TODO: move to rhashtable */ - DECLARE_HASHTABLE(hashtable, 13); + /* + * TSWG: 2^17 buckets (1MB), for a handshake and up to three keypairs + * each of 50,000 peers, rather than upstream's 2^13. + */ + DECLARE_HASHTABLE(hashtable, 17); spinlock_t lock; }; diff --git a/kernel/linux/src/socket.c b/kernel/linux/src/socket.c index f2effea39..a88e3b825 100644 --- a/kernel/linux/src/socket.c +++ b/kernel/linux/src/socket.c @@ -185,11 +185,30 @@ out: #endif } +/* + * TSWG: tswg_endpoint_cache_init allocates peer's endpoint_cache, on its + * first send. Upstream allocates it with the peer, but it's per CPU, 40 + * bytes on each: 5KB a peer on a 128-CPU host, 250MB for a 50,000-node + * tailnet, most of whose peers never send. If it fails, sends go uncached + * (the dst_cache functions allow for that) until it doesn't. + */ +static void tswg_endpoint_cache_init(struct wg_peer *peer) +{ + struct dst_cache tmp; + + if (dst_cache_init(&tmp, GFP_ATOMIC)) + return; + if (cmpxchg(&peer->endpoint_cache.cache, NULL, tmp.cache)) + dst_cache_destroy(&tmp); /* another CPU won */ +} + int wg_socket_send_skb_to_peer(struct wg_peer *peer, struct sk_buff *skb, u8 ds) { size_t skb_len = skb->len; int ret = -EAFNOSUPPORT; + if (unlikely(!READ_ONCE(peer->endpoint_cache.cache))) /* TSWG */ + tswg_endpoint_cache_init(peer); read_lock_bh(&peer->endpoint_lock); if (peer->endpoint.addr.sa_family == AF_INET) ret = send4(peer->device, skb, &peer->endpoint, ds,