diff --git a/kernel/linux/DESIGN.md b/kernel/linux/DESIGN.md index e8c2e5f78..30ffe9117 100644 --- a/kernel/linux/DESIGN.md +++ b/kernel/linux/DESIGN.md @@ -1348,8 +1348,9 @@ Other TODOs: * Only magicsock's first socket per family is shared; `UDPSockets` > 1 isn't supported. -* A full sync (on any config change) still walks every peer; fine for - now, but worth making incremental for huge tailnets. +* Full syncs walk every peer, but now happen only at startup, on a new + private key, and when the config source is (re)installed; each peer + change is synced alone (see "Lazy peer configuration"). ### Hybrid post-quantum handshakes @@ -1403,10 +1404,66 @@ asked for it. tswg does it (2026-11-08): tailscaled only programs recently active peers into wireguard-go, mostly because of per-peer cost in wireguard-go. Kernel peers are cheap and -hashed, so first try programming every peer. If that doesn't scale, add an -unknown-peer upcall from `wg_noise_handshake_consume_initiation` with a -short stash of the initiation for replay (to avoid a 5s `REKEY_TIMEOUT` -stall). +hashed, so the kernel data plane programs every peer, which holds up to +50,000 (2026-11-09; `TestLeanScale`, in a lab VM, adding 50,000 peers +with an endpoint and two allowed IPs each, 5,000 at a time): + +| | 500 peers | 5,000 | 50,000 | +|---|---|---|---| +| kernel memory per peer | | 2.2 KB slab | 2.1 KB slab, no per-CPU (106 MB total) | +| add, per peer in bulk | 48 us | 35 us | 37 us | +| add one / remove one | 48 / 58 us | 59 / 54 us | 39 / 65 us | +| update one's endpoint / replace its allowed IPs | 10 / 13 us | 6 / 15 us | 26 / 80 us | +| rename one (key rotation) | 73 us | | 83 us | +| dump the device | 4 ms | 17 ms | 0.5-0.7 s | +| delete the device | 38 ms | 32 ms | 0.1-0.2 s | + +The single-peer operations' growth from 5,000 to 50,000 is cache misses +in bigger tables and tries, not O(N) work: at 20,000 and 50,000, at 16 +vCPUs, they're the same (22 and 23 us to update an endpoint, 64 and 79 +to replace allowed IPs). Fixed for it: deleting a device +removed its peers one at a time, each waiting for an RCU grace period +(`netif_napi_del`) and three workqueue flushes, 6ms a peer, so 5 minutes +for 50,000; now all of them wait together (`peer_remove_dead_batch`), +1000x faster. Each peer's route cache (`dst_cache`) was per CPU, 40 +bytes on each, so 5KB a peer on a 128-CPU host, 250MB for 50,000; it's +now allocated on the peer's first send. And the pubkey and index hash +tables have 2^16 and 2^17 buckets (1.5MB a device), where upstream's +2^11 and 2^13 had chains of 25 at 50,000 peers. + +In tailscaled (tailscale.com `wgengine/kerneldp_linux.go`), each changed +peer (`Engine.SyncDevicePeer`, called for each peer a netmap delta +changes) used to trigger a full sync, walking every peer through +magicsock and the config source: O(N) per change, so O(N^2) for a +netmap of N changes. Now `syncPeer` syncs just that peer, O(1): its +magicsock path and config decide whether to add, update or remove it. +magicsock's kernel path hook now also fires when it adds or removes a +peer, so a peer magicsock learns of after its `SyncDevicePeer` is still +synced. + +Key rotation is atomic: a peer that rotates its node key comes to +tailscaled as the removal of the old key and the addition of the new, +in either order, perhaps in different syncs, or as a full netmap. Done +as such, the peer's allowed IPs and traffic would be gone in between. +So tailscaled matches the two by the peer's Tailscale IP (`peerAddr`; +by its own map of pushed peers' addresses, and LocalBackend's peer by IP +lookup for the other order) and tells the kernel to rename the peer +(`TSWGPEER_A_NEW_PUBLIC_KEY`, `tswg_peer_rename`): in one step, under +`device_update_lock`, the peer moves in the pubkey table to its new key, +with its precomputed static-static and cookie keys recomputed and its +sessions cleared, keeping its allowed IPs, endpoint and queues; O(1). +`TestTSWGTailscaledKeyRotation` rotates a peer's key (`tailscale up +--force-reauth`) and checks that the other node's kernel always had +exactly one peer, that it renamed it, and that the tunnel works. + +Not done: the device dump is O(N) by nature (0.5-0.7s at 50,000), and +tailscaled does one after each full sync (`refreshListenPort`); the +kernel's peer events, rate limited to one per peer per 2 seconds each +way, could be 50,000 a second with every peer busy; magicsock and +LocalBackend at 50,000 peers are as in TUN mode. If programming every +peer stops scaling, the fallback is an unknown-peer upcall from +`wg_noise_handshake_consume_initiation`, with a short stash of the +initiation for replay (to avoid a 5s `REKEY_TIMEOUT` stall). ### DKMS and fallback @@ -1658,9 +1715,12 @@ Everything else: (2026-11-08; it left a new kernel device without its key until a later reconcile), in 29 rounds of the tests above, each with the tunnel working on the first ping both ways. -- [ ] Scale to a 50,000-node tailnet: memory per peer, in the module and - in tailscaled's kernel data plane, and no O(N) work per peer change - (netlink pushes, reconcile, allowedips). +- [x] Scale to a 50,000-node tailnet (2026-11-09; see "Lazy peer + configuration"): 2.1 KB of kernel memory per peer, whatever the CPU + count; O(1) adds, removals, edits and atomic key rotations, in the + module and in tailscaled. +- [ ] tailscaled: skip the full device dump after each full sync, and + measure the kernel data plane's own memory per peer at 50,000. - [x] Fallback to wireguard-go when tswg is unavailable (module not loaded). - [ ] Fallback on version or feature mismatch between tailscaled and diff --git a/kernel/linux/src/generated/netlink.c b/kernel/linux/src/generated/netlink.c index 6975cbcd3..fc97cf764 100644 --- a/kernel/linux/src/generated/netlink.c +++ b/kernel/linux/src/generated/netlink.c @@ -28,6 +28,7 @@ const struct nla_policy wireguard_wgallowedip_nl_policy[WGALLOWEDIP_A_FLAGS + 1] const struct nla_policy tswg_peer_nl_policy[TSWGPEER_A_MAX + 1] = { [TSWGPEER_A_FLAGS] = NLA_POLICY_MASK(NLA_U32, TSWGPEER_F_NO_ROAM | TSWGPEER_F_HYBRID), [TSWGPEER_A_CLEAR_SESSION] = { .type = NLA_FLAG, }, + [TSWGPEER_A_NEW_PUBLIC_KEY] = NLA_POLICY_EXACT_LEN(WG_KEY_LEN), }; const struct nla_policy wireguard_wgpeer_nl_policy[WGPEER_A_TSWG + 1] = { /* TSWG */ diff --git a/kernel/linux/src/include/uapi/linux/tswg.h b/kernel/linux/src/include/uapi/linux/tswg.h index 11ecea045..951ce820f 100644 --- a/kernel/linux/src/include/uapi/linux/tswg.h +++ b/kernel/linux/src/include/uapi/linux/tswg.h @@ -121,6 +121,15 @@ enum tswgpeer_attribute { * disco key. */ TSWGPEER_A_CLEAR_SESSION, + /* + * NLA_BINARY (32 bytes), set only: change the peer's public key to + * this one, atomically, keeping its allowed IPs, endpoint and other + * settings and dropping its sessions (as TSWGPEER_A_CLEAR_SESSION), + * for when a peer rotates its key. The rest of the peer's attributes + * in the same message apply after. Fails with EEXIST if another peer + * has the key. + */ + TSWGPEER_A_NEW_PUBLIC_KEY, __TSWGPEER_A_LAST }; diff --git a/kernel/linux/src/netlink.c b/kernel/linux/src/netlink.c index 07594d67b..a4e842f35 100644 --- a/kernel/linux/src/netlink.c +++ b/kernel/linux/src/netlink.c @@ -9,6 +9,7 @@ #include "socket.h" #include "queueing.h" #include "messages.h" +#include "cookie.h" /* TSWG */ #include "generated/netlink.h" #include "tswg.h" @@ -643,6 +644,9 @@ int tswg_set_device(struct wg_device *wg, struct nlattr *attr, return 0; } +static void tswg_peer_clear_session(struct wg_peer *peer); +static int tswg_peer_rename(struct wg_peer *peer, const u8 key[NOISE_PUBLIC_KEY_LEN]); + /* TSWG: tswg_set_peer applies the nested WGPEER_A_TSWG attribute to peer. */ int tswg_set_peer(struct wg_peer *peer, struct nlattr *attr) { @@ -653,23 +657,75 @@ int tswg_set_peer(struct wg_peer *peer, struct nlattr *attr) NULL); if (ret < 0) return ret; + if (tb[TSWGPEER_A_NEW_PUBLIC_KEY]) { + ret = tswg_peer_rename(peer, nla_data(tb[TSWGPEER_A_NEW_PUBLIC_KEY])); + if (ret < 0) + return ret; + } if (tb[TSWGPEER_A_FLAGS]) { u32 flags = nla_get_u32(tb[TSWGPEER_A_FLAGS]); WRITE_ONCE(peer->tswg_no_roam, !!(flags & TSWGPEER_F_NO_ROAM)); tswg_noise_set_hybrid(&peer->handshake, flags & TSWGPEER_F_HYBRID); } - if (tb[TSWGPEER_A_CLEAR_SESSION]) { - /* As wg_pm_notification does for every peer before suspend. */ - timer_delete(&peer->timer_zero_key_material); - wg_noise_handshake_clear(&peer->handshake); - wg_noise_keypairs_clear(&peer->keypairs); - /* Don't make the next handshake wait for REKEY_TIMEOUT. */ - wg_noise_reset_last_sent_handshake(&peer->last_sent_handshake); - } + if (tb[TSWGPEER_A_CLEAR_SESSION]) + tswg_peer_clear_session(peer); return 0; } +/* TSWG: tswg_peer_clear_session forgets peer's sessions; see tswg_set_peer. */ +static void tswg_peer_clear_session(struct wg_peer *peer) +{ + /* As wg_pm_notification does for every peer before suspend. */ + timer_delete(&peer->timer_zero_key_material); + wg_noise_handshake_clear(&peer->handshake); + wg_noise_keypairs_clear(&peer->keypairs); + /* Don't make the next handshake wait for REKEY_TIMEOUT. */ + wg_noise_reset_last_sent_handshake(&peer->last_sent_handshake); +} + +/* + * TSWG: tswg_peer_rename changes peer's public key to key, atomically: the + * peer stays in allowedips, with its endpoint, settings and queues, so + * traffic to it isn't dropped, but its sessions, made with the old key, + * are cleared. All of it is O(1). The caller holds device_update_lock. + */ +static int tswg_peer_rename(struct wg_peer *peer, const u8 key[NOISE_PUBLIC_KEY_LEN]) +{ + struct wg_device *wg = peer->device; + struct wg_peer *other; + int ret = 0; + + lockdep_assert_held(&wg->device_update_lock); + if (!crypto_memneq(key, peer->handshake.remote_static, NOISE_PUBLIC_KEY_LEN)) + return 0; + other = wg_pubkey_hashtable_lookup(wg->peer_hashtable, key); + if (other) { + wg_peer_put(other); + return -EEXIST; + } + down_read(&wg->static_identity.lock); + if (wg->static_identity.has_identity && + !memcmp(key, wg->static_identity.static_public, NOISE_PUBLIC_KEY_LEN)) { + ret = -EINVAL; /* our own key */ + goto out; + } + tswg_peer_clear_session(peer); + wg_pubkey_hashtable_remove(wg->peer_hashtable, peer); + down_write(&peer->handshake.lock); + memcpy(peer->handshake.remote_static, key, NOISE_PUBLIC_KEY_LEN); + up_write(&peer->handshake.lock); + wg_noise_precompute_static_static(peer); + wg_cookie_init(&peer->latest_cookie); + wg_cookie_checker_precompute_peer_keys(peer); + wg_pubkey_hashtable_add(wg->peer_hashtable, peer); + /* An in-flight handshake may have raced the clearing above. */ + tswg_peer_clear_session(peer); +out: + up_read(&wg->static_identity.lock); + return ret; +} + /* TSWG: tswg_get_peer adds peer's WGPEER_A_TSWG attribute to skb. */ int tswg_get_peer(struct sk_buff *skb, struct wg_peer *peer) { diff --git a/kernel/linux/src/noise.h b/kernel/linux/src/noise.h index 0b83644a1..6c4d722f4 100644 --- a/kernel/linux/src/noise.h +++ b/kernel/linux/src/noise.h @@ -96,6 +96,9 @@ struct noise_handshake { /* Protects all members except the immutable (after noise_handshake_ * init): remote_static, precomputed_static_static, static_identity. + * TSWG: but tswg_peer_rename changes remote_static (and so + * precomputed_static_static), under this lock and the device's + * device_update_lock, having first cleared the handshake and keypairs. */ struct rw_semaphore lock; }; diff --git a/kernel/linux/src/peer.c b/kernel/linux/src/peer.c index 55764e927..277d0a425 100644 --- a/kernel/linux/src/peer.c +++ b/kernel/linux/src/peer.c @@ -33,8 +33,7 @@ struct wg_peer *wg_peer_create(struct wg_device *wg, peer = kmem_cache_zalloc(peer_cache, GFP_KERNEL); if (unlikely(!peer)) return ERR_PTR(ret); - if (unlikely(dst_cache_init(&peer->endpoint_cache, GFP_KERNEL))) - goto err; + /* TSWG: endpoint_cache is allocated on first send; see socket.c. */ peer->device = wg; wg_noise_handshake_init(&peer->handshake, &wg->static_identity, @@ -65,9 +64,6 @@ struct wg_peer *wg_peer_create(struct wg_device *wg, pr_debug("%s: Peer %llu created\n", wg->dev->name, peer->internal_id); return peer; -err: - kmem_cache_free(peer_cache, peer); - return ERR_PTR(ret); } struct wg_peer *wg_peer_get_maybe_zero(struct wg_peer *peer) @@ -166,6 +162,37 @@ void wg_peer_remove(struct wg_peer *peer) peer_remove_after_dead(peer); } +/* + * TSWG: peer_remove_dead_batch is peer_remove_after_dead for all of the + * dead peers on list, with each of its waits (workqueue flushes and RCU + * grace periods) done once for all of them rather than once per peer. + * Removing a device's 5,000 peers took 30 seconds, 6ms per peer, mostly + * netif_napi_del's synchronize_net, so 50,000 would have taken 5 minutes. + */ +static void peer_remove_dead_batch(struct wg_device *wg, struct list_head *list) +{ + struct wg_peer *peer, *temp; + + list_for_each_entry(peer, list, peer_list) { + WARN_ON(!peer->is_dead); + wg_noise_keypairs_clear(&peer->keypairs); + wg_timers_stop(peer); + } + /* As in peer_remove_after_dead. */ + flush_workqueue(wg->packet_crypt_wq); + flush_workqueue(wg->packet_crypt_wq); + list_for_each_entry(peer, list, peer_list) { + napi_disable(&peer->napi); + __netif_napi_del(&peer->napi); + } + synchronize_net(); /* for all the __netif_napi_dels */ + flush_workqueue(wg->handshake_send_wq); + list_for_each_entry_safe(peer, temp, list, peer_list) { + --wg->num_peers; + wg_peer_put(peer); + } +} + void wg_peer_remove_all(struct wg_device *wg) { struct wg_peer *peer, *temp; @@ -181,8 +208,7 @@ void wg_peer_remove_all(struct wg_device *wg) list_add_tail(&peer->peer_list, &dead_peers); } synchronize_net(); - list_for_each_entry_safe(peer, temp, &dead_peers, peer_list) - peer_remove_after_dead(peer); + peer_remove_dead_batch(wg, &dead_peers); /* TSWG */ } static void rcu_release(struct rcu_head *rcu) diff --git a/kernel/linux/src/peerlookup.h b/kernel/linux/src/peerlookup.h index ced811797..392156ecc 100644 --- a/kernel/linux/src/peerlookup.h +++ b/kernel/linux/src/peerlookup.h @@ -15,8 +15,11 @@ struct wg_peer; struct pubkey_hashtable { - /* TODO: move to rhashtable */ - DECLARE_HASHTABLE(hashtable, 11); + /* + * TSWG: 2^16 buckets (512KB), for a 50,000-node tailnet, rather than + * upstream's 2^11, at which lookups walked chains of 25. + */ + DECLARE_HASHTABLE(hashtable, 16); siphash_key_t key; struct mutex lock; }; @@ -31,8 +34,11 @@ wg_pubkey_hashtable_lookup(struct pubkey_hashtable *table, const u8 pubkey[NOISE_PUBLIC_KEY_LEN]); struct index_hashtable { - /* TODO: move to rhashtable */ - DECLARE_HASHTABLE(hashtable, 13); + /* + * TSWG: 2^17 buckets (1MB), for a handshake and up to three keypairs + * each of 50,000 peers, rather than upstream's 2^13. + */ + DECLARE_HASHTABLE(hashtable, 17); spinlock_t lock; }; diff --git a/kernel/linux/src/socket.c b/kernel/linux/src/socket.c index f2effea39..a88e3b825 100644 --- a/kernel/linux/src/socket.c +++ b/kernel/linux/src/socket.c @@ -185,11 +185,30 @@ out: #endif } +/* + * TSWG: tswg_endpoint_cache_init allocates peer's endpoint_cache, on its + * first send. Upstream allocates it with the peer, but it's per CPU, 40 + * bytes on each: 5KB a peer on a 128-CPU host, 250MB for a 50,000-node + * tailnet, most of whose peers never send. If it fails, sends go uncached + * (the dst_cache functions allow for that) until it doesn't. + */ +static void tswg_endpoint_cache_init(struct wg_peer *peer) +{ + struct dst_cache tmp; + + if (dst_cache_init(&tmp, GFP_ATOMIC)) + return; + if (cmpxchg(&peer->endpoint_cache.cache, NULL, tmp.cache)) + dst_cache_destroy(&tmp); /* another CPU won */ +} + int wg_socket_send_skb_to_peer(struct wg_peer *peer, struct sk_buff *skb, u8 ds) { size_t skb_len = skb->len; int ret = -EAFNOSUPPORT; + if (unlikely(!READ_ONCE(peer->endpoint_cache.cache))) /* TSWG */ + tswg_endpoint_cache_init(peer); read_lock_bh(&peer->endpoint_lock); if (peer->endpoint.addr.sa_family == AF_INET) ret = send4(peer->device, skb, &peer->endpoint, ds,