kernel/linux/src: scale to 50,000 peers, and rename peers atomically

For a 50,000-node tailnet, every peer operation must be O(1), and a peer
that rotates its key must keep its traffic. Measured in a lab VM with
up to 50,000 peers (tailscale.com's TestLeanScale), three things were
off, plus key rotation:

Deleting a device removed its peers one at a time, each waiting for an
RCU grace period (netif_napi_del) and three workqueue flushes: 6ms a
peer, 5 minutes for 50,000, on every tailscaled restart. Now all of a
device's peers wait together (peer_remove_dead_batch): 50,000 in
0.1-0.2s.

Each peer's route cache was per CPU, 40 bytes on each: 5KB a peer on a
128-CPU host, 250MB for 50,000 peers, most of which never send. It's now
allocated on a peer's first send. A peer is 2.1KB, whatever the CPUs.

The pubkey and index hash tables had 2^11 and 2^13 buckets, chains of
25 at 50,000 peers; now 2^16 and 2^17 (1.5MB a device).

And TSWGPEER_A_NEW_PUBLIC_KEY renames a peer: in one step, under
device_update_lock, it moves to its new key in the pubkey table, with
its static-static and cookie keys recomputed and its sessions cleared,
keeping its allowed IPs, endpoint and queues. Removing it and adding
it anew would drop its traffic in between.

At 50,000 peers, adding, removing, renaming and editing one take 26-83
microseconds, as at 500.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I2abc5369ce8321267b39df9d9dc137ef7dc040a7
This commit is contained in:
Brad Fitzpatrick committed 2026-11-09 03:10:05 +00:00
1 parent 867eaea830
commit 2ebb76b1af
8 files changed
+208 -28

No files matched your search

+69 -9
View File
@@ -1348,8 +1348,9 @@ Other TODOs:
* Only magicsock's first socket per family is shared; `UDPSockets` > 1
isn't supported.
* A full sync (on any config change) still walks every peer; fine for
now, but worth making incremental for huge tailnets.
* Full syncs walk every peer, but now happen only at startup, on a new
private key, and when the config source is (re)installed; each peer
change is synced alone (see "Lazy peer configuration").
### Hybrid post-quantum handshakes
@@ -1403,10 +1404,66 @@ asked for it. tswg does it (2026-11-08):
tailscaled only programs recently active peers into wireguard-go, mostly
because of per-peer cost in wireguard-go. Kernel peers are cheap and
hashed, so first try programming every peer. If that doesn't scale, add an
unknown-peer upcall from `wg_noise_handshake_consume_initiation` with a
short stash of the initiation for replay (to avoid a 5s `REKEY_TIMEOUT`
stall).
hashed, so the kernel data plane programs every peer, which holds up to
50,000 (2026-11-09; `TestLeanScale`, in a lab VM, adding 50,000 peers
with an endpoint and two allowed IPs each, 5,000 at a time):
| | 500 peers | 5,000 | 50,000 |
|---|---|---|---|
| kernel memory per peer | | 2.2 KB slab | 2.1 KB slab, no per-CPU (106 MB total) |
| add, per peer in bulk | 48 us | 35 us | 37 us |
| add one / remove one | 48 / 58 us | 59 / 54 us | 39 / 65 us |
| update one's endpoint / replace its allowed IPs | 10 / 13 us | 6 / 15 us | 26 / 80 us |
| rename one (key rotation) | 73 us | | 83 us |
| dump the device | 4 ms | 17 ms | 0.5-0.7 s |
| delete the device | 38 ms | 32 ms | 0.1-0.2 s |
The single-peer operations' growth from 5,000 to 50,000 is cache misses
in bigger tables and tries, not O(N) work: at 20,000 and 50,000, at 16
vCPUs, they're the same (22 and 23 us to update an endpoint, 64 and 79
to replace allowed IPs). Fixed for it: deleting a device
removed its peers one at a time, each waiting for an RCU grace period
(`netif_napi_del`) and three workqueue flushes, 6ms a peer, so 5 minutes
for 50,000; now all of them wait together (`peer_remove_dead_batch`),
1000x faster. Each peer's route cache (`dst_cache`) was per CPU, 40
bytes on each, so 5KB a peer on a 128-CPU host, 250MB for 50,000; it's
now allocated on the peer's first send. And the pubkey and index hash
tables have 2^16 and 2^17 buckets (1.5MB a device), where upstream's
2^11 and 2^13 had chains of 25 at 50,000 peers.
In tailscaled (tailscale.com `wgengine/kerneldp_linux.go`), each changed
peer (`Engine.SyncDevicePeer`, called for each peer a netmap delta
changes) used to trigger a full sync, walking every peer through
magicsock and the config source: O(N) per change, so O(N^2) for a
netmap of N changes. Now `syncPeer` syncs just that peer, O(1): its
magicsock path and config decide whether to add, update or remove it.
magicsock's kernel path hook now also fires when it adds or removes a
peer, so a peer magicsock learns of after its `SyncDevicePeer` is still
synced.
Key rotation is atomic: a peer that rotates its node key comes to
tailscaled as the removal of the old key and the addition of the new,
in either order, perhaps in different syncs, or as a full netmap. Done
as such, the peer's allowed IPs and traffic would be gone in between.
So tailscaled matches the two by the peer's Tailscale IP (`peerAddr`;
by its own map of pushed peers' addresses, and LocalBackend's peer by IP
lookup for the other order) and tells the kernel to rename the peer
(`TSWGPEER_A_NEW_PUBLIC_KEY`, `tswg_peer_rename`): in one step, under
`device_update_lock`, the peer moves in the pubkey table to its new key,
with its precomputed static-static and cookie keys recomputed and its
sessions cleared, keeping its allowed IPs, endpoint and queues; O(1).
`TestTSWGTailscaledKeyRotation` rotates a peer's key (`tailscale up
--force-reauth`) and checks that the other node's kernel always had
exactly one peer, that it renamed it, and that the tunnel works.
Not done: the device dump is O(N) by nature (0.5-0.7s at 50,000), and
tailscaled does one after each full sync (`refreshListenPort`); the
kernel's peer events, rate limited to one per peer per 2 seconds each
way, could be 50,000 a second with every peer busy; magicsock and
LocalBackend at 50,000 peers are as in TUN mode. If programming every
peer stops scaling, the fallback is an unknown-peer upcall from
`wg_noise_handshake_consume_initiation`, with a short stash of the
initiation for replay (to avoid a 5s `REKEY_TIMEOUT` stall).
### DKMS and fallback
@@ -1658,9 +1715,12 @@ Everything else:
(2026-11-08; it left a new kernel device without its key until a
later reconcile), in 29 rounds of the tests above, each with the
tunnel working on the first ping both ways.
- [ ] Scale to a 50,000-node tailnet: memory per peer, in the module and
in tailscaled's kernel data plane, and no O(N) work per peer change
(netlink pushes, reconcile, allowedips).
- [x] Scale to a 50,000-node tailnet (2026-11-09; see "Lazy peer
configuration"): 2.1 KB of kernel memory per peer, whatever the CPU
count; O(1) adds, removals, edits and atomic key rotations, in the
module and in tailscaled.
- [ ] tailscaled: skip the full device dump after each full sync, and
measure the kernel data plane's own memory per peer at 50,000.
- [x] Fallback to wireguard-go when tswg is unavailable (module not
loaded).
- [ ] Fallback on version or feature mismatch between tailscaled and
+1
View File
@@ -28,6 +28,7 @@ const struct nla_policy wireguard_wgallowedip_nl_policy[WGALLOWEDIP_A_FLAGS + 1]
const struct nla_policy tswg_peer_nl_policy[TSWGPEER_A_MAX + 1] = {
[TSWGPEER_A_FLAGS] = NLA_POLICY_MASK(NLA_U32, TSWGPEER_F_NO_ROAM | TSWGPEER_F_HYBRID),
[TSWGPEER_A_CLEAR_SESSION] = { .type = NLA_FLAG, },
[TSWGPEER_A_NEW_PUBLIC_KEY] = NLA_POLICY_EXACT_LEN(WG_KEY_LEN),
};
const struct nla_policy wireguard_wgpeer_nl_policy[WGPEER_A_TSWG + 1] = { /* TSWG */
@@ -121,6 +121,15 @@ enum tswgpeer_attribute {
* disco key.
*/
TSWGPEER_A_CLEAR_SESSION,
/*
* NLA_BINARY (32 bytes), set only: change the peer's public key to
* this one, atomically, keeping its allowed IPs, endpoint and other
* settings and dropping its sessions (as TSWGPEER_A_CLEAR_SESSION),
* for when a peer rotates its key. The rest of the peer's attributes
* in the same message apply after. Fails with EEXIST if another peer
* has the key.
*/
TSWGPEER_A_NEW_PUBLIC_KEY,
__TSWGPEER_A_LAST
};
+64 -8
View File
@@ -9,6 +9,7 @@
#include "socket.h"
#include "queueing.h"
#include "messages.h"
#include "cookie.h" /* TSWG */
#include "generated/netlink.h"
#include "tswg.h"
@@ -643,6 +644,9 @@ int tswg_set_device(struct wg_device *wg, struct nlattr *attr,
return 0;
}
static void tswg_peer_clear_session(struct wg_peer *peer);
static int tswg_peer_rename(struct wg_peer *peer, const u8 key[NOISE_PUBLIC_KEY_LEN]);
/* TSWG: tswg_set_peer applies the nested WGPEER_A_TSWG attribute to peer. */
int tswg_set_peer(struct wg_peer *peer, struct nlattr *attr)
{
@@ -653,23 +657,75 @@ int tswg_set_peer(struct wg_peer *peer, struct nlattr *attr)
NULL);
if (ret < 0)
return ret;
if (tb[TSWGPEER_A_NEW_PUBLIC_KEY]) {
ret = tswg_peer_rename(peer, nla_data(tb[TSWGPEER_A_NEW_PUBLIC_KEY]));
if (ret < 0)
return ret;
}
if (tb[TSWGPEER_A_FLAGS]) {
u32 flags = nla_get_u32(tb[TSWGPEER_A_FLAGS]);
WRITE_ONCE(peer->tswg_no_roam, !!(flags & TSWGPEER_F_NO_ROAM));
tswg_noise_set_hybrid(&peer->handshake, flags & TSWGPEER_F_HYBRID);
}
if (tb[TSWGPEER_A_CLEAR_SESSION]) {
/* As wg_pm_notification does for every peer before suspend. */
timer_delete(&peer->timer_zero_key_material);
wg_noise_handshake_clear(&peer->handshake);
wg_noise_keypairs_clear(&peer->keypairs);
/* Don't make the next handshake wait for REKEY_TIMEOUT. */
wg_noise_reset_last_sent_handshake(&peer->last_sent_handshake);
}
if (tb[TSWGPEER_A_CLEAR_SESSION])
tswg_peer_clear_session(peer);
return 0;
}
/* TSWG: tswg_peer_clear_session forgets peer's sessions; see tswg_set_peer. */
static void tswg_peer_clear_session(struct wg_peer *peer)
{
/* As wg_pm_notification does for every peer before suspend. */
timer_delete(&peer->timer_zero_key_material);
wg_noise_handshake_clear(&peer->handshake);
wg_noise_keypairs_clear(&peer->keypairs);
/* Don't make the next handshake wait for REKEY_TIMEOUT. */
wg_noise_reset_last_sent_handshake(&peer->last_sent_handshake);
}
/*
* TSWG: tswg_peer_rename changes peer's public key to key, atomically: the
* peer stays in allowedips, with its endpoint, settings and queues, so
* traffic to it isn't dropped, but its sessions, made with the old key,
* are cleared. All of it is O(1). The caller holds device_update_lock.
*/
static int tswg_peer_rename(struct wg_peer *peer, const u8 key[NOISE_PUBLIC_KEY_LEN])
{
struct wg_device *wg = peer->device;
struct wg_peer *other;
int ret = 0;
lockdep_assert_held(&wg->device_update_lock);
if (!crypto_memneq(key, peer->handshake.remote_static, NOISE_PUBLIC_KEY_LEN))
return 0;
other = wg_pubkey_hashtable_lookup(wg->peer_hashtable, key);
if (other) {
wg_peer_put(other);
return -EEXIST;
}
down_read(&wg->static_identity.lock);
if (wg->static_identity.has_identity &&
!memcmp(key, wg->static_identity.static_public, NOISE_PUBLIC_KEY_LEN)) {
ret = -EINVAL; /* our own key */
goto out;
}
tswg_peer_clear_session(peer);
wg_pubkey_hashtable_remove(wg->peer_hashtable, peer);
down_write(&peer->handshake.lock);
memcpy(peer->handshake.remote_static, key, NOISE_PUBLIC_KEY_LEN);
up_write(&peer->handshake.lock);
wg_noise_precompute_static_static(peer);
wg_cookie_init(&peer->latest_cookie);
wg_cookie_checker_precompute_peer_keys(peer);
wg_pubkey_hashtable_add(wg->peer_hashtable, peer);
/* An in-flight handshake may have raced the clearing above. */
tswg_peer_clear_session(peer);
out:
up_read(&wg->static_identity.lock);
return ret;
}
/* TSWG: tswg_get_peer adds peer's WGPEER_A_TSWG attribute to skb. */
int tswg_get_peer(struct sk_buff *skb, struct wg_peer *peer)
{
+3
View File
@@ -96,6 +96,9 @@ struct noise_handshake {
/* Protects all members except the immutable (after noise_handshake_
* init): remote_static, precomputed_static_static, static_identity.
* TSWG: but tswg_peer_rename changes remote_static (and so
* precomputed_static_static), under this lock and the device's
* device_update_lock, having first cleared the handshake and keypairs.
*/
struct rw_semaphore lock;
};
+33 -7
View File
@@ -33,8 +33,7 @@ struct wg_peer *wg_peer_create(struct wg_device *wg,
peer = kmem_cache_zalloc(peer_cache, GFP_KERNEL);
if (unlikely(!peer))
return ERR_PTR(ret);
if (unlikely(dst_cache_init(&peer->endpoint_cache, GFP_KERNEL)))
goto err;
/* TSWG: endpoint_cache is allocated on first send; see socket.c. */
peer->device = wg;
wg_noise_handshake_init(&peer->handshake, &wg->static_identity,
@@ -65,9 +64,6 @@ struct wg_peer *wg_peer_create(struct wg_device *wg,
pr_debug("%s: Peer %llu created\n", wg->dev->name, peer->internal_id);
return peer;
err:
kmem_cache_free(peer_cache, peer);
return ERR_PTR(ret);
}
struct wg_peer *wg_peer_get_maybe_zero(struct wg_peer *peer)
@@ -166,6 +162,37 @@ void wg_peer_remove(struct wg_peer *peer)
peer_remove_after_dead(peer);
}
/*
* TSWG: peer_remove_dead_batch is peer_remove_after_dead for all of the
* dead peers on list, with each of its waits (workqueue flushes and RCU
* grace periods) done once for all of them rather than once per peer.
* Removing a device's 5,000 peers took 30 seconds, 6ms per peer, mostly
* netif_napi_del's synchronize_net, so 50,000 would have taken 5 minutes.
*/
static void peer_remove_dead_batch(struct wg_device *wg, struct list_head *list)
{
struct wg_peer *peer, *temp;
list_for_each_entry(peer, list, peer_list) {
WARN_ON(!peer->is_dead);
wg_noise_keypairs_clear(&peer->keypairs);
wg_timers_stop(peer);
}
/* As in peer_remove_after_dead. */
flush_workqueue(wg->packet_crypt_wq);
flush_workqueue(wg->packet_crypt_wq);
list_for_each_entry(peer, list, peer_list) {
napi_disable(&peer->napi);
__netif_napi_del(&peer->napi);
}
synchronize_net(); /* for all the __netif_napi_dels */
flush_workqueue(wg->handshake_send_wq);
list_for_each_entry_safe(peer, temp, list, peer_list) {
--wg->num_peers;
wg_peer_put(peer);
}
}
void wg_peer_remove_all(struct wg_device *wg)
{
struct wg_peer *peer, *temp;
@@ -181,8 +208,7 @@ void wg_peer_remove_all(struct wg_device *wg)
list_add_tail(&peer->peer_list, &dead_peers);
}
synchronize_net();
list_for_each_entry_safe(peer, temp, &dead_peers, peer_list)
peer_remove_after_dead(peer);
peer_remove_dead_batch(wg, &dead_peers); /* TSWG */
}
static void rcu_release(struct rcu_head *rcu)
+10 -4
View File
@@ -15,8 +15,11 @@
struct wg_peer;
struct pubkey_hashtable {
/* TODO: move to rhashtable */
DECLARE_HASHTABLE(hashtable, 11);
/*
* TSWG: 2^16 buckets (512KB), for a 50,000-node tailnet, rather than
* upstream's 2^11, at which lookups walked chains of 25.
*/
DECLARE_HASHTABLE(hashtable, 16);
siphash_key_t key;
struct mutex lock;
};
@@ -31,8 +34,11 @@ wg_pubkey_hashtable_lookup(struct pubkey_hashtable *table,
const u8 pubkey[NOISE_PUBLIC_KEY_LEN]);
struct index_hashtable {
/* TODO: move to rhashtable */
DECLARE_HASHTABLE(hashtable, 13);
/*
* TSWG: 2^17 buckets (1MB), for a handshake and up to three keypairs
* each of 50,000 peers, rather than upstream's 2^13.
*/
DECLARE_HASHTABLE(hashtable, 17);
spinlock_t lock;
};
+19
View File
@@ -185,11 +185,30 @@ out:
#endif
}
/*
* TSWG: tswg_endpoint_cache_init allocates peer's endpoint_cache, on its
* first send. Upstream allocates it with the peer, but it's per CPU, 40
* bytes on each: 5KB a peer on a 128-CPU host, 250MB for a 50,000-node
* tailnet, most of whose peers never send. If it fails, sends go uncached
* (the dst_cache functions allow for that) until it doesn't.
*/
static void tswg_endpoint_cache_init(struct wg_peer *peer)
{
struct dst_cache tmp;
if (dst_cache_init(&tmp, GFP_ATOMIC))
return;
if (cmpxchg(&peer->endpoint_cache.cache, NULL, tmp.cache))
dst_cache_destroy(&tmp); /* another CPU won */
}
int wg_socket_send_skb_to_peer(struct wg_peer *peer, struct sk_buff *skb, u8 ds)
{
size_t skb_len = skb->len;
int ret = -EAFNOSUPPORT;
if (unlikely(!READ_ONCE(peer->endpoint_cache.cache))) /* TSWG */
tswg_endpoint_cache_init(peer);
read_lock_bh(&peer->endpoint_lock);
if (peer->endpoint.addr.sa_family == AF_INET)
ret = send4(peer->device, skb, &peer->endpoint, ds,