From 867eaea830bc429eb71aa931de84d1d31cde078e Mon Sep 17 00:00:00 2001 From: Brad Fitzpatrick Date: Sun, 8 Nov 2026 23:20:47 +0000 Subject: [PATCH] kernel/linux/src: recruit fewer crypt workers, and go inline for slow peers One 7 Gbps tunnel between 128-CPU hosts had its crypt work spread over 33 CPUs, each about 15% busy, mostly waking up and going back to sleep, for up to 1.6x wireguard-go's CPU per GB: a worker was recruited whenever more than one GRO packet's worth waited, which GRO's bursts always did. Now one is recruited only when more than recruit_factor (default 16) times that waits for each worker draining, capped at a quarter of the ring, and from a compact set of CPUs after the caller's, twice as many as were draining at once in the last 10ms, rather than round robin over every CPU. And for request/response traffic, the handoff to a worker is on each response's critical path, so a local socket's bulk sends to a peer sending under tx_inline_mbps (default 2000) are encrypted inline, and rx_inline_mbps drops from 4000 to 1000, so faster peers' receives go the parallel way sooner. Between two c6in.32xlarge hosts on EC2, at EC2's per-flow limit, tswg now uses 0.52-1.02x wireguard-go's CPU per GB (was up to 1.6x), and one HTTP-like stream gets 3.0 Gbps to its 3.3 (was 2.5). Crawling 37 peers goes from 76 to 89 Gbps; routing and serving are unchanged. Updates #426 Signed-off-by: Brad Fitzpatrick Change-Id: I777adf1eadb4f309c62111ed3b2695a816350c56 --- kernel/linux/DESIGN.md | 82 +++++++++++++++++++++++++++++++------ kernel/linux/src/device.h | 6 +++ kernel/linux/src/peer.h | 3 ++ kernel/linux/src/queueing.c | 5 +++ kernel/linux/src/queueing.h | 70 ++++++++++++++++++++++++++++--- kernel/linux/src/receive.c | 11 +++-- kernel/linux/src/send.c | 53 +++++++++++++++++++++--- 7 files changed, 203 insertions(+), 27 deletions(-) diff --git a/kernel/linux/DESIGN.md b/kernel/linux/DESIGN.md index 898a0eab2..e8c2e5f78 100644 --- a/kernel/linux/DESIGN.md +++ b/kernel/linux/DESIGN.md @@ -242,7 +242,17 @@ the host's CPU-s/GB: (4.1). And one fetch stream gets 2.7 Gbps to wireguard-go's 3.5. On a 128-CPU host, tswg's parallel paths (crypt workers, a fast peer's receive) spread one tunnel's work over far more CPUs than 7 Gbps - needs; see the TODO list. + needs. **Fixed by changes 17-18** (same day, same pair, interleaved + runs, Gbps and CPU-s/GB, sender / receiver): + + | load | tswg | wireguard-go | tswg/wggo CPU | + |---|---|---|---| + | bulk, 1 stream | 7.2; 2.1 / 2.6 | 7.1; 4.1 / 4.9 | 0.52x / 0.52x | + | bulk, 16 | 7.3; 3.1 / 4.2 | 7.1; 4.2 / 5.3 | 0.74x / 0.79x | + | bulk, 128 | 7.4; 4.1 / 5.9 | 7.4; 4.4 / 5.8 | 0.94x / 1.02x | + | fetch, 1 stream | 3.0; 3.4 | 3.3; 5.5 | 0.63x (0.91x the throughput) | + | fetch, 16 | 7.3; 3.0 | 7.2; 3.7 | 0.80x | + | fetch, 128 | 7.2; 3.9 | 6.9; 4.3 | 0.91x | tswg in the same sweep (after a reboot, before change 16 below, so on the unlucky NIC queue spread described there) reached 7.3, 29, 71 and @@ -256,7 +266,15 @@ each boot, and with only 37 peers' UDP flows, one boot spread them over results more than any of the work after change 12 (uploads 68 -> 28 Gbps, serving 133 -> 99, crawling 81 -> 42), and doesn't affect wireguard-go's, which don't reach the NIC's limits. Benchmarks with few -peers should check the spread (`ethtool -S`'s `queue_N_rx_cnt`). +peers should check the spread (`ethtool -S`'s `queue_N_rx_cnt`); change +19 rebalances it. + +After changes 17-19 (another boot; `TSWG_SUBNET_RSS_BALANCE` had +rebalanced the router's NIC): upload 74, fetch 70, serve 131 and crawl +97 Gbps, at 4.8, 4.3, 3.3 and 6.7 CPU-s/GB. Squeezing the router's RSS +table onto 11 queues (`ethtool -X equal 11`, like the unlucky boot), +upload fell to 64 and crawl to 72; balancing brought them back to 72 +and 78. ### What we inherit from upstream WireGuard @@ -457,6 +475,47 @@ this order. Each step's throughput is bulk TCP, 1 stream, IPv4, 4 vCPUs. and a peer's home is the least loaded of 8 spread from its hash, kept while under `rx_steer_busy`. On the unlucky boot's 11 queues: uploads 28 -> 44 Gbps, crawling 42 -> 51, serving 99 -> 101. +17. Fewer crypt workers for a tunnel (2026-11-08). One 7 Gbps tunnel on + a 128-CPU host had its crypt work on 33 CPUs, each about 15% busy, + mostly waking up and going back to sleep, with twice wireguard-go's + context switches: up to 1.6x its CPU per GB. A worker was recruited + whenever more than a batch's worth (one GRO packet) waited, which + GRO's bursts always did. Now another is recruited only when more + than `recruit_factor` (default 16) times that waits for each worker + draining (but never more than a quarter of the ring), and from a + compact set of CPUs after the caller's, twice as many as were + draining at once in the last 10ms (`tswg_crypt_recruit_cpu`), rather + than round robin over all CPUs. Bulk, 16 streams: sender 4.4 -> 2.9 + CPU-s/GB, receiver 7.1 -> 4.2, at the same throughput. Tried and not + kept: crypt workers polling for 20-100us before sleeping (more CPU); + sorting a peer's decrypted packets by flow before GRO (GRO already + merged better than wireguard-go's: 38 TCP segments per MB to its + 50); running the peer's NAPI straight from the decrypt worker. +18. Inline sends and receives for slow peers. A local socket's bulk + (GSO) sends to a peer sending under `tx_inline_mbps` (default 2000) + over 10ms are encrypted and sent inline, as forwarded packets are + (change 12), rather than handed to a worker: for request/response + traffic, that handoff is on every response's critical path. And + `rx_inline_mbps` is now 1000 (from 4000), so a peer receiving over + that goes the parallel way sooner. One HTTP-like stream between big + hosts: 2.5 -> 3.0 Gbps (wireguard-go 3.3); crawling 37 peers, 76 -> + 89 (at 5.9 -> 7.1 CPU-s/GB); routers and servers unchanged. Not + TCP small queues: without `tswg_move_owner`, the stream was no + faster. +19. Balancing the NIC's RSS table (tailscale.com + `wgengine/tswg/rss`). With few peers, the RSS hash key decides how + many queues their flows get (change 16's boots: 11 or 23 of 32), and + the busiest queues, at ENA's per-queue packet rate limit (about + 700k/s), cap the host. ENA's hash couldn't be reproduced from the key + and table it reports (no variant of Toeplitz on the 2- or 4-tuple + matched), so `rss.Balance` measures: per-queue packet rates from the + driver's counters, then each busy queue's table entries' loads by + pointing them at idle queues for 300ms, then a bin-packing of the + entries onto the queues. Entries carry several flows (37 flows used + 19 of 128 entries), so first it tries 8 random hash keys with an even + table, keeping the one whose busiest queue is lightest. Each move can + reorder a few of a flow's packets. The bench agent's + `/rss-balance` runs it; tailscaled doesn't yet (see the TODO list). The lean harness's VMs have guest USO off (decided 2026-11-07): with it, the host hands VM-to-VM UDP GSO packets to the receiving guest still @@ -1505,16 +1564,15 @@ Performance (the primary motivation; see "Performance"): - [ ] Forwarded packets' transmit could steer like received ones (change 15) when the NIC's CPUs are busy, instead of RPS: fetches through a 128-CPU router with 32 NIC queues gained 13% from RPS. -- [ ] One tunnel on a big host: tswg uses up to 2x wireguard-go's CPU per - GB at EC2's per-flow limit, more with more streams, and one fetch - stream gets 0.77x its throughput (see "wireguard-go at its best"). - The crypt workers and a fast peer's parallel receive spread over - many more CPUs than the rate needs; recruit fewer, and look at - request/response latency. -- [ ] Few peers on a NIC with RSS: the random hash key decides how many - queues they get (11 or 23 of 32, on two boots with 37 peers), and - so tswg's throughput; steering helps past the NIC, but not the - queues themselves. +- [x] One tunnel on a big host: tswg used up to 2x wireguard-go's CPU + per GB, and one fetch stream got 0.77x its throughput; now + 0.52-1.02x, and 0.91x (changes 17-18). +- [x] Few peers on a NIC with RSS: the random hash key decided how many + queues they got; `rss.Balance` (change 19) rebalances. +- [ ] tailscaled: run `rss.Balance` when the kernel data plane is busy + and its NIC's queues are uneven (needs CAP_NET_ADMIN, which it + has), at most every minute or so, since each pass moves flows. +- [ ] One request/response stream is still 0.91x wireguard-go's. - [ ] Page zeroing (`init_on_alloc`, on in Ubuntu's kernels) is 5-13% of a big host's CPU, for pages we fill entirely: a page pool for the encrypted and decrypted packets' pages would recycle them unzeroed. diff --git a/kernel/linux/src/device.h b/kernel/linux/src/device.h index d37e44e42..c2d0fdb28 100644 --- a/kernel/linux/src/device.h +++ b/kernel/linux/src/device.h @@ -53,6 +53,12 @@ struct crypt_queue { struct ptr_ring ring; struct multicore_worker __percpu *worker; int last_cpu; + /* TSWG: the workers' CPUs; see tswg_crypt_recruit_cpu */ + atomic_t tswg_active; /* workers draining now */ + unsigned int tswg_peak; /* most draining at once, this window */ + unsigned int tswg_width; /* CPUs to recruit from */ + unsigned int tswg_next; /* the next of them to recruit */ + unsigned long tswg_width_jiffy; /* when tswg_width was last set */ }; struct prev_queue { diff --git a/kernel/linux/src/peer.h b/kernel/linux/src/peer.h index 8e9693044..410ce4e47 100644 --- a/kernel/linux/src/peer.h +++ b/kernel/linux/src/peer.h @@ -74,6 +74,9 @@ struct wg_peer { spinlock_t tswg_tx_lock; bool tswg_tx_more; /* tx_queue may have more to send */ /* TSWG: bytes received since jiffy tswg_rx_jiffy; see tswg_rx_inline_ok */ + /* TSWG: local bytes sent since jiffy tswg_tx_jiffy; see tswg_tx_slow */ + unsigned long tswg_tx_jiffy; + unsigned int tswg_tx_bytes; unsigned long tswg_rx_jiffy; unsigned int tswg_rx_bytes; unsigned long tswg_rx_parallel_until; /* jiffies; see tswg_rx_path */ diff --git a/kernel/linux/src/queueing.c b/kernel/linux/src/queueing.c index ae950573f..4f19a8c47 100644 --- a/kernel/linux/src/queueing.c +++ b/kernel/linux/src/queueing.c @@ -6,6 +6,11 @@ #include "queueing.h" #include +/* TSWG: recruit_factor scales the backlog that recruits a crypt worker. */ +unsigned int tswg_recruit_factor = 16; +module_param_named(recruit_factor, tswg_recruit_factor, uint, 0644); +MODULE_PARM_DESC(recruit_factor, "entries waiting per draining crypt worker, times the queue's backlog, to recruit another"); + struct multicore_worker __percpu * wg_packet_percpu_multicore_worker_alloc(work_func_t function, void *ptr) { diff --git a/kernel/linux/src/queueing.h b/kernel/linux/src/queueing.h index 6d35bcda1..d9c7b2415 100644 --- a/kernel/linux/src/queueing.h +++ b/kernel/linux/src/queueing.h @@ -159,6 +159,7 @@ static inline int wg_cpumask_next_online(int *last_cpu) * are lists of packets, and decrypt entries can be packets merged by GRO). */ #define TSWG_PACKET_BACKLOG 16 +extern unsigned int tswg_recruit_factor; /* see tswg_crypt_kick */ #define TSWG_BATCH_BACKLOG 1 /* @@ -178,6 +179,39 @@ static inline bool tswg_worker_active(struct multicore_worker *w) return READ_ONCE(w->tswg_busy) || work_pending(&w->work); } +/* + * TSWG: tswg_crypt_recruit_cpu returns the CPU of the next worker to + * recruit: one of the tswg_width CPUs after the caller's, in turn. Upstream + * recruits the next CPU's worker every time, round robin, so on a 128-CPU + * host, one 7 Gbps tunnel's work spread over all 128, each a few percent + * busy and paying the wakeups, IPIs and cold caches of that, for up to 2x + * wireguard-go's CPU per GB. tswg_width is twice the most workers that + * were draining at once in the last TSWG_WIDTH_WINDOW, plus one (and at + * least 2), so it grows as fast as demand does, and shrinks after. + */ +#define TSWG_WIDTH_WINDOW max(HZ / 100, 1) + +static inline int tswg_crypt_recruit_cpu(struct crypt_queue *queue) +{ + unsigned int n = num_online_cpus(), self = raw_smp_processor_id(); + unsigned int width, k, idx; + unsigned long now = jiffies; + + if (n <= 1) + return self; + if (time_after_eq(now, READ_ONCE(queue->tswg_width_jiffy) + TSWG_WIDTH_WINDOW)) { + WRITE_ONCE(queue->tswg_width, 2 * READ_ONCE(queue->tswg_peak) + 1); + WRITE_ONCE(queue->tswg_peak, atomic_read(&queue->tswg_active)); + WRITE_ONCE(queue->tswg_width_jiffy, now); + } + width = clamp(READ_ONCE(queue->tswg_width), 2U, n - 1); + k = READ_ONCE(queue->tswg_next) + 1; + WRITE_ONCE(queue->tswg_next, k); + /* The caller's index among the online CPUs. */ + idx = bitmap_weight(cpumask_bits(cpu_online_mask), self); + return cpumask_nth((idx + 1 + k % width) % n, cpu_online_mask); +} + /* * tswg_crypt_kick makes sure some worker will consume what was just added * to queue's ring, recruiting another one if more than backlog entries @@ -201,12 +235,21 @@ static inline void tswg_crypt_kick(struct crypt_queue *queue, queue_work_on(cpu, wq, &w->work); return; } - if (tswg_crypt_backlog(&queue->ring) <= backlog) + /* + * TSWG: recruit only when more than recruit_factor times + * backlog is waiting for each worker draining. GRO hands us + * its packets in bursts, and recruiting a worker for each one + * spread one 7 Gbps tunnel over 33 CPUs of a 128-CPU host, + * each 15% busy, mostly waking and going back to sleep. + */ + if (tswg_crypt_backlog(&queue->ring) <= + min_t(unsigned int, backlog * READ_ONCE(tswg_recruit_factor) * + max_t(int, atomic_read(&queue->tswg_active), 1), + queue->ring.size / 4)) return; } - cpu = wg_cpumask_next_online(&queue->last_cpu); - if (cpu == raw_smp_processor_id() && num_online_cpus() > 1) - cpu = wg_cpumask_next_online(&queue->last_cpu); + cpu = tswg_crypt_recruit_cpu(queue); + WRITE_ONCE(queue->last_cpu, cpu); w = per_cpu_ptr(queue->worker, cpu); if (!tswg_worker_active(w)) queue_work_on(cpu, wq, &w->work); @@ -222,12 +265,29 @@ static inline bool tswg_crypt_worker_idle(struct multicore_worker *w, { WRITE_ONCE(w->tswg_busy, false); smp_mb(); /* clear tswg_busy before checking the ring; see above */ - if (likely(__ptr_ring_empty(&queue->ring))) + if (likely(__ptr_ring_empty(&queue->ring))) { + atomic_dec(&queue->tswg_active); return true; + } WRITE_ONCE(w->tswg_busy, true); return false; } +/* + * tswg_crypt_worker_start is called by a crypt worker as it starts to + * drain queue's ring; tswg_crypt_worker_idle undoes it. + */ +static inline void tswg_crypt_worker_start(struct multicore_worker *w, + struct crypt_queue *queue) +{ + unsigned int n; + + WRITE_ONCE(w->tswg_busy, true); + n = atomic_inc_return(&queue->tswg_active); + if (n > READ_ONCE(queue->tswg_peak)) + WRITE_ONCE(queue->tswg_peak, n); +} + /* * TSWG: inline crypto. Handing a packet to a crypt worker on another CPU * costs an IPI, and, if that CPU is idle, waking it, which in a VM means a diff --git a/kernel/linux/src/receive.c b/kernel/linux/src/receive.c index e556970e7..fc88ff363 100644 --- a/kernel/linux/src/receive.c +++ b/kernel/linux/src/receive.c @@ -932,7 +932,7 @@ void wg_packet_decrypt_worker(struct work_struct *work) struct crypt_queue *queue = w->ptr; struct sk_buff *skb; - WRITE_ONCE(w->tswg_busy, true); /* TSWG */ + tswg_crypt_worker_start(w, queue); /* TSWG */ again: while ((skb = ptr_ring_consume_bh(&queue->ring)) != NULL) { enum packet_state state = @@ -963,8 +963,11 @@ again: * But that's one CPU per peer, which a single fast peer outruns: one TCP * stream through a tunnel fell from 12.2 to 6.6 Gbps in the lab. So a peer * goes the parallel way while it's receiving more than rx_inline_mbps - * (module parameter; 0 for never inline), over TSWG_RX_WINDOW (a jiffy was - * too short: GRO's bursts put peers of a router averaging 2 Gbps over 4), + * (module parameter, default 1000; 0 for never inline), over TSWG_RX_WINDOW + * (a jiffy was too short: GRO's bursts put peers of a router averaging 2 + * Gbps over 4). At 4000, one HTTP-like stream between big hosts got 2.5 + * Gbps to wireguard-go's 3.3, and a crawler of 37 peers 76 Gbps; at 1000, + * 2.9 and 89, with routers and servers unchanged. * and for TSWG_RX_PARALLEL_FOR after (switching back and forth often cost * more than either way), and back only once that way has delivered all it * had, so that switching doesn't reorder the peer's packets; see @@ -974,7 +977,7 @@ again: #define TSWG_RX_WINDOW max(HZ / 100, 1) #define TSWG_RX_PARALLEL_FOR (HZ / 10) -static unsigned int tswg_rx_inline_mbps = 4000; +static unsigned int tswg_rx_inline_mbps = 1000; module_param_named(rx_inline_mbps, tswg_rx_inline_mbps, uint, 0644); MODULE_PARM_DESC(rx_inline_mbps, "receive a peer's packets where they arrive while it's below this rate (0: never)"); diff --git a/kernel/linux/src/send.c b/kernel/linux/src/send.c index 3e18107f4..15f80e69d 100644 --- a/kernel/linux/src/send.c +++ b/kernel/linux/src/send.c @@ -870,7 +870,7 @@ void wg_packet_encrypt_worker(struct work_struct *work) struct crypt_queue *queue = w->ptr; struct sk_buff *first; - WRITE_ONCE(w->tswg_busy, true); /* TSWG */ + tswg_crypt_worker_start(w, queue); /* TSWG */ again: while ((first = ptr_ring_consume_bh(&queue->ring)) != NULL) { enum packet_state state = PACKET_STATE_CRYPTED; @@ -972,6 +972,39 @@ static int tswg_tx_inline = 1; module_param_named(tx_inline, tswg_tx_inline, int, 0644); MODULE_PARM_DESC(tx_inline, "encrypt and send where packets arrive: 0 never, 1 forwarded ones, 2 always"); +/* + * TSWG: a local socket's bulk (GSO) sends to a peer sending under + * tx_inline_mbps go inline too. Handing them to a crypt worker costs each + * burst a wakeup and a handoff, which for request and response traffic is + * on every response's critical path: one HTTP-like stream between two big + * hosts got 2.5 Gbps to wireguard-go's 3.3 that way, 3.0 inline. But one + * CPU can't keep up with a fast stream (7.2 -> 6.2 Gbps), so above the + * rate, over TSWG_TX_WINDOW, the workers share it. 0 turns it off. + */ +static unsigned int tswg_tx_inline_mbps = 2000; +module_param_named(tx_inline_mbps, tswg_tx_inline_mbps, uint, 0644); +MODULE_PARM_DESC(tx_inline_mbps, "encrypt local bulk sends inline to a peer sending below this rate (0: never)"); + +#define TSWG_TX_WINDOW max(HZ / 100, 1) + +/* + * tswg_tx_slow counts len bytes sent to peer and reports whether it's + * sending below tx_inline_mbps. + */ +static bool tswg_tx_slow(struct wg_peer *peer, unsigned int len) +{ + unsigned int mbps = READ_ONCE(tswg_tx_inline_mbps); + unsigned long now = jiffies; + + if (now - READ_ONCE(peer->tswg_tx_jiffy) >= TSWG_TX_WINDOW) { + WRITE_ONCE(peer->tswg_tx_jiffy, now); + WRITE_ONCE(peer->tswg_tx_bytes, 0); + } + WRITE_ONCE(peer->tswg_tx_bytes, READ_ONCE(peer->tswg_tx_bytes) + len); + return mbps && (u64)READ_ONCE(peer->tswg_tx_bytes) * 8 * HZ <= + (u64)mbps * 1000000 * TSWG_TX_WINDOW; +} + bool tswg_xmit_now(struct wg_peer *peer, struct sk_buff_head *packets) { int mode = READ_ONCE(tswg_tx_inline); @@ -982,11 +1015,19 @@ bool tswg_xmit_now(struct wg_peer *peer, struct sk_buff_head *packets) if (!mode) return false; - if (mode == 1 && packets->next->sk && - (!skb_queue_is_last(packets, packets->next) || - skb_is_gso(packets->next) || - atomic_read(&peer->tx_queue.count))) - return false; + if (mode == 1 && packets->next->sk) { + unsigned int len = 0; + bool lone; + + skb_queue_walk(packets, skb) + len += skb->len; + lone = skb_queue_is_last(packets, packets->next) && + !skb_is_gso(packets->next); + /* Either way, not past what's on its way through the workers. */ + if ((!tswg_tx_slow(peer, len) && !lone) || + atomic_read(&peer->tx_queue.count)) + return false; + } /* Not past packets waiting for a handshake, nor without a key. */ if (!skb_queue_empty_lockless(&peer->staged_packet_queue)) return false;