Commit Graph
6 Commits
Author SHA1 Message Date
Brad Fitzpatrick 867eaea830 kernel/linux/src: recruit fewer crypt workers, and go inline for slow peers
One 7 Gbps tunnel between 128-CPU hosts had its crypt work spread over
33 CPUs, each about 15% busy, mostly waking up and going back to sleep,
for up to 1.6x wireguard-go's CPU per GB: a worker was recruited
whenever more than one GRO packet's worth waited, which GRO's bursts
always did. Now one is recruited only when more than recruit_factor
(default 16) times that waits for each worker draining, capped at a
quarter of the ring, and from a compact set of CPUs after the caller's,
twice as many as were draining at once in the last 10ms, rather than
round robin over every CPU.

And for request/response traffic, the handoff to a worker is on each
response's critical path, so a local socket's bulk sends to a peer
sending under tx_inline_mbps (default 2000) are encrypted inline, and
rx_inline_mbps drops from 4000 to 1000, so faster peers' receives go
the parallel way sooner.

Between two c6in.32xlarge hosts on EC2, at EC2's per-flow limit, tswg
now uses 0.52-1.02x wireguard-go's CPU per GB (was up to 1.6x), and one
HTTP-like stream gets 3.0 Gbps to its 3.3 (was 2.5). Crawling 37 peers
goes from 76 to 89 Gbps; routing and serving are unchanged.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I777adf1eadb4f309c62111ed3b2695a816350c56
2026-11-08 23:20:47 +00:00
Brad Fitzpatrick 96695d4464 kernel/linux/src: scale a big host serving and crawling many peers
A 128-CPU host on the tailnet, serving HTTP-like responses to 37 peers
or crawling them all at once, was held back by three things, each fixed
here:

Sending by combining: a peer's encrypted packets went out from its tx
worker, one work item on one CPU at a time, which on a NIC without UDP
segmentation offload (EC2's ENA) also segments them, so the CPUs the
tx workers landed on ran at 90% and more. Now the encrypt worker that
makes an entry ready sends what's ready, in order, itself, unless
another CPU is sending the peer's (a trylock), which then sends it.

Local sockets' lone packets (TCP's ACKs, small requests) are encrypted
and sent inline, when nothing of the peer's is on its way through the
workers to be overtaken: a crawler spent 7% of its CPU on the encrypt
ring's lock for its ACKs.

Steering: with more peers than NIC queues, RSS puts several peers' UDP
flows on some CPUs, and receiving inline left those at 100% and the
rest idle. When a CPU is over rx_steer_busy (default 90%) busy, by its
idle time, or its softirq work runs out of budget, its peers' packets
go to each peer's home CPU (one not receiving from a NIC, by hash)
through a lockless per-CPU list and that CPU's NAPI: RPS, per peer, for
our packets, when needed. A peer changes between inline, steered and the
parallel way only when the way it was going is empty, so as not to
reorder.

On EC2 (c6in.32xlarge, 32 NIC queues, MTU 1280): serving goes from
84-98 to 133-137 Gbps (plain TCP without a tunnel: 152; wireguard-go:
5.5), at 3.9 CPU-s/GB, and crawling from 41-51 to 80-83 (plain: 128;
wireguard-go: 8.5), where the NIC's busiest queues, with two or three
peers each, are at ENA's limit of about 680k packets per second.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I011ab98d61296b5beef172ae1581d463820179a5
2026-11-08 11:52:01 +00:00
Brad Fitzpatrick 925022517f kernel/linux/src: run forwarded packets and most received ones to completion
On a 128-CPU subnet router sending big responses to hundreds of flows,
75-83% of the router's CPU spun on the encrypt ring's consumer lock.
Upstream's design hands every packet to a crypt worker on another CPU,
through a ring all CPUs share, and back through the peer's in-order
queue to a third, so that one flow's packets can be encrypted in
parallel and still go out in order. Forwarded packets don't need that:
RSS already spreads flows over CPUs and keeps each in order on one.

So tswg_xmit_now encrypts and sends packets with no socket (forwarded)
right in wg_xmit, on the CPU they arrived on, as IPsec does: no queues,
rings or other CPUs. Local sockets' packets still go to the workers, so
that one flow is encrypted on several CPUs; much of a TCP sender's
sending is in softirq too, as ACKs arrive, and inlining it cost a
stream half its throughput. Module parameter tx_inline: 0 never, 1
forwarded (the default), 2 always.

And tswg_receive_now decrypts a peer's packets in encap_rcv, on the CPU
its UDP flow arrives on, delivering them through the device's per-CPU
GRO cells, as VXLAN does, rather than through the decrypt ring, a
worker, the peer's in-order queue and its NAPI. That's one CPU per peer,
which a fast one outruns (a stream: 12.2 -> 6.4 Gbps in the lab), so a
peer receiving more than rx_inline_mbps (default 4000) over 10ms goes
the parallel way for at least 100ms, coming back once that way has
delivered what it had, so that switching doesn't reorder.

On EC2's 128-CPU c6in router with 37 clients: HTTP-like fetches go from
37 to 69 Gbps, and 27 to 3.7 CPU-s/GB; uploads from 66 to 68 Gbps, and
6.8 to 3.1 CPU-s/GB; wireguard-go does 6.2 either way. The 32 CPUs
taking the NIC's interrupts are now the limit, which RPS lifts for
fetches to 78 Gbps. In the lab, one stream through a tunnel is
unchanged at 12 Gbps, and RTT under load is down from 1.5 to 1.1ms.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I8b1cbbed3c87a8e6a9a7cf8a91a2c18dcfb4e62a
2026-11-08 06:48:49 +00:00
Brad Fitzpatrick 1b969a6b3a kernel/linux/src: multicast peer events so that userspace needn't poll
tailscaled was dumping the device every second to notice new sessions
(to send its disco key advertisement, as wireguard-go does) and traffic
on direct paths (to keep magicsock heartbeating active sessions).

Instead, multicast TSWG_CMD_PEER_EVENT messages on a new "events"
genetlink group, in the device's network namespace: one when a new
session is established (from wg_timers_session_derived), and TX and RX
activity notices, each rate limited to one per peer every 2 seconds with
a single comparison on the packet path. Messages are only built when
someone is listening.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I08bdaa23a9f9b01a0bb3a4404296cae5961cde86
2026-11-07 05:37:16 +00:00
Brad Fitzpatrick 189b463b86 kernel/linux/src: add a per-peer flag to stop roaming
Upstream WireGuard moves a peer's endpoint to the source of each
authenticated packet. In Tailscale, disco picks the path, so that would
fight it: a packet that arrives via DERP would steal the endpoint from a
good direct path.

Add TSWGPEER_F_NO_ROAM, set through TSWGPEER_A_FLAGS nested in a new
WGPEER_A_TSWG attribute (numbered 64, like WGDEVICE_A_TSWG) and reported
in dumps. With it set, the endpoint learned from handshake initiations,
handshake responses and data packets is ignored once the peer has an
endpoint, and only userspace changes it. A peer with no endpoint yet
still gets one from its first authenticated packet, so it stays
reachable if it contacts us before userspace has configured one.

The gate, tswg_peer_may_roam, sits at the two learned-endpoint paths
rather than in wg_socket_set_peer_endpoint, which netlink also uses for
explicit updates. The flag is applied before WGPEER_A_ENDPOINT in the
same request, so pinning and setting an endpoint are one step.

Tested by TestTSWG{Gokrazy,Ubuntu}NoRoam in tailscale.com's natlab
vmtests, which fail as expected with the gate disabled.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I621a6e86d3947e154c4cfa582aa845bb49d42b2f
2026-11-06 23:07:32 +00:00
Brad Fitzpatrick af92b849a7 kernel/linux/src: fork WireGuard as the out-of-tree tswg module
Copy upstream/drivers/net/wireguard into src and rename only the
user-visible identifiers: the module (and thus the rtnl link kind and
device type, which come from KBUILD_MODNAME) is now tswg, and the
genetlink family is "tswg". The uapi header moves to
src/include/uapi/linux/tswg.h so it can't collide with the kernel's.
Internal wg_ symbols keep their names to keep upstream merges easy; they
aren't exported, so tswg.ko can coexist with wireguard.ko.

Add an out-of-tree Kbuild and top-level Makefile, plus
scripts/ubuntu-headers.sh, which fetches and unpacks an Ubuntu kernel's
headers so we can build for the natlab Ubuntu guest without installing
anything.

This still only builds against 7.4-era kernels; compat shims follow.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ia36ef82a53efbbcdb301ed069785162372efda55
2026-11-06 20:55:25 +00:00