Commit Graph
18 Commits
Author SHA1 Message Date
Brad Fitzpatrick 003fef170a kernel/linux/DESIGN.md: record the 24-peer, MTU 1280 subnet router results
With 24 clients at MTU 1280, tswg forwards 40 Gbps through a c7i.48xlarge
(5.6x wireguard-go, 3x kernel WireGuard), using about 22 of its 192 CPUs,
and is then limited by EC2's packet rate allowance for the instance.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ifdb78f7d85d182a89a42c2f0f9f77bd14344fb59
2026-11-07 19:21:46 +00:00
Brad Fitzpatrick 95c00517eb kernel/linux/{scripts,DESIGN.md}: resize EC2 instances, and record 50 Gbps results
ec2.sh gets "resize TYPE N..." (for stopped instances) and "start N..."
(some instances), and numbers new instances after all existing ones,
stopped ones included.

With a c7i.48xlarge router (50 Gbps) and 8 clients, tswg forwards 31.5
Gbps at MTU 1280 (wireguard-go 7.2, kernel WireGuard 12.3 using 137 of
192 CPUs), limited by EC2's packet rate per NIC receive queue (about
715k pps, 7.3 Gbps of tunnel packets) with only 8 tunnel flows to spread
over 16 queues; and at line rate with a jumbo tunnel MTU, a data point
only, since MTU 1280 is the target.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I70744651f3385343bf8890607911d9b4e0f70620
2026-11-07 19:16:21 +00:00
Brad Fitzpatrick f810784988 kernel/linux/DESIGN.md: record the 20-peer subnet router results and what was tried
With 20 clients (160 flows), tswg as the subnet router forwards 2.2x (4
CPUs) and 3.6x (8 CPUs) what wireguard-go does, at 38-43% less CPU per
byte. wireguard-go stays at about 6.4 Gbps however many peers or CPUs it
has, so its limit is global, most likely its one UDP-reading goroutine.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I9274a49b47cdbdf9af06f1f9a1f6c8d04f126442
2026-11-07 18:58:15 +00:00
Brad Fitzpatrick df191a1199 kernel/linux/{scripts,DESIGN.md}: add the subnet router benchmark and its results
subnetbench.sh runs tailscale.com's TestLeanSubnetRouter on the EC2
instances for each router CPU count given. ec2.sh now turns off source
and destination checks on its instances, so that one can forward as a
subnet router does, and installs nftables (for the router's masquerade).

DESIGN.md records the results: with 4 clients, tswg as the subnet
router forwards 2.1x (4 CPUs) to 3.2x (8 CPUs) what wireguard-go does,
which uses about 3 CPUs however many it has, at the least CPU per byte
of any tunnel, and reaches the 25 Gbps link at 16 CPUs. Zero-copy
receive was tried there and was 7% worse, so isn't in.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I6b1da8bb8eaa17f9b20cbe36896426411b8b905f
2026-11-07 18:02:37 +00:00
Brad Fitzpatrick f889883537 kernel/linux/{scripts,DESIGN.md}: add EC2 instances for benchmarks, stopped when idle
The QEMU lab runs on a host that's itself a VM, which distorted latency
in particular. ec2.sh manages benchmark instances in the sandbox AWS
account (us-west-2; the org's SCP allows only some regions and requires
IMDSv2): a key pair, a security group open to this host and between its
members, a cluster placement group, and Ubuntu 24.04 instances with IPv6
that install the build tools for their own kernel. Everything is named
bradfitz-tswg... and tagged owner=claude-tswg. It writes the hosts file
that tailscale.com's lean harness takes in remote mode.

To keep costs down, ec2-idle-stop.sh, run every 5 minutes from a systemd
user timer on the always-on dev host, stops the instances once nothing
has touched ~/.cache/tslinux/ec2/last-used for 30 minutes. ec2.sh and the
harness touch it while in use, and "ec2.sh start" brings stopped
instances back.

First EC2 results (c7i.xlarge pair) are in DESIGN.md: throughput is
capped by EC2 at about 7.25 Gbps for both tswg and wireguard-go, with
tswg's receiver using up to 17% less CPU per byte; 1.6x the small
packets; a quarter of the RTT under load at p50; twice the idle RTT.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ibe770a30bea31cb79ac514aeb545f468576b5f05
2026-11-07 17:26:49 +00:00
Brad Fitzpatrick 06bdac96d3 kernel/linux/{DESIGN.md,scripts}: record the performance work, add leancompare.sh
DESIGN.md's Performance section now describes what tswg changes in the
data path and why, the results against wireguard-go (parity to 1.05x on
bulk TCP at 6-10% less CPU per byte, much better small-packet delivery,
worse idle and p99-under-load latency in VMs) and the tools for this
work. The TODO list gets the open performance items.

leancompare.sh runs the lean harness's interleaved A/B comparison for
RTT and TCP with 1 and 4 streams.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Icf8c7ca8bb3711a8172284382af9ce8c3e25ff3d
2026-11-07 14:38:57 +00:00
Brad Fitzpatrick 7648a28d11 kernel/linux/{scripts,DESIGN.md}: add leanbench.sh and record lean benchmark results
leanbench.sh runs tailscale.com's new lean data plane benchmark, which
compares wireguard-go, tswg, the kernel's WireGuard and no tunnel between
two QEMU VMs on a host bridge with vhost-net, without natlab, over IPv4
and IPv6. With -smoke it runs the quick correctness test instead.

The results overturn the natlab ones, which natlab's userspace network
had capped: kernel WireGuard (and tswg, identically) gets about a third
of wireguard-go's bulk TCP throughput, at 1.5x to 2.5x the CPU per byte,
without being CPU-bound. So outer UDP GSO/GRO and finding where the
kernel serializes are now the critical performance work.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I2a24de8d82370fc8dfb6a9af0971da20831ea307
2026-11-07 06:06:16 +00:00
Brad Fitzpatrick ac26c5bfce kernel/linux/{scripts,DESIGN.md}: add bench.sh and record first natlab benchmarks
bench.sh runs tailscale.com's TestTSWGBenchmark, which compares
tailscaled in kernel mode against tailscaled in TUN mode (wireguard-go)
on the same pair of natlab gokrazy VMs, and saves the results as JSON.

Record the first results: bulk TCP throughput and CPU per byte are about
the same (natlab caps throughput, and the kernel lacks outer UDP
GSO/GRO), while the kernel receives about 1.7x the small packets per
second at about 30% less CPU per packet and roughly halves latency under
load. Also record the plan for a lean benchmark harness without natlab,
whose userspace switch is expensive, and a flake seen when both peers
restart at once.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I69e696817f98e685576a30d766c42740d4d04810
2026-11-07 05:52:04 +00:00
Brad Fitzpatrick 0b23989672 kernel/linux/DESIGN.md: document the event-driven kernel data plane
Record how the kernel data plane now runs without polling: config
changes trigger full syncs, magicsock's kernel path hook (with a trust
expiry timer for idle sessions) triggers per-peer endpoint updates, and
the module's peer events drive disco key advertisements and activity
feedback, all applied by one goroutine.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I8c555a623e8bba7cf7561fe94e4bc9d17fd40003
2026-11-07 05:37:17 +00:00
Brad Fitzpatrick f7c80e0a32 kernel/linux/DESIGN.md: document TSMP and peer restarts in kernel mode
Record how tailscaled exchanges TSMP and its other generated packets with
peers in kernel mode (raw sockets on the tswg device, and a tstun
outbound inject hook), how it emulates wireguard-go's disco key
advertisement on session establishment, how it clears the kernel session
when a peer restarts, and that wireguard-go must never get peers in
kernel mode. Mark the packet filter and MagicDNS as deliberately
deferred.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I632c5a75b1c2e1dc74680faaa34bbd799e52fe74
2026-11-07 05:00:05 +00:00
Brad Fitzpatrick fe0c4864e9 kernel/linux/DESIGN.md: document the loopback bridge and tailscaled integration
DERP via 127.0.0.0/8 loopback endpoints turned out to need no kernel
changes; record how the bridge works (including peer relays and the
untrusted-path spray) and how tailscaled now drives tswg on the
tailscale.com bradfitz/tswg branch, with its known gaps, the biggest
being that kernel mode has no packet filter yet.

Also record that Tailscale's wireguard-go is about to start using a
hybrid post-quantum handshake, which tswg doesn't implement, so the
assumption that tswg is wire-compatible with every Tailscale peer no
longer holds, and the options for dealing with that.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I2447db2494bfd1a572f8ce4d093a42a1949aa3d4
2026-11-07 00:09:41 +00:00
Brad Fitzpatrick 189b463b86 kernel/linux/src: add a per-peer flag to stop roaming
Upstream WireGuard moves a peer's endpoint to the source of each
authenticated packet. In Tailscale, disco picks the path, so that would
fight it: a packet that arrives via DERP would steal the endpoint from a
good direct path.

Add TSWGPEER_F_NO_ROAM, set through TSWGPEER_A_FLAGS nested in a new
WGPEER_A_TSWG attribute (numbered 64, like WGDEVICE_A_TSWG) and reported
in dumps. With it set, the endpoint learned from handshake initiations,
handshake responses and data packets is ignored once the peer has an
endpoint, and only userspace changes it. A peer with no endpoint yet
still gets one from its first authenticated packet, so it stays
reachable if it contacts us before userspace has configured one.

The gate, tswg_peer_may_roam, sits at the two learned-endpoint paths
rather than in wg_socket_set_peer_endpoint, which netlink also uses for
explicit updates. The flag is applied before WGPEER_A_ENDPOINT in the
same request, so pinning and setting an endpoint are one step.

Tested by TestTSWG{Gokrazy,Ubuntu}NoRoam in tailscale.com's natlab
vmtests, which fail as expected with the gate disabled.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I621a6e86d3947e154c4cfa582aa845bb49d42b2f
2026-11-06 23:07:32 +00:00
Brad Fitzpatrick 7398e67950 kernel/linux/DESIGN.md: make performance the primary goal, with its own TODOs
Performance is why we're doing this, so say so in the Goal section and
add a Performance section: what we inherit from upstream WireGuard (inner
GSO on send, inner GRO via NAPI on receive, parallel crypto), what's
missing (outer UDP GSO on send and outer UDP GRO on receive, the kernel
analogues of what makes wireguard-go fast), why tailscaled shouldn't set
UDP_GRO on the shared socket in kernel mode, and how to measure.

Group the performance work at the top of the TODO list.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: If844d30a81f5f2fe52bc2fbf2b31d7bbbb9359c7
2026-11-06 23:00:12 +00:00
Brad Fitzpatrick 4c916c40ab kernel/linux/src: let userspace share its UDP socket with a tswg device
Tailscale needs disco, STUN and WireGuard to share one UDP 4-tuple, so
that the NAT mappings disco discovers are the ones WireGuard uses. So let
tailscaled create and bind the socket itself and hand its fd to the
device, as L2TP allows for its tunnel sockets.

The new TSWGDEVICE_A_SOCK_FD4 and TSWGDEVICE_A_SOCK_FD6 attributes, nested
in a new WGDEVICE_A_TSWG attribute numbered 64 to stay clear of future
upstream attributes, take the fd of a bound UDP socket (IPV6_V6ONLY for
IPv6). The device hooks the socket's UDP encap_rcv, consumes WireGuard
messages (a first byte of 1 through 4 and three zero bytes), and returns
everything else to the normal UDP path for userspace to read. Userspace
can send its own datagrams on the socket as usual.

The device keeps a reference to the struct sock, not the file, and drops
the socket from its encap_destroy hook when userspace closes it, so the
socket lives exactly as long as userspace's fd. Passing -1 detaches the
socket, restoring it to plain UDP, and the device goes back to creating
its own sockets. User sockets persist across ifdown/ifup, and changing
the listen port while one is attached fails with EBUSY. Dumps report
which families have a user socket in TSWGDEVICE_A_FLAGS.

A global mutex serializes attach and detach, including detach from
encap_destroy, which runs from the user's close() with none of our locks
held.

Tested by TestTSWGGokrazySharedSocket (7.2.8) and
TestTSWGUbuntuSharedSocket (6.8) in tailscale.com's natlab vmtests.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I2e7a97952ced03df4bc954ba398a49daf9128d41
2026-11-06 22:13:43 +00:00
Brad Fitzpatrick 7fccee622e kernel/linux/DESIGN.md: note that the gokrazy test kernel is bumped by bots
A gokrazy bot bumps github.com/gokrazy/kernel.amd64 to new kernel
releases and a tailscale.com cron bot bumps that go.mod dependency, so
the gokrazy vmtests follow upstream kernels automatically. Record that,
and that a failure after such a bump most likely means kernel API churn
needing a new compat shim.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I12428a124604edb65a61f101b72c87caed6c9332
2026-11-06 21:59:18 +00:00
Brad Fitzpatrick d3c49a3c40 kernel/linux/DESIGN.md: make gokrazy the primary test guest
Record the decision to test primarily on natlab's gokrazy image, which
boots a recent upstream kernel and has no userspace tools, as in the end
state where tailscaled drives the kernel entirely. Describe how its
module build tree is recreated and how the test agent loads and
configures tswg. Keep Ubuntu 24.04 as a secondary guest for distro and
DKMS-style coverage.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I89714b58c944be4f373f23471e3aeef11942b1f6
2026-11-06 21:50:15 +00:00
Brad Fitzpatrick d095c5b6ef kernel/linux/DESIGN.md: record compat, Go client and testbed decisions
Document the feature-probe compat framework, that the Go netlink client
and tswgctl live in tailscale.com on the bradfitz/tswg branch, and how
the natlab vmtests build, install and exercise tswg.ko. Update the TODO
list: TestTSWGManualTunnel now passes against Ubuntu 24.04's stock
6.8.0-106-generic kernel.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ie279a07e251d1ee81cdc91478f3eec9d0dd46afb
2026-11-06 21:07:52 +00:00
Brad Fitzpatrick 24131ef7cd kernel/linux/DESIGN.md: add initial design and TODO list
Record the decisions made so far: an out-of-tree C fork of in-tree
WireGuard (no Rust, no upstreaming), a shared UDP socket passed in from
tailscaled, explicit endpoints with auto-roaming disabled, DERP via
127.0.0.0/8 loopback endpoints, and userspace spray. Also describe the
natlab vmtest testbed and the TODO list.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ie18d356974a0a0d8fce19674b590861144be8cea
2026-11-06 20:53:38 +00:00