kernel/linux/{scripts,DESIGN.md}: add the subnet router benchmark and its results

subnetbench.sh runs tailscale.com's TestLeanSubnetRouter on the EC2
instances for each router CPU count given. ec2.sh now turns off source
and destination checks on its instances, so that one can forward as a
subnet router does, and installs nftables (for the router's masquerade).

DESIGN.md records the results: with 4 clients, tswg as the subnet
router forwards 2.1x (4 CPUs) to 3.2x (8 CPUs) what wireguard-go does,
which uses about 3 CPUs however many it has, at the least CPU per byte
of any tunnel, and reaches the 25 Gbps link at 16 CPUs. Zero-copy
receive was tried there and was 7% worse, so isn't in.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I6b1da8bb8eaa17f9b20cbe36896426411b8b905f
This commit is contained in:
Brad Fitzpatrick committed 2026-11-07 18:02:37 +00:00
1 parent f889883537
commit df191a1199
3 files changed
+97 -7

No files matched your search

+59 -6
View File
@@ -296,6 +296,51 @@ virtualization instead of the lab's two:
vCPU-wakeup cost of tswg's handoffs (see above) is real on Nitro
too, and is the main latency item.
#### Subnet router (2026-11-07)
Tailscale's most important server role for customers: clients on the
tailnet reach a private LAN (10.0.0.0/24, say) through a subnet router,
which forwards between its tunnel and the LAN, masquerading the clients
(the default, `--snat-subnet-routes`); the LAN's machines aren't on
Tailscale. wireguard-go is known to fall over with many clients and
flows. `TestLeanSubnetRouter` (`scripts/subnetbench.sh [ROUTER_CPUS...]`)
sets this up on EC2: a router (c7i.16xlarge, 25 Gbps, with CPUs taken
offline to make CPU the limit), a backend on the same VPC subnet (also
c7i.16xlarge), and 4 clients (c7i.4xlarge) running tswg, each a peer of
the router, each with 16 TCP flows to the backend at once. `plain`
routes the clients through the router without a tunnel: the ceiling.
| router CPUs | tswg | wireguard-go | kernel WG | plain | tswg/wg |
|---|---|---|---|---|---|
| 4 | 13976 Mbps (96% CPU) | 6627 (68%) | 9756 (91%) | 24890 | 2.1x |
| 8 | 21868 (86%) | 6808 (39%) | 10009 (70%) | 24897 | 3.2x |
| 16 | 22654 (49%) | 8217 (19%) | 11489 (65%) | 24898 | 2.8x |
| 8, 64 flows per client | 21088 (91%) | 6490 (41%) | 9665 (74%) | | 3.2x |
Router CPU seconds per GB forwarded: tswg 2.3-2.9, wireguard-go 2.9-4.2,
kernel WireGuard 3.2-7.3, plain 0.2-0.4. Medians of 3 interleaved runs.
* tswg scales with the router's CPUs until the network is the limit (at
16 CPUs, 22.7 Gbps of inner traffic is about the 25 Gbps link with
WireGuard's overhead), at the lowest CPU cost per byte.
* wireguard-go uses about 3 CPUs however many it has (2.7 of 4, 3.1 of 8,
3.0 of 16), with the rest idle: each peer is capped at about 1.6 Gbps
by a serial stage (its profile is mostly idle CPUs and context
switches; most likely each peer's receive goroutine, which writes to
TUN), so it only scales with more peers. Every tswg client got at
least 4.6 Gbps.
* Upstream kernel WireGuard does better than wireguard-go but well below
tswg, and least fairly (clients from 0.8 to 2.8 Gbps).
Profile of the tswg router at 4 CPUs (CPU-bound): ChaCha20-Poly1305
about 23%, the ENA driver about 20%, packet allocation and freeing about
15%, copying decrypted messages out 8%, zeroing new pages 5% (Ubuntu
enables `init_on_alloc`). Zero-copy receive (decrypting in place in the
received pages and handing out packets that reference them) was tried
and was 7% worse: scatterlist decryption costs more than linear
decryption of an L1-hot copy, and page-fragment packets cost more
downstream (header pulls for NAT, more transmit descriptors).
EC2 tooling: `scripts/ec2.sh` (`up`, `start`, `stop`, `down`, `ssh`,
`status`) manages instances in the sandbox account (profile
`tswg-sandbox`, us-west-2: the org's SCP allows only some regions, and
@@ -831,8 +876,9 @@ install the module.
(a KASAN/lockdep build of a gokrazy kernel), `extract-ikconfig`
(vendored from Linux), `vmtest.sh` (run the natlab vmtests),
`bench.sh` (natlab benchmark), `leanbench.sh` (lean benchmark and
smoke test), `leancompare.sh` (quick A/B medians), and `ec2.sh` and
`ec2-idle-stop.sh` (EC2 instances for the harness's remote mode).
smoke test), `leancompare.sh` (quick A/B medians), `ec2.sh` and
`ec2-idle-stop.sh` (EC2 instances for the harness's remote mode), and
`subnetbench.sh` (the subnet router benchmark).
## TODO
@@ -865,16 +911,23 @@ Performance (the primary motivation; see "Performance"):
when nothing is busy (encrypt or decrypt inline, send from the
encrypt worker), without bringing back softirq starvation of
workers on the NIC's CPU under load.
- [ ] Zero-copy receive: decrypt in place in the GRO packet's page frags
(when not shared) and hand out page-frag packets, instead of
copying each message out (`tswg_decrypt_gso`).
- [x] Zero-copy receive: tried, 7% worse for the subnet router (see
"Subnet router"); kept out of tree.
- [ ] TSO from the inner GSO packet: encrypt straight from it, building
each segment's headers, instead of `skb_gso_segment` in `wg_xmit`
(per-segment skb allocation).
- [ ] Device-coalesced input: UDP packets that arrive already GSO from
the NIC (VM-to-VM with guest USO) skip GRO, so UDP segments them,
with software checksums, before `encap_rcv`.
- [ ] Benchmark subnet router and exit node forwarding.
- [x] Benchmark a subnet router with many clients and flows
(`TestLeanSubnetRouter`): 2.1-3.2x wireguard-go.
- [ ] Subnet router with many more peers than client machines (several
tunnels per client machine, in network namespaces), and an exit
node benchmark.
- [ ] tailscaled: loading tswg.ko from an image (rather than modprobe, as
DKMS would) must load its dependencies first on distro kernels
(udp_tunnel, ip6_udp_tunnel, libcurve25519 are modules on Ubuntu);
the harness agent does (`loadModuleDeps`).
- [x] Benchmarks on cloud VMs (EC2, harness remote mode, `scripts/ec2.sh`).
- [ ] EC2: c7i.large (2 vCPUs, CPU-bound), and the QEMU lab on a
c7i.metal-24xl (to see what nesting cost the lab's numbers).
+4 -1
View File
@@ -143,7 +143,7 @@ userdata() {
set -e
export DEBIAN_FRONTEND=noninteractive
apt-get update -q
apt-get install -qy build-essential gcc-13 gcc-14 "linux-headers-$(uname -r)"
apt-get install -qy build-essential gcc-13 gcc-14 nftables "linux-headers-$(uname -r)"
touch /var/lib/tswg-ready
EOF
}
@@ -201,6 +201,9 @@ up)
--tag-specifications "$(tags "$name-$i" instance)" "$(tags "$name-$i" volume)" \
--query 'Instances[0].InstanceId' --output text)
echo "launched $name-$i ($type): $id" >&2
# Let it forward packets that aren't its own, as a subnet router does.
ec2 wait instance-exists --instance-ids "$id"
ec2 modify-instance-attribute --instance-id "$id" --no-source-dest-check
ids="$ids $id"
i=$((i + 1))
done
+34
View File
@@ -0,0 +1,34 @@
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0
#
# subnetbench.sh runs tailscale.com's TestLeanSubnetRouter (a subnet router
# with many clients and flows; see tstest/tswgbench/subnet_test.go) on the
# EC2 instances from ec2.sh, once for each router CPU count given, and
# prints the medians. Logs go to $TSWG_SUBNET_LOGS (default /tmp/subnetbench).
#
# Environment (see subnet_test.go for more):
# TSWG_LEAN_SUBNET router data planes (default plain,tswg,wggo,kernelwg)
# TSWG_COMPARE_RUNS runs of each (default 3)
# TAILSCALE_DIR the tailscale.com checkout (default ~/src/tailscale.com-tswg)
#
# usage: subnetbench.sh [ROUTER_CPUS...] (default: all of the router's CPUs)
set -eu
top=$(cd "$(dirname "$0")/.." && pwd)
ts=${TAILSCALE_DIR:-$HOME/src/tailscale.com-tswg}
logs=${TSWG_SUBNET_LOGS:-/tmp/subnetbench}
mkdir -p "$logs"
export TSWG_LEAN=1 TSWG_DIR="$top"
export TSWG_REMOTE="${TSWG_REMOTE:-$HOME/.cache/tslinux/ec2/hosts.json}"
export TSWG_LEAN_SUBNET="${TSWG_LEAN_SUBNET:-plain,tswg,wggo,kernelwg}"
export TSWG_COMPARE_RUNS="${TSWG_COMPARE_RUNS:-3}"
cd "$ts"
[ $# -gt 0 ] || set -- 0
for cpus in "$@"; do
echo "== router CPUs: $cpus (0 is all)" >&2
TSWG_SUBNET_ROUTER_CPUS=$cpus go test -count=1 ./tstest/tswgbench/ \
-run 'TestLeanSubnetRouter$' -v -timeout=60m > "$logs/cpus$cpus.log" 2>&1 || true
grep -E -e '--- FAIL|router .* CPUs|median|kernel reported' "$logs/cpus$cpus.log" |
sed 's/^ *[a-z_]*_test.go:[0-9]*: //'
done