mirror of
https://github.com/tailscale/tailscale.git
synced 2026-11-20 16:51:18 -08:00
kernel/linux/{scripts,DESIGN.md}: add the subnet router benchmark and its results
subnetbench.sh runs tailscale.com's TestLeanSubnetRouter on the EC2 instances for each router CPU count given. ec2.sh now turns off source and destination checks on its instances, so that one can forward as a subnet router does, and installs nftables (for the router's masquerade). DESIGN.md records the results: with 4 clients, tswg as the subnet router forwards 2.1x (4 CPUs) to 3.2x (8 CPUs) what wireguard-go does, which uses about 3 CPUs however many it has, at the least CPU per byte of any tunnel, and reaches the 25 Gbps link at 16 CPUs. Zero-copy receive was tried there and was 7% worse, so isn't in. Updates #426 Signed-off-by: Brad Fitzpatrick <[email protected]> Change-Id: I6b1da8bb8eaa17f9b20cbe36896426411b8b905f
This commit is contained in:
1 parent
f889883537
commit
df191a1199
3 files changed
+97
-7
No files matched your search
+59
-6
@@ -296,6 +296,51 @@ virtualization instead of the lab's two:
|
||||
vCPU-wakeup cost of tswg's handoffs (see above) is real on Nitro
|
||||
too, and is the main latency item.
|
||||
|
||||
#### Subnet router (2026-11-07)
|
||||
|
||||
Tailscale's most important server role for customers: clients on the
|
||||
tailnet reach a private LAN (10.0.0.0/24, say) through a subnet router,
|
||||
which forwards between its tunnel and the LAN, masquerading the clients
|
||||
(the default, `--snat-subnet-routes`); the LAN's machines aren't on
|
||||
Tailscale. wireguard-go is known to fall over with many clients and
|
||||
flows. `TestLeanSubnetRouter` (`scripts/subnetbench.sh [ROUTER_CPUS...]`)
|
||||
sets this up on EC2: a router (c7i.16xlarge, 25 Gbps, with CPUs taken
|
||||
offline to make CPU the limit), a backend on the same VPC subnet (also
|
||||
c7i.16xlarge), and 4 clients (c7i.4xlarge) running tswg, each a peer of
|
||||
the router, each with 16 TCP flows to the backend at once. `plain`
|
||||
routes the clients through the router without a tunnel: the ceiling.
|
||||
|
||||
| router CPUs | tswg | wireguard-go | kernel WG | plain | tswg/wg |
|
||||
|---|---|---|---|---|---|
|
||||
| 4 | 13976 Mbps (96% CPU) | 6627 (68%) | 9756 (91%) | 24890 | 2.1x |
|
||||
| 8 | 21868 (86%) | 6808 (39%) | 10009 (70%) | 24897 | 3.2x |
|
||||
| 16 | 22654 (49%) | 8217 (19%) | 11489 (65%) | 24898 | 2.8x |
|
||||
| 8, 64 flows per client | 21088 (91%) | 6490 (41%) | 9665 (74%) | | 3.2x |
|
||||
|
||||
Router CPU seconds per GB forwarded: tswg 2.3-2.9, wireguard-go 2.9-4.2,
|
||||
kernel WireGuard 3.2-7.3, plain 0.2-0.4. Medians of 3 interleaved runs.
|
||||
|
||||
* tswg scales with the router's CPUs until the network is the limit (at
|
||||
16 CPUs, 22.7 Gbps of inner traffic is about the 25 Gbps link with
|
||||
WireGuard's overhead), at the lowest CPU cost per byte.
|
||||
* wireguard-go uses about 3 CPUs however many it has (2.7 of 4, 3.1 of 8,
|
||||
3.0 of 16), with the rest idle: each peer is capped at about 1.6 Gbps
|
||||
by a serial stage (its profile is mostly idle CPUs and context
|
||||
switches; most likely each peer's receive goroutine, which writes to
|
||||
TUN), so it only scales with more peers. Every tswg client got at
|
||||
least 4.6 Gbps.
|
||||
* Upstream kernel WireGuard does better than wireguard-go but well below
|
||||
tswg, and least fairly (clients from 0.8 to 2.8 Gbps).
|
||||
|
||||
Profile of the tswg router at 4 CPUs (CPU-bound): ChaCha20-Poly1305
|
||||
about 23%, the ENA driver about 20%, packet allocation and freeing about
|
||||
15%, copying decrypted messages out 8%, zeroing new pages 5% (Ubuntu
|
||||
enables `init_on_alloc`). Zero-copy receive (decrypting in place in the
|
||||
received pages and handing out packets that reference them) was tried
|
||||
and was 7% worse: scatterlist decryption costs more than linear
|
||||
decryption of an L1-hot copy, and page-fragment packets cost more
|
||||
downstream (header pulls for NAT, more transmit descriptors).
|
||||
|
||||
EC2 tooling: `scripts/ec2.sh` (`up`, `start`, `stop`, `down`, `ssh`,
|
||||
`status`) manages instances in the sandbox account (profile
|
||||
`tswg-sandbox`, us-west-2: the org's SCP allows only some regions, and
|
||||
@@ -831,8 +876,9 @@ install the module.
|
||||
(a KASAN/lockdep build of a gokrazy kernel), `extract-ikconfig`
|
||||
(vendored from Linux), `vmtest.sh` (run the natlab vmtests),
|
||||
`bench.sh` (natlab benchmark), `leanbench.sh` (lean benchmark and
|
||||
smoke test), `leancompare.sh` (quick A/B medians), and `ec2.sh` and
|
||||
`ec2-idle-stop.sh` (EC2 instances for the harness's remote mode).
|
||||
smoke test), `leancompare.sh` (quick A/B medians), `ec2.sh` and
|
||||
`ec2-idle-stop.sh` (EC2 instances for the harness's remote mode), and
|
||||
`subnetbench.sh` (the subnet router benchmark).
|
||||
|
||||
## TODO
|
||||
|
||||
@@ -865,16 +911,23 @@ Performance (the primary motivation; see "Performance"):
|
||||
when nothing is busy (encrypt or decrypt inline, send from the
|
||||
encrypt worker), without bringing back softirq starvation of
|
||||
workers on the NIC's CPU under load.
|
||||
- [ ] Zero-copy receive: decrypt in place in the GRO packet's page frags
|
||||
(when not shared) and hand out page-frag packets, instead of
|
||||
copying each message out (`tswg_decrypt_gso`).
|
||||
- [x] Zero-copy receive: tried, 7% worse for the subnet router (see
|
||||
"Subnet router"); kept out of tree.
|
||||
- [ ] TSO from the inner GSO packet: encrypt straight from it, building
|
||||
each segment's headers, instead of `skb_gso_segment` in `wg_xmit`
|
||||
(per-segment skb allocation).
|
||||
- [ ] Device-coalesced input: UDP packets that arrive already GSO from
|
||||
the NIC (VM-to-VM with guest USO) skip GRO, so UDP segments them,
|
||||
with software checksums, before `encap_rcv`.
|
||||
- [ ] Benchmark subnet router and exit node forwarding.
|
||||
- [x] Benchmark a subnet router with many clients and flows
|
||||
(`TestLeanSubnetRouter`): 2.1-3.2x wireguard-go.
|
||||
- [ ] Subnet router with many more peers than client machines (several
|
||||
tunnels per client machine, in network namespaces), and an exit
|
||||
node benchmark.
|
||||
- [ ] tailscaled: loading tswg.ko from an image (rather than modprobe, as
|
||||
DKMS would) must load its dependencies first on distro kernels
|
||||
(udp_tunnel, ip6_udp_tunnel, libcurve25519 are modules on Ubuntu);
|
||||
the harness agent does (`loadModuleDeps`).
|
||||
- [x] Benchmarks on cloud VMs (EC2, harness remote mode, `scripts/ec2.sh`).
|
||||
- [ ] EC2: c7i.large (2 vCPUs, CPU-bound), and the QEMU lab on a
|
||||
c7i.metal-24xl (to see what nesting cost the lab's numbers).
|
||||
|
||||
@@ -143,7 +143,7 @@ userdata() {
|
||||
set -e
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -q
|
||||
apt-get install -qy build-essential gcc-13 gcc-14 "linux-headers-$(uname -r)"
|
||||
apt-get install -qy build-essential gcc-13 gcc-14 nftables "linux-headers-$(uname -r)"
|
||||
touch /var/lib/tswg-ready
|
||||
EOF
|
||||
}
|
||||
@@ -201,6 +201,9 @@ up)
|
||||
--tag-specifications "$(tags "$name-$i" instance)" "$(tags "$name-$i" volume)" \
|
||||
--query 'Instances[0].InstanceId' --output text)
|
||||
echo "launched $name-$i ($type): $id" >&2
|
||||
# Let it forward packets that aren't its own, as a subnet router does.
|
||||
ec2 wait instance-exists --instance-ids "$id"
|
||||
ec2 modify-instance-attribute --instance-id "$id" --no-source-dest-check
|
||||
ids="$ids $id"
|
||||
i=$((i + 1))
|
||||
done
|
||||
|
||||
Executable
+34
@@ -0,0 +1,34 @@
|
||||
#!/bin/sh
|
||||
# SPDX-License-Identifier: GPL-2.0
|
||||
#
|
||||
# subnetbench.sh runs tailscale.com's TestLeanSubnetRouter (a subnet router
|
||||
# with many clients and flows; see tstest/tswgbench/subnet_test.go) on the
|
||||
# EC2 instances from ec2.sh, once for each router CPU count given, and
|
||||
# prints the medians. Logs go to $TSWG_SUBNET_LOGS (default /tmp/subnetbench).
|
||||
#
|
||||
# Environment (see subnet_test.go for more):
|
||||
# TSWG_LEAN_SUBNET router data planes (default plain,tswg,wggo,kernelwg)
|
||||
# TSWG_COMPARE_RUNS runs of each (default 3)
|
||||
# TAILSCALE_DIR the tailscale.com checkout (default ~/src/tailscale.com-tswg)
|
||||
#
|
||||
# usage: subnetbench.sh [ROUTER_CPUS...] (default: all of the router's CPUs)
|
||||
|
||||
set -eu
|
||||
|
||||
top=$(cd "$(dirname "$0")/.." && pwd)
|
||||
ts=${TAILSCALE_DIR:-$HOME/src/tailscale.com-tswg}
|
||||
logs=${TSWG_SUBNET_LOGS:-/tmp/subnetbench}
|
||||
mkdir -p "$logs"
|
||||
export TSWG_LEAN=1 TSWG_DIR="$top"
|
||||
export TSWG_REMOTE="${TSWG_REMOTE:-$HOME/.cache/tslinux/ec2/hosts.json}"
|
||||
export TSWG_LEAN_SUBNET="${TSWG_LEAN_SUBNET:-plain,tswg,wggo,kernelwg}"
|
||||
export TSWG_COMPARE_RUNS="${TSWG_COMPARE_RUNS:-3}"
|
||||
cd "$ts"
|
||||
[ $# -gt 0 ] || set -- 0
|
||||
for cpus in "$@"; do
|
||||
echo "== router CPUs: $cpus (0 is all)" >&2
|
||||
TSWG_SUBNET_ROUTER_CPUS=$cpus go test -count=1 ./tstest/tswgbench/ \
|
||||
-run 'TestLeanSubnetRouter$' -v -timeout=60m > "$logs/cpus$cpus.log" 2>&1 || true
|
||||
grep -E -e '--- FAIL|router .* CPUs|median|kernel reported' "$logs/cpus$cpus.log" |
|
||||
sed 's/^ *[a-z_]*_test.go:[0-9]*: //'
|
||||
done
|
||||
Reference in new issue
Block a user