kernel/linux/DESIGN.md: record the 20-peer subnet router results and what was tried

With 20 clients (160 flows), tswg as the subnet router forwards 2.2x (4
CPUs) and 3.6x (8 CPUs) what wireguard-go does, at 38-43% less CPU per
byte. wireguard-go stays at about 6.4 Gbps however many peers or CPUs it
has, so its limit is global, most likely its one UDP-reading goroutine.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I9274a49b47cdbdf9af06f1f9a1f6c8d04f126442
This commit is contained in:
Brad Fitzpatrick committed 2026-11-07 18:58:15 +00:00
1 parent 34e785a933
commit f810784988
1 file changed
+36 -8
+36 -8
View File
@@ -332,14 +332,42 @@ kernel WireGuard 3.2-7.3, plain 0.2-0.4. Medians of 3 interleaved runs.
* Upstream kernel WireGuard does better than wireguard-go but well below
tswg, and least fairly (clients from 0.8 to 2.8 Gbps).
Profile of the tswg router at 4 CPUs (CPU-bound): ChaCha20-Poly1305
about 23%, the ENA driver about 20%, packet allocation and freeing about
15%, copying decrypted messages out 8%, zeroing new pages 5% (Ubuntu
enables `init_on_alloc`). Zero-copy receive (decrypting in place in the
received pages and handing out packets that reference them) was tried
and was 7% worse: scatterlist decryption costs more than linear
decryption of an L1-hot copy, and page-fragment packets cost more
downstream (header pulls for NAT, more transmit descriptors).
With many more peers (20 clients: the 4 above plus 16 c7i.large, 8
flows each, 160 flows; after the optimizations below):
| router CPUs | tswg | wireguard-go | kernel WG | plain | tswg/wg |
|---|---|---|---|---|---|
| 4 | 13893 Mbps (2.38 CPU-s/GB) | 6268 (3.87) | 10155 (3.35) | 24908 | 2.2x |
| 8 | 23031 (2.62) | 6444 (4.56) | 16935 (3.98) | 24940 | 3.6x |
wireguard-go doesn't scale with peers either (6.4 Gbps with 4 or 20, at
45% of 8 CPUs), so its limit is global rather than per peer: most likely
its one goroutine reading the UDP socket. Upstream kernel WireGuard
scales with peers (each has its own tx worker and NAPI), from 10.0 to
16.9 Gbps at 8 CPUs.
Profile of the tswg router at 4 CPUs (CPU-bound), before the
optimizations: ChaCha20-Poly1305 about 23% (of which 5.5% was a scalar
ChaCha20 block per message for its Poly1305 key), the ENA driver about
20%, packet allocation and freeing about 15%, copying decrypted messages
out 8%, zeroing new pages 5% (Ubuntu enables `init_on_alloc`).
* Kept: decrypting straight from the received packet when a message is
contiguous there (`tswg_contig`), and our own ChaCha20-Poly1305
(`tswg_aead.h`), which gets the Poly1305 key in the same SIMD pass as
the message by running ChaCha20 from block 0 over the 64 bytes before
it too and XORing them back out. With 4 clients: 14.1 to 14.9 Gbps,
2.30 to 2.18 CPU-s/GB.
* Tried, not kept: zero-copy receive (decrypting in place in the received
pages, handing out packets that reference them), 7% worse:
scatterlist decryption costs more than linear decryption of an L1-hot
copy, and page-fragment packets cost more downstream (header pulls for
NAT, more transmit descriptors). A page pool per decrypt worker for
decrypted packets, to avoid allocating and zeroing pages: they didn't
come back in time, so it still allocated, and was no better. Capping
wg0's `gro_max_size` at 17 segments (so forwarded GRO packets have no
frag_list and the NIC does TSO in hardware): worse, since bigger GRO
packets save more in forwarding than software segmentation costs.
EC2 tooling: `scripts/ec2.sh` (`up`, `start`, `stop`, `down`, `ssh`,
`status`) manages instances in the sandbox account (profile