Files
tailscale/kernel/linux/src
Brad Fitzpatrick 34e785a933 kernel/linux/src: decrypt straight from received packets, and in one ChaCha20 pass
Profiling a CPU-bound tswg subnet router (EC2, 4 CPUs) found two costs
in decrypting merged packets:

Copying. Each message was copied out of the received packet and then
decrypted in place in the copy, two passes over it. When the message is
contiguous in the received packet (in its head or one page frag, as a
message received in one datagram is), decrypt straight from it into the
new packet instead (tswg_contig).

The Poly1305 key. lib/crypto makes it from a separate 32 bytes of
ChaCha20 keystream (block 0), and x86's ChaCha20 does anything up to a
block with its portable code: a scalar ChaCha20 block per message, 5.5%
of the router's CPU. tswg_aead.h has our own ChaCha20-Poly1305 for data
messages, from the library's primitives. When decrypting from one buffer
to another, it runs ChaCha20 from block 0 over the 64 bytes before the
message too (packet headers in the source, headroom in the destination),
and recovers block 0's keystream, the Poly1305 key, as the two XORed:
one SIMD pass per message (tswg_aead_decrypt_ahead). Otherwise, and for
encryption, it asks for two blocks of keystream, which takes the SIMD
path, and sets the counter back.

The 4-CPU router, 4 clients with 16 flows each, medians of 3 runs: 14.1
to 14.9 Gbps, 2.30 to 2.18 CPU-seconds per GB (wireguard-go: 6.6 Gbps,
3.36). Interop with wireguard-go is unchanged; KASAN is quiet.

Also tried, and not kept: zero-copy receive (decrypting in place in the
received pages, 7% worse), and a page pool for decrypted packets (its
pages didn't come back in time to save allocations).

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I98456d17b82d5ef640c55ca1a7fa7a018dc5a279
2026-11-07 18:51:47 +00:00
..