Files
tailscale/kernel/linux
Brad Fitzpatrick a444291254 kernel/linux/src: encrypt straight from GSO packets, rather than segmenting them
wg_xmit segmented GSO packets with skb_gso_segment(skb, 0), as upstream
does, which allocates a packet per segment and, with no features, also
copies and checksums each segment's payload into it, after which
tswg_encrypt_list copied it all again to encrypt it.

Now TCP and UDP GSO packets, from local sockets or made by GRO and being
forwarded, stay whole (tswg_gso_whole_ok), get a block of nonces, and
tswg_encrypt_list writes each segment's plaintext straight into its
pages: the headers, fixed up as TCP's, UDP's, IPv4's and IPv6's
segmentation would (tswg_copy_segment), and the segment's slice of the
payload, checksummed while it's copied. Every packet a GSO packet is
encrypted into gets its socket ownership, as segments would. Others
(untrusted, tunnels, frag lists, TX timestamps) are still segmented,
but with NETIF_F_SG | NETIF_F_HW_CSUM, so that segments share pages and
get their checksums in our copy.

In the lab, with 1 TCP stream, throughput goes from 10.5 to 12.2 Gbps
and the sender's CPU from 1.82 to 1.21 CPU-s/GB (wireguard-go: 9.8
Gbps, 1.97); the features alone were 11.7 Gbps and 1.65. Forwarding
GRO's packets goes from 5.3 to 5.8 Gbps.

Module parameter gso_whole turns it off. verify_gso, for tests, checks
every segment against skb_gso_segment's, counting into gso_verified and
gso_verify_errors; tailscale.com's TestLeanGSOVerify and
TestLeanGSOVerifyForwarded find millions of segments byte-identical, on
the KASAN and lockdep kernel too, and catch a planted bug.

SKB_GSO_TCP_ACCECN gets a compat probe, for older kernels.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I0370d74be47bcb7a3833f721465fa33cd00aac7d
2026-11-08 03:33:16 +00:00
..