mirror of
https://github.com/tailscale/tailscale.git
synced 2026-11-20 08:41:19 -08:00
wg_xmit segmented GSO packets with skb_gso_segment(skb, 0), as upstream does, which allocates a packet per segment and, with no features, also copies and checksums each segment's payload into it, after which tswg_encrypt_list copied it all again to encrypt it. Now TCP and UDP GSO packets, from local sockets or made by GRO and being forwarded, stay whole (tswg_gso_whole_ok), get a block of nonces, and tswg_encrypt_list writes each segment's plaintext straight into its pages: the headers, fixed up as TCP's, UDP's, IPv4's and IPv6's segmentation would (tswg_copy_segment), and the segment's slice of the payload, checksummed while it's copied. Every packet a GSO packet is encrypted into gets its socket ownership, as segments would. Others (untrusted, tunnels, frag lists, TX timestamps) are still segmented, but with NETIF_F_SG | NETIF_F_HW_CSUM, so that segments share pages and get their checksums in our copy. In the lab, with 1 TCP stream, throughput goes from 10.5 to 12.2 Gbps and the sender's CPU from 1.82 to 1.21 CPU-s/GB (wireguard-go: 9.8 Gbps, 1.97); the features alone were 11.7 Gbps and 1.65. Forwarding GRO's packets goes from 5.3 to 5.8 Gbps. Module parameter gso_whole turns it off. verify_gso, for tests, checks every segment against skb_gso_segment's, counting into gso_verified and gso_verify_errors; tailscale.com's TestLeanGSOVerify and TestLeanGSOVerifyForwarded find millions of segments byte-identical, on the KASAN and lockdep kernel too, and catch a planted bug. SKB_GSO_TCP_ACCECN gets a compat probe, for older kernels. Updates #426 Signed-off-by: Brad Fitzpatrick <[email protected]> Change-Id: I0370d74be47bcb7a3833f721465fa33cd00aac7d