Files
tailscale/kernel/linux
Brad Fitzpatrick 2c3ba4df08 kernel/linux/src: encrypt into page frags, and kick crypt workers on demand
Three changes to the data path, from profiling bulk TCP in the lean
harness:

Encryption. Upstream encrypts each packet in place, which first means
making it linear and writable (skb_cow_data, usually a copy), and the
previous commit then chained the encrypted packets as a GSO frag_list,
which segmentation copied again (skb_segment must unshare each clone's
head to write its headers). Now the encrypt worker copies each inner
packet (finishing its checksum on the way, if needed) into pages of its
own, encrypts it there, and gathers each run of equal-size messages into
a UDP GSO packet whose payload is those pages. Segmenting that needs only
page references, and a device with UDP segmentation offload takes it
whole: virtio with USO now sends 64KB superpackets. Sender CPU at the same
throughput went from about 65% to 43% per vCPU.

Worker kicks. Instead of kicking a crypt worker per packet (or per 16),
kick only when no worker is draining the ring, or when the backlog shows
the busy ones aren't keeping up, and prefer other CPUs than the one
taking packets from the NIC. A tswg_busy flag, paired with the ring
under full barriers, closes the race with a worker going idle.

Queue length. The peer's in-order queue overflowed at 1024 packets (1.5ms
at 7 Gbps) whenever a decrypt worker was held up, and TCP took those
silent drops (0.2% of packets) as congestion. Allow 4096 entries, and
count drops in rx_dropped and rx_fifo_errors, which upstream doesn't.

The receiver is now the bottleneck (about 6.5 Gbps): UDP packets arriving
coalesced are segmented back, with software checksums, before reaching
encap_rcv. That's next.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ib8d9cf9e5541b665e06d57a16b46f653d49bc3ea
2026-11-07 12:26:13 +00:00
..