mirror of
https://github.com/tailscale/tailscale.git
synced 2026-11-20 16:51:18 -08:00
Three changes to the data path, from profiling bulk TCP in the lean harness: Encryption. Upstream encrypts each packet in place, which first means making it linear and writable (skb_cow_data, usually a copy), and the previous commit then chained the encrypted packets as a GSO frag_list, which segmentation copied again (skb_segment must unshare each clone's head to write its headers). Now the encrypt worker copies each inner packet (finishing its checksum on the way, if needed) into pages of its own, encrypts it there, and gathers each run of equal-size messages into a UDP GSO packet whose payload is those pages. Segmenting that needs only page references, and a device with UDP segmentation offload takes it whole: virtio with USO now sends 64KB superpackets. Sender CPU at the same throughput went from about 65% to 43% per vCPU. Worker kicks. Instead of kicking a crypt worker per packet (or per 16), kick only when no worker is draining the ring, or when the backlog shows the busy ones aren't keeping up, and prefer other CPUs than the one taking packets from the NIC. A tswg_busy flag, paired with the ring under full barriers, closes the race with a worker going idle. Queue length. The peer's in-order queue overflowed at 1024 packets (1.5ms at 7 Gbps) whenever a decrypt worker was held up, and TCP took those silent drops (0.2% of packets) as congestion. Allow 4096 entries, and count drops in rx_dropped and rx_fifo_errors, which upstream doesn't. The receiver is now the bottleneck (about 6.5 Gbps): UDP packets arriving coalesced are segmented back, with software checksums, before reaching encap_rcv. That's next. Updates #426 Signed-off-by: Brad Fitzpatrick <[email protected]> Change-Id: Ib8d9cf9e5541b665e06d57a16b46f653d49bc3ea