mirror of
https://github.com/tailscale/tailscale.git
synced 2026-11-20 08:41:19 -08:00
On a 128-CPU subnet router sending big responses to hundreds of flows, 75-83% of the router's CPU spun on the encrypt ring's consumer lock. Upstream's design hands every packet to a crypt worker on another CPU, through a ring all CPUs share, and back through the peer's in-order queue to a third, so that one flow's packets can be encrypted in parallel and still go out in order. Forwarded packets don't need that: RSS already spreads flows over CPUs and keeps each in order on one. So tswg_xmit_now encrypts and sends packets with no socket (forwarded) right in wg_xmit, on the CPU they arrived on, as IPsec does: no queues, rings or other CPUs. Local sockets' packets still go to the workers, so that one flow is encrypted on several CPUs; much of a TCP sender's sending is in softirq too, as ACKs arrive, and inlining it cost a stream half its throughput. Module parameter tx_inline: 0 never, 1 forwarded (the default), 2 always. And tswg_receive_now decrypts a peer's packets in encap_rcv, on the CPU its UDP flow arrives on, delivering them through the device's per-CPU GRO cells, as VXLAN does, rather than through the decrypt ring, a worker, the peer's in-order queue and its NAPI. That's one CPU per peer, which a fast one outruns (a stream: 12.2 -> 6.4 Gbps in the lab), so a peer receiving more than rx_inline_mbps (default 4000) over 10ms goes the parallel way for at least 100ms, coming back once that way has delivered what it had, so that switching doesn't reorder. On EC2's 128-CPU c6in router with 37 clients: HTTP-like fetches go from 37 to 69 Gbps, and 27 to 3.7 CPU-s/GB; uploads from 66 to 68 Gbps, and 6.8 to 3.1 CPU-s/GB; wireguard-go does 6.2 either way. The 32 CPUs taking the NIC's interrupts are now the limit, which RPS lifts for fetches to 78 Gbps. In the lab, one stream through a tunnel is unchanged at 12 Gbps, and RTT under load is down from 1.5 to 1.1ms. Updates #426 Signed-off-by: Brad Fitzpatrick <[email protected]> Change-Id: I8b1cbbed3c87a8e6a9a7cf8a91a2c18dcfb4e62a