Files
tailscale/kernel/linux
Brad Fitzpatrick 925022517f kernel/linux/src: run forwarded packets and most received ones to completion
On a 128-CPU subnet router sending big responses to hundreds of flows,
75-83% of the router's CPU spun on the encrypt ring's consumer lock.
Upstream's design hands every packet to a crypt worker on another CPU,
through a ring all CPUs share, and back through the peer's in-order
queue to a third, so that one flow's packets can be encrypted in
parallel and still go out in order. Forwarded packets don't need that:
RSS already spreads flows over CPUs and keeps each in order on one.

So tswg_xmit_now encrypts and sends packets with no socket (forwarded)
right in wg_xmit, on the CPU they arrived on, as IPsec does: no queues,
rings or other CPUs. Local sockets' packets still go to the workers, so
that one flow is encrypted on several CPUs; much of a TCP sender's
sending is in softirq too, as ACKs arrive, and inlining it cost a
stream half its throughput. Module parameter tx_inline: 0 never, 1
forwarded (the default), 2 always.

And tswg_receive_now decrypts a peer's packets in encap_rcv, on the CPU
its UDP flow arrives on, delivering them through the device's per-CPU
GRO cells, as VXLAN does, rather than through the decrypt ring, a
worker, the peer's in-order queue and its NAPI. That's one CPU per peer,
which a fast one outruns (a stream: 12.2 -> 6.4 Gbps in the lab), so a
peer receiving more than rx_inline_mbps (default 4000) over 10ms goes
the parallel way for at least 100ms, coming back once that way has
delivered what it had, so that switching doesn't reorder.

On EC2's 128-CPU c6in router with 37 clients: HTTP-like fetches go from
37 to 69 Gbps, and 27 to 3.7 CPU-s/GB; uploads from 66 to 68 Gbps, and
6.8 to 3.1 CPU-s/GB; wireguard-go does 6.2 either way. The 32 CPUs
taking the NIC's interrupts are now the limit, which RPS lifts for
fetches to 78 Gbps. In the lab, one stream through a tunnel is
unchanged at 12 Gbps, and RTT under load is down from 1.5 to 1.1ms.

Updates #426

Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I8b1cbbed3c87a8e6a9a7cf8a91a2c18dcfb4e62a
2026-11-08 06:48:49 +00:00
..