Tailscale's wireguard-go has a hybrid handshake,
Noise_IKhfs+psk2_25519+MLKEM768, which both peers must agree to use,
with no fallback to the classic one. A kernel-mode node without it
would be cut off from any peer that asked for it.
The initiation now may carry an ephemeral ML-KEM-768 encapsulation
key, sealed under the es key (with the static key then using nonce
1), and the response a ciphertext for it, sealed under the ee key,
whose shared secret is mixed into the chaining key before se. That's
per peer, with the new TSWGPEER_F_HYBRID flag. Hybrid retries carry
wireguard-go's retry bit (bit 31 of the type), which we accept, set and
echo, and which the socket's message classifier now allows.
Linux has no ML-KEM, and its SHA-3 is too new for the kernels we build
for, so src/mlkem.c is a FIPS 203 ML-KEM-768 with its own Keccak.
tests/mlkem checks it against Go's crypto/mlkem: the same keys from the
same seeds, each decapsulating the other's ciphertexts, and the same
implicit rejection. The natlab tests in tailscale.com check the
handshake against wireguard-go, in both roles.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ic524483d6cd27d36900a730553e1b21e4530c399
Handing a packet to a crypt worker, the tx worker or a NAPI kthread on
another CPU can wake a halted vCPU, which in a VM is a VM exit and a
trip through the host's scheduler. tswg's idle round trip time was
1.5-2x wireguard-go's in VMs because of it.
So when the crypt workers look idle, a packet that isn't part of a
batch is encrypted or decrypted where it is, after going on the peer's
in-order queue as before. To keep TCP ACKs under load from moving
crypto onto the CPUs taking the NIC's interrupts (8% off a subnet
router), it's limited to 8 packets per CPU per jiffy, and never done
for encryption from softirq (forwarding). And peers' NAPI runs in
softirq again, not kthreads, unless napi_threaded is set.
Idle RTT p50 is now 0.51ms in the lab (wireguard-go 0.60) and 0.75ms
on EC2 (0.97), with subnet router throughput unchanged at the same
NAPI setting. Threaded NAPI is still 4-5% faster for a CPU-bound
router.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ib5ebd1edcda1a515dac0ae4e55540bcfe52d1d7f
rx_zerocopy (writable at /sys/module/tswg/parameters/rx_zerocopy) makes
tswg_decrypt_gso decrypt data messages in place in the received packet's
pages, when nothing else can see them, and hand out packets that
reference those pages, rather than copy each message into a new packet.
It's off because it doesn't pay: on a CPU-bound subnet router it was 7%
slower, scatterlist decryption costing more than linear decryption of a
cache-hot copy, and on a 200 Gbps c6in.32xlarge router, where the NIC
interrupt CPUs' page allocation is the limit, it took tswg's own page
allocations out but was still no faster (62.5-63.3 Gbps against
63.4-64.7) at 10% more CPU per byte, since ENA's page per received
packet is what binds. It stays for NICs and loads where it might.
The copying path, unchanged, moves into tswg_decrypt_message_copy.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ieb6aa3531f869dd47d768fe367c424fbd22aa3f1
Profiling a CPU-bound tswg subnet router (EC2, 4 CPUs) found two costs
in decrypting merged packets:
Copying. Each message was copied out of the received packet and then
decrypted in place in the copy, two passes over it. When the message is
contiguous in the received packet (in its head or one page frag, as a
message received in one datagram is), decrypt straight from it into the
new packet instead (tswg_contig).
The Poly1305 key. lib/crypto makes it from a separate 32 bytes of
ChaCha20 keystream (block 0), and x86's ChaCha20 does anything up to a
block with its portable code: a scalar ChaCha20 block per message, 5.5%
of the router's CPU. tswg_aead.h has our own ChaCha20-Poly1305 for data
messages, from the library's primitives. When decrypting from one buffer
to another, it runs ChaCha20 from block 0 over the 64 bytes before the
message too (packet headers in the source, headroom in the destination),
and recovers block 0's keystream, the Poly1305 key, as the two XORed:
one SIMD pass per message (tswg_aead_decrypt_ahead). Otherwise, and for
encryption, it asks for two blocks of keystream, which takes the SIMD
path, and sets the counter back.
The 4-CPU router, 4 clients with 16 flows each, medians of 3 runs: 14.1
to 14.9 Gbps, 2.30 to 2.18 CPU-seconds per GB (wireguard-go: 6.6 Gbps,
3.36). Interop with wireguard-go is unchanged; KASAN is quiet.
Also tried, and not kept: zero-copy receive (decrypting in place in the
received pages, 7% worse), and a page pool for decrypted packets (its
pages didn't come back in time to save allocations).
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I98456d17b82d5ef640c55ca1a7fa7a018dc5a279
RTT under a TCP load had grown to about 11ms at p99 (wireguard-go: 3.7).
Instrumenting the queues found three causes:
Backpressure. Upstream encrypts in place, so a packet stays charged to
its socket (for TCP, counted by its small queues, TSQ) until the NIC has
sent it, and TCP is held to the NIC's pace. Our encrypted packets are
new (and earlier ones were orphaned), so TCP could fill the NIC's qdisc
as fast as we encrypt: on a slow uplink with a FIFO qdisc, that would be
massive bufferbloat. Now each encrypted packet takes over its packets'
socket ownership and charge (tswg_move_owner), for destructors that
just uncharge truesize (sock_wfree, and TCP's), settled to its real
size (segments' truesize is about twice their length). TSQ then allowed
TCP only about 1ms in flight, which cost 15-20% of throughput, so
sockets sending over tswg get sk_pacing_shift 9 (about 2ms), as Wi-Fi
drivers set 8 for their aggregation.
The tx worker. Upstream pins a peer's tx worker to one CPU by peer ID.
When that's the CPU taking the NIC's interrupts, softirq work starves
it, and encrypted packets waited up to 20ms to be sent. Queue it on the
CPU that finished encrypting instead (also saving an IPI).
Queue length. The peer queue's limit counted entries, but a GRO entry
is up to 64 packets; count packets, with a limit of 2048.
Medians of interleaved runs in the lean harness, tswg vs. wireguard-go:
TCP 1 stream 10.1 vs. 9.7 Gbps, 4 streams 12.3 vs. 10.8; RTT under load
p50 1.7 vs. 1.7ms. p99 under load is still about 10ms on this testbed:
at full load, the receiving VM's workers sometimes don't run for tens of
milliseconds. That's next to investigate.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ief80aac9a1e443727c89b990c86c7c8f71690954
The encrypt workers freed each list's original packets as soon as they
were encrypted. For TCP, freeing releases the socket's small queues
accounting (tcp_wfree), whose TSQ handlers then ran on all the encrypt
workers' CPUs at once and contended for the socket lock (about 5% of
each sender CPU in queued_spin_lock_slowpath). Free them instead on the
peer's tx worker once their encrypted packets are sent, which is also
when upstream releases them.
Bulk TCP in the lean harness (medians of interleaved runs): 1 stream
10.4 Gbps (wireguard-go 10.0), 4 streams 12.7 Gbps (wireguard-go 11.1).
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I904dea12ccfd7bc0534d616eb319fd692eefd832
Our UDP sockets now have gro_receive/gro_complete callbacks, as UDP
tunnels do. They merge consecutive data messages of a flow that use the
same receiving key and are all the same size but perhaps the last, and
UDP hands the merged packet (SKB_GSO_UDP_TUNNEL) to encap_rcv whole. It
goes through the decrypt ring and the peer's queue as one entry: one
worker decrypts each message into a new packet (copying it out and
decrypting in place there, rather than linearizing in place), and the
peer's NAPI poll delivers them in order. So the IP/UDP input path, the
ring and queue handoffs and the worker kick are paid once per merged
packet rather than per packet. Upstream gets no outer GRO at all.
skb_gro_receive isn't exported to modules, so socket.c has a copy
(tswg_gro_merge). gro_receive_network_flush and udp_tunnel_update_gro_rcv
are newer than 6.8, so they're probed. The latter matters: 7.x may make
the socket's gro_receive a static call, which detaching a user socket
must undo, or it would outlive the module.
Bulk TCP in the lean harness, now with guest USO off (so the receiving
VM sees individual datagrams, as from a physical network): about 9.8
Gbps, against 6.5 before, and within a few percent of wireguard-go
(about 10.0), at about 18% less receiver CPU. Making workers poll the
ring briefly before going idle (to save wakeup IPIs) was tried and
didn't help, so isn't included.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ic22482120c6ce75b9a4d896f16b6cf77aca28b4c
gokrazy-debug-kernel.sh builds the gokrazy kernel package's kernel again
with KASAN, lockdep, DEBUG_VM and DEBUG_NET, for tailscale.com's lean
harness to boot (TSWG_DEBUG_KERNEL=1). build-for-gokrazy.sh builds
against it when given TSWG_KDIR, since those options change kernel data
structures.
Its first find: our slab caches had upstream WireGuard's names (via
KMEM_CACHE), which clash with wireguard.ko or a built-in WireGuard, as
gokrazy has, and make slab_common warn at load. Give them tswg_ names.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I2e7d5448bcf8b2fb68810486f7e5d4bf2af4558d
Three changes to the data path, from profiling bulk TCP in the lean
harness:
Encryption. Upstream encrypts each packet in place, which first means
making it linear and writable (skb_cow_data, usually a copy), and the
previous commit then chained the encrypted packets as a GSO frag_list,
which segmentation copied again (skb_segment must unshare each clone's
head to write its headers). Now the encrypt worker copies each inner
packet (finishing its checksum on the way, if needed) into pages of its
own, encrypts it there, and gathers each run of equal-size messages into
a UDP GSO packet whose payload is those pages. Segmenting that needs only
page references, and a device with UDP segmentation offload takes it
whole: virtio with USO now sends 64KB superpackets. Sender CPU at the same
throughput went from about 65% to 43% per vCPU.
Worker kicks. Instead of kicking a crypt worker per packet (or per 16),
kick only when no worker is draining the ring, or when the backlog shows
the busy ones aren't keeping up, and prefer other CPUs than the one
taking packets from the NIC. A tswg_busy flag, paired with the ring
under full barriers, closes the race with a worker going idle.
Queue length. The peer's in-order queue overflowed at 1024 packets (1.5ms
at 7 Gbps) whenever a decrypt worker was held up, and TCP took those
silent drops (0.2% of packets) as congestion. Allow 4096 entries, and
count drops in rx_dropped and rx_fifo_errors, which upstream doesn't.
The receiver is now the bottleneck (about 6.5 Gbps): UDP packets arriving
coalesced are segmented back, with software checksums, before reaching
encap_rcv. That's next.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ib8d9cf9e5541b665e06d57a16b46f653d49bc3ea
Profiling bulk TCP in the lean harness (tailscale.com's tstest/tswgbench)
showed kernel WireGuard serializing on two per-packet costs:
On receive, the single softirq CPU spent half its time in queue_work_on:
upstream kicks the next CPU's crypt worker for every packet, round robin,
which usually means waking it with an IPI (a VM exit in a VM). All the
workers drain the same ring, so instead stay on one CPU for 16 packets at
a time, and skip queue_work_on when the work is already pending (a read,
rather than queue_work_on's atomic test-and-set on a cache line the
worker writes). The same pending check applies to the per-peer transmit
work.
On send, the per-peer transmit worker sent each encrypted packet through
the UDP/IP stack and the device on its own, which with virtio is a device
kick, a VM exit, per packet. Packets from one GSO packet encrypt to the
same size (WireGuard pads to 16 bytes) except perhaps the last, so chain
each such run, without copying, as the frag_list of its first packet and
send it as one UDP GSO (SKB_GSO_UDP_L4) packet, with its checksum set up
as for a UDP_SEGMENT socket. The stack (or a capable device) segments it,
and the segments go to the device in one batch.
The coalesced packets are orphaned first: segmenting a frag_list of
packets still owned by the inner TCP socket uncharged the socket more
than it was charged (refcount underflow in tcp_wfree). wireguard-go has
the same effect, since the TUN device frees the inner packet once read.
Bulk TCP over tswg goes from 3.6 to 5.8 Gbps (wireguard-go: about 10.5).
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I649a7f1790c35d1af584d2c14f588195232d5ef4
tailscaled was dumping the device every second to notice new sessions
(to send its disco key advertisement, as wireguard-go does) and traffic
on direct paths (to keep magicsock heartbeating active sessions).
Instead, multicast TSWG_CMD_PEER_EVENT messages on a new "events"
genetlink group, in the device's network namespace: one when a new
session is established (from wg_timers_session_derived), and TX and RX
activity notices, each rate limited to one per peer every 2 seconds with
a single comparison on the packet path. Messages are only built when
someone is listening.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I08bdaa23a9f9b01a0bb3a4404296cae5961cde86
When tailscaled learns that a peer restarted (its disco key changed), the
peer has lost its WireGuard keys, but the kernel would keep sending with
the dead session until WireGuard's own timers start a new handshake about
15 seconds later.
Add TSWGPEER_A_CLEAR_SESSION, an NLA_FLAG in the nested WGPEER_A_TSWG
attribute, which forgets the peer's keypairs and any handshake in
progress, as wg_pm_notification does for every peer before suspend, and
resets the handshake rate limit, so that the next packet to the peer is
staged and starts a new handshake right away.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ia9821a47e6c8114bbabd81d8d5a2be528295e74f
Upstream WireGuard moves a peer's endpoint to the source of each
authenticated packet. In Tailscale, disco picks the path, so that would
fight it: a packet that arrives via DERP would steal the endpoint from a
good direct path.
Add TSWGPEER_F_NO_ROAM, set through TSWGPEER_A_FLAGS nested in a new
WGPEER_A_TSWG attribute (numbered 64, like WGDEVICE_A_TSWG) and reported
in dumps. With it set, the endpoint learned from handshake initiations,
handshake responses and data packets is ignored once the peer has an
endpoint, and only userspace changes it. A peer with no endpoint yet
still gets one from its first authenticated packet, so it stays
reachable if it contacts us before userspace has configured one.
The gate, tswg_peer_may_roam, sits at the two learned-endpoint paths
rather than in wg_socket_set_peer_endpoint, which netlink also uses for
explicit updates. The flag is applied before WGPEER_A_ENDPOINT in the
same request, so pinning and setting an endpoint are one step.
Tested by TestTSWG{Gokrazy,Ubuntu}NoRoam in tailscale.com's natlab
vmtests, which fail as expected with the gate disabled.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I621a6e86d3947e154c4cfa582aa845bb49d42b2f
Tailscale needs disco, STUN and WireGuard to share one UDP 4-tuple, so
that the NAT mappings disco discovers are the ones WireGuard uses. So let
tailscaled create and bind the socket itself and hand its fd to the
device, as L2TP allows for its tunnel sockets.
The new TSWGDEVICE_A_SOCK_FD4 and TSWGDEVICE_A_SOCK_FD6 attributes, nested
in a new WGDEVICE_A_TSWG attribute numbered 64 to stay clear of future
upstream attributes, take the fd of a bound UDP socket (IPV6_V6ONLY for
IPv6). The device hooks the socket's UDP encap_rcv, consumes WireGuard
messages (a first byte of 1 through 4 and three zero bytes), and returns
everything else to the normal UDP path for userspace to read. Userspace
can send its own datagrams on the socket as usual.
The device keeps a reference to the struct sock, not the file, and drops
the socket from its encap_destroy hook when userspace closes it, so the
socket lives exactly as long as userspace's fd. Passing -1 detaches the
socket, restoring it to plain UDP, and the device goes back to creating
its own sockets. User sockets persist across ifdown/ifup, and changing
the listen port while one is attached fails with EBUSY. Dumps report
which families have a user socket in TSWGDEVICE_A_FLAGS.
A global mutex serializes attach and detach, including detach from
encap_destroy, which runs from the user's close() with none of our locks
held.
Tested by TestTSWGGokrazySharedSocket (7.2.8) and
TestTSWGUbuntuSharedSocket (6.8) in tailscale.com's natlab vmtests.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I2e7a97952ced03df4bc954ba398a49daf9128d41
The fork tracks the newest upstream WireGuard (7.4-rc1), but has to build
on distro kernels. Our first target is the natlab Ubuntu 24.04 guest
(6.8.0-106-generic) and the dev host is on 7.0.
Rather than LINUX_VERSION_CODE checks, which lie on heavily backported
kernels such as RHEL's, scripts/probe.sh compile-tests each
src/compat/probes/HAVE_*.c against the target kernel's build directory
(in parallel, about one second in total) and writes
src/compat/generated/autoconf.h. Kbuild force-includes src/compat/compat.h,
which shims whatever is missing: udp_get_len_short, the struct sock
udp_tunnel APIs, the ipcb_flags xmit arguments, the blake2s_ctx API,
WQ_PERCPU, netif_threaded_enable, rtnl_newlink_params, sockaddr_inet, the
kmalloc_obj family and timer_container_of.
Three spots in device.c can't be shimmed from a header (the lltx bit, the
old rtnl newlink signature, and pernet exit_rtnl) and get small #ifdefs
marked TSWG_COMPAT. device.c also always sets ndo_get_stats64 to
dev_get_tstats64, since older kernels don't fold tstats in
dev_get_stats(). That is equivalent on new kernels.
tswg.ko now builds against both 6.8.0-106-generic and 7.0.0-1009-aws.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I0ed279318b4f623688d1f178a9b8fbcc0b585219
Copy upstream/drivers/net/wireguard into src and rename only the
user-visible identifiers: the module (and thus the rtnl link kind and
device type, which come from KBUILD_MODNAME) is now tswg, and the
genetlink family is "tswg". The uapi header moves to
src/include/uapi/linux/tswg.h so it can't collide with the kernel's.
Internal wg_ symbols keep their names to keep upstream merges easy; they
aren't exported, so tswg.ko can coexist with wireguard.ko.
Add an out-of-tree Kbuild and top-level Makefile, plus
scripts/ubuntu-headers.sh, which fetches and unpacks an Ubuntu kernel's
headers so we can build for the natlab Ubuntu guest without installing
anything.
This still only builds against 7.4-era kernels; compat shims follow.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ia36ef82a53efbbcdb301ed069785162372efda55