mirror of
https://github.com/tailscale/tailscale.git
synced 2026-11-21 09:11:18 -08:00
Record the decisions made so far: an out-of-tree C fork of in-tree WireGuard (no Rust, no upstreaming), a shared UDP socket passed in from tailscaled, explicit endpoints with auto-roaming disabled, DERP via 127.0.0.0/8 loopback endpoints, and userspace spray. Also describe the natlab vmtest testbed and the TODO list. Updates #426 Signed-off-by: Brad Fitzpatrick <[email protected]> Change-Id: Ie18d356974a0a0d8fce19674b590861144be8cea
218 lines
9.5 KiB
Markdown
218 lines
9.5 KiB
Markdown
# tswg: Tailscale-flavored kernel WireGuard
|
|
|
|
This is a living design document. It records requirements, decisions (with
|
|
their reasons), and the TODO list. Update it whenever something is decided
|
|
or changes.
|
|
|
|
Tracking issue: https://github.com/tailscale/tailscale/issues/426
|
|
|
|
## Goal
|
|
|
|
Move the Tailscale data plane on Linux into the kernel. Packets between
|
|
Tailscale peers should be encrypted, decrypted and forwarded by a kernel
|
|
module without crossing into userspace on the direct (UDP) path. tailscaled
|
|
keeps doing everything else: control plane, disco, NAT traversal (STUN,
|
|
hole punching), port mapping (UPnP, NAT-PMP, PCP), DERP, and programming
|
|
the kernel module.
|
|
|
|
What we expect to win, and should measure, compared to wireguard-go with
|
|
TUN GSO/GRO:
|
|
|
|
* CPU cost per byte and per packet.
|
|
* Latency and small-packet rate.
|
|
* Forwarding performance for subnet routers and exit nodes.
|
|
|
|
Raw single-flow throughput is not the main argument; wireguard-go is
|
|
already fast there.
|
|
|
|
## Non-goals
|
|
|
|
* Upstreaming. tswg is an out-of-tree module, installed via DKMS or
|
|
similar. We do not depend on upstream WireGuard accepting changes.
|
|
* Rust. Decided 2026-11-06: the in-tree Rust networking abstractions are
|
|
too thin (no sk_buff, net_device, UDP tunnel, or lib/crypto bindings) and
|
|
out-of-tree Rust modules need a kernel built with CONFIG_RUST and a
|
|
matching rustc, which most distro kernels can't provide. Revisit later.
|
|
* Changing the WireGuard wire protocol. tswg must interoperate with every
|
|
existing Tailscale peer running wireguard-go.
|
|
* Multipath or "spray" sending in the kernel. When disco hasn't yet
|
|
trusted a direct path, tailscaled sprays from userspace (see "DERP and
|
|
spray").
|
|
|
|
## Architecture
|
|
|
|
```
|
|
tailscaled (userspace)
|
|
control, disco, STUN, portmapper, DERP client
|
|
| genetlink "tswg" | owns the UDP socket(s)
|
|
v v
|
|
+---------------------------------------------+
|
|
| tswg.ko (fork of drivers/net/wireguard) |
|
|
| tailscale0 netdev, noise, allowedips, ... |
|
|
| encap_rcv on tailscaled's UDP socket: |
|
|
| WG messages -> kernel |
|
|
| everything else -> socket receive queue |
|
|
+---------------------------------------------+
|
|
```
|
|
|
|
### Fork, not patches
|
|
|
|
tswg is a fork of the in-tree WireGuard driver, kept as close to upstream as
|
|
possible so upstream security fixes merge cleanly.
|
|
|
|
* `upstream/` holds a pristine copy of the upstream files. It is only ever
|
|
replaced wholesale by an import of a newer upstream; see
|
|
`upstream/UPSTREAM_COMMIT`.
|
|
* `src/` holds the fork. `diff -r upstream/drivers/net/wireguard src` is
|
|
the fork's delta (plus build files and compat headers).
|
|
* Internal symbols keep their `wg_` names. Unexported symbols in two
|
|
modules don't conflict, so tswg.ko and the in-tree wireguard.ko (built-in
|
|
or modular) can be loaded at the same time. Renaming them would make every
|
|
upstream merge painful.
|
|
* User-visible identifiers are renamed: module `tswg`, rtnl link kind
|
|
`tswg` (from `KBUILD_MODNAME`), genetlink family `tswg`.
|
|
* The uapi stays attribute-compatible with upstream WireGuard. New
|
|
attributes are only ever appended. That way wgctrl-go and a lightly
|
|
patched `wg(8)` keep working by changing only the family name.
|
|
|
|
### Kernel compatibility
|
|
|
|
The fork's base is the newest upstream (currently 7.4-rc1). Older kernels
|
|
are supported through compat shims in `src/compat/`.
|
|
|
|
* Prefer feature probes (configure-style compile tests, as in OpenZFS or
|
|
NVIDIA's conftest) over `LINUX_VERSION_CODE` checks. RHEL kernels carry
|
|
years of backports, so version numbers lie.
|
|
* Use the kernel's lib/crypto (chacha20poly1305, blake2s, curve25519). Every
|
|
distro kernel that ships WireGuard builds those libraries. Do not vendor
|
|
crypto.
|
|
* Minimum kernel: TBD. The first target is Ubuntu 24.04 (6.8).
|
|
|
|
### Shared UDP socket
|
|
|
|
Tailscale needs disco, STUN and WireGuard to share one UDP 4-tuple so NAT
|
|
mappings discovered by disco are the ones WireGuard uses. Plan, modeled on
|
|
ESP-in-UDP (`UDP_ENCAP_ESPINUDP`) and L2TP:
|
|
|
|
1. tailscaled creates and binds the UDP socket(s) and sets its socket
|
|
options (SO_MARK, buffer sizes, and so on).
|
|
2. tailscaled passes the fd to tswg over genetlink (new attributes
|
|
`WGDEVICE_A_TSWG_SOCK_FD4` and `..._FD6`, names TBD).
|
|
3. tswg calls `setup_udp_tunnel_sock()` on it with its `encap_rcv`.
|
|
4. `encap_rcv` consumes WireGuard messages (types 1-4 with three zero
|
|
reserved bytes) and returns >0 for everything else (disco, STUN), which
|
|
then lands in the socket's normal receive queue for tailscaled's
|
|
`recvmsg`.
|
|
5. tailscaled sends disco and STUN with plain `sendmsg` on the same socket.
|
|
|
|
Socket lifetime: tswg holds a reference to the socket, so the data plane
|
|
survives a tailscaled restart. The new tailscaled can't bind the same port
|
|
while the kernel holds it, so either put the fd in systemd's fd store
|
|
(`FileDescriptorStoreMax=`) or add an explicit "detach socket" op. TBD.
|
|
|
|
### Endpoints and roaming
|
|
|
|
Upstream WireGuard roams a peer's endpoint to the source of the latest
|
|
authenticated packet. That fights disco's path selection: a packet that
|
|
arrives via DERP would steal the endpoint from a good direct path. tswg
|
|
adds a per-peer flag (or device-wide default) that disables auto-roaming;
|
|
tailscaled sets endpoints explicitly.
|
|
|
|
Changing a peer's endpoint does not re-key or reset the noise session in
|
|
upstream WireGuard either; that is already fine.
|
|
|
|
### DERP and spray
|
|
|
|
DERP stays in tailscaled. A peer whose current path is DERP gets a
|
|
loopback endpoint (as planned back in 2020):
|
|
|
|
* Each DERP-pathed peer gets its own address in 127.0.0.0/8 with a common
|
|
port P, so tailscaled can bind one socket on `*:P`, read the destination
|
|
with `IP_PKTINFO` and map it back to the peer. That mapping is needed
|
|
because WireGuard transport messages only carry a receiver index.
|
|
* For inbound DERP traffic, tailscaled sends to tswg's socket from
|
|
127.x.y.z:P (setting the source with `ipi_spec_dst`).
|
|
* Spray mode (untrusted direct path): tailscaled points the peer at its
|
|
loopback endpoint, forwards each ciphertext packet to DERP, and also
|
|
sends a copy to the candidate direct address via the shared socket (same
|
|
source port). Once disco trusts the direct path, tailscaled switches the
|
|
kernel endpoint to it.
|
|
|
|
### What doesn't need kernel work
|
|
|
|
* ACLs: nftables (tailscaled already has nftables and iptables code;
|
|
conntrack handles return traffic).
|
|
* MagicDNS, peerapi, Tailscale SSH, Taildrop: kernel sockets on the
|
|
tailscale0 address.
|
|
* TSMP: raw socket on IP protocol 99.
|
|
* Subnet routers and exit nodes: kernel forwarding.
|
|
|
|
### Lazy peer configuration
|
|
|
|
tailscaled only programs recently active peers into wireguard-go, mostly
|
|
because of per-peer cost in wireguard-go. Kernel peers are cheap and
|
|
hashed, so first try programming every peer. If that doesn't scale, add an
|
|
unknown-peer upcall from `wg_noise_handshake_consume_initiation` with a
|
|
short stash of the initiation for replay (to avoid a 5s `REKEY_TIMEOUT`
|
|
stall).
|
|
|
|
### DKMS and fallback
|
|
|
|
Out-of-tree modules on Secure Boot systems need signing and MOK
|
|
enrollment. tailscaled must always fall back to wireguard-go when tswg is
|
|
missing, fails to load, or reports an incompatible version (the genetlink
|
|
family version or a feature attribute).
|
|
|
|
## Testbed
|
|
|
|
Integration tests run as natlab vmtests in tailscale.com
|
|
(`tstest/natlab/vmtest`), which boot QEMU VMs connected by vnet, a
|
|
simulated network with NATs, DERP and a test control server.
|
|
|
|
* Guest: the natlab Ubuntu 24.04 cloud image, stock kernel
|
|
`6.8.0-106-generic`. The host is also Ubuntu 24.04 with the same gcc
|
|
(13.3), so the host builds tswg.ko against that kernel's headers,
|
|
fetched with `apt-get download` and unpacked into a cache directory
|
|
(never installed).
|
|
* The test copies tswg.ko into the guest (over the debug SSH NIC or the
|
|
vnet file server) and loads it.
|
|
* Later: also boot a custom debug kernel (KASAN, lockdep) built from a
|
|
Linux tree, and more distro images (Debian, Fedora) for compat coverage.
|
|
|
|
## Repo layout
|
|
|
|
* `DESIGN.md`: this file.
|
|
* `upstream/`: pristine upstream sources (see "Fork, not patches").
|
|
* `src/`: the tswg module sources, Kbuild and compat headers.
|
|
* `scripts/`: helpers (building against a distro kernel's headers,
|
|
importing upstream).
|
|
|
|
## TODO
|
|
|
|
- [x] Create repo; import pristine upstream WireGuard (Linux 7.4-rc1).
|
|
- [x] Write DESIGN.md.
|
|
- [ ] Fork into `src/`: rename module, link kind and genetlink family to
|
|
`tswg`; out-of-tree Kbuild/Makefile.
|
|
- [ ] Build against the host's 7.0 headers.
|
|
- [ ] Script: fetch and unpack Ubuntu kernel headers for a given version;
|
|
build tswg.ko against them.
|
|
- [ ] Compat shims so it builds and loads on Ubuntu 24.04 (6.8).
|
|
- [ ] vmtest: two Ubuntu VMs, load tswg.ko, configure a tswg tunnel by
|
|
hand, ping through it.
|
|
- [ ] Userspace config tool or Go package (fork of wgctrl's netlink code
|
|
with family `tswg`).
|
|
- [ ] Socket fd passing plus `encap_rcv` classifier (WG vs. disco/STUN).
|
|
- [ ] vmtest: disco/STUN still reach userspace on the shared socket.
|
|
- [ ] Per-peer no-auto-roam flag.
|
|
- [ ] Loopback DERP endpoints; vmtest with DERP-only path.
|
|
- [ ] tailscaled integration (wgengine backend that programs tswg) on a
|
|
tailscale.com branch.
|
|
- [ ] Fallback to wireguard-go when tswg is unavailable.
|
|
- [ ] Decide socket lifetime across tailscaled restarts.
|
|
- [ ] Decide minimum supported kernel; feature-probe compat framework.
|
|
- [ ] dkms.conf and packaging.
|
|
- [ ] Benchmarks versus wireguard-go (CPU/byte, pps, latency, forwarding).
|
|
- [ ] Custom debug kernel (KASAN, lockdep) in the testbed.
|
|
- [ ] UDP GSO/GRO on the encap socket (later perf work).
|
|
- [ ] Measure scaling with many peers; unknown-peer upcall only if needed.
|