mirror of
https://github.com/tailscale/tailscale.git
synced 2026-11-21 09:11:18 -08:00
Record the decision to test primarily on natlab's gokrazy image, which boots a recent upstream kernel and has no userspace tools, as in the end state where tailscaled drives the kernel entirely. Describe how its module build tree is recreated and how the test agent loads and configures tswg. Keep Ubuntu 24.04 as a secondary guest for distro and DKMS-style coverage. Updates #426 Signed-off-by: Brad Fitzpatrick <[email protected]> Change-Id: I89714b58c944be4f373f23471e3aeef11942b1f6
311 lines
14 KiB
Markdown
311 lines
14 KiB
Markdown
# tswg: Tailscale-flavored kernel WireGuard
|
|
|
|
This is a living design document. It records requirements, decisions (with
|
|
their reasons), and the TODO list. Update it whenever something is decided
|
|
or changes.
|
|
|
|
Tracking issue: https://github.com/tailscale/tailscale/issues/426
|
|
|
|
## Goal
|
|
|
|
Move the Tailscale data plane on Linux into the kernel. Packets between
|
|
Tailscale peers should be encrypted, decrypted and forwarded by a kernel
|
|
module without crossing into userspace on the direct (UDP) path. tailscaled
|
|
keeps doing everything else: control plane, disco, NAT traversal (STUN,
|
|
hole punching), port mapping (UPnP, NAT-PMP, PCP), DERP, and programming
|
|
the kernel module.
|
|
|
|
What we expect to win, and should measure, compared to wireguard-go with
|
|
TUN GSO/GRO:
|
|
|
|
* CPU cost per byte and per packet.
|
|
* Latency and small-packet rate.
|
|
* Forwarding performance for subnet routers and exit nodes.
|
|
|
|
Raw single-flow throughput is not the main argument; wireguard-go is
|
|
already fast there.
|
|
|
|
## Non-goals
|
|
|
|
* Upstreaming. tswg is an out-of-tree module, installed via DKMS or
|
|
similar. We do not depend on upstream WireGuard accepting changes.
|
|
* Rust. Decided 2026-11-06: the in-tree Rust networking abstractions are
|
|
too thin (no sk_buff, net_device, UDP tunnel, or lib/crypto bindings) and
|
|
out-of-tree Rust modules need a kernel built with CONFIG_RUST and a
|
|
matching rustc, which most distro kernels can't provide. Revisit later.
|
|
* Changing the WireGuard wire protocol. tswg must interoperate with every
|
|
existing Tailscale peer running wireguard-go.
|
|
* Multipath or "spray" sending in the kernel. When disco hasn't yet
|
|
trusted a direct path, tailscaled sprays from userspace (see "DERP and
|
|
spray").
|
|
|
|
## Architecture
|
|
|
|
```
|
|
tailscaled (userspace)
|
|
control, disco, STUN, portmapper, DERP client
|
|
| genetlink "tswg" | owns the UDP socket(s)
|
|
v v
|
|
+---------------------------------------------+
|
|
| tswg.ko (fork of drivers/net/wireguard) |
|
|
| tailscale0 netdev, noise, allowedips, ... |
|
|
| encap_rcv on tailscaled's UDP socket: |
|
|
| WG messages -> kernel |
|
|
| everything else -> socket receive queue |
|
|
+---------------------------------------------+
|
|
```
|
|
|
|
### Fork, not patches
|
|
|
|
tswg is a fork of the in-tree WireGuard driver, kept as close to upstream as
|
|
possible so upstream security fixes merge cleanly.
|
|
|
|
* `upstream/` holds a pristine copy of the upstream files. It is only ever
|
|
replaced wholesale by an import of a newer upstream; see
|
|
`upstream/UPSTREAM_COMMIT`.
|
|
* `src/` holds the fork. `diff -r upstream/drivers/net/wireguard src` is
|
|
the fork's delta (plus build files and compat headers).
|
|
* Internal symbols keep their `wg_` names. Unexported symbols in two
|
|
modules don't conflict, so tswg.ko and the in-tree wireguard.ko (built-in
|
|
or modular) can be loaded at the same time. Renaming them would make every
|
|
upstream merge painful.
|
|
* User-visible identifiers are renamed: module `tswg`, rtnl link kind
|
|
`tswg` (from `KBUILD_MODNAME`), genetlink family `tswg`.
|
|
* The uapi stays attribute-compatible with upstream WireGuard. New
|
|
attributes are only ever appended. That way wgctrl-go and a lightly
|
|
patched `wg(8)` keep working by changing only the family name.
|
|
* The uapi header is `src/include/uapi/linux/tswg.h`. Upstream generates
|
|
it, and `src/generated/netlink.[ch]`, from
|
|
`Documentation/netlink/specs/wireguard.yaml` with YNL. For now we
|
|
hand-edit the generated files; once we add attributes, keep a forked
|
|
spec and regenerate with a kernel tree's `tools/net/ynl/ynl-regen.sh`
|
|
(TODO).
|
|
|
|
### Kernel compatibility
|
|
|
|
The fork's base is the newest upstream (currently 7.4-rc1). Older kernels
|
|
are supported through compat shims in `src/compat/`.
|
|
|
|
* Feature probes, not `LINUX_VERSION_CODE` checks (decided 2026-11-06;
|
|
RHEL kernels carry years of backports, so version numbers lie).
|
|
`scripts/probe.sh` compiles each `src/compat/probes/HAVE_*.c` against
|
|
the target kernel's build directory, in parallel (about one second in
|
|
total), and writes `src/compat/generated/autoconf.h`. The top-level
|
|
Makefile runs it before every build.
|
|
* `src/compat/compat.h` is force-included into every file and shims
|
|
whatever is missing, mostly as macros or static inlines named like the
|
|
new API, so upstream code stays untouched. The few spots that can't be
|
|
shimmed from a header get small `#ifdef HAVE_...` blocks marked
|
|
`TSWG_COMPAT` in the `.c` files (currently only `device.c`).
|
|
* To add a shim: add a probe file, use its `HAVE_` macro in compat.h, and
|
|
build against both an old and a new kernel.
|
|
* Use the kernel's lib/crypto (chacha20poly1305, blake2s, curve25519). Every
|
|
distro kernel that ships WireGuard builds those libraries. Do not vendor
|
|
crypto.
|
|
* Builds today: Ubuntu 6.8.0-106-generic (the natlab guest) and
|
|
7.0.0-1009-aws (the dev host). The 7.4 path itself (all probes
|
|
succeeding) isn't verified yet, since that needs a configured and built
|
|
7.4 tree.
|
|
* Minimum kernel: TBD.
|
|
|
|
### Userspace (Go) side
|
|
|
|
Decided 2026-11-06: the Go code lives in tailscale.com, because tailscaled
|
|
is what will program the module.
|
|
|
|
* `tailscale.com/wgengine/tswg`: netlink client for the `tswg` genetlink
|
|
family (configure and dump devices).
|
|
* `tailscale.com/cmd/tswgctl`: a wg(8)-like debugging tool built on it.
|
|
* This work lives on the `bradfitz/tswg` branch, in a worktree at
|
|
`~/src/tailscale.com-tswg`.
|
|
|
|
### Shared UDP socket
|
|
|
|
Tailscale needs disco, STUN and WireGuard to share one UDP 4-tuple so NAT
|
|
mappings discovered by disco are the ones WireGuard uses. Plan, modeled on
|
|
ESP-in-UDP (`UDP_ENCAP_ESPINUDP`) and L2TP:
|
|
|
|
1. tailscaled creates and binds the UDP socket(s) and sets its socket
|
|
options (SO_MARK, buffer sizes, and so on).
|
|
2. tailscaled passes the fd to tswg over genetlink (new attributes
|
|
`WGDEVICE_A_TSWG_SOCK_FD4` and `..._FD6`, names TBD).
|
|
3. tswg calls `setup_udp_tunnel_sock()` on it with its `encap_rcv`.
|
|
4. `encap_rcv` consumes WireGuard messages (types 1-4 with three zero
|
|
reserved bytes) and returns >0 for everything else (disco, STUN), which
|
|
then lands in the socket's normal receive queue for tailscaled's
|
|
`recvmsg`.
|
|
5. tailscaled sends disco and STUN with plain `sendmsg` on the same socket.
|
|
|
|
Socket lifetime: tswg holds a reference to the socket, so the data plane
|
|
survives a tailscaled restart. The new tailscaled can't bind the same port
|
|
while the kernel holds it, so either put the fd in systemd's fd store
|
|
(`FileDescriptorStoreMax=`) or add an explicit "detach socket" op. TBD.
|
|
|
|
### Endpoints and roaming
|
|
|
|
Upstream WireGuard roams a peer's endpoint to the source of the latest
|
|
authenticated packet. That fights disco's path selection: a packet that
|
|
arrives via DERP would steal the endpoint from a good direct path. tswg
|
|
adds a per-peer flag (or device-wide default) that disables auto-roaming;
|
|
tailscaled sets endpoints explicitly.
|
|
|
|
Changing a peer's endpoint does not re-key or reset the noise session in
|
|
upstream WireGuard either; that is already fine.
|
|
|
|
### DERP and spray
|
|
|
|
DERP stays in tailscaled. A peer whose current path is DERP gets a
|
|
loopback endpoint (as planned back in 2020):
|
|
|
|
* Each DERP-pathed peer gets its own address in 127.0.0.0/8 with a common
|
|
port P, so tailscaled can bind one socket on `*:P`, read the destination
|
|
with `IP_PKTINFO` and map it back to the peer. That mapping is needed
|
|
because WireGuard transport messages only carry a receiver index.
|
|
* For inbound DERP traffic, tailscaled sends to tswg's socket from
|
|
127.x.y.z:P (setting the source with `ipi_spec_dst`).
|
|
* Spray mode (untrusted direct path): tailscaled points the peer at its
|
|
loopback endpoint, forwards each ciphertext packet to DERP, and also
|
|
sends a copy to the candidate direct address via the shared socket (same
|
|
source port). Once disco trusts the direct path, tailscaled switches the
|
|
kernel endpoint to it.
|
|
|
|
### What doesn't need kernel work
|
|
|
|
* ACLs: nftables (tailscaled already has nftables and iptables code;
|
|
conntrack handles return traffic).
|
|
* MagicDNS, peerapi, Tailscale SSH, Taildrop: kernel sockets on the
|
|
tailscale0 address.
|
|
* TSMP: raw socket on IP protocol 99.
|
|
* Subnet routers and exit nodes: kernel forwarding.
|
|
|
|
### Lazy peer configuration
|
|
|
|
tailscaled only programs recently active peers into wireguard-go, mostly
|
|
because of per-peer cost in wireguard-go. Kernel peers are cheap and
|
|
hashed, so first try programming every peer. If that doesn't scale, add an
|
|
unknown-peer upcall from `wg_noise_handshake_consume_initiation` with a
|
|
short stash of the initiation for replay (to avoid a 5s `REKEY_TIMEOUT`
|
|
stall).
|
|
|
|
### DKMS and fallback
|
|
|
|
Out-of-tree modules on Secure Boot systems need signing and MOK
|
|
enrollment. tailscaled must always fall back to wireguard-go when tswg is
|
|
missing, fails to load, or reports an incompatible version (the genetlink
|
|
family version or a feature attribute).
|
|
|
|
## Testbed
|
|
|
|
Integration tests run as natlab vmtests in tailscale.com
|
|
(`tstest/natlab/vmtest`), which boot QEMU VMs connected by vnet, a
|
|
simulated network with NATs, DERP and a test control server. The tests
|
|
live in the tailscale.com worktree, in `tstest/natlab/vmtest/tswg_test.go`.
|
|
Run them with `scripts/vmtest.sh [TEST_REGEXP]`; it sets `TSWG_DIR` to this
|
|
tree, and the tests skip if the tree is missing.
|
|
|
|
### Primary guest: gokrazy
|
|
|
|
Decided 2026-11-06: the natlab gokrazy image is the primary test guest.
|
|
|
|
* It boots a recent upstream kernel (7.2.8 today), from the
|
|
`github.com/gokrazy/kernel.amd64` package pinned in tailscale.com's
|
|
go.mod, which gets bumped regularly. That keeps the fork honest against
|
|
new kernel APIs, which is where the fork's base is anyway.
|
|
* gokrazy has no userspace tools (no modprobe, iproute2 or shell), just
|
|
like the end state, where tailscaled drives the kernel entirely. So
|
|
everything is done from Go: `tswg.LoadModule` calls init_module(2) on
|
|
the tswg.ko image, and links and addresses are set up over rtnetlink.
|
|
* gokrazy kernels are kernel.org releases plus the patches in
|
|
`_build/series`, with an embedded config (`CONFIG_IKCONFIG`) and without
|
|
MODVERSIONS or module signing. `scripts/gokrazy-kdir.sh` recreates a
|
|
matching module build tree (download, patch, embedded config,
|
|
`modules_prepare`; about 20 seconds, cached). We build with the host's
|
|
gcc 13 while gokrazy uses Debian 12's gcc 12; the only resulting config
|
|
differences are toolchain capability bits. If that ever matters, build
|
|
in a Debian 12 container instead.
|
|
* Such a tree has no `Module.symvers`, so modpost can't check the kernel's
|
|
exports (we set `KBUILD_MODPOST_WARN=1`); a missing symbol shows up as a
|
|
load failure instead.
|
|
* The test agent (`cmd/tta`) has `/tswg/load` (POST tswg.ko),
|
|
`/tswg/up` (create, configure and bring up a one-peer device),
|
|
`/tswg/show` (device JSON) and `/tswg/dmesg` (kernel log lines
|
|
mentioning tswg, read from /dev/kmsg).
|
|
* `TestTSWGGokrazyManualTunnel`: two gokrazy VMs on one LAN, tunnel set up
|
|
through tta, ping across it, and check handshake and byte counters.
|
|
|
|
### Secondary guest: Ubuntu 24.04
|
|
|
|
Kept for distro and DKMS-style coverage, since that's how real users will
|
|
install the module.
|
|
|
|
* The natlab Ubuntu 24.04 cloud image, stock kernel `6.8.0-106-generic`.
|
|
The host is also Ubuntu 24.04 with the same gcc (13.3), so
|
|
`scripts/build-for-ubuntu.sh` builds tswg.ko against that kernel's
|
|
headers, fetched with `apt-get download` and unpacked into a cache
|
|
directory (never installed).
|
|
* The test asks the guest for `uname -r`, builds tswg.ko and `tswgctl`,
|
|
registers both with the vnet file server, and has the guest curl them
|
|
from `http://52.52.0.6/tswg/<kver>/`, then `depmod` and `modprobe tswg`.
|
|
Commands run as root over the debug SSH NIC.
|
|
* `TestTSWGManualTunnel`: the same tunnel check, configured with
|
|
`ip link` and `tswgctl`.
|
|
|
|
### Later
|
|
|
|
* A custom debug kernel (KASAN, lockdep) built from a Linux tree, which
|
|
would also cover the 7.4 compat path.
|
|
* More distro images (Debian, Fedora) for compat coverage.
|
|
|
|
## Repo layout
|
|
|
|
* `DESIGN.md`: this file.
|
|
* `upstream/`: pristine upstream sources (see "Fork, not patches").
|
|
* `src/`: the tswg module sources, Kbuild and compat headers.
|
|
* `scripts/`: `probe.sh` (compat probes), `build.sh` (build tswg.ko
|
|
against a KDIR in a private copy of `src/`), `ubuntu-headers.sh` and
|
|
`build-for-ubuntu.sh` (Ubuntu kernels), `gokrazy-kdir.sh` and
|
|
`build-for-gokrazy.sh` (gokrazy kernels), `extract-ikconfig` (vendored
|
|
from Linux), `vmtest.sh` (run the natlab vmtests).
|
|
|
|
## TODO
|
|
|
|
- [x] Create repo; import pristine upstream WireGuard (Linux 7.4-rc1).
|
|
- [x] Write DESIGN.md.
|
|
- [x] Fork into `src/`: rename module, link kind and genetlink family to
|
|
`tswg`; out-of-tree Kbuild/Makefile.
|
|
- [x] Feature-probe compat framework.
|
|
- [x] Build against the host's 7.0 headers.
|
|
- [x] Script: fetch and unpack Ubuntu kernel headers for a given version;
|
|
build tswg.ko against them.
|
|
- [x] Compat shims so it builds on Ubuntu 24.04 (6.8).
|
|
- [x] Go netlink client (`tailscale.com/wgengine/tswg`) and `tswgctl`.
|
|
- [x] vmtest: two Ubuntu VMs, load tswg.ko, configure a tswg tunnel by
|
|
hand, ping through it (`TestTSWGManualTunnel`, passes in about 80s).
|
|
- [x] Same on gokrazy, with no userspace tools
|
|
(`TestTSWGGokrazyManualTunnel`, passes in about 23s).
|
|
- [ ] Generate a Module.symvers for gokrazy build trees (for example from
|
|
the kernel's exported-symbol list) so missing exports fail at build
|
|
time rather than load time.
|
|
- [ ] Verify the 7.4 path (all probes succeed) against a built 7.4 tree.
|
|
(Partly covered: gokrazy's 7.2.8 has every probed API except
|
|
udp_get_len_short.)
|
|
- [ ] Forked YNL spec plus regen script once we add attributes.
|
|
- [ ] Look at the allowedips.c frame-size warning (1048 > 1024 bytes),
|
|
present in upstream code too.
|
|
- [ ] Split large peer lists across several netlink messages in the Go
|
|
client.
|
|
- [ ] Socket fd passing plus `encap_rcv` classifier (WG vs. disco/STUN).
|
|
- [ ] vmtest: disco/STUN still reach userspace on the shared socket.
|
|
- [ ] Per-peer no-auto-roam flag.
|
|
- [ ] Loopback DERP endpoints; vmtest with DERP-only path.
|
|
- [ ] tailscaled integration (wgengine backend that programs tswg) on a
|
|
tailscale.com branch.
|
|
- [ ] Fallback to wireguard-go when tswg is unavailable.
|
|
- [ ] Decide socket lifetime across tailscaled restarts.
|
|
- [ ] Decide minimum supported kernel.
|
|
- [ ] dkms.conf and packaging.
|
|
- [ ] Benchmarks versus wireguard-go (CPU/byte, pps, latency, forwarding).
|
|
- [ ] Custom debug kernel (KASAN, lockdep) in the testbed.
|
|
- [ ] UDP GSO/GRO on the encap socket (later perf work).
|
|
- [ ] Measure scaling with many peers; unknown-peer upcall only if needed.
|