Record how the kernel data plane now runs without polling: config changes trigger full syncs, magicsock's kernel path hook (with a trust expiry timer for idle sessions) triggers per-peer endpoint updates, and the module's peer events drive disco key advertisements and activity feedback, all applied by one goroutine. Updates #426 Signed-off-by: Brad Fitzpatrick <[email protected]> Change-Id: I8c555a623e8bba7cf7561fe94e4bc9d17fd40003
33 KiB
tswg: Tailscale-flavored kernel WireGuard
This is a living design document. It records requirements, decisions (with their reasons), and the TODO list. Update it whenever something is decided or changes.
Tracking issue: https://github.com/tailscale/tailscale/issues/426
Goal
Move the Tailscale data plane on Linux into the kernel. Packets between Tailscale peers should be encrypted, decrypted and forwarded by a kernel module without crossing into userspace on the direct (UDP) path. tailscaled keeps doing everything else: control plane, disco, NAT traversal (STUN, hole punching), port mapping (UPnP, NAT-PMP, PCP), DERP, and programming the kernel module.
Performance is the primary motivation for this work (stated 2026-11-06). Everything else is in service of getting the data plane out of userspace, so every change should be judged by its effect on performance, and performance must be measured, not assumed.
What we expect to win, compared to wireguard-go with TUN GSO/GRO and UDP GSO/GRO:
- CPU cost per byte and per packet.
- Latency and small-packet rate.
- Forwarding performance for subnet routers and exit nodes.
- Throughput, especially on small or many-core machines, once outer UDP GSO/GRO is in place (see "Performance" below).
Performance
What we inherit from upstream WireGuard
- Inner GSO on send: the device advertises GSO (
WG_NETDEV_FEATURESindevice.c), so the stack hands it large TCP super-packets, whichwg_xmitsegments (skb_gso_segment) before encrypting each segment. - Inner GRO on receive: decrypted packets go up through each peer's NAPI
context with
napi_gro_receive(receive.c), so the stack sees coalesced inner packets. - Parallel encryption and decryption across CPUs (per-device crypt workqueue), with per-peer ordering.
What's missing: outer UDP GSO/GRO
- Send: each encrypted packet is sent as its own UDP datagram through
udp_tunnel_xmit_skb, so every MTU-sized packet pays the full IP/UDP output and qdisc/driver cost. Building UDP GSO super-datagrams (SKB_GSO_UDP_L4) out of a peer's consecutive encrypted packets would amortize that, and let NICs with UDP segmentation offload do the splitting. - Receive: the encap socket has no
gro_receive, so encrypted datagrams aren't coalesced; each one goes through the IP/UDP input path separately. Agro_receive/gro_completepair (as VXLAN, Geneve and FOU have) could batch them per flow before decryption. - These are the kernel analogues of what makes wireguard-go fast. In the kernel there are no syscalls to amortize, so the expected win is smaller, but it is likely the biggest remaining throughput lever. Measure before and after.
Shared socket and GRO
The UDP stack segments GRO'd UDP packets before handing them to a socket
with encap_rcv set (udp_unexpected_gso, checked in both 6.8 and 7.4),
so UDP_GRO on the shared socket is safe but wasteful: datagrams get
coalesced and then immediately split again. In kernel mode, tailscaled
should not enable UDP_GRO (or rely on UDP_SEGMENT) on the shared
socket; only low-volume disco and STUN traffic reaches userspace there,
and the module's sends don't go through the socket send path at all.
Measuring
- A benchmark harness in the natlab testbed (iperf3-style TCP and UDP
throughput, pps with small packets, latency under load, CPU use via
/proc/stator perf), comparing tswg against wireguard-go (tailscaled in its usual TUN mode) on the same VMs. - natlab's vnet is a userspace network simulator, so it caps absolute numbers. Use it for relative comparisons and regressions; for real numbers, also benchmark on bare metal or between cloud VMs with real NICs.
Non-goals
- Upstreaming. tswg is an out-of-tree module, installed via DKMS or similar. We do not depend on upstream WireGuard accepting changes.
- Rust. Decided 2026-11-06: the in-tree Rust networking abstractions are too thin (no sk_buff, net_device, UDP tunnel, or lib/crypto bindings) and out-of-tree Rust modules need a kernel built with CONFIG_RUST and a matching rustc, which most distro kernels can't provide. Revisit later.
- Changing the WireGuard wire protocol. tswg must interoperate with every existing Tailscale peer running wireguard-go. (But see "Hybrid post-quantum handshakes": wireguard-go is about to grow a handshake tswg doesn't have.)
- Multipath or "spray" sending in the kernel. When disco hasn't yet trusted a direct path, tailscaled sprays from userspace (see "DERP and spray").
Architecture
tailscaled (userspace)
control, disco, STUN, portmapper, DERP client
| genetlink "tswg" | owns the UDP socket(s)
v v
+---------------------------------------------+
| tswg.ko (fork of drivers/net/wireguard) |
| tailscale0 netdev, noise, allowedips, ... |
| encap_rcv on tailscaled's UDP socket: |
| WG messages -> kernel |
| everything else -> socket receive queue |
+---------------------------------------------+
Fork, not patches
tswg is a fork of the in-tree WireGuard driver, kept as close to upstream as possible so upstream security fixes merge cleanly.
upstream/holds a pristine copy of the upstream files. It is only ever replaced wholesale by an import of a newer upstream; seeupstream/UPSTREAM_COMMIT.src/holds the fork.diff -r upstream/drivers/net/wireguard srcis the fork's delta (plus build files and compat headers).- Internal symbols keep their
wg_names. Unexported symbols in two modules don't conflict, so tswg.ko and the in-tree wireguard.ko (built-in or modular) can be loaded at the same time. Renaming them would make every upstream merge painful. - User-visible identifiers are renamed: module
tswg, rtnl link kindtswg(fromKBUILD_MODNAME), genetlink familytswg. - The uapi stays attribute-compatible with upstream WireGuard. New
attributes are only ever appended. That way wgctrl-go and a lightly
patched
wg(8)keep working by changing only the family name. - The uapi header is
src/include/uapi/linux/tswg.h. Upstream generates it, andsrc/generated/netlink.[ch], fromDocumentation/netlink/specs/wireguard.yamlwith YNL. For now we hand-edit the generated files; once we add attributes, keep a forked spec and regenerate with a kernel tree'stools/net/ynl/ynl-regen.sh(TODO).
Kernel compatibility
The fork's base is the newest upstream (currently 7.4-rc1). Older kernels
are supported through compat shims in src/compat/.
- Feature probes, not
LINUX_VERSION_CODEchecks (decided 2026-11-06; RHEL kernels carry years of backports, so version numbers lie).scripts/probe.shcompiles eachsrc/compat/probes/HAVE_*.cagainst the target kernel's build directory, in parallel (about one second in total), and writessrc/compat/generated/autoconf.h. The top-level Makefile runs it before every build. src/compat/compat.his force-included into every file and shims whatever is missing, mostly as macros or static inlines named like the new API, so upstream code stays untouched. The few spots that can't be shimmed from a header get small#ifdef HAVE_...blocks markedTSWG_COMPATin the.cfiles (currently onlydevice.c).- To add a shim: add a probe file, use its
HAVE_macro in compat.h, and build against both an old and a new kernel. - Use the kernel's lib/crypto (chacha20poly1305, blake2s, curve25519). Every distro kernel that ships WireGuard builds those libraries. Do not vendor crypto.
- Builds today: Ubuntu 6.8.0-106-generic (the natlab guest) and 7.0.0-1009-aws (the dev host). The 7.4 path itself (all probes succeeding) isn't verified yet, since that needs a configured and built 7.4 tree.
- Minimum kernel: TBD.
Userspace (Go) side
Decided 2026-11-06: the Go code lives in tailscale.com, because tailscaled is what will program the module.
tailscale.com/wgengine/tswg: netlink client for thetswggenetlink family (configure and dump devices).tailscale.com/cmd/tswgctl: a wg(8)-like debugging tool built on it.- This work lives on the
bradfitz/tswgbranch, in a worktree at~/src/tailscale.com-tswg.
Shared UDP socket
Tailscale needs disco, STUN and WireGuard to share one UDP 4-tuple so NAT
mappings discovered by disco are the ones WireGuard uses. Implemented
2026-11-06 (src/socket.c, "TSWG: user-supplied sockets"), modeled on
L2TP's user-supplied tunnel sockets and ESP-in-UDP:
- tailscaled creates and binds the UDP socket(s) and sets its socket options (SO_MARK, buffer sizes, and so on).
- tailscaled passes the fds to tswg in
WG_CMD_SET_DEVICE, asTSWGDEVICE_A_SOCK_FD4/TSWGDEVICE_A_SOCK_FD6(NLA_S32) nested inWGDEVICE_A_TSWG. The netlink request is processed in the caller's context, sosockfd_lookupsees the caller's fd table. - tswg validates the socket (UDP, the right family, bound, IPv6 sockets
must be
IPV6_V6ONLY, not already used by a tunnel), callssetup_udp_tunnel_sock()with itsencap_rcvandencap_destroy, and keeps a reference to thestruct sock(not the file). encap_rcvconsumes WireGuard messages (a first byte of 1-4 followed by three zero bytes) and returns 1 for everything else, which lands in the socket's normal receive queue for tailscaled. Nothing else Tailscale sends matches: disco starts with "TS", STUN with 0x00, or 0x01 followed by a nonzero byte, and Geneve (peer relay) with 0x00.- tailscaled sends disco and STUN with plain
sendmsgon the same socket.
Details and decisions:
- uapi numbering: all tswg device attributes live in one nested attribute,
WGDEVICE_A_TSWG= 64, far above upstream's (8 today), so upstream can keep appending its own without colliding. Peer attributes will get the same treatment (WGPEER_A_TSWG= 64). Dumps reportTSWGDEVICE_A_FLAGS(TSWGDEVICE_F_USER_SOCK4/6). - Socket lifetime is the lifetime of tailscaled's fd (decided
2026-11-06). When the process closes the socket (or exits), UDP calls
our
encap_destroyand the device drops it, leaving the device with no socket for that family until tailscaled passes a new one. No explicit detach op is needed for restarts. A restart that keeps the data plane up needs the fd to survive it, via systemd's fd store (FileDescriptorStoreMax=); that's a tailscaled-side TODO. - Passing -1 detaches the socket, and the device goes back to creating its own on its listen port (which fails with EADDRINUSE if tailscaled still holds that port).
- The user sockets are kept per device (
tswg_user_sock4/6) across ifdown/ifup; while the device is up they're also its activesock4/sock6. WireGuard messages arriving while the device is down are dropped; everything else still reaches userspace. - Setting
WGDEVICE_A_LISTEN_PORTto a different port while a user socket is attached fails with EBUSY. - Detaching restores the socket to plain UDP (
encap_rcv,encap_type,sk_user_data, checksum conversion). It leaves multicast loopback disabled, since we don't know its previous value, and leaves the encap static key reference for UDP to drop when the socket is destroyed. - A global mutex,
tswg_user_sock_lock, serializes attach and detach, including detach fromencap_destroy, which runs from the user'sclose()without any of our locks held. Lock order:device_update_lock, thentswg_user_sock_lock, thensocket_update_lock. UDP_GROon the shared socket is safe: the UDP stack segments GRO'd packets before handing them to a socket withencap_rcvset (checked in both 6.8 and 7.4). Making WireGuard itself use GRO is separate perf work.- We set
sk_allocation = GFP_ATOMICon the user socket, as WireGuard and L2TP do for their sockets, since we send from atomic context.
Endpoints and roaming
Upstream WireGuard roams a peer's endpoint to the source of the latest authenticated packet. That fights disco's path selection: a packet that arrives via DERP would steal the endpoint from a good direct path. Implemented 2026-11-06:
- A per-peer flag,
TSWGPEER_F_NO_ROAM, inTSWGPEER_A_FLAGSnested inWGPEER_A_TSWG(= 64, like the device attribute). SettingTSWGPEER_A_FLAGSreplaces all of the peer's tswg flags. Dumps report it. Go:tswg.PeerConfig.NoRoamandtswg.Peer.NoRoam. - Upstream learns endpoints in three places: on a valid handshake
initiation and a valid handshake response (via
wg_socket_set_peer_endpoint_from_skb) and on each authenticated data packet (wg_packet_consume_data_donecallswg_socket_set_peer_endpointdirectly).tswg_peer_may_roam()(inpeer.h) gates the learned updates, notwg_socket_set_peer_endpointitself, which netlink also uses for explicit updates. - With the flag set, the first authenticated packet still sets the endpoint if the peer doesn't have one yet; after that only userspace changes it. That keeps a peer reachable if it contacts us before tailscaled has given it an endpoint.
- The flag is applied before
WGPEER_A_ENDPOINTin the same request, so pinning and setting an endpoint are one step. - Per-peer rather than device-wide, for now. tailscaled will set it on every peer; a device-wide default is easy to add if per-peer turns out to be a hassle.
- Pinning also stops learning the local source address that upstream records alongside the endpoint (for replying from the same local IP on multihomed hosts); the kernel picks the source address by routing, as in userspace Tailscale.
Changing a peer's endpoint does not re-key or reset the noise session in upstream WireGuard either; that is already fine.
DERP, peer relays and spray: the loopback bridge
Implemented 2026-11-07 in tailscaled (wgengine/kerneldp_linux.go); it
needed no kernel changes. The rule: the kernel handles direct UDP paths
by itself; everything else (DERP, peer relays, and the untrusted-path
spray) goes through tailscaled's loopback bridge.
- The bridge is one UDP socket bound to
0.0.0.0:PwithSO_BINDTODEVICE lo(so only local traffic reaches it) andIP_PKTINFO. - Each peer gets its own address in 127.0.0.0/8 (127.0.0.2 upwards; the
mapping lives for the process's lifetime). When the kernel shouldn't
send to a peer directly, the peer's kernel endpoint is set to that
address and port P. The bridge reads what the kernel sends there, maps
the destination (from
IP_PKTINFO) back to the peer, and hands the ciphertext to magicsock'sendpoint.send, which sends it the way it would a packet from wireguard-go: DERP, a peer relay, or both DERP and the candidate UDP path while that isn't trusted yet (the spray). - In kernel mode, magicsock passes every WireGuard packet it would have
handed to wireguard-go (from DERP, or Geneve-wrapped from a peer relay)
to the bridge instead (
magicsock.Options.KernelWGRecv), which injects it into the kernel by sending it to127.0.0.1:<kernel listen port>from the peer's 127.x.y.z:P (ipi_spec_dst). The kernel finds the peer by receiver index (or, for handshake initiations, by decrypting the static key), so the source only matters for handshake rate limiting and cookie replies, which stay per-peer because each peer has its own address. Packets from unknown peers are injected from 127.0.0.1. - Each peer's kernel endpoint is magicsock's best address if it's direct
and trusted (
magicsock.Conn.KernelPeerPath), else the peer's bridge address; see "No polling" for how changes reach the kernel.
tailscaled integration
Implemented 2026-11-07 on the tailscale.com bradfitz/tswg branch. Opt
in with TS_DEBUG_TSWG=1 on Linux; the code is behind the tswg feature
tag (ts_omit_tswg drops it, and the minimal builds don't include it).
-
Startup: if the module is loaded, tailscaled creates a tswg device named tailscale0 (replacing any stale one) with the usual TUN MTU, and gives the engine and router a fake
tun.Devicethat only reports that name (tswg.NewTailscaleDevice), so the router configures addresses and routes on the tswg device. If the module isn't loaded, tailscaled logs why and falls back to TUN and wireguard-go. tailscaled doesn't load the module itself. -
The engine (
wgengine.Config.KernelWG) keeps wireguard-go running with no peers, since its receive loops are what drive magicsock's socket reads (disco, STUN). The kernel data plane:- sets the device fwmark to Tailscale's bypass mark
(
tsconst.LinuxBypassMarkNum), so the kernel's encrypted packets don't route into tailscale0 (for example via an exit node's default route), as magicsock's sockets do; - shares magicsock's first UDP socket per family with the kernel, again
after every rebind (
magicsock.Conn.SetKernelSocketHook); - reconciles the kernel's peers with magicsock's peers and the live
per-peer config (
SetPeerConfigFunc, whose peer set is otherwise lazy): allowed IPs, preshared key, endpoint, and no-roam on all of them, in netlink messages of at most 32 peers; - feeds direct-path activity, from the kernel's peer events, to
magicsock (
NoteKernelActivity), so it keeps heartbeating active sessions and the direct path stays trusted, and reports per-peer status (bytes, last handshake) in the engine status, from a dump when asked.
- sets the device fwmark to Tailscale's bypass mark
(
-
Tests:
TestTSWGTailscaledDirect(two gokrazy VMs behind easy NATs; the kernel ends up sending directly to the peer's NAT-traversed public address) andTestTSWGTailscaledDERP(hard NATs on both sides; the kernel peer stays on its bridge address and everything flows over DERP). Both start tailscaled before the module is loaded, so they also cover the TUN fallback, then load the module and restart tailscaled. -
wireguard-go must never get peers in kernel mode: if it did (from its lazy peer lookup on an outbound packet or an incoming handshake), it would handshake over magicsock with our keys and fight the kernel's sessions. So in kernel mode the engine doesn't install wireguard-go's
PeerLookupFuncorPeerByIPPacketFunc(decided 2026-11-07).
TSMP and other packets tailscaled sends to peers
Implemented 2026-11-07 (wgengine/kerneldp_tsmp_linux.go). With
wireguard-go, TSMP (IP protocol 99) and the other packets tailscaled
itself exchanges with peers pass through the tstun.Wrapper between the
TUN device and wireguard-go. In kernel mode:
- Outbound: tstun gets an outbound inject hook
(
tstun.Wrapper.SetOutboundInjectHook). In kernel mode, everything tailscaled injects towards peers (TSMP pings, pongs and disco key advertisements, ICMP echo replies, netstack output) goes to it instead of wireguard-go's queue, and the kernel data plane writes it, header and all, to a rawIPPROTO_RAWsocket bound to tailscale0 (IPv4 and IPv6), so the kernel routes it into the tswg device, which encrypts it. - Inbound: raw
IPPROTO_TSMPsockets bound to tailscale0 receive the TSMP packets peers send, and the data plane feeds them totstun.Wrapper.Writeas wireguard-go would, which handles them as usual (answering pings, delivering pongs, learning disco keys). IPv6 raw sockets don't return the IP header, so it's rebuilt from the source address and theIPV6_PKTINFOandIPV6_HOPLIMITcontrol messages. With such a raw socket open, the kernel doesn't answer TSMP with ICMP protocol unreachable. - wireguard-go sends a "priority message" (a TSMP disco key advertisement) whenever it establishes a session. The kernel data plane does the same on the kernel's session event for the peer (see "No polling").
- Tested by
TestTSWGTailscaledTSMP: TSMP pings both ways, and each side receives the other's disco key advertisement (magicsock_tsmp_disco_key_advertisement_received).
No polling
Done 2026-11-07 (at first the data plane polled every second). Nothing in the kernel data plane runs on a timer now; everything is driven by events:
- Peer set and config:
SyncDevicePeer,SetPeerConfigFuncand private key changes request a full sync. - Paths: magicsock calls a hook (
Conn.SetKernelPathHook) whenever a peer's kernel path flips between direct-and-trusted and not. All changes tobestAddrgo throughsetBestAddrLockedand all changes totrustBestAddrUntilthroughsetTrustBestAddrUntilLocked, which check for a flip. Trust lapsing is an event with no packet attached, so while a peer's kernel path is direct, magicsock keeps a timer armed fortrustBestAddrUntil(re-armed if trust was extended meanwhile). The data plane then updates just that peer's endpoint. - While a session is active, the fallback when a direct path dies comes
from magicsock's heartbeat ping timing out after trust lapsed
(
discoPingTimeoutclears the best address). The trust timer is what covers idle sessions: once magicsock stops heartbeating (45 seconds without traffic), trust lapses silently, and the kernel must stop sending directly so that the next packets go through magicsock (DERP plus the candidate path) until the path is trusted again, as with wireguard-go. - Kernel to tailscaled: the module multicasts peer events
(
TSWG_CMD_PEER_EVENTon the "events" genetlink group): a new session (TSWGEVENT_F_SESSION, fromwg_timers_session_derived), and TX and RX activity (TSWGEVENT_F_TX,TSWGEVENT_F_RX), each rate limited to one per peer every 2 seconds (TSWG_ACTIVITY_EVENT_INTERVAL) with a single comparison on the packet path. Messages are only built when someone listens. tailscaled sends the disco key advertisement on session events and feeds direct-path activity to magicsock. If the event socket overruns, events are lost but harmlessly: activity recurs, and the advertisement is an optimization. - All changes to the kernel's peers are applied by one goroutine: hooks only record what to do (full sync, dirty paths, session clears) and kick it, so a sync that read paths a moment ago can't overwrite a newer endpoint, and callers holding LocalBackend locks never wait on a sync that needs them.
- Tests:
TestTSWGTailscaledPathFallback(a direct path dies mid-session; the kernel falls back to the bridge and DERP about 8 seconds later) andTestTSWGTailscaledIdleTrustExpiry(a session goes idle; the kernel moves to the bridge once trust lapses, and back to direct when traffic resumes; fails without the trust timer).
Peer restarts
Implemented 2026-11-07. When a peer's disco key changes in a way that
means it restarted (and so lost its WireGuard keys), ipnlocal calls
Engine.MarkDevicePeerForHandshake. wireguard-go then handshakes after
the next outbound packet. In kernel mode, the data plane instead clears
the kernel's session with the peer (TSWGPEER_A_CLEAR_SESSION: forget
the keypairs and any handshake in progress, as on suspend, and reset the
handshake rate limit), so the next packet is staged and starts a new
handshake right away. In the same request it updates the peer's endpoint
from magicsock's current path, since a restarted peer usually has a new
port, and the handshake should go there rather than wait for the next
reconcile.
Without this, the kernel keeps sending with dead keys until WireGuard's
own timers start a new handshake about 15 seconds later.
TestTSWGTailscaledPeerRestart checks that the first pings after a peer
restart get through (and fails without the session clearing).
Known gaps in kernel mode
Deliberately deferred (2026-11-07: "easy later, not a priority yet"):
- No packet filter: the engine's filter runs on packets passing through userspace, which kernel-mode packets don't. Peers can reach anything on the node. Must be done (nftables, or in the module) before anyone uses this for real.
- No MagicDNS at 100.100.100.100 (or other netstack-provided services).
Other TODOs:
- Only magicsock's first socket per family is shared;
UDPSockets> 1 isn't supported. - A full sync (on any config change) still walks every peer; fine for now, but worth making incremental for huge tailnets.
Hybrid post-quantum handshakes
Found 2026-11-07: Tailscale's wireguard-go fork has a hybrid
post-quantum handshake (wgcfg.PeerConfig.Hybrid, different message
sizes), wired up in tailscale.com on 2026-10-29, with a control
"disable hybrid" capability for the initial rollout (2026-11-03). Nothing
sets Hybrid yet, but once it's in use between two nodes, a kernel-mode
node can't talk to a hybrid-only peer, since tswg only does classic
WireGuard. So the "wire-compatible with every Tailscale peer" assumption
no longer holds. Options:
- a kernel-mode node advertises (to control) that it doesn't do hybrid handshakes, so peers use classic handshakes with it; or
- implement the hybrid handshake in tswg (needs a kernel ML-KEM, or a vendored one).
For now the kernel data plane logs any peer that asks for a hybrid handshake and uses a classic handshake.
What doesn't need kernel work
- ACLs: nftables (tailscaled already has nftables and iptables code; conntrack handles return traffic).
- MagicDNS, peerapi, Tailscale SSH, Taildrop: kernel sockets on the tailscale0 address.
- TSMP: raw socket on IP protocol 99.
- Subnet routers and exit nodes: kernel forwarding.
Lazy peer configuration
tailscaled only programs recently active peers into wireguard-go, mostly
because of per-peer cost in wireguard-go. Kernel peers are cheap and
hashed, so first try programming every peer. If that doesn't scale, add an
unknown-peer upcall from wg_noise_handshake_consume_initiation with a
short stash of the initiation for replay (to avoid a 5s REKEY_TIMEOUT
stall).
DKMS and fallback
Out-of-tree modules on Secure Boot systems need signing and MOK enrollment. tailscaled must always fall back to wireguard-go when tswg is missing, fails to load, or reports an incompatible version (the genetlink family version or a feature attribute).
Testbed
Integration tests run as natlab vmtests in tailscale.com
(tstest/natlab/vmtest), which boot QEMU VMs connected by vnet, a
simulated network with NATs, DERP and a test control server. The tests
live in the tailscale.com worktree, in tstest/natlab/vmtest/tswg_test.go.
Run them with scripts/vmtest.sh [TEST_REGEXP]; it sets TSWG_DIR to this
tree, and the tests skip if the tree is missing.
Primary guest: gokrazy
Decided 2026-11-06: the natlab gokrazy image is the primary test guest.
- It boots a recent upstream kernel (7.2.8 today), from the
github.com/gokrazy/kernel.amd64package pinned in tailscale.com's go.mod. A gokrazy bot bumps that package to new kernel releases, and a tailscale.com cron bot bumps the go.mod dependency, so the test guest follows upstream kernels without anyone doing anything. That keeps the fork honest against new kernel APIs, which is where the fork's base is anyway. - Consequence: a gokrazy vmtest failure after a dependency bump most likely means kernel API churn that needs a new compat probe and shim (or an upstream WireGuard change worth importing), not a test flake.
- gokrazy has no userspace tools (no modprobe, iproute2 or shell), just
like the end state, where tailscaled drives the kernel entirely. So
everything is done from Go:
tswg.LoadModulecalls init_module(2) on the tswg.ko image, and links and addresses are set up over rtnetlink. - gokrazy kernels are kernel.org releases plus the patches in
_build/series, with an embedded config (CONFIG_IKCONFIG) and without MODVERSIONS or module signing.scripts/gokrazy-kdir.shrecreates a matching module build tree (download, patch, embedded config,modules_prepare; about 20 seconds, cached). We build with the host's gcc 13 while gokrazy uses Debian 12's gcc 12; the only resulting config differences are toolchain capability bits. If that ever matters, build in a Debian 12 container instead. - Such a tree has no
Module.symvers, so modpost can't check the kernel's exports (we setKBUILD_MODPOST_WARN=1); a missing symbol shows up as a load failure instead. - The test agent (
cmd/tta) has/tswg/load(POST tswg.ko),/tswg/up(create, configure and bring up a one-peer device),/tswg/show(device JSON) and/tswg/dmesg(kernel log lines mentioning tswg, read from /dev/kmsg). TestTSWGGokrazyManualTunnel: two gokrazy VMs on one LAN, tunnel set up through tta, ping across it, and check handshake and byte counters.
Secondary guest: Ubuntu 24.04
Kept for distro and DKMS-style coverage, since that's how real users will install the module.
- The natlab Ubuntu 24.04 cloud image, stock kernel
6.8.0-106-generic. The host is also Ubuntu 24.04 with the same gcc (13.3), soscripts/build-for-ubuntu.shbuilds tswg.ko against that kernel's headers, fetched withapt-get downloadand unpacked into a cache directory (never installed). - The test asks the guest for
uname -r, builds tswg.ko andtswgctl, registers both with the vnet file server, and has the guest curl them fromhttp://52.52.0.6/tswg/<kver>/, thendepmodandmodprobe tswg. Commands run as root over the debug SSH NIC. TestTSWGManualTunnel: the same tunnel check, configured withip linkandtswgctl.
Later
- A custom debug kernel (KASAN, lockdep) built from a Linux tree, which would also cover the 7.4 compat path.
- More distro images (Debian, Fedora) for compat coverage.
Repo layout
DESIGN.md: this file.upstream/: pristine upstream sources (see "Fork, not patches").src/: the tswg module sources, Kbuild and compat headers.scripts/:probe.sh(compat probes),build.sh(build tswg.ko against a KDIR in a private copy ofsrc/),ubuntu-headers.shandbuild-for-ubuntu.sh(Ubuntu kernels),gokrazy-kdir.shandbuild-for-gokrazy.sh(gokrazy kernels),extract-ikconfig(vendored from Linux),vmtest.sh(run the natlab vmtests).
TODO
Performance (the primary motivation; see "Performance"):
- Benchmark harness: tswg vs. wireguard-go (tailscaled TUN mode) on the same VMs: TCP/UDP throughput, small-packet pps, latency under load, CPU per byte and per packet, subnet router and exit node forwarding.
- Benchmarks on real NICs (bare metal or cloud VMs), not just natlab.
- Outer UDP GSO on send: coalesce a peer's consecutive encrypted
packets into
SKB_GSO_UDP_L4super-datagrams. - Outer UDP GRO on receive:
gro_receive/gro_completeon the encap socket to batch encrypted datagrams before decryption. - vmtest: shared socket with
UDP_GROenabled still works (exercises the segment-before-encap_rcv path at runtime). - tailscaled: don't enable
UDP_GRO/UDP_SEGMENTon the shared socket in kernel mode. - Profile the encrypt/decrypt path (workqueue fan-out, per-peer ordering) under many-flow and many-peer loads.
- Measure scaling with many peers; unknown-peer upcall only if needed.
Everything else:
- Create repo; import pristine upstream WireGuard (Linux 7.4-rc1).
- Write DESIGN.md.
- Fork into
src/: rename module, link kind and genetlink family totswg; out-of-tree Kbuild/Makefile. - Feature-probe compat framework.
- Build against the host's 7.0 headers.
- Script: fetch and unpack Ubuntu kernel headers for a given version; build tswg.ko against them.
- Compat shims so it builds on Ubuntu 24.04 (6.8).
- Go netlink client (
tailscale.com/wgengine/tswg) andtswgctl. - vmtest: two Ubuntu VMs, load tswg.ko, configure a tswg tunnel by
hand, ping through it (
TestTSWGManualTunnel, passes in about 80s). - Same on gokrazy, with no userspace tools
(
TestTSWGGokrazyManualTunnel, passes in about 23s). - Generate a Module.symvers for gokrazy build trees (for example from the kernel's exported-symbol list) so missing exports fail at build time rather than load time.
- Verify the 7.4 path (all probes succeed) against a built 7.4 tree. (Partly covered: gokrazy's 7.2.8 has every probed API except udp_get_len_short.)
- Forked YNL spec plus regen script. (We now have hand-written
additions in
src/generated/netlink.[ch], marked TSWG.) - Look at the allowedips.c frame-size warning (1048 > 1024 bytes), present in upstream code too.
- Split large peer lists across several netlink messages in the Go client.
- Socket fd passing plus
encap_rcvclassifier (WG vs. disco/STUN). - vmtest: disco/STUN still reach userspace on the shared socket, from
the same 4-tuple; no WireGuard leaks to userspace; closing the fd
detaches; a new fd reattaches (
TestTSWG{Gokrazy,Ubuntu}SharedSocket). - tailscaled: keep the shared sockets in systemd's fd store across restarts.
- Per-peer no-auto-roam flag (
TestTSWG{Gokrazy,Ubuntu}NoRoam). - Loopback DERP endpoints; vmtest with DERP-only path
(
TestTSWGTailscaledDERP). - tailscaled integration on the tailscale.com
bradfitz/tswgbranch (TestTSWGTailscaledDirect). - Kernel mode: packet filter (security; required before real use; deferred).
- Kernel mode: hybrid PQ handshakes (advertise no support, or implement them in tswg).
- Kernel mode: TSMP (
TestTSWGTailscaledTSMP). - Kernel mode: MagicDNS and the other netstack-provided services (deferred).
- Kernel mode: clear the session on MarkDevicePeerForHandshake
(
TestTSWGTailscaledPeerRestart). - Kernel mode: event-driven everything, no polling
(
TestTSWGTailscaledPathFallback,TestTSWGTailscaledIdleTrustExpiry). - Kernel mode: IPv6-only and dual-stack vmtests.
- Fallback to wireguard-go when tswg is unavailable (module not loaded).
- Fallback on version or feature mismatch between tailscaled and the loaded module.
- Decide minimum supported kernel.
- dkms.conf and packaging.
- Custom debug kernel (KASAN, lockdep) in the testbed.