ec2.sh gets "resize TYPE N..." (for stopped instances) and "start N..."
(some instances), and numbers new instances after all existing ones,
stopped ones included.
With a c7i.48xlarge router (50 Gbps) and 8 clients, tswg forwards 31.5
Gbps at MTU 1280 (wireguard-go 7.2, kernel WireGuard 12.3 using 137 of
192 CPUs), limited by EC2's packet rate per NIC receive queue (about
715k pps, 7.3 Gbps of tunnel packets) with only 8 tunnel flows to spread
over 16 queues; and at line rate with a jumbo tunnel MTU, a data point
only, since MTU 1280 is the target.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I70744651f3385343bf8890607911d9b4e0f70620
gokrazy-debug-kernel.sh hashed sha256sum's output for its cache key,
which includes the config's random mktemp path, so every run built
another 3.5GB debug kernel; seven of them filled the disk. Hash the
files' contents instead.
ec2.sh sorted hosts by name, which put bradfitz-tswg-10 before
bradfitz-tswg-2; sort by number.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ia758dae0077c54a2ef8f6112ae3c43d6604f66e1
subnetbench.sh runs tailscale.com's TestLeanSubnetRouter on the EC2
instances for each router CPU count given. ec2.sh now turns off source
and destination checks on its instances, so that one can forward as a
subnet router does, and installs nftables (for the router's masquerade).
DESIGN.md records the results: with 4 clients, tswg as the subnet
router forwards 2.1x (4 CPUs) to 3.2x (8 CPUs) what wireguard-go does,
which uses about 3 CPUs however many it has, at the least CPU per byte
of any tunnel, and reaches the 25 Gbps link at 16 CPUs. Zero-copy
receive was tried there and was 7% worse, so isn't in.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I6b1da8bb8eaa17f9b20cbe36896426411b8b905f
The QEMU lab runs on a host that's itself a VM, which distorted latency
in particular. ec2.sh manages benchmark instances in the sandbox AWS
account (us-west-2; the org's SCP allows only some regions and requires
IMDSv2): a key pair, a security group open to this host and between its
members, a cluster placement group, and Ubuntu 24.04 instances with IPv6
that install the build tools for their own kernel. Everything is named
bradfitz-tswg... and tagged owner=claude-tswg. It writes the hosts file
that tailscale.com's lean harness takes in remote mode.
To keep costs down, ec2-idle-stop.sh, run every 5 minutes from a systemd
user timer on the always-on dev host, stops the instances once nothing
has touched ~/.cache/tslinux/ec2/last-used for 30 minutes. ec2.sh and the
harness touch it while in use, and "ec2.sh start" brings stopped
instances back.
First EC2 results (c7i.xlarge pair) are in DESIGN.md: throughput is
capped by EC2 at about 7.25 Gbps for both tswg and wireguard-go, with
tswg's receiver using up to 17% less CPU per byte; 1.6x the small
packets; a quarter of the RTT under load at p50; twice the idle RTT.
Updates #426
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: Ibe770a30bea31cb79ac514aeb545f468576b5f05