From 003fef170a21eebd349ccebbde0eed915ec695df Mon Sep 17 00:00:00 2001 From: Brad Fitzpatrick Date: Sat, 7 Nov 2026 19:21:46 +0000 Subject: [PATCH] kernel/linux/DESIGN.md: record the 24-peer, MTU 1280 subnet router results With 24 clients at MTU 1280, tswg forwards 40 Gbps through a c7i.48xlarge (5.6x wireguard-go, 3x kernel WireGuard), using about 22 of its 192 CPUs, and is then limited by EC2's packet rate allowance for the instance. Updates #426 Signed-off-by: Brad Fitzpatrick Change-Id: Ifdb78f7d85d182a89a42c2f0f9f77bd14344fb59 --- kernel/linux/DESIGN.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/kernel/linux/DESIGN.md b/kernel/linux/DESIGN.md index ee8ed6bb0..864699d06 100644 --- a/kernel/linux/DESIGN.md +++ b/kernel/linux/DESIGN.md @@ -369,8 +369,19 @@ At 50 Gbps (2026-11-07): router and backend c7i.48xlarge (192 vCPUs, c7i sizes), and with 8 peers, flows collide on queues (4 clients on 3 queues: the two sharing got half each). More peers spread better; a larger MTU needs 7x fewer packets. The router's CPUs weren't the limit. -* For 100 Gbps, the next step: c6in instances (up to 200 Gbps), which - the IAM policy doesn't allow yet. +* With more peers at MTU 1280 (24 clients: the 8 above plus 16 + c7i.2xlarge, 8 flows each, 192 flows), traffic spreads over all the + queues: tswg 39996 Mbps (4.49 CPU-s/GB, about 22 of the 192 CPUs busy), + wireguard-go 7107 (5.6x less), kernel WireGuard 13463 (78% of 192 + CPUs). There the limit is EC2's packet rate allowance for the + instance (ENA's `pps_allowance_exceeded` rises during tswg's runs): at + MTU 1280 everything the router handles is small (tunnel packets in, + 1280-byte TCP segments out to the backend, ACKs both ways), about 12M + packets per second at 40 Gbps. (The plain baseline isn't comparable + here: without a tunnel, the clients' TCP uses the VPC's 9001-byte MTU.) +* For more: instances with higher packet rate allowances, such as c6in + (network optimized, up to 200 Gbps), which the IAM policy doesn't allow + yet. Profile of the tswg router at 4 CPUs (CPU-bound), before the optimizations: ChaCha20-Poly1305 about 23% (of which 5.5% was a scalar