Adds a test that issueACMECert, against tstest/acmetest, retries an
order naming "replaces" once without it when the CA rejects it, and
stores the cert from the retried order. It also checks that rate limits
and server errors are not retried, and that the error from the plain
order is returned if that is rejected too.
Updates #22042
Signed-off-by: chaosinthecrd <[email protected]>
When a previous cert exists, the renewal order names it in the ARI
"replaces" field. Let's Encrypt rejects that order if the current ACME
account did not issue the cert, which is the case once the account key
has been recreated but the cert kept (for example after
acme-account.key.pem is deleted), so renewal fails until the cert
expires or is deleted. This fixes this by retrying the order once
without "replaces" when the CA rejects it, unless it was rate limited.
The new cert belongs to the current account, so later renewals use
"replaces" again.
Fixes#22042
Signed-off-by: chaosinthecrd <[email protected]>
Tests that the localapi cert-status/ handler checks read access and returns
the status as JSON, that LocalBackend.CertStatus goes through the hook, and
that feature/acme's hook reports the same missing, valid and expired results
as the C2N /tls-cert-status handler. Also tests that Client.CertStatus
returns ErrCertStatusUnsupported for a 404 and not for other errors.
Updates tailscale/corp#38175
Signed-off-by: chaosinthecrd <[email protected]>
The localapi cert/ endpoint fetches a cert when none is cached, so polling it
triggers issuance. feature/acme can already report a cached cert without
contacting the CA, but only over the C2N /tls-cert-status path. This moves
that check into an extension method and serves it as localapi cert-status/,
with a matching Client.CertStatus, reporting Missing on a cache miss instead
of issuing. Client.CertStatus returns ErrCertStatusUnsupported (wrapping
errors.ErrUnsupported) when tailscaled has no cert-status endpoint.
Updates tailscale/corp#38175
Signed-off-by: chaosinthecrd <[email protected]>
feature/acme uses LocalBackend.ServeConfig to find hostnames needing
cert refresh, which broke builds with ts_omit_serve but not
ts_omit_acme. The serveConfig field it reads exists in all builds, so
move the method out of the serve-only file rather than making acme
depend on serve, and skip the refresh loop hook when serve is omitted
so the linker can drop it.
Updates #22283
Change-Id: I2cef6683cd6e6144306ddec95255d606f7a539ae
Signed-off-by: Brad Fitzpatrick <[email protected]>
When validation fails, WaitOrder returns only the order's status, so
tailscale cert printed "acme: order ... status: invalid" with no reason.
The CA's reason is on the authorization's challenge. Wait on each
authorization before the order and return that error instead.
To test this, move natlab's fake ACME CA to tstest/acmetest, where unit
tests can import it, and teach it tls-alpn-01, a validation hook, and
the "invalid" state a CA reports when validation fails.
Updates #21759
Signed-off-by: Brendan Creane <[email protected]>
The cert refresh loop treated every Web entry in the ServeConfig as
HTTPS, so a node serving only plain HTTP fetched and renewed a Let's
Encrypt cert for its name. Check the port's TCP handler for HTTPS
before considering a Web entry, for node and Service entries alike.
Fixes#21693
Signed-off-by: Brendan Creane <[email protected]>
The ts_omit_<name> build tags omit a feature at build time; there has
been no way to do the same at runtime. Some users (either proactively
or in response to a security announcement) might like a way to disable
a feature that's linked-in in their binaries that they're not using.
Then a mitigation announcement can say "set this env var" without
asking users to rebuild or wait for a new release.
This adds env var TS_DISABLE_FEATURE, a comma-separated list of
feature names to disable, and the listed set is reported by the
debug-optional-features LocalAPI endpoint next to the registered set.
The legacy per-feature knobs such as TS_DISABLE_SSH_SERVER and
TS_DISABLE_TAILDROP keep working independently.
A disabled feature behaves as if it had not been linked: it is absent
from feature.IsRegistered, its hooks are unset, and its extensions and
handlers are not registered. Three pieces make that happen:
* feature.Register now returns bool, false when disabled, and
feature packages gate their registration init on it. It was added
to the feature packages that never called it (including taildrop
and ssh), which also completes the picture reported by
debug-optional-features. taildrop, routecheck, favorites, and
serviceclientprefs had registration split across several inits and
now register from one gated init.
* ipnext.RegisterExtension ignores a disabled feature's extension.
* feature.Hook.Set and feature.Hooks.Add walk the call stack and
silently skip when the calling package under feature/<name> is
disabled. This covers sub-packages such as
feature/captiveportal/netcheckhook, which cannot call Register
themselves without colliding with their parent, and future
packages whose authors forget the gate.
ssh/tailssh's registrations moved from its inits into tailssh.Register,
called from feature/ssh's gated init. The aws and kube state stores and
syspolicy's Windows store registration are gated too.
feature/register_disable_test.go runs this test binary as a child
process (it links condregister, as tailscaled does) with
TS_DISABLE_FEATURE set to every registered feature at once, and fails
if any of them register anyway, so a feature that ignores the variable
cannot land.
Updates #12614
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I720af6ccab844ae060a9dfd1539fee577fd483e3
Our tests are inconsistent in how they set environment variables and cleanup.
Missing cleanup logic can leak environment variable state across tests and
change the behaviour of subsequent tests.
This recently caused flakes in feature/acme, where `TestGetCertPEMWithValidity`
leaked `TS_CERT_SHARE_MODE` and `TestAsyncRenewalDedup` to fail inconsistently.
To fix the immediate flake and prevent future leaks, introduce a `SetenvForTest`
helper that handles setting and cleaning up environment variables in tests.
Fixes#20902
Change-Id: I9f8ef45ec875d66034cfd0054311bd69b67e2243
Signed-off-by: Alex Chan <[email protected]>
Currently we serialise ACME account setup and ACME issuances for domains
which don't yet have a valid certificate, but not for async renewals of
still-valid certificates.
This patch adds a check that we only have one async renewal for a domain
in-flight at a time, and a test that ensures we de-dupe these renewals.
Updates tailscale/corp#46420
Change-Id: Ibbaa537cd28b8adf238b5c9396403e845f95b7e7
Signed-off-by: Alex Chan <[email protected]>
Package tailcfg defines the types and constants used by the Tailscale
protocol, but since everything is all in one package, it’s difficult
to sift through the docs: https://pkg.go.dev/tailscale.com/tailcfg
We define and enumerate capabilities as string constants for
tailcfg.NodeCapability and tailcfg.PeerCapability. This PR extracts
them into their own packages:
- tailcfg.CapabilityFileSharing becomes nodecap.FileSharing
- tailcfg.NodeAttrOnlyTCP443 becomes nodecap.OnlyTCP443
- tailcfg.PeerCapabilityTaildrive becomes peercap.Taildrive
We originally intended for CapabilityFoo to grant an entitlement or
permission for Foo, and for NodeAttrBar to configure Bar in the
nodeAttrs section of the policy file. However, there was no technical
enforcement of this convention, so new capabilities have used the
NodeAttr prefix regardless of meaning. Therefore, this PR unifies
tailcfg.CapabilityFoo and tailcfg.NodeAttrBar into a single package as
nodecap.Foo and nodecap.Bar.
Ran `go fix -inline ./...` and committed the changes that replaced
uses of the tailcfg aliases with the authoritative ones.
Updates #20259
Change-Id: Ieb7e7e6c8247c39faf42fdf15c68cdc7c621c730
Signed-off-by: Simon Law <[email protected]>
Introduce a per-tailnet shared ACME account key so that all ingress
ProxyGroup replicas on a tailnet present the same account identity to
Let's Encrypt. This lets renewals claim the ARI "replaces" exemption
from the 50-certs-per-week rate limit, surviving Pod restarts,
ProxyGroup recreation, and cluster migrations.
The operator provisions a "tailscale-acme-accounts" Secret in its
namespace, guarded by a finalizer and a deletion warning event, and
watched so it is recreated promptly if removed. Proxies migrate any
pre-existing per-pod key into the shared Secret on first boot, adopt
the shared key on subsequent boots, and restore it on cert writes if
the Secret was recreated empty. Certs are stamped with the fingerprint
of the issuing account so renewals skip the "replaces" claim when the
account doesn't match.
Opt-in per-ProxyGroup via the tailscale.com/share-acme-account
annotation, or operator-wide via OPERATOR_SHARED_ACME_ACCOUNT_KEY.
Updates #18251
Updates #20288
Signed-off-by: chaosinthecrd <[email protected]>
An SNI ServerName with a trailing dot (e.g. "host.ts.net.") failed
cert lookup because stored cert names have no trailing dot. Per RFC
6066 section 3 the SNI HostName carries no trailing dot, but some
clients send a fully-qualified name with one.
Trim the trailing dot at the boundary in getCertPEMWithValidity so all
lookup paths (the GetCertificate hook, Serve, and the localapi) resolve
the dotted and dotless forms to the same certificate.
Fixes#10233
Signed-off-by: Saleh <[email protected]>
The extension's acmeMu was a single lock around getCertPEM. Any
in-flight ACME flow blocked every other domain. With many domains
(ProxyGroup ingress) the queue would back up and per-call timeouts
started firing while we were just waiting on the lock -- the cert
loop treated that as a failure.
Replace with one mutex per domain. Different domains run at the
same time. Same domain still queues so the first run fills the
cache and the rest read from it.
The old global lock also kept ACME account setup safe by accident.
Two goroutines could both find no account key, both generate one,
both write -- last one wins on disk but each carries on with its
own. Add acmeAccountMu around acmeKey and ensureACMEAccount to
keep that path single-file. Otherwise two first-time issuances for
different domains end up with separate accounts at LE.
Updates #20288
Updates tailscale/corp#42164
Signed-off-by: chaosinthecrd <[email protected]>
* ipn/localapi,ipnlocal,feature/acme,client/local: honour Retry-After on cert rate-limit
serveCert now responds with 429 + Retry-After when the underlying ACME
error is a rate limit, instead of a generic 500. client/local surfaces
this as a typed RateLimitedError with the parsed hint so callers can
back off intelligently.
Updates tailscale/corp#42164
Signed-off-by: chaosinthecrd <[email protected]>
* tsweb,feature/acme,ipn/localapi,ipnlocal: generalise cert error → HTTP mapping via tsweb.HTTPStatuser
Introduces a tsweb.HTTPStatuser interface, any error can implement
to describe its intended HTTP response (code, message, headers).
Moves CertRateLimitedError from ipnlocal to feature/acme where it's
constructed, and it now uses HTTPStatuser to return 429 + Retry-After.
serveCert now checks for tsweb.HTTPStatuser rather than the specific
error type, so it no longer needs to know about the ACME rate-limit
type.
Signed-off-by: chaosinthecrd <[email protected]>
---------
Signed-off-by: chaosinthecrd <[email protected]>
f5eac39ea ("feature/acme, ipn/ipnlocal: start moving ACME/cert state
into an extension") started to move the cert code into feature/acme
but was meant as a baby step.
This goes further, moving almost everything, leaving only some hooks
in ipnlocal.
When we later move "serve" support out to feature/serve, this will
look a bit different in that the hooks currently in ipnlocal will move
to feature/serve (cert support already depends on serve).
As part of this, cert-related tests move to feaure/acme too, which
means some test infra from ipnlocal now moves to shared ipnlocaltest.
(it's not big at the moment, but I imagine it growing)
Updates #12614
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I9ea89aa9754f12d54b81751b6bd830f2664241ff
The ACME serialization mutex (acmeMu) was a package-level global, and
several ACME-related fields lived on LocalBackend even though the
cert code is conditional and not linked into every binary. With
multiple tsnet.Servers in one process (each its own LocalBackend),
a process-wide acmeMu also serialized unrelated backends.
Introduce a new feature/acme extension that owns the per-LocalBackend
ACME/cert state in an ipnlocal.CertState value:
- acmeMu, renewMu, renewCertAt (previously package globals)
- pendingACMETLSALPNCerts, pendingCertDomains{,Mu},
getCertForTest, certRefreshCancel (previously LocalBackend
fields, only meaningful when ACME was compiled in)
ipnlocal/cert.go now reaches the state through b.certState(), which
is routed by a feature.Hook installed at init by feature/acme. The
CertState type lives in ipnlocal so cert.go can access its fields
directly without a method explosion; the extension in feature/acme
constructs and owns it.
This is a baby step. The end goal is for the entire cert/ACME code
to live in feature/acme, with ipnlocal only retaining whatever thin
hooks the rest of LocalBackend needs to call into it. The current
split (CertState and most of cert.go in ipnlocal, extension wrapper
in feature/acme) is a deliberately temporary middle ground that
keeps this PR small while making the next moves mechanical.
The package is named feature/acme to match the existing HasACME /
ts_omit_acme naming. condregister/maybe_acme.go wires it in for
non-js builds.
Updates #12614
Updates #20248
Updates #20249
Signed-off-by: Brad Fitzpatrick <[email protected]>
Change-Id: I520909f24ad11a9622ef33c2290fe36ad44d6f71