fix(san-cert): serve the held certificate while its replacement issues

[mhenrixon]

Summary

• internal/server/san_cert_manager.go — GetCertificate no longer blocks the TLS handshake on a synchronous ACME order for a registered domain whose certificate is merely due for replacement (inside 24h of expiry, or issued by a directory its service has moved away from). While the certificate is still valid it keeps serving, and the replacement is queued on the domain issuer — which already brings dedup, quarantine, preflight-skip for held certs, the shared rate bucket, and directory-aware issuable checks. Synchronous handshake provisioning remains only where it buys anything: first issuance and actual expiry.
• Evicted domains behave exactly as before (serve out the held cert, no replacement, refusal after expiry) — now expressed by the same branch instead of a parallel one.

The failure mode this removes: reaching the <24h window means the renewer has been failing for weeks (ACME outage, quarantine-deferred batch), and the old code chose that exact moment to gamble every handshake on a live order — failing connections while a valid certificate sat in memory.

Closes #101

Test plan

• TestSANCertManager_GetCertificate_ServesExpiringRegisteredCertAndQueuesReplacement — held cert served, no synchronous order, replacement queued with the owning service
• TestSANCertManager_GetCertificate_ExpiredRegisteredCertReprovisionsSynchronously — an expired cert is replaced on the spot, handshake gets the fresh cert
• TestSANCertManager_GetCertificate_MismatchedDirectoryCertServedWhileReplacementQueues — post---tls-staging-flip cert keeps serving while the replacement queues
• make test (1827 tests), go test -race, make lint, gofmt, go vet all clean

Deviations & judgment calls

• Replacement rides the domain issuer, accepting single-domain orders. For a deploy-registered service the issuer's batch size is 1, so a large SAN cert nearing expiry could in principle replace member-by-member rather than as one batch. Chosen anyway: the issuer already provides dedup, quarantine, rate limiting, and directory-awareness; the renewer remains the primary whole-batch mechanism, and this path only fires when renewal has already been failing. A second background batch-provisioner for a failsafe path wasn't worth the machinery.
• Discovery: the old synchronous path could hit live ACME from a unit test. The RED run of the directory-mismatch test reached acme-v02.api.letsencrypt.org/acme/new-acct (rejected for the example.com contact) because the sync order lazily built a real client bundle for the overridden directory. The fix removes the network call from that path; noted as a caution for future cert tests — clientsForDirectory registers live accounts wherever an overridden-directory order escapes a test without an injected fake bundle.

---

Summary by cubic

Keep serving a still‑valid certificate during TLS handshakes while its replacement issues asynchronously to avoid outages when renewal is degraded. Previously handshakes blocked for expiring (<24h) or directory‑mismatched certs; now only first issuance, expired certs, and registered mismatches block, and waiters refuse expired or mismatched held certs after failed orders.

• Expiring registered (<24h) and valid dynamic certs: serve the held cert and queue replacement on the domain issuer; evicted domains unchanged.
• Directory mismatch: registered reprovisions synchronously; dynamic domains keep serving and queue replacement to avoid tenant outages.
• Waiters now use getServableCertForDomain and refuse still‑mismatched or expired certs; getCertForDomain removed.
• Changes in internal/server/san_cert_manager.go; tests in internal/server/san_cert_manager_test.go cover expiring, expired, mismatch, and waiter cases.
• Fixes SAN manager fails handshakes in the last 24h of a cert's life instead of serving the still-valid certificate #101.

^{Written for commit 2899d3c. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 2 files

Confidence score: 2/5

• In internal/server/san_cert_manager.go, the staging→production ACME transition can keep serving the staging cert while a new one is queued, causing public TLS handshake failures for all clients until issuance finishes; this is a concrete availability risk during cutover—switch to a valid production-chain fallback or force immediate cert replacement before serving traffic.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="internal/server/san_cert_manager.go">

<violation number="1" location="internal/server/san_cert_manager.go:474">
P1: When a service moves from the staging ACME directory to production, this branch still serves the staging certificate while replacement is queued, so public TLS clients reject every handshake until issuance completes. Do not serve a certificate with `directoryMismatch`; let registered domains reprovision and let dynamic domains fail fast while the issuer replaces it.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}