singhsanjay12 · singhsanjay12 · Jun 30, 2026
diff --git a/_posts/2026-06-01-mtls-at-scale-spiffe-spire.md b/_posts/2026-06-01-mtls-at-scale-spiffe-spire.md
@@ -0,0 +1,172 @@
+---
+title: "mTLS at Scale Is a Certificate Problem: SPIFFE, SPIRE, and Rotation"
+description: "mTLS gives you mutual cryptographic identity, but the hard part is issuing and rotating certificates for thousands of workloads. Here is how SPIFFE and SPIRE turn that into automatic, short-lived identity, and why short TTLs beat revocation lists."
+date: 2026-06-01 12:00:00 +0000
+categories: [Security, Zero Trust]
+tags: [mtls, spiffe, spire, certificates, identity, zero-trust, security]
+image:
+  path: /assets/img/posts/mtls-spiffe-spire/hero.svg
+  alt: "SPIRE attesting a workload, issuing a short-lived SVID, and that identity being used in a mutual TLS handshake with continuous rotation"
+---
+
+Everyone agrees on the goal: **two services should prove who they are to each other before they exchange a single byte.** That is mutual TLS, and it is the right answer to the question Zero Trust keeps asking, which I have written about from the [reverse proxy](/2025/08/03/zero-trust-with-reverse-proxy.html) and [control plane](/2025/10/20/zero-trust-control-plane-and-sessions.html) angles. mTLS is the cryptographic floor under all of it. But here is the hidden assumption almost every "just turn on mTLS" plan makes: that the certificates already exist, are in the right place, are trusted, and are still valid. At three services that is true. At three thousand, it is the entire problem.
+
+The protocol is the easy part. The hard part is the certificate lifecycle: who an identity belongs to, how it gets there, and what happens when it expires. mTLS at scale is not a TLS problem. It is a **certificate distribution and rotation problem** wearing a TLS costume.
+
+## The Part Nobody Budgets For
+
+A certificate is just a signed claim: "the holder of this private key is `payments`." For that claim to mean anything, four things have to be true at once, for every workload, continuously.
+
+The key has to be **on the right host and only there**. The certificate has to be **signed by a CA the other side trusts**. The identity in it has to **match what the workload actually is**. And it has to be **unexpired**. Get any one wrong and the handshake fails, or worse, succeeds for the wrong party.
+
+Now multiply by a fleet that autoscales, reschedules, and redeploys constantly. Pods come and go in seconds (the same churn that makes [service discovery in Kubernetes](/2026/06/30/service-discovery-in-kubernetes.html) a live-registry problem rather than a config-file one). Every one of those ephemeral workloads needs an identity it did not have a moment ago. The naive answer, baking a long-lived certificate into an image or a secret, is exactly the liability you do not want, and it is where most homegrown mTLS efforts quietly rot.
+
+## Long-Lived Certs Are a Liability, Not a Convenience
+
+The instinct is to issue a cert that lasts a year so you "do not have to think about it." That instinct is backwards.
+
+A one-year certificate is a one-year window of exposure. If the private key leaks (a logged secret, a copied volume, a compromised node) the attacker holds a valid identity until that cert expires or you successfully revoke it. And revocation, as anyone who has run a CRL or OCSP responder knows, is the part of PKI that never quite works at scale: clients cache, responders go down, and the revocation list is always a little behind reality.
+
+Flip the model. If a certificate lives for **one hour**, a leaked key is useful for at most an hour, and you never have to revoke anything: the cert dies on its own schedule and is simply not renewed. This is the same argument I made for [bounded sessions in the control plane post](/2025/10/20/zero-trust-control-plane-and-sessions.html), pushed all the way down to the identity itself. Short TTLs turn revocation from a distributed-consistency nightmare into a non-event.
+
+The catch is obvious: a one-hour cert means rotation is **constant**, not occasional. You cannot do that by hand. You need a system whose entire job is to attest workloads and hand them fresh identities on a tight loop. That system needs a common language for "what is an identity," and that language is SPIFFE.
+
+## SPIFFE: a Standard Name for a Workload
+
+SPIFFE (Secure Production Identity Framework For Everyone) is not software. It is a **specification for what a workload identity is**, so that different systems can agree on it. Two pieces matter.
+
+The **SPIFFE ID** is a URI that names a workload, scoped to a trust domain:
+
+<div style="border-radius:8px;overflow:hidden;margin:1.5em 0;box-shadow:0 1px 6px rgba(0,0,0,0.15);">
+  <div style="padding:7px 14px;background:#0f172a;display:flex;align-items:center;gap:8px;">
+    <svg xmlns="http://www.w3.org/2000/svg" width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="#6366f1" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="8" r="4"/><path d="M6 21v-2a4 4 0 0 1 4-4h4a4 4 0 0 1 4 4v2"/></svg>
+    <span style="font-family:system-ui,sans-serif;font-size:11px;color:#94a3b8;font-weight:500;letter-spacing:0.3px;">a SPIFFE ID</span>
+  </div>
+  <pre style="margin:0;padding:16px 18px;background:#1e293b;overflow-x:auto;"><code style="font-family:'SFMono-Regular',Consolas,'Liberation Mono',Menlo,monospace;font-size:13px;color:#e2e8f0;line-height:1.65;">spiffe://prod.example.com/ns/shop/sa/payments
+<span style="color:#64748b;">       \________________/  \___________________/</span>
+<span style="color:#64748b;">         trust domain          path: who this is</span>
+
+<span style="color:#64748b;"># trust domain  = the root of trust (one CA hierarchy)</span>
+<span style="color:#64748b;"># path          = namespace + service account, here</span></code></pre>
+</div>
+
+The **SVID** (SPIFFE Verifiable Identity Document) is how that ID is presented and proven. The common form is an X.509 certificate that carries the SPIFFE ID in its URI SAN field. So an SVID is an ordinary X.509 cert that every TLS stack already understands, with the workload's name in a place verifiers know to look. (There is also a JWT-SVID for cases where you cannot do mTLS end to end, but the X.509 SVID is what feeds the handshake.)
+
+That is the whole point of SPIFFE: a **portable, verifiable name** that means the same thing to a proxy, a sidecar, a database, and a mesh, regardless of who issued it. SPIFFE says what an identity is. It does not issue one. For that you need SPIRE.
+
+## SPIRE: the System That Actually Issues Identity
+
+SPIRE (the SPIFFE Runtime Environment) is the reference implementation that issues and rotates SVIDs. It has two parts: a central **SPIRE Server** that holds the trust domain's CA and the registration rules, and a **SPIRE Agent** on every node that workloads talk to over a local socket.
+
+The hard question SPIRE answers is the one every identity system has to: **how do you give a workload its first credential without already having a credential to authenticate it?** The bootstrap problem. SPIRE solves it with two-layer attestation.
+
+**Node attestation** establishes that an agent is running on a legitimate node. The agent proves the node's identity to the server using something the node already has and the platform vouches for: an AWS instance identity document, a GCP instance token, a Kubernetes node's projected token, a TPM. The server checks it and binds an identity to that agent.
+
+**Workload attestation** happens locally, on the node, when a process asks the agent for its SVID. The agent inspects the calling process through the kernel (its UID, its Kubernetes pod via the kubelet, its container labels) and matches those properties against registration entries. No secret is presented by the workload. Its identity is derived from **what it verifiably is**, not from a token it carries.
+
+<div style="border-radius:8px;overflow:hidden;margin:1.5em 0;box-shadow:0 1px 6px rgba(0,0,0,0.15);">
+  <div style="padding:7px 14px;background:#0f172a;display:flex;align-items:center;gap:8px;">
+    <svg xmlns="http://www.w3.org/2000/svg" width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="#6366f1" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"/><polyline points="14 2 14 8 20 8"/></svg>
+    <span style="font-family:system-ui,sans-serif;font-size:11px;color:#94a3b8;font-weight:500;letter-spacing:0.3px;">register a workload with SPIRE</span>
+  </div>
+  <pre style="margin:0;padding:16px 18px;background:#1e293b;overflow-x:auto;"><code style="font-family:'SFMono-Regular',Consolas,'Liberation Mono',Menlo,monospace;font-size:13px;color:#e2e8f0;line-height:1.65;">$ spire-server entry create \
+    -spiffeID  spiffe://prod.example.com/ns/shop/sa/payments \
+    -parentID  spiffe://prod.example.com/spire/agent/k8s/node-7 \
+    -selector  k8s:ns:shop \
+    -selector  k8s:sa:payments \
+    -ttl <span style="color:#f0abfc;">3600</span>     <span style="color:#64748b;"># SVID lives one hour, then rotates</span>
+
+<span style="color:#64748b;"># Meaning: any pod on node-7, in namespace 'shop', running as</span>
+<span style="color:#64748b;"># service account 'payments', GETS this identity. Nothing else does.</span></code></pre>
+</div>
+
+The selectors are the policy: identity is granted to a process **because** it runs in a given namespace as a given service account, verified by the agent against the kubelet, not because it shipped with a password. That is the property that makes the whole thing safe to automate.
+
+## The Workload API and Continuous Rotation
+
+A workload never generates a CSR by hand or writes a cert to disk. It connects to the SPIRE Agent's local Workload API (a Unix domain socket) and **streams** its identity. The agent pushes the current SVID, the trust bundle (the CA certs to verify peers), and, crucially, a fresh SVID before the old one expires.
+
+<div style="border-radius:8px;overflow:hidden;margin:1.5em 0;box-shadow:0 1px 6px rgba(0,0,0,0.15);">
+  <div style="padding:7px 14px;background:#0f172a;display:flex;align-items:center;gap:8px;">
+    <svg xmlns="http://www.w3.org/2000/svg" width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="#6366f1" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><polyline points="4 17 10 11 4 5"/><line x1="12" y1="19" x2="20" y2="19"/></svg>
+    <span style="font-family:system-ui,sans-serif;font-size:11px;color:#94a3b8;font-weight:500;letter-spacing:0.3px;">fetch and watch the SVID</span>
+  </div>
+  <pre style="margin:0;padding:16px 18px;background:#1e293b;overflow-x:auto;"><code style="font-family:'SFMono-Regular',Consolas,'Liberation Mono',Menlo,monospace;font-size:13px;color:#e2e8f0;line-height:1.65;"><span style="color:#64748b;"># the agent exposes a local socket; no network, no secret to mount</span>
+SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/sockets/agent.sock
+
+<span style="color:#64748b;"># the workload subscribes and gets a NEW SVID on every rotation</span>
+$ spiffe-watcher
+received SVID  spiffe://prod.example.com/ns/shop/sa/payments
+  not_after    <span style="color:#a3e635;">2026-06-01T13:00:00Z</span>   <span style="color:#64748b;"># 1h out</span>
+... 50 min later, well before expiry ...
+rotated  SVID  spiffe://prod.example.com/ns/shop/sa/payments
+  not_after    <span style="color:#a3e635;">2026-06-01T14:00:00Z</span>   <span style="color:#64748b;"># new cert, same ID</span></code></pre>
+</div>
+
+This inverts the usual cert lifecycle. There is no cron job racing expiry, no secret to provision, no "the cert expired over the weekend" incident. Rotation is the **steady state**, not an event. The SPIFFE ID is stable while the certificate behind it is replaced continuously, which is the same pattern as a stable name in front of an ever-changing set of endpoints, just applied to identity. With a one-hour TTL, a leaked key is worthless almost immediately, and you have deleted CRLs and OCSP from your architecture entirely.
+
+## Trust Domains and Federation
+
+A **trust domain** is a single root of trust: one SPIRE Server, one CA hierarchy, one namespace of SPIFFE IDs (`prod.example.com` above). Everything inside it can verify everything else because they share a trust bundle.
+
+But real systems span boundaries: prod and staging, two business units, an acquisition, a partner. You do not want one giant trust domain, and you do not want to copy private keys around. **Federation** is the answer: two trust domains exchange only their public trust bundles, so a workload in `prod.example.com` can verify an SVID from `partner.other.com` without either side sharing a CA or trusting the other's issuance.
+
+<div style="border-radius:8px;overflow:hidden;margin:1.5em 0;box-shadow:0 1px 6px rgba(0,0,0,0.15);">
+  <div style="padding:7px 14px;background:#0f172a;display:flex;align-items:center;gap:8px;">
+    <svg xmlns="http://www.w3.org/2000/svg" width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="#6366f1" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="2" y1="12" x2="22" y2="12"/><path d="M12 2a15.3 15.3 0 0 1 4 10 15.3 15.3 0 0 1-4 10 15.3 15.3 0 0 1-4-10 15.3 15.3 0 0 1 4-10z"/></svg>
+    <span style="font-family:system-ui,sans-serif;font-size:11px;color:#94a3b8;font-weight:500;letter-spacing:0.3px;">federate two trust domains</span>
+  </div>
+  <pre style="margin:0;padding:16px 18px;background:#1e293b;overflow-x:auto;"><code style="font-family:'SFMono-Regular',Consolas,'Liberation Mono',Menlo,monospace;font-size:13px;color:#e2e8f0;line-height:1.65;"><span style="color:#64748b;"># each server publishes its bundle at a SPIFFE bundle endpoint;</span>
+<span style="color:#64748b;"># the other fetches and refreshes it (only PUBLIC keys cross)</span>
+federation:
+  bundle_endpoint:
+    address: <span style="color:#a3e635;">"0.0.0.0"</span>
+    port: <span style="color:#f0abfc;">8443</span>
+  federates_with <span style="color:#a3e635;">"partner.other.com"</span> {
+    bundle_endpoint_url: <span style="color:#a3e635;">"https://spire.other.com:8443"</span>
+    bundle_endpoint_profile <span style="color:#a3e635;">"https_spiffe"</span> {
+      endpoint_spiffe_id: <span style="color:#a3e635;">"spiffe://partner.other.com/spire/server"</span>
+    }
+  }</code></pre>
+</div>
+
+Federation is how mTLS scales across organizational lines without flattening everyone into one CA, which is the cryptographic equivalent of the trust-zone boundaries I described in the [control plane post](/2025/10/20/zero-trust-control-plane-and-sessions.html): cross a boundary only with explicit, verifiable trust.
+
+## Where This Meets a Service Mesh
+
+You can call the Workload API from your application directly, but most teams do not want to. This is where a **service mesh** earns its keep. In a mesh like Istio, the Envoy sidecar is the SPIFFE workload: it fetches the SVID over the Workload API (Istio's identity model is SPIFFE-shaped), terminates mTLS on both sides of every connection, and rotates certs underneath the application, which never touches a key at all.
+
+<div style="border-radius:8px;overflow:hidden;margin:1.5em 0;box-shadow:0 1px 6px rgba(0,0,0,0.15);">
+  <div style="padding:7px 14px;background:#0f172a;display:flex;align-items:center;gap:8px;">
+    <svg xmlns="http://www.w3.org/2000/svg" width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="#6366f1" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M12 22s8-4 8-10V5l-8-3-8 3v7c0 6 8 10 8 10z"/></svg>
+    <span style="font-family:system-ui,sans-serif;font-size:11px;color:#94a3b8;font-weight:500;letter-spacing:0.3px;">authorize by SPIFFE identity, not IP</span>
+  </div>
+  <pre style="margin:0;padding:16px 18px;background:#1e293b;overflow-x:auto;"><code style="font-family:'SFMono-Regular',Consolas,'Liberation Mono',Menlo,monospace;font-size:13px;color:#e2e8f0;line-height:1.65;">apiVersion: security.istio.io/v1
+kind: AuthorizationPolicy
+metadata: { name: payments-allow-checkout, namespace: shop }
+spec:
+  selector: { matchLabels: { app: payments } }
+  action: ALLOW
+  rules:
+    - from:
+        - source:
+            <span style="color:#64748b;"># policy keys off the verified SPIFFE ID of the caller,</span>
+            <span style="color:#64748b;"># NOT a source IP that any pod could spoof or inherit</span>
+            principals: [<span style="color:#a3e635;">"prod.example.com/ns/shop/sa/checkout"</span>]</code></pre>
+</div>
+
+This is the payoff. Once every workload has a verifiable identity, authorization stops keying off network location (an IP, a subnet, a security group) and starts keying off **who the caller cryptographically is**. That is the deepest version of the Zero Trust idea: the network is untrusted, and identity travels with the workload. The mesh handles the mechanics that the [proxy concurrency](/2026/03/09/concurrent-requests-reverse-proxy.html) and discovery layers already taught us to expect: a control plane pushing live state to data-plane proxies, here pushing identity instead of routes.
+
+## What to Take Away
+
+If you remember one thing: **turning on mTLS is a day. Operating it is forever, and the operating cost is certificate lifecycle.** SPIFFE gives you a name for a workload that every layer agrees on. SPIRE earns that name through attestation, so a process gets identity because of what it verifiably is, then keeps that identity fresh on a tight rotation loop you never have to touch.
+
+Make the certificates short-lived and you trade the unsolved problem (revocation at scale) for a solved one (automated issuance). Make identity portable with SPIFFE and you can federate across boundaries without merging CAs. Put it behind a mesh and your applications get mutual identity for free. The certificate problem does not disappear. It just stops being yours to babysit.
+
+<p style="font-family: Georgia, serif; font-style: italic; font-size: 1.05em; border-left: 3px solid #c7d2fe; padding-left: 1em; margin-top: 2em; color: #1e293b;">mTLS is not hard because the handshake is hard. It is hard because a valid identity, in the right place, that is still good right now, is a logistics problem at fleet scale. SPIFFE and SPIRE are what turn that logistics problem back into a protocol detail.</p>
+
+---
+
+*This builds on my earlier writing on [Zero Trust with a reverse proxy](/2025/08/03/zero-trust-with-reverse-proxy.html), the [Zero Trust control plane and sessions](/2025/10/20/zero-trust-control-plane-and-sessions.html), and [service discovery in Kubernetes](/2026/06/30/service-discovery-in-kubernetes.html).*
+
+*Rolling out workload identity or untangling certificate rotation at scale? I am on [LinkedIn](https://www.linkedin.com/in/singhsanjay12) or reachable by [email](mailto:hello@singh-sanjay.com).*