docs: security roadmap

docs/security/roadmap.md

@@ -0,0 +1,245 @@
+# EastGuard Security Roadmap
+
+**Goal:** Provide a secure, default-deny production boundary for all nodes and clients without sacrificing online certificate rotation, node restarts, or network partition recovery.
+
+**Depends on:** SWIM membership, Raft consensus, data-plane placement, client protocol, and client redirect handling.
+
+---
+
+## 1. Overview & Threat Model
+
+EastGuard operates in two distinct security modes:
+
+- **Secure Mode (Default):** Mutual TLS 1.3 / DTLS 1.3 enforced across all listeners. Unauthenticated or unauthorized traffic is immediately rejected. Plaintext connections and protocol downgrades are forbidden; invalid configuration prevents startup.
+- **Trusted Development Mode:** Plaintext protocols enabled strictly via explicit opt-in configuration for isolated test environments.
+
+### Threat Model
+The production boundary defends against external network attackers attempting eavesdropping, packet modification, replay, UDP spoofing, connection injection, or TLS downgrade.
+
+> **Scope Note:** Compromised brokers or authorized clients are outside this threat model. Their impact is constrained by short-lived credentials, scoped ACLs, and strict principal binding. Byzantine consensus is out of scope.
+
+### Listener Architecture
+
+| Listener | Port | Protocol | Peer Authentication | Purpose |
+| :--- | :--- | :--- | :--- | :--- |
+| **Client** | TCP 2921 | TLS 1.3 | Mutual X.509 | Metadata queries, administration, produce, fetch |
+| **Raft** | TCP 2922 | TLS 1.3 | Mutual X.509 | Metadata shard consensus log replication |
+| **Data** | TCP 2923 | TLS 1.3 | Mutual X.509 | Segment replication, repair, and coordination |
+| **SWIM** | UDP 2922 | DTLS 1.3 | Mutual X.509 | Membership gossip & failure detection |
+
+---
+
+## 2. Layered Architecture
+
+Security checks are split between the transport layer and application state machines to keep state machines free of security I/O:
+
+```
+        [ TLS 1.3 / DTLS 1.3 Transport Layer ]
+  - Authenticate peer X.509 certificates
+  - Enforce framing, datagram MTU, and resource limits
+  - Bind connection envelope sender to verified identity
+                     │
+                     │ (Drop connection on transport failure)
+                     ▼
+        [ Application Layer State Machines ]
+  - Authorize requested operation against cached ACLs / placement
+  - Execute SWIM / Raft / Data-Plane state transitions
+                     │
+                     │ (Drop denied envelope only; connection stays open)
+```
+
+- **Transport Responsibility:** Performs cryptographic handshakes, validates certificates, tracks cache expiry, and binds envelope senders to verified identities. Transport errors close the connection.
+- **Application Responsibility:** Synchronous state machines (SWIM, Raft, Topology, Data Plane) process only pre-validated envelopes. An authorization failure drops the specific denied envelope without tearing down the underlying connection.
+- **Placement-Based Data-Plane Authorization:** Data-plane nodes authorize incoming replication and repair requests against local committed placement state, ignoring sender-asserted replica lists.
+- **Redirects:** A redirect is only an address hint. The destination authenticates the peer and repeats the authorization check.
+
+---
+
+## 3. Node Identity & Admission Model
+
+To allow safe node restarts and hardware replacement without exposing the cluster to stale process replay, EastGuard decouples reusable identity from running process instances.
+
+### Identity Hierarchy
+
+| Term | Scope | Lifetime / Ordering | Function |
+| :--- | :--- | :--- | :--- |
+| **Certificate Node ID** | Configuration | Long-lived / Reused | Operator-assigned node name embedded in the X.509 certificate. |
+| **Admission Epoch** | Metadata Shard | Monotonically increasing `u64` | Assigned by metadata Raft upon restart; higher epoch **fences** older instances. |
+| **NodeId** | Running Process | Single process lifetime | Unique ID generated on startup; used by SWIM, topology ring, Raft, and data placement. |
+| **Process Key** | Running Process | Single process lifetime | Proves that the connection belongs to the process admitted for this epoch. |
+| **SWIM Incarnation** | Running Process | Monotonically increasing counter | Incremented by the *same* process instance to refute false `Suspect`/`Dead` gossip. |
+
+### Resolution Rules
+
+Node identity conflicts and stale gossip are resolved in this order:
+
+```
+ 1. Admission Epoch   ──► Higher epoch always wins & fences older processes
+          │ (If equal)
+          ▼
+ 2. SWIM Incarnation  ──► Higher incarnation refutes stale liveness facts
+          │ (If equal)
+          ▼
+ 3. Liveness State    ──► Dead > Suspect > Alive
+```
+
+### Partition Recovery vs. Node Restart
+
+An **authorized operator** is a person or trusted automation using a client
+certificate whose principal has the `security/cluster` grant. In an orchestrated
+deployment, automation may approve restarts and scaling, but metadata Raft still
+commits every admission.
+
+- **Healed Partition (Same Process):** Retains its existing `NodeId` and `Admission Epoch`. Increments its `SWIM Incarnation` counter to refute `Suspect` or `Dead` rumors spread during the partition.
+- **Node Restart / Replacement:** The process generates a new `NodeId` and process key. An authorized operator approves that exact process, and metadata Raft commits the new `Admission Epoch`, `NodeId`, and process public key. The higher epoch fences older processes immediately where observed and everywhere else within 60 seconds.
+
+### Admission Gate & SWIM Separation
+
+SWIM liveness gossip is decoupled from cluster admission authority to prevent network partitions from admitting unauthorized nodes:
+
+```
+  [ Authorized Operator ] ──► Approve Joining Node's NodeId & Process Key
+                                             │
+                                             ▼
+  [ Joining Node ] ──► Limited Admission Endpoint ──► Metadata Raft
+                                                        │
+                                                        ▼
+                                  [ Epoch + NodeId + Process Public Key ]
+                                                        │
+                                                        ▼
+  [ SWIM Actor ] ◄──────────── Check Admission Cache ───┘
+           │
+           ▼ (Accepted)
+  [ SWIM State Machine ] ──► Liveness Gossip (Alive / Suspect / Dead)
+           │
+           ▼
+  [ Raft Reconciliation ] ──► Commit AddPeer / RemovePeer
+```
+
+1. **Process Proof:** A node connection proves possession of the process private key bound to its admitted epoch. The process key signs the admitted identity and a connection-specific value produced by TLS or DTLS, so the proof cannot be replayed on another connection. The reusable node certificate alone cannot create or claim a newer epoch.
+2. **Admission Gate:** The SWIM actor checks incoming packets against a local admission cache backed by metadata Raft before passing facts to the SWIM state machine.
+3. **Gossip Rule:** DTLS authenticates the immediate sender. Every relayed membership fact is separately accepted only when its subject `NodeId` and `Admission Epoch` match an active admission record.
+4. **Cache Policy:** Admission records are cached locally with a maximum TTL of 60 seconds. If a cache entry expires while the owning metadata shard is unreachable, the gate fails closed.
+
+---
+
+## 4. Authorization & Sharded Security Records
+
+EastGuard enforces exact, default-deny access control lists (ACLs) without wildcards or inheritance.
+A principal is the client ID read from an authenticated certificate and used for
+permission checks; its text grants no authority by itself.
+
+### ACL Resource Catalog
+
+| Resource Key Format | Granted Actions |
+| :--- | :--- |
+| `cluster` | Membership inspection, topology lookup, operator diagnostics |
+| `topic-admin/{topic}` | Create, delete, describe topic metadata |
+| `topic-data/{topic}` | Produce, fetch, list offsets for topic |
+| `consumer-group/{topic}/{group}` | Consume messages, read/commit consumer offsets |
+| `producer-session/{topic}/{session}` | Renew producer session (permanently bound to creator principal) |
+| `security/cluster` | Read/write ACLs, manage admissions and revocations, inspect security audit |
+
+Consumer-group access permits group coordination and offset read/commit. Reading
+records separately requires `Fetch` on `topic-data/{topic}`.
+
+### Sharded Metadata Storage
+
+A security record is one durable admission, ACL, or revocation entry. Its record
+path selects one metadata shard; its revision lets brokers detect stale cached
+copies.
+
+Security records (`security/node/{id}`, `security/acl/{resource}`, `security/revocation/{issuer}/{serial}`) do not rely on a centralized security controller. Instead, they hash to standard metadata shards and replicate via Raft:
+
+```
+  Client/Node Request ──► Any Broker ──► Hash Record Path ──► Hosts Shard? ─┬─► Yes ──► Commit via Raft
+                                                                           └─► No  ──► Return Owner Redirect
+```
+
+- **Local Authorization:** Brokers evaluate ACLs against local cached security records.
+- **Freshness & Expiry:** Cached records include a monotonic deadline (max 60s) and revision counter. Expired entries require re-validation from the owner metadata shard; if the owner is offline, authorization fails closed.
+
+---
+
+## 5. Operations & Credential Lifecycle
+
+### Bootstrap & Node Joining
+
+1. Operator initializes a cluster trust root and issues the first node certificate.
+2. The first node generates its `NodeId` and process key. Initial metadata state
+   stores that admission, the first operator principal, and its
+   `security/cluster` grant.
+3. A later joining node generates a new `NodeId` and process key.
+4. An authorized operator approves that exact `NodeId` and process public key. The reusable node certificate alone cannot authorize replacement.
+5. The joining node connects to a **limited admission endpoint** using its X.509 certificate.
+6. The endpoint routes to the owning metadata shard, which atomically commits the next `Admission Epoch`, `NodeId`, and process public key.
+7. The joining node proves possession of the process private key before entering SWIM gossip and Raft membership reconciliation.
+
+### Online Credential Rotation
+
+- **Zero-Downtime CA Rotation:** Brokers support dual trust chain loading. New root CAs can be added and leaf certificates reloaded online without restarting brokers or changing `Admission Epoch` / `NodeId`.
+- **Revocation & Expiry:** Certificate revocations commit to metadata Raft records. Active TLS/DTLS sessions are terminated within the cache enforcement window. Expired certificates are rejected with clock-skew tolerance.
+- **Recovery:** Runbooks cover lost authorized-operator access, lost issuing keys, expiry, accidental revocation, trust-root replacement, and cold-cluster restart.
+
+---
+
+## 6. Resource Limits & Security Audit
+
+### Rate & Memory Bounds
+- Every listener enforces strict limits on unauthenticated handshakes, concurrent connections, in-flight frames, memory allocations, and per-source request rates.
+- **DTLS Datagram Size:** Encrypted SWIM datagram payloads are capped at **1200 bytes** to ensure the total packet fits within the 1280-byte minimum IPv6 MTU without IP fragmentation.
+
+### Audit Subsystem
+- **Non-Blocking Execution:** Security audit events (authentication success/failure, ACL denials, admissions) are queued asynchronously. Audit backpressure never blocks protocol execution or consensus.
+- **Rate-Limited Flood Protection:** High-frequency audit events use aggregate counters and sampled detail logging.
+- **Credential Hygiene:** Audit logs never record private keys, tokens, credentials, or message payloads.
+
+---
+
+## 7. Delivery Plan (S0–S6)
+
+```
+S0 ──► S1 ──► S2 ──► S3 ──► S4 ──► S5 ──► S6
+config  records  TCP     SWIM    clients  operations  production
+                 mTLS    DTLS    + ACLs   + audit     gate
+```
+
+| Phase | Target Scope | Key Deliverable | Exit Criteria |
+| :--- | :--- | :--- | :--- |
+| **S0** | Configuration | Security modes, cert loader, DTLS 1.3 feasibility | Secure mode opens no plaintext listeners; a maintained DTLS 1.3 implementation satisfies certificate, replay, cookie, and resource-bound requirements |
+| **S1** | Metadata Storage | Security record schema, sharded Raft state | Security records survive snapshot & recovery |
+| **S2** | Cluster Transport | TLS 1.3 on TCP 2922/2923, Raft D8 RPC authorization | Authenticated and authorized cluster TCP traffic |
+| **S3** | Membership | DTLS 1.3 on UDP 2922 & SWIM admission gate | Secure gossip & partition-safe admission fencing |
+| **S4** | Client API | Client mTLS on TCP 2921, principal binding, ACLs | Default-deny enforcement on all client APIs |
+| **S5** | Operations | Certificate rotation, revocation, expiry, recovery, audit logging | Online credential operations and recovery runbooks |
+| **S6** | Production Gate | Adversarial testing, fuzzing, partition stress | Passes all production readiness checks |
+
+S6 must verify node and client impersonation, stale-process replay, unauthorized
+operations, protocol downgrade, rotation under live traffic, expired and revoked
+credentials, cold-cluster restart, handshake and datagram fuzzing, resource
+bounds, and secret-free diagnostics.
+
+---
+
+## 8. Invariants & Security Rules
+
+### System Invariants
+
+1. **Single Connection Identity:** Every established client connection has exactly one authenticated principal; every node connection has exactly one `(Certificate Node ID, Admission Epoch, NodeId, Process Public Key)`.
+2. **Unique Active Node Admission:** Metadata state maintains at most one active `(Admission Epoch, NodeId, Process Public Key)` per Certificate Node ID.
+3. **Immutable Producer Session Principal:** Every producer session is immutably bound to the principal that created it.
+4. **Explicit Cache Bounding:** Every cached security entry specifies its source metadata shard, revision, and expiry measured with a monotonic clock (≤ 60 seconds).
+5. **Bounded Audit Footprint:** Audit queues and aggregate rate counters remain within configured capacity.
+
+### Operational Rules
+
+1. Secure mode never downgrades to plaintext or unauthenticated protocols.
+2. Transport layers authenticate identity; application state machines authorize actions.
+3. Transport identity mismatches close the connection immediately before payload dispatch.
+4. Application authorization denials drop only the denied envelope, preserving the connection for valid traffic.
+5. A higher admission epoch fences every older process immediately where observed and no later than admission-cache expiry elsewhere.
+6. SWIM controls liveness; metadata Raft controls admission.
+7. Missing or expired security records cause authorization and admission checks to fail closed.
+8. Resource permits are acquired before allocating memory or spawning async tasks.
+9. Audit logging backpressure must never block network protocol processing or consensus.
+10. Only an authorized operator may approve a higher admission epoch.