diff --git a/.claude/rules/code-convention.md b/.claude/rules/code-convention.md index f272c3db..6399db5b 100644 --- a/.claude/rules/code-convention.md +++ b/.claude/rules/code-convention.md @@ -6,7 +6,7 @@ Functions taking `&self`, `&mut self`, or a reference to a struct as their first ## Enum struct pattern -Enum variants that carry data must use the tuple-variant + named-struct pattern. Never use inline fields on enum variants. See `/.claude/skills/enum-struct-pattern/SKILL.md` for the full pattern including `impl_from_variant!` usage. +Enum variants that carry data must use the tuple-variant + named-struct pattern. Prefer not using inline fields on enum variants. See `/.claude/skills/enum-struct-pattern/SKILL.md` for the full pattern including `impl_from_variant!` usage. ## Result over Option for observability diff --git a/.claude/rules/metadata-state-machine.md b/.claude/rules/metadata-state-machine.md index 956d58c5..91386e54 100644 --- a/.claude/rules/metadata-state-machine.md +++ b/.claude/rules/metadata-state-machine.md @@ -21,7 +21,11 @@ MetadataStateMachine (one per shard group) │ │ SegmentMeta (nested inside RangeMeta) │ -└── topic_name_index: HashMap +├── topic_name_index: HashMap +└── security + ├── admissions: Node Certificate Principal → Admission Record + ├── ACLs: Resource → ACL Record + └── revocations: (Issuer, Serial) → Revocation Record ``` ## Commands @@ -80,3 +84,5 @@ MetadataStateMachine (one per shard group) 21. **`ReassignSegment` only re-points a sealed segment.** `apply_reassign_segment()` accepts only a `Sealed` segment, swaps `replica_set`, and changes nothing else — state stays `Sealed`; data, offsets, lineage, and timestamps stay frozen (invariant 3). An active, deleting, or unknown segment is rejected (`SegmentNotSealed` / `SegmentNotFound`), logged but not fatal (invariant 11). Re-applying with the same `replica_set` succeeds without raising a metadata event, tolerating duplicate death detection and no-leader re-proposals (cf. invariant 19). The swap runs through `apply`, so the umbrella `assert_invariants` re-checks every other invariant afterward — a reassignment cannot leave the machine inconsistent. 22. **A committed consumer-group generation assigns each active range exactly once.** When a group has members, its assignment keys exactly equal the topic's active ranges and every assignment names a current member. When it has no members, it has no assignments. Membership or range-topology changes advance the generation and recompute the full desired assignment through the Raft log; heartbeat refreshes that do not change membership leave the generation unchanged. + +23. **Security map keys match their records.** Admission keys equal the record's Node Certificate Principal, ACL keys equal the record's resource, and revocation keys equal the record's issuer and serial. A map therefore contains at most one current admission per Node Certificate Principal and snapshot restore cannot silently associate a security record with a different lookup key. diff --git a/.claude/rules/raft-transport.md b/.claude/rules/raft-transport.md index 5498b237..2de0e2cd 100644 --- a/.claude/rules/raft-transport.md +++ b/.claude/rules/raft-transport.md @@ -1,27 +1,56 @@ # Raft Transport (Invariants) -`RaftTransportActor` — async TCP transport for Raft RPCs. Manages persistent bidirectional connections between nodes. Each connection split into a reader task and a writer half held in the per-node `writers` map. +`RaftTransportActor` — async TCP transport for Raft RPCs. It manages persistent +bidirectional Raft connections between nodes. Each connection splits into a +reader task and a writer half held in the per-node `writers` map. +The same authenticated cluster listener also serves a one-shot ACL snapshot read used to +refresh a broker's local authorization cache and a limited admission-record read used to authenticate a connecting process. Separate from SWIM's UDP transport. Raft uses TCP for reliable, ordered delivery. ## Architecture (brief) ``` -RaftTransportActor -├── listener (raft_port, TCP) -├── writers: HashMap (one write half per peer) -└── reader tasks (one per accepted/established connection) +cluster listener (TCP) + │ + ├── limited admission read ──► security actor → Raft → reply → close + │ + └── request with process proof + │ + ├── both sides verify a TLS-session-bound process proof + ├── Raft ──► persistent reader + one writer per peer + └── ACL ───► security actor → Raft → reply → close ``` ## Wire Protocol -Length-prefixed bincode frames: -1. **Handshake** (first frame on every connection): `[len: u32][NodeId: bincode]` -2. **Messages**: `[len: u32][WireRaftMessage: bincode]`, where `WireRaftMessage` carries `shard_group_id` so transport can dispatch to the correct Raft group. +Length-prefixed Borsh frames: + +1. **Initial message:** `InitialClusterMessage` carries one `ClusterRequest` and + an optional process-admission proof. Secure Raft and ACL requests include the + proof; the limited admission lookup omits it. +2. **Mutual admission:** the acceptor verifies the connecting node, then replies + with its own `AdmissionProof`. Both proofs sign the same TLS exporter value and are + checked against the peer's current admission record. The exporter lets both + ends derive identical connection-specific bytes without sending those bytes; + another TLS connection derives a different value. +3. **After mutual admission:** + - The first Raft message carries its sender. Later frames are raw + `WireRaftMessage` values until close. Each carries `shard_group_id`. + - An ACL snapshot request carries its requesting node, shard, and resource. + Its response is one `AclSnapshotResponse`, then the connection closes. +4. **Trusted-development initial message:** no cryptographic admission exchange; + the Raft or ACL request carries no admission proof. ## Invariants -1. **Connection identity is established by handshake before any RPCs.** First frame on every connection is the initiator's `NodeId`. The acceptor uses this to key the writer slot and to detect the simultaneous-connect race. Without the handshake, the acceptor cannot route inbound messages to a peer-identified slot. +1. **Secure connection identity comes from mTLS plus mutual process admission.** + TLS supplies each stable Node Certificate Principal. A signature over the TLS + exporter value proves possession of the current process key and cannot be + replayed on another TLS session. Mutual proof is required because the Raft + connection carries traffic in both directions. The Raft sender or ACL requester + must equal its admitted `NodeId`. The first Raft sender then keys the writer slot + and detects the simultaneous-connect race. 2. **At most one writer per peer.** `writers` is keyed by `NodeId`. Coexisting writers would split messages to the same peer across two TCP connections; per-connection ordering would let later messages overtake earlier ones in unpredictable patterns, causing the leader to chase its own retries. @@ -29,10 +58,35 @@ Length-prefixed bincode frames: 4. **Address resolution is always live.** Every connect attempt queries SWIM for the peer's current address; the transport keeps no local address cache. A stale local cache would connect to the wrong host after a peer moves or restarts on a different address. -5. **Frame sizes are bounded.** Handshake frames capped at 1KB; message frames at 4MB. Without bounds, a malicious or buggy peer can exhaust memory by sending a giant length prefix before any payload. +5. **Handshake work is bounded.** The listener acquires a permit before +spawning a handshake task, applies a total handshake deadline, and uses a +bounded queue to return verified Raft streams to the dispatcher. A slow TLS, +admission lookup, or proof exchange never blocks the transport select loop. + +6. **Frame sizes are bounded.** Initial, process-admission, Raft, proof, and response +frames are capped before allocating their payload. -6. **Transport validates envelope identity but never interprets the RPC.** The -connection peer must match the envelope `sender`; a mismatch closes that +7. **Transport validates message identity but never interprets the RPC.** The +connection peer must match the message `sender`; a mismatch closes that connection. The transport routes by `shard_group_id` and passes the authenticated peer onward, but the RPC remains opaque. Voter, learner, leader, term, and log checks belong to the target Raft state machine. + +## Limited Admission Lookup Rule + +Admission records are sharded, so the acceptor may need another broker to read +the record required for its proof check. Requiring process admission for that +read would recurse. `AdmissionLookup(AdmissionRecordKey)` is therefore accepted +after mTLS but before process admission. It can read one named admission record +from one shard, returns one `AdmissionLookupResponse`, and closes. It cannot +carry Raft, ACL, client, or admission-write traffic. + +## ACL Snapshot Rule + +An ACL snapshot request is not a Raft RPC and never enters a Raft state machine. +In secure mode it is served only after the requester completes process +admission. It asks the local multi-Raft actor for the selected shard's committed +ACL record through the broker security actor, returns that record on the same +connection, then closes. It carries no client data request and cannot proxy one. +A connection admitted for Raft carries only raw Raft frames after its first +message; an invalid frame closes the connection. diff --git a/Cargo.lock b/Cargo.lock index e4634bd1..6bacc64d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -47,7 +47,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys", + "windows-sys 0.61.2", ] [[package]] @@ -58,7 +58,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys", + "windows-sys 0.61.2", ] [[package]] @@ -76,6 +76,45 @@ dependencies = [ "rustversion", ] +[[package]] +name = "asn1-rs" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7f43a50ac4fdca5df8e885c21b835997f0a1cdee65494a6847694a98652d9d8" +dependencies = [ + "asn1-rs-derive", + "asn1-rs-impl", + "displaydoc", + "nom", + "num-traits", + "rusticata-macros", + "thiserror", + "time", +] + +[[package]] +name = "asn1-rs-derive" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3109e49b1e4909e9db6515a30c633684d68cdeaa252f215214cb4fa1a5bfee2c" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", + "synstructure", +] + +[[package]] +name = "asn1-rs-impl" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "autocfg" version = "1.5.0" @@ -97,7 +136,16 @@ dependencies = [ "regex", "rustc-hash", "shlex 1.3.0", - "syn", + "syn 2.0.117", +] + +[[package]] +name = "bit-vec" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b71798fca2c1fe1086445a7258a4bc81e6e49dcd24c8d0dd9a1e57395b603f51" +dependencies = [ + "serde", ] [[package]] @@ -127,7 +175,7 @@ dependencies = [ "proc-macro-crate", "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -238,7 +286,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -300,6 +348,43 @@ dependencies = [ "parking_lot_core", ] +[[package]] +name = "data-encoding" +version = "2.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" + +[[package]] +name = "der-parser" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07da5016415d5a3c4dd39b11ed26f915f52fc4e0dc197d87908bc916e51bc1a6" +dependencies = [ + "asn1-rs", + "displaydoc", + "nom", + "num-bigint", + "num-traits", + "rusticata-macros", +] + +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" + +[[package]] +name = "displaydoc" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "east-guard" version = "0.1.0" @@ -317,7 +402,11 @@ dependencies = [ "lz4_flex", "murmur3", "rand 0.10.1", + "rcgen", + "ring", "rocksdb", + "rustls", + "rustls-pemfile", "rustyline", "serde", "serde_json", @@ -327,10 +416,12 @@ dependencies = [ "tempfile", "thiserror", "tokio", + "tokio-rustls", "tracing", "tracing-subscriber", "turmoil", "uuid", + "x509-parser", "zstd", ] @@ -359,7 +450,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys", + "windows-sys 0.61.2", ] [[package]] @@ -457,7 +548,7 @@ checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -567,7 +658,7 @@ version = "0.5.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cc627f471c528ff0c4a49e1d5e60450c8f6461dd6d10ba9dcd3a61d3dff7728d" dependencies = [ - "windows-sys", + "windows-sys 0.61.2", ] [[package]] @@ -722,9 +813,9 @@ dependencies = [ [[package]] name = "lz4_flex" -version = "0.13.1" +version = "0.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ef0d4ed8669f8f8826eb00dc878084aa8f253506c4fd5e8f58f5bce72ddb97e" +checksum = "ecbdfe44b1bd960b68170b417450a628c43f7cf56bb3c5317e61cb230ee7f226" dependencies = [ "twox-hash", ] @@ -758,7 +849,7 @@ checksum = "50b7e5b27aa02a74bac8c3f23f448f8d87ff11f92d3aac1a6ed369ee08cc56c1" dependencies = [ "libc", "wasi", - "windows-sys", + "windows-sys 0.61.2", ] [[package]] @@ -804,7 +895,32 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "num-bigint" +version = "0.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", ] [[package]] @@ -817,6 +933,15 @@ dependencies = [ "libm", ] +[[package]] +name = "oid-registry" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12f40cff3dde1b6087cc5d5f5d4d65712f34016a03ed60e9c08dcc392736b5b7" +dependencies = [ + "asn1-rs", +] + [[package]] name = "once_cell" version = "1.21.4" @@ -864,6 +989,12 @@ version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + [[package]] name = "ppv-lite86" version = "0.2.21" @@ -880,7 +1011,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn", + "syn 2.0.117", ] [[package]] @@ -982,6 +1113,19 @@ dependencies = [ "rand 0.9.2", ] +[[package]] +name = "rcgen" +version = "0.14.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57f6d249aad744e274e682777a50283a225a32705394ee6d5fcc01efa25e4055" +dependencies = [ + "ring", + "rustls-pki-types", + "time", + "x509-parser", + "yasna", +] + [[package]] name = "redox_syscall" version = "0.5.18" @@ -1020,6 +1164,20 @@ version = "0.8.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted", + "windows-sys 0.52.0", +] + [[package]] name = "rocksdb" version = "0.24.0" @@ -1036,6 +1194,15 @@ version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" +[[package]] +name = "rusticata-macros" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632" +dependencies = [ + "nom", +] + [[package]] name = "rustix" version = "1.1.4" @@ -1046,7 +1213,50 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustls" +version = "0.23.42" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c54fcab019b409d04215d3a17cb438fd7fbf192ee61461f20f4fe18704bc138" +dependencies = [ + "once_cell", + "ring", + "rustls-pki-types", + "rustls-webpki", + "subtle", + "zeroize", +] + +[[package]] +name = "rustls-pemfile" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "rustls-pki-types" +version = "1.15.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96" +dependencies = [ + "zeroize", +] + +[[package]] +name = "rustls-webpki" +version = "0.103.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +dependencies = [ + "ring", + "rustls-pki-types", + "untrusted", ] [[package]] @@ -1074,7 +1284,7 @@ dependencies = [ "unicode-segmentation", "unicode-width", "utf8parse", - "windows-sys", + "windows-sys 0.61.2", ] [[package]] @@ -1085,7 +1295,7 @@ checksum = "64e5587417a3c4e16a4415e8d7d07f80998ed835ade621d19dfbe9fbe3205b0f" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -1139,7 +1349,7 @@ checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -1170,9 +1380,9 @@ dependencies = [ [[package]] name = "serial_test" -version = "3.5.0" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "699f4197115b8a7e7ff19c9a315a4bd6fffec26cc4626ef45ecaea389e081c6d" +checksum = "a6df5ed973ad8d834e09f824f9e9f449af6b9a3745f78dec7cc752770bd3bf11" dependencies = [ "futures-executor", "futures-util", @@ -1184,13 +1394,13 @@ dependencies = [ [[package]] name = "serial_test_derive" -version = "3.5.0" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94e153fc76e1c6a068703d6d29c508a0b15c061c4b7e43da59cc097bc342673c" +checksum = "a22144e767da4ddd8416dbf383700542ffd8a5dc493dfecedfe1fe3ad03c98ae" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 3.0.3", ] [[package]] @@ -1243,7 +1453,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3a766e1110788c36f4fa1c2b71b387a7815aa65f88ce0229841826633d93723e" dependencies = [ "libc", - "windows-sys", + "windows-sys 0.61.2", ] [[package]] @@ -1261,6 +1471,12 @@ version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + [[package]] name = "syn" version = "2.0.117" @@ -1272,6 +1488,28 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "tempfile" version = "3.27.0" @@ -1282,7 +1520,7 @@ dependencies = [ "getrandom 0.4.1", "once_cell", "rustix", - "windows-sys", + "windows-sys 0.61.2", ] [[package]] @@ -1302,7 +1540,7 @@ checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -1314,6 +1552,36 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "time" +version = "0.3.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3e1d5e639ff6bab73cb6885cc7e7b1de96c3f32c68ec55f3952614bec1092244" +dependencies = [ + "deranged", + "num-conv", + "powerfmt", + "serde_core", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" + +[[package]] +name = "time-macros" +version = "0.2.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e689342a48d2ea927c87ea50cabf8594854bf940e9310208848d680d668ed85" +dependencies = [ + "num-conv", + "time-core", +] + [[package]] name = "tokio" version = "1.52.3" @@ -1328,7 +1596,7 @@ dependencies = [ "signal-hook-registry", "socket2", "tokio-macros", - "windows-sys", + "windows-sys 0.61.2", ] [[package]] @@ -1339,7 +1607,17 @@ checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", +] + +[[package]] +name = "tokio-rustls" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" +dependencies = [ + "rustls", + "tokio", ] [[package]] @@ -1391,7 +1669,7 @@ checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] [[package]] @@ -1484,6 +1762,12 @@ version = "0.2.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + [[package]] name = "utf8parse" version = "0.2.2" @@ -1571,7 +1855,7 @@ dependencies = [ "bumpalo", "proc-macro2", "quote", - "syn", + "syn 2.0.117", "wasm-bindgen-shared", ] @@ -1624,6 +1908,15 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets", +] + [[package]] name = "windows-sys" version = "0.61.2" @@ -1633,6 +1926,70 @@ dependencies = [ "windows-link", ] +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm", + "windows_aarch64_msvc", + "windows_i686_gnu", + "windows_i686_gnullvm", + "windows_i686_msvc", + "windows_x86_64_gnu", + "windows_x86_64_gnullvm", + "windows_x86_64_msvc", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + [[package]] name = "winnow" version = "1.0.3" @@ -1672,7 +2029,7 @@ dependencies = [ "heck", "indexmap", "prettyplease", - "syn", + "syn 2.0.117", "wasm-metadata", "wit-bindgen-core", "wit-component", @@ -1688,7 +2045,7 @@ dependencies = [ "prettyplease", "proc-macro2", "quote", - "syn", + "syn 2.0.117", "wit-bindgen-core", "wit-bindgen-rust", ] @@ -1730,6 +2087,34 @@ dependencies = [ "wasmparser", ] +[[package]] +name = "x509-parser" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d43b0f71ce057da06bc0851b23ee24f3f86190b07203dd8f567d0b706a185202" +dependencies = [ + "asn1-rs", + "data-encoding", + "der-parser", + "lazy_static", + "nom", + "oid-registry", + "ring", + "rusticata-macros", + "thiserror", + "time", +] + +[[package]] +name = "yasna" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5f6765e852b9b4dc8e2a76843e4d64d1cea8e79bcde0b6901aea8e7c7f08282" +dependencies = [ + "bit-vec", + "time", +] + [[package]] name = "zerocopy" version = "0.8.40" @@ -1747,9 +2132,15 @@ checksum = "f65c489a7071a749c849713807783f70672b28094011623e200cb86dcb835953" dependencies = [ "proc-macro2", "quote", - "syn", + "syn 2.0.117", ] +[[package]] +name = "zeroize" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" + [[package]] name = "zmij" version = "1.0.21" diff --git a/Cargo.toml b/Cargo.toml index a36c9b8e..8654f8cc 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -23,7 +23,7 @@ dashmap = "6.2.1" crc32fast = "1.4" arc-swap = "1" flume = "0.12" -lz4_flex = "0.13.1" +lz4_flex = "0.14.0" zstd = "0.13" rustyline = { version = "18.0.1", features = ["derive"] } shlex = "2.0.1" @@ -31,13 +31,19 @@ serde_yaml = "0.9.34" serde = "1" serde_json = "1" futures = "0.3" +rustls = { version = "0.23.42", default-features = false, features = ["ring", "std"] } +rustls-pemfile = "2.2.0" +tokio-rustls = { version = "0.26.4", default-features = false, features = ["ring"] } +x509-parser = "0.18.1" +ring = "0.17.14" [target.'cfg(target_os = "linux")'.dependencies] libc = "0.2" [dev-dependencies] +rcgen = { version = "0.14.8", default-features = false, features = ["crypto", "ring"] } tempfile = "3.27.0" -serial_test = "3.5.0" +serial_test = "4.0.1" turmoil = { version = "0.7.2", package = "turmoil" } serde = { version = "1", features = ["derive"] } serde_json = "1" diff --git a/docs/security/roadmap.md b/docs/security/roadmap.md index c9525ce8..7bb7a7c9 100644 --- a/docs/security/roadmap.md +++ b/docs/security/roadmap.md @@ -8,50 +8,68 @@ ## 1. Overview & Threat Model -EastGuard operates in two distinct security modes: - -- **Secure Mode (Default):** Mutual TLS 1.3 / DTLS 1.3 enforced across all listeners. Unauthenticated or unauthorized traffic is immediately rejected. Plaintext connections and protocol downgrades are forbidden; invalid configuration prevents startup. -- **Trusted Development Mode:** Plaintext protocols enabled strictly via explicit opt-in configuration for isolated test environments. +| Mode | Boundary | +| :--- | :--- | +| **Secure (default)** | TLS 1.3 protects TCP. Plaintext, downgrade, invalid configuration, and unauthenticated or unauthorized traffic are rejected. Startup fails while secure SWIM UDP is unavailable. | +| **Trusted development** | Explicit opt-in permits plaintext only in isolated test environments. | ### Threat Model -The production boundary defends against external network attackers attempting eavesdropping, packet modification, replay, UDP spoofing, connection injection, or TLS downgrade. -> **Scope Note:** Compromised brokers or authorized clients are outside this threat model. Their impact is constrained by short-lived credentials, scoped ACLs, and strict principal binding. Byzantine consensus is out of scope. +The boundary covers eavesdropping, modification, replay, UDP spoofing, +connection injection, and downgrade. Compromised brokers, authorized clients, +and Byzantine consensus are out of scope. ### Listener Architecture | Listener | Port | Protocol | Peer Authentication | Purpose | | :--- | :--- | :--- | :--- | :--- | | **Client** | TCP 2921 | TLS 1.3 | Mutual X.509 | Metadata queries, administration, produce, fetch | -| **Raft** | TCP 2922 | TLS 1.3 | Mutual X.509 | Metadata shard consensus log replication | +| **Raft** | TCP 2922 | TLS 1.3 | Mutual X.509 + process admission | Metadata shard consensus, one-shot ACL-cache refreshes, and limited admission-record reads between brokers | | **Data** | TCP 2923 | TLS 1.3 | Mutual X.509 | Segment replication, repair, and coordination | -| **SWIM** | UDP 2922 | DTLS 1.3 | Mutual X.509 | Membership gossip & failure detection | +| **SWIM** | UDP 2922 | Secure datagrams (deferred) | Mutual X.509 | Membership gossip and failure detection | --- ## 2. Layered Architecture -Security checks are split between the transport layer and application state machines to keep state machines free of security I/O: +Security checks are split across transport, one broker security actor, and the +application state machines. This keeps security I/O out of state machines: ``` - [ TLS 1.3 / DTLS 1.3 Transport Layer ] +[ Authenticated Transport Layer ] + TCP: TLS 1.3 + UDP: secure datagrams (deferred) - Authenticate peer X.509 certificates + - Prove that a node connection belongs to the currently admitted process - Enforce framing, datagram MTU, and resource limits - Bind connection envelope sender to verified identity │ │ (Drop connection on transport failure) ▼ - [ Application Layer State Machines ] - - Authorize requested operation against cached ACLs / placement - - Execute SWIM / Raft / Data-Plane state transitions +[ Broker Security Actor ] + - Cache admission and ACL records + - Route record reads to local Raft or one remote broker + - Authorize the authenticated principal │ - │ (Drop denied envelope only; connection stays open) + │ (Deny one request on authorization failure) + ▼ +[ Application Layer State Machines ] + - Execute SWIM / Raft / Data-Plane state transitions ``` -- **Transport Responsibility:** Performs cryptographic handshakes, validates certificates, tracks cache expiry, and binds envelope senders to verified identities. Transport errors close the connection. -- **Application Responsibility:** Synchronous state machines (SWIM, Raft, Topology, Data Plane) process only pre-validated envelopes. An authorization failure drops the specific denied envelope without tearing down the underlying connection. -- **Placement-Based Data-Plane Authorization:** Data-plane nodes authorize incoming replication and repair requests against local committed placement state, ignoring sender-asserted replica lists. -- **Redirects:** A redirect is only an address hint. The destination authenticates the peer and repeats the authorization check. +| Boundary | Rule | +| :--- | :--- | +| Transport | Authenticates, bounds frames and handshakes, and binds senders to verified identities. Failure closes the connection. | +| Security actor | Owns caches, record reads, and authorization. Denial drops one request, not the connection. | +| State machine | Applies only authenticated and authorized operations. | +| Data placement | Replication and repair use local committed placement, never a sender-asserted replica list. | +| Redirect | Carries only an address hint. The destination repeats authentication and authorization. | + +The actor owns only shared, non-durable state: both caches, record routing, and +identical-read combining. Certificates, connection lifetimes, frame parsing, +and durable records stay outside. Slow remote reads run in the background. +Admission and ACL refreshes share one active-read limit; saturation fails +closed instead of creating another queue or actor. --- @@ -63,11 +81,15 @@ To allow safe node restarts and hardware replacement without exposing the cluste | Term | Scope | Lifetime / Ordering | Function | | :--- | :--- | :--- | :--- | -| **Certificate Node ID** | Configuration | Long-lived / Reused | Operator-assigned node name embedded in the X.509 certificate. | -| **Admission Epoch** | Metadata Shard | Monotonically increasing `u64` | Assigned by metadata Raft upon restart; higher epoch **fences** older instances. | -| **NodeId** | Running Process | Single process lifetime | Unique ID generated on startup; used by SWIM, topology ring, Raft, and data placement. | +| **Node Certificate Principal** | X.509 Certificate | Long-lived / Reused | Stable admission-record key read from the certificate. | +| **Admission Epoch** | Node Certificate Principal | Increasing `u64` | Metadata Raft assigns it; a higher value fences older processes. | +| **NodeId** | Running Process | Single process lifetime | Startup-generated ID used by SWIM, topology, Raft, and placement. | | **Process Key** | Running Process | Single process lifetime | Proves that the connection belongs to the process admitted for this epoch. | -| **SWIM Incarnation** | Running Process | Monotonically increasing counter | Incremented by the *same* process instance to refute false `Suspect`/`Dead` gossip. | +| **SWIM Incarnation** | Running Process | Increasing counter | The same process increments it to refute stale liveness gossip. | + +The Node Certificate Principal is the value after `urn:eastguard:node:` in one +URI Subject Alternative Name. A node certificate must contain exactly one such +URI; missing or repeated values fail authentication. ### Resolution Rules @@ -85,13 +107,13 @@ Node identity conflicts and stale gossip are resolved in this order: ### Partition Recovery vs. Node Restart -An **authorized operator** is a person or trusted automation using a client -certificate whose principal has the `security/cluster` grant. In an orchestrated -deployment, automation may approve restarts and scaling, but metadata Raft still -commits every admission. +An **authorized operator** is a person or automation whose client-certificate +principal has the `security/cluster` grant. -- **Healed Partition (Same Process):** Retains its existing `NodeId` and `Admission Epoch`. Increments its `SWIM Incarnation` counter to refute `Suspect` or `Dead` rumors spread during the partition. -- **Node Restart / Replacement:** The process generates a new `NodeId` and process key. An authorized operator approves that exact process, and metadata Raft commits the new `Admission Epoch`, `NodeId`, and process public key. The higher epoch fences older processes immediately where observed and everywhere else within 60 seconds. +| Event | Identity and recovery | +| :--- | :--- | +| Healed partition; same process | Keep the `NodeId` and epoch. Increase the SWIM incarnation to refute stale `Suspect` or `Dead` gossip. | +| Restart or replacement | Create a new `NodeId` and process key. The operator approves them; metadata Raft commits a higher epoch. Cache expiry fences the old process everywhere within 60 seconds. | ### Admission Gate & SWIM Separation @@ -116,48 +138,134 @@ SWIM liveness gossip is decoupled from cluster admission authority to prevent ne [ Raft Reconciliation ] ──► Commit AddPeer / RemovePeer ``` -1. **Process Proof:** A node connection proves possession of the process private key bound to its admitted epoch. The process key signs the admitted identity and a connection-specific value produced by TLS or DTLS, so the proof cannot be replayed on another connection. The reusable node certificate alone cannot create or claim a newer epoch. -2. **Admission Gate:** The SWIM actor checks incoming packets against a local admission cache backed by metadata Raft before passing facts to the SWIM state machine. -3. **Gossip Rule:** DTLS authenticates the immediate sender. Every relayed membership fact is separately accepted only when its subject `NodeId` and `Admission Epoch` match an active admission record. -4. **Cache Policy:** Admission records are cached locally with a maximum TTL of 60 seconds. If a cache entry expires while the owning metadata shard is unreachable, the gate fails closed. +- Metadata Raft decides admission; SWIM reports only liveness. +- The transport authenticates the immediate sender. A relayed fact is accepted + only when its subject `NodeId` and epoch match an active admission. +- The local admission cache expires within 60 seconds and fails closed when its + owning shard is unavailable. + +### Why a TLS Session Proof Is Necessary + +A node certificate identifies a reusable broker principal, not one process +start. A UUID prevents accidental identity collisions, but it is public cluster +data. An old process that still has the reusable certificate can observe and +claim the current `NodeId` and `Admission Epoch`. + +``` +NodeId ──► which process the record names +Epoch ──► which admission is newer +Process key ──► proof that the speaker owns that admission +TLS value ──► proof is valid only on this connection +``` + +The admission record therefore stores a public key for one process start. It is +approved and committed before the connection. The broker sends a signature, not +a replacement key. Both sides prove their keys because Raft traffic is +bidirectional. + +``` +Connecting broker Accepting broker + │ │ + │◄────────────── mTLS ───────────────────►│ + │ derive the same fresh TLS session value │ + │── process proof + Raft / ACL ──────────►│ + │ │ verify current admission + │◄──────────── process proof ─────────────│ + │ verify current admission │ + │◄──────── admitted connection ──────────►│ +``` + +A TLS exporter derives application-specific bytes from a completed handshake: + +``` +connection 1: broker A derives X broker B derives X +connection 2: broker A derives Y broker B derives Y + X != Y +network: sends signatures over X or Y, never X or Y itself +``` + +“Shared” means both ends of one connection derive the same value. A signature +over `X` fails on a connection using `Y`; this removes the need for another +challenge. Missing or stale admission, identity mismatch, or bad signature +closes the connection before Raft or ACL dispatch. + +### Why Admission Lookup Has a Narrow Wire Path + +The admission record may live on another broker. A normal cluster connection +would recurse: + +``` +Need record ──► open admitted connection ──► need record ──► loop +``` + +A narrow pre-admission path breaks the loop: + +``` +Accepting broker Admission shard host + │ │ + 1. Open ─────────────── mTLS ──────────────────►│ + │ │ + 2. Ask ─────── one admission-record key ───────►│ + │ │ read committed state + 3. Return ◄────────── record or no record ──────│ + │ │ + └──────────────── connection closes ───────┘ +``` + +- **Authentication:** Secure mode requires mTLS. It authenticates the reusable + node certificate, not the running process. +- **One purpose:** The connection reads one admission record. It cannot carry + Raft messages, ACL reads, client requests, or admission writes. +- **Bounded work:** The broker security actor combines simultaneous reads for + the same record. Admission and ACL reads share one active-read limit. +- **Cache result:** A record or confirmed missing record is cached for at most + 60 seconds. A missing record denies admission. +- **Do not cache failure:** Timeout, routing failure, or an unavailable shard + denies the current connection but is retried by a later lookup. --- ## 4. Authorization & Sharded Security Records -EastGuard enforces exact, default-deny access control lists (ACLs) without wildcards or inheritance. -A principal is the client ID read from an authenticated certificate and used for -permission checks; its text grants no authority by itself. +ACLs are exact and default deny, with no wildcards or inheritance. A principal +comes from the client certificate; its text grants no authority by itself. ### ACL Resource Catalog | Resource Key Format | Granted Actions | | :--- | :--- | -| `cluster` | Membership inspection, topology lookup, operator diagnostics | -| `topic-admin/{topic}` | Create, delete, describe topic metadata | -| `topic-data/{topic}` | Produce, fetch, list offsets for topic | -| `consumer-group/{topic}/{group}` | Consume messages, read/commit consumer offsets | -| `producer-session/{topic}/{session}` | Renew producer session (permanently bound to creator principal) | +| `cluster` | Create and list topics, membership inspection, topology lookup, operator diagnostics | +| `topic-admin/{topic-id}` | Delete and describe topic metadata | +| `topic-data/{topic-id}` | Produce, fetch, list offsets for topic | +| `consumer-group/{topic-id}/{group-id}` | Coordinate the group and read/commit its offsets | +| `producer-session/{topic-id}/{producer-id}` | Renew the session, bound to its creator for the session lifetime | | `security/cluster` | Read/write ACLs, manage admissions and revocations, inspect security audit | -Consumer-group access permits group coordination and offset read/commit. Reading -records separately requires `Fetch` on `topic-data/{topic}`. +Consumer-group access covers coordination and offsets. Fetching records also +requires `topic-data/{topic-id}`. Text keys exist only at routing and +administrative boundaries; replicated state stores typed resources. ### Sharded Metadata Storage -A security record is one durable admission, ACL, or revocation entry. Its record -path selects one metadata shard; its revision lets brokers detect stale cached -copies. +Admission, ACL, and revocation paths hash to standard metadata shards: -Security records (`security/node/{id}`, `security/acl/{resource}`, `security/revocation/{issuer}/{serial}`) do not rely on a centralized security controller. Instead, they hash to standard metadata shards and replicate via Raft: +``` +Request ──► hash record path ──► shard host? ─┬─► yes: commit through Raft + └─► no: return owner redirect +``` ``` - Client/Node Request ──► Any Broker ──► Hash Record Path ──► Hosts Shard? ─┬─► Yes ──► Commit via Raft - └─► No ──► Return Owner Redirect +security/node/{node-certificate-principal} + │ + └── Admission Epoch + NodeId + Process Public Key ``` -- **Local Authorization:** Brokers evaluate ACLs against local cached security records. -- **Freshness & Expiry:** Cached records include a monotonic deadline (max 60s) and revision counter. Expired entries require re-validation from the owner metadata shard; if the owner is offline, authorization fails closed. +| Property | Rule | +| :--- | :--- | +| No controller | Every record is owned and replicated by its ordinary metadata shard. | +| Stable admission key | Restart changes the `NodeId` and process key, not the certificate principal, so one record atomically replaces the old process. | +| Local authorization | Cached ACLs and stable topic IDs let a data replica authorize without hosting topic metadata. | +| Freshness | Cache entries carry source shard, revision, and a monotonic deadline no later than 60 seconds. Expiry triggers refresh or denial. | --- @@ -165,81 +273,165 @@ Security records (`security/node/{id}`, `security/acl/{resource}`, `security/rev ### Bootstrap & Node Joining -1. Operator initializes a cluster trust root and issues the first node certificate. -2. The first node generates its `NodeId` and process key. Initial metadata state - stores that admission, the first operator principal, and its - `security/cluster` grant. -3. A later joining node generates a new `NodeId` and process key. -4. An authorized operator approves that exact `NodeId` and process public key. The reusable node certificate alone cannot authorize replacement. -5. The joining node connects to a **limited admission endpoint** using its X.509 certificate. -6. The endpoint routes to the owning metadata shard, which atomically commits the next `Admission Epoch`, `NodeId`, and process public key. -7. The joining node proves possession of the process private key before entering SWIM gossip and Raft membership reconciliation. +| Moment | Operator and cluster action | +| :--- | :--- | +| First broker | Create the trust root and node certificate. Initial metadata stores the first process admission, operator principal, and `security/cluster` grant. | +| Every later start | The process creates a new `NodeId` and process key. An authorized operator approves both. The owning shard increments the admission epoch and replaces the old process atomically. | +| Cluster connection | Each side follows the admission gate in Section 3. Raft or ACL traffic starts only after mutual process proof succeeds. | + +The reusable node certificate alone cannot replace an admitted process. SWIM +also remains blocked until the secure datagram admission gate in S3 exists. ### Online Credential Rotation -- **Zero-Downtime CA Rotation:** Brokers support dual trust chain loading. New root CAs can be added and leaf certificates reloaded online without restarting brokers or changing `Admission Epoch` / `NodeId`. -- **Revocation & Expiry:** Certificate revocations commit to metadata Raft records. Active TLS/DTLS sessions are terminated within the cache enforcement window. Expired certificates are rejected with clock-skew tolerance. -- **Recovery:** Runbooks cover lost authorized-operator access, lost issuing keys, expiry, accidental revocation, trust-root replacement, and cold-cluster restart. +| Operation | Required behavior | +| :--- | :--- | +| CA or leaf rotation | Load old and new trust chains together; reload leaves without restart or process-identity change. | +| Revocation or expiry | Commit revocations through metadata Raft; close active sessions within cache expiry; allow bounded clock skew. | +| Recovery | Cover lost operator access or issuing keys, accidental revocation, trust-root replacement, and cold restart. | --- ## 6. Resource Limits & Security Audit ### Rate & Memory Bounds -- Every listener enforces strict limits on unauthenticated handshakes, concurrent connections, in-flight frames, memory allocations, and per-source request rates. -- **DTLS Datagram Size:** Encrypted SWIM datagram payloads are capped at **1200 bytes** to ensure the total packet fits within the 1280-byte minimum IPv6 MTU without IP fragmentation. -### Audit Subsystem -- **Non-Blocking Execution:** Security audit events (authentication success/failure, ACL denials, admissions) are queued asynchronously. Audit backpressure never blocks protocol execution or consensus. -- **Rate-Limited Flood Protection:** High-frequency audit events use aggregate counters and sampled detail logging. -- **Credential Hygiene:** Audit logs never record private keys, tokens, credentials, or message payloads. +| Boundary | Limit | +| :--- | :--- | +| Listener | Bound unauthenticated handshakes, connections, in-flight frames, and allocations. | +| Client requests | Rate limiting is deferred until principal sharing and node-wide limits are defined for autoscaling workloads. | +| Future secure UDP | Keep protected payloads below the IP-fragmentation threshold. | ---- +### Client Request Boundary -## 7. Delivery Plan (S0–S6) +Clients normally route directly to the data replica named by their current topic +metadata. A redirect is only recovery from stale routing; brokers never proxy a +produce or fetch to another data node. ``` -S0 ──► S1 ──► S2 ──► S3 ──► S4 ──► S5 ──► S6 -config records TCP SWIM clients operations production - mTLS DTLS + ACLs + audit gate +Client request + │ + ▼ +Authenticate certificate + │ + ▼ +Check local ACL cache + ├── Current grant + ├── Current denial ─► Return unauthorized + └── Missing / expired + │ + ▼ + Read committed ACL state + ├── Local shard ─────────► Local metadata read + └── Remote shard ────────► One authenticated node request + │ + ▼ + Refresh cache + │ + ┌─────┴─────┐ + ▼ ▼ + Grant Deny / unavailable ──► Fail closed + │ + ▼ + Does this node serve the requested data? + ├── No ──► Return data-node redirect + └── Yes ──► Execute locally ``` -| Phase | Target Scope | Key Deliverable | Exit Criteria | -| :--- | :--- | :--- | :--- | -| **S0** | Configuration | Security modes, cert loader, DTLS 1.3 feasibility | Secure mode opens no plaintext listeners; a maintained DTLS 1.3 implementation satisfies certificate, replay, cookie, and resource-bound requirements | -| **S1** | Metadata Storage | Security record schema, sharded Raft state | Security records survive snapshot & recovery | -| **S2** | Cluster Transport | TLS 1.3 on TCP 2922/2923, Raft D8 RPC authorization | Authenticated and authorized cluster TCP traffic | -| **S3** | Membership | DTLS 1.3 on UDP 2922 & SWIM admission gate | Secure gossip & partition-safe admission fencing | -| **S4** | Client API | Client mTLS on TCP 2921, principal binding, ACLs | Default-deny enforcement on all client APIs | -| **S5** | Operations | Certificate rotation, revocation, expiry, recovery, audit logging | Online credential operations and recovery runbooks | -| **S6** | Production Gate | Adversarial testing, fuzzing, partition stress | Passes all production readiness checks | - -S6 must verify node and client impersonation, stale-process replay, unauthorized -operations, protocol downgrade, rotation under live traffic, expired and revoked -credentials, cold-cluster restart, handshake and datagram fuzzing, resource -bounds, and secret-free diagnostics. +| Decision | Rule | +| :--- | :--- | +| Authorize before redirect | An ungranted client cannot discover placement through stale-route responses. | +| Refresh on cache miss | Read local committed state or make one authenticated ACL-only request to the shard host. Never proxy client data. | +| Fail closed | Cache a missing record as a bounded denial. An unavailable owner denies without caching the failure. | ---- +Pull-on-miss avoids a second connection pool for reads needed at most once per +cache window. A future push or hybrid design may update only the same cache and +must remain fail closed. -## 8. Invariants & Security Rules +The broker security actor owns these remote reads and combines simultaneous +requests for the same ACL record. When its shared read limit is full, stopped, +or too slow, callers deny rather than opening more connections: -### System Invariants +``` +many cache misses + │ + ▼ + broker security actor + ├── same shard + resource ──► one read, reply to all waiters + ├── different records ──────► bounded background reads + └── full / unavailable ─────► deny +``` + +Only identical records combine. If distinct records to one shard become costly, +batch those resource keys in one request; do not delay mailbox reads or add a +connection pool. -1. **Single Connection Identity:** Every established client connection has exactly one authenticated principal; every node connection has exactly one `(Certificate Node ID, Admission Epoch, NodeId, Process Public Key)`. -2. **Unique Active Node Admission:** Metadata state maintains at most one active `(Admission Epoch, NodeId, Process Public Key)` per Certificate Node ID. -3. **Immutable Producer Session Principal:** Every producer session is immutably bound to the principal that created it. -4. **Explicit Cache Bounding:** Every cached security entry specifies its source metadata shard, revision, and expiry measured with a monotonic clock (≤ 60 seconds). -5. **Bounded Audit Footprint:** Audit queues and aggregate rate counters remain within configured capacity. +Client rate limiting remains deferred: + +| Principal model | Problem with a fixed per-principal limit | +| :--- | :--- | +| Shared by application replicas | One limit represents an autoscaling workload. | +| Unique per replica | Principal state grows with replica count. | -### Operational Rules +Identity granularity, node-wide capacity, and cache distribution must be chosen +together before adding a limit. -1. Secure mode never downgrades to plaintext or unauthenticated protocols. -2. Transport layers authenticate identity; application state machines authorize actions. -3. Transport identity mismatches close the connection immediately before payload dispatch. -4. Application authorization denials drop only the denied envelope, preserving the connection for valid traffic. -5. A higher admission epoch fences every older process immediately where observed and no later than admission-cache expiry elsewhere. -6. SWIM controls liveness; metadata Raft controls admission. -7. Missing or expired security records cause authorization and admission checks to fail closed. -8. Resource permits are acquired before allocating memory or spawning async tasks. -9. Audit logging backpressure must never block network protocol processing or consensus. -10. Only an authorized operator may approve a higher admission epoch. +### Secure UDP Decision + +EastGuard retains UDP for SWIM because connection-oriented transport does not +fit membership at cluster scale: + +``` + SWIM probes one peer per interval + │ + ▼ + stateless UDP + │ + ┌────────────┴────────────┐ + ▼ ▼ +constant socket count packet loss remains +per node visible to SWIM +``` + +| Alternative | Why it is not selected now | +| :--- | :--- | +| TCP | Either keeps a connection mesh or causes handshake churn, kernel tracking, and head-of-line blocking. | +| QUIC datagrams | Preserve loss, but add per-peer connection state and complexity for sparse probes. | +| Current DTLS libraries | Do not yet combine maturity, permissive licensing, Rust integration, and deterministic simulation. | + +Secure SWIM remains deferred. Trusted development may use plaintext UDP in +isolation; secure mode fails startup and never falls back to it. + +### Acceptance Criteria for a Future Secure UDP Transport + +The selected transport must: + +| Requirement | Reason | +| :--- | :--- | +| Preserve datagram boundaries and loss | SWIM timeouts and indirect probes must observe loss rather than transport retransmission delays | +| Keep per-node transport state bounded independently of cluster size | Membership must remain viable for clusters with thousands of nodes | +| Authenticate node certificates and expose the certificate principal | Admission must bind each packet source to a verified node identity | +| Reject replay and spoofed source traffic | Old or forged membership packets must not alter liveness | +| Avoid IP fragmentation | One lost fragment must not discard an oversized protected packet | +| Run over EastGuard's UDP abstraction | Production and turmoil must exercise the same protocol state machine | +| Use virtual time in deterministic tests | Handshake retry, expiry, and packet loss must be reproducible | +| Use a mature, maintainable, permissively licensed dependency | Cluster security must not rely on an unaudited or incompatible implementation | + +### Audit Subsystem +- **Non-Blocking Execution:** Security audit events (authentication success/failure, ACL denials, admissions) are queued asynchronously. Audit backpressure never blocks protocol execution or consensus. +- **Rate-Limited Flood Protection:** High-frequency audit events use aggregate counters and sampled detail logging. +- **Credential Hygiene:** Audit logs never record private keys, tokens, credentials, or message payloads. + +--- + +## 7. Delivery Plan (S0–S6) + +| Phase | Complete when | +| :--- | :--- | +| **S0 — Configuration** | Secure configuration loads certificates, opens no plaintext listener, and fails startup when a required secure listener is unavailable. | +| **S1 — Records** | Sharded admission, ACL, and revocation records survive snapshot and recovery. | +| **S2 — Cluster TCP** | TLS 1.3 and session-bound process proof admit Raft, ACL, and data traffic only from the current process. | +| **S3 — SWIM (deferred)** | A transport meeting Section 6's UDP criteria provides authenticated gossip and partition-safe admission fencing. | +| **S4 — Clients** | Client mTLS and ACLs enforce default deny on every API. | +| **S5 — Operations** | Rotation, revocation, expiry, recovery, and non-blocking audit work online. | +| **S6 — Production** | Impersonation, replay, downgrade, fuzzing, resource bounds, recovery, and reproducible secure-SWIM tests pass. S3 must be complete. | diff --git a/eastguard.yaml.example b/eastguard.yaml.example index ef670971..5a821630 100644 --- a/eastguard.yaml.example +++ b/eastguard.yaml.example @@ -1,5 +1,12 @@ # EastGuard Configuration +# Secure is the production default. Use trusted-development only on isolated +# development and test networks until secure transports are configured. +security_mode: "secure" +certificate_chain_path: "./eastguard/config/node-chain.pem" +private_key_path: "./eastguard/config/node-key.pem" +trust_root_path: "./eastguard/config/cluster-ca.pem" + # Sockets and networking client_port: 2921 cluster_port: 2922 diff --git a/src/client/consumer/group.rs b/src/client/consumer/group.rs index d0c55202..cc7305db 100644 --- a/src/client/consumer/group.rs +++ b/src/client/consumer/group.rs @@ -8,9 +8,9 @@ use dashmap::DashMap; use uuid::Uuid; use crate::client::{Client, ClientError, ClientSuccess}; -use crate::connections::protocol::{ClientResponse, ConsumerGroupSyncAction}; +use crate::connections::protocol::{ClientResponse, ConsumerGroupMemberAction}; use crate::control_plane::metadata::consumer_group::GenerationId; -use crate::control_plane::metadata::{EntryId, RangeId, SyncConsumerGroupRequest, TopicId}; +use crate::control_plane::metadata::{EntryId, RangeId, TopicId, UpdateConsumerGroupMemberRequest}; use crate::data_plane::auxiliary_states::consumer_offsets::state::{ ConsumerOffsetKey, ConsumerOffsetPosition, ConsumerOffsetUpdate, }; @@ -190,11 +190,11 @@ impl ConsumerGroup { } pub(crate) async fn request_assignment(&self) -> Result, ClientError> { - let request = SyncConsumerGroupRequest { + let request = UpdateConsumerGroupMemberRequest { topic_name: self.topic.to_string(), group_id: self.group_id.to_string(), member_id: self.consumer_id, - action: ConsumerGroupSyncAction::Heartbeat, + action: ConsumerGroupMemberAction::Heartbeat, }; let assignment = match self @@ -219,11 +219,11 @@ impl ConsumerGroup { if self.quit.load(AtomicOrdering::Acquire) { return Ok(()); } - let request = SyncConsumerGroupRequest { + let request = UpdateConsumerGroupMemberRequest { topic_name: self.topic.clone(), group_id: self.group_id.clone(), member_id: self.consumer_id, - action: ConsumerGroupSyncAction::Leave, + action: ConsumerGroupMemberAction::Leave, }; let served = self .client @@ -294,11 +294,11 @@ impl Drop for ConsumerGroup { return; } let client = self.client.clone(); - let req = SyncConsumerGroupRequest { + let req = UpdateConsumerGroupMemberRequest { topic_name: self.topic.clone(), group_id: self.group_id.clone(), member_id: self.consumer_id, - action: ConsumerGroupSyncAction::Leave, + action: ConsumerGroupMemberAction::Leave, }; if let Ok(handle) = tokio::runtime::Handle::try_current() { handle.spawn(async move { diff --git a/src/client/mod.rs b/src/client/mod.rs index a545188d..a51c88e5 100644 --- a/src/client/mod.rs +++ b/src/client/mod.rs @@ -36,7 +36,7 @@ use crate::control_plane::NodeAddressInfo; use crate::control_plane::metadata::consumer_group::GenerationId; pub use crate::control_plane::metadata::strategy::{PartitionStrategy, StoragePolicy}; pub use crate::control_plane::metadata::{EntryId, RangeId}; -use crate::control_plane::metadata::{SyncConsumerGroupRequest, TopicId}; +use crate::control_plane::metadata::{TopicId, UpdateConsumerGroupMemberRequest}; use crate::data_plane::auxiliary_states::consumer_offsets::state::{ ConsumerOffsetKey, ConsumerOffsetPosition, }; @@ -53,7 +53,7 @@ use uuid::Uuid; use crate::connections::protocol::{ ClientDataPlaneRequest, ClientRequest, ClientResponse, CommitConsumerOffsetRequest, - ConsumerGroupAssignmentResponse, ConsumerGroupSyncAction, ControlPlaneRequest, + ConsumerGroupAssignmentResponse, ConsumerGroupMemberAction, ControlPlaneRequest, FetchConsumerOffsetRequest, OpenProducerSessionRequest, ProduceRequest, ProducerSessionOpened, RangeOffsetRequest, }; @@ -506,6 +506,7 @@ impl Client { ServerError::TopicNotFound => Redirect::NotFound, ServerError::SegmentNotLocal | ServerError::Internal(_) => Redirect::Reresolve, ServerError::AlreadyExists + | ServerError::Unauthorized | ServerError::StaleRange | ServerError::ProduceRejected(_) | ServerError::EntryIdOutOfRange diff --git a/src/config.rs b/src/config.rs index 11cd88ae..b795d50d 100644 --- a/src/config.rs +++ b/src/config.rs @@ -4,7 +4,7 @@ use std::sync::{Arc, LazyLock}; use std::fs::{self, OpenOptions}; -use clap::Parser; +use clap::{Parser, ValueEnum}; use uuid::Uuid; use crate::control_plane::membership::peer_discovery::JoinAttempt; @@ -15,9 +15,32 @@ use crate::data_plane::sparse_index::SparseIndex; use crate::schedulers::ticker::TICK_PERIOD_100_MS; pub static ENV: LazyLock = LazyLock::new(Environment::init); +#[derive(Debug, Clone, Copy, PartialEq, Eq, ValueEnum)] +pub enum SecurityMode { + Secure, + TrustedDevelopment, +} + #[derive(Parser, Debug, Clone)] #[command(version, about, long_about = None)] pub struct Environment { + /// Secure mode requires authenticated encrypted transports. Trusted-development + /// mode keeps the existing plaintext protocols for isolated tests and local work. + #[arg(long, env = "SECURITY_MODE", value_enum, default_value = "secure")] + pub security_mode: SecurityMode, + + /// PEM certificate chain presented by this node in secure mode. + #[arg(long, env = "CERTIFICATE_CHAIN_PATH")] + pub certificate_chain_path: Option, + + /// PEM private key matching this node's leaf certificate. + #[arg(long, env = "PRIVATE_KEY_PATH")] + pub private_key_path: Option, + + /// PEM certificate authorities trusted for node and client authentication. + #[arg(long, env = "TRUST_ROOT_PATH")] + pub trust_root_path: Option, + #[arg(long, env = "CONFIG_DIR", default_value = "./eastguard/config")] pub config_dir: String, @@ -428,6 +451,10 @@ mod tests { fn make_env() -> Environment { Environment { + security_mode: SecurityMode::TrustedDevelopment, + certificate_chain_path: None, + private_key_path: None, + trust_root_path: None, config_dir: "./eastguard/config".to_string(), config_file: None, data_dir: "./eastguard/data".to_string(), @@ -494,6 +521,18 @@ mod tests { assert_eq!(env.host, "0.0.0.0"); assert_eq!(env.data_dir, "/tmp/test"); assert_eq!(env.vnodes_per_node, 8); + assert_eq!(env.security_mode, SecurityMode::Secure); + } + + #[test] + fn security_mode_requires_explicit_trusted_development_opt_in() { + let secure = Environment::try_parse_from(["eastguard"]).unwrap(); + let trusted = + Environment::try_parse_from(["eastguard", "--security-mode", "trusted-development"]) + .unwrap(); + + assert_eq!(secure.security_mode, SecurityMode::Secure); + assert_eq!(trusted.security_mode, SecurityMode::TrustedDevelopment); } #[test] diff --git a/src/connections/controller.rs b/src/connections/controller.rs index 732cbbeb..f5683752 100644 --- a/src/connections/controller.rs +++ b/src/connections/controller.rs @@ -4,13 +4,14 @@ use crate::connections::{protocol::*, run_client_writer}; use crate::control_plane::NodeAddressInfo; use crate::control_plane::consensus::raft::errors::ProposalError; use crate::control_plane::metadata::{ - OpenProducerSession, RangeMeta, SyncConsumerGroup, SyncConsumerGroupRequest, TopicState, + AclResource, ConsumerGroupResource, OpenProducerSession, ProducerSessionOwner, RangeMeta, + TopicState, UpdateConsumerGroupMember, UpdateConsumerGroupMemberRequest, }; use crate::control_plane::{ NodeId, SwimNodeState, consensus::actor::MutlRaftSender, membership::{ - ShardGroupId, + ShardGroup, ShardGroupId, actor::{ShardRouting, SwimSender}, }, metadata::{ @@ -28,7 +29,8 @@ use crate::data_plane::messages::command::{ use crate::data_plane::messages::query::{ DataPlaneQuery, Fetch, ListOffsets, ReadConsumerOffset, ReadConsumerOffsetResult, }; -use crate::net::TcpStream; +use crate::net::TransportTcpStream; +use crate::security::{CertificatePrincipal, SecurityHandle}; use tokio::sync::mpsc; /// # Client ↔ Server request_id protocol @@ -47,24 +49,30 @@ use tokio::sync::mpsc; /// with responses arriving in any order. #[derive(Clone)] pub struct ClientController { + certificate_principal: Option, node_id: NodeId, swim_sender: SwimSender, raft_sender: MutlRaftSender, data_plane_tx: DataPlaneSender, + security: SecurityHandle, } impl ClientController { fn new( + certificate_principal: Option, node_id: NodeId, swim_sender: SwimSender, raft_sender: MutlRaftSender, data_plane_tx: DataPlaneSender, + security: SecurityHandle, ) -> Self { Self { + certificate_principal, node_id, swim_sender, raft_sender, data_plane_tx, + security, } } @@ -103,10 +111,14 @@ impl ClientController { } pub async fn dispatch(&self, request: ClientRequest) -> ClientResponse { + tracing::trace!( + certificate_principal = ?self.certificate_principal, + "dispatching client request" + ); match request { ClientRequest::ControlPlane(cp) => self.handle_control_plane(cp).await, ClientRequest::DataPlane(dp) => self.handle_data_plane(dp).await, - ClientRequest::Admin(admin) => self.handle_admin(admin).await, + ClientRequest::Admin(admin) => self.handle_admin(admin).await.into(), } } @@ -121,7 +133,7 @@ impl ClientController { DeleteTopic { name } => self.delete_topic(name).await, ListHostedTopics => self.list_hosted_topics().await, DescribeTopic { name } => self.describe_topic(name).await, - SyncConsumerGroup(req) => self.sync_consumer_group(req).await, + SyncConsumerGroup(req) => self.update_consumer_group_member(req).await, OpenProducerSession(req) => self.open_producer_session(req).await, }; res.into() @@ -131,29 +143,39 @@ impl ClientController { &self, req: OpenProducerSessionRequest, ) -> Result { - let command: OpenProducerSession = req.into_command(); + // Convert ephemeral authentication evidence into the durable owner + // carried by the Raft command. The persisted type intentionally does + // not depend on TLS or stream implementation details. + let owner = ProducerSessionOwner::from(self.certificate_principal.as_ref()); - let group = match self.route(command.topic_name.as_bytes().to_vec()).await? { - ShardRouting::Local(group) => group, - ShardRouting::Redirect(member) => { - return Err(self.control_plane_redirect(member)); - } - }; + let command: OpenProducerSession = req.into_command(owner.clone()); - self.propose_topic_write(group.id, command.clone()).await?; + let group = self + .route_local(command.topic_name.as_bytes().to_vec()) + .await?; let topic_meta = self .raft_sender - .get_topic_metadata(command.topic_name) + .get_topic_metadata(command.topic_name.to_string()) .await?; + self.authorize_acl_resource(AclResource::TopicData(topic_meta.id)) + .await?; + topic_meta + .producer_sessions + .get_for_owner(&command.producer_id, &owner)?; + + self.propose_topic_write(group.id, command.clone()).await?; - let session = topic_meta + let committed_topic = self + .raft_sender + .get_topic_metadata(command.topic_name.to_string()) + .await?; + + let session = committed_topic .producer_sessions - .get(&command.producer_id) - .copied() - .ok_or_else(|| { - ServerError::Internal("committed producer session is unavailable".into()) - })?; + .get_for_owner(&command.producer_id, &owner)? + .cloned() + .ok_or(ServerError::Unauthorized)?; Ok(ClientSuccess::ProducerSessionOpened( ProducerSessionOpened { @@ -163,21 +185,26 @@ impl ClientController { )) } - async fn sync_consumer_group( + async fn update_consumer_group_member( &self, - req: SyncConsumerGroupRequest, + req: UpdateConsumerGroupMemberRequest, ) -> Result { - let group = match self.route(req.topic_name.as_bytes().to_vec()).await? { - ShardRouting::Local(group) => group, - ShardRouting::Redirect(member) => { - return Err(self.control_plane_redirect(member)); - } - }; + let group = self.route_local(req.topic_name.as_bytes().to_vec()).await?; - self.propose_topic_write(group.id, SyncConsumerGroup::new(req.clone())) + let topic = self + .raft_sender + .get_topic_metadata(req.topic_name.clone()) .await?; + self.authorize_acl_resource(AclResource::ConsumerGroup(ConsumerGroupResource { + topic_id: topic.id, + group_id: req.group_id.clone(), + })) + .await?; - if req.action == ConsumerGroupSyncAction::Leave { + self.propose_topic_write(group.id, UpdateConsumerGroupMember::new(req.clone())) + .await?; + + if req.action == ConsumerGroupMemberAction::Leave { return Ok(ClientSuccess::ConsumerGroupLeft); } @@ -202,11 +229,11 @@ impl ClientController { /// otherwise it redirects to a member so the consumer retries against the right /// node — no server-side proxying. async fn describe_topic(&self, topic_name: String) -> Result { - if let ShardRouting::Redirect(member) = self.route(topic_name.as_bytes().to_vec()).await? { - return Err(self.control_plane_redirect(member)); - } + self.route_local(topic_name.as_bytes().to_vec()).await?; let topic = self.raft_sender.get_topic_metadata(topic_name).await?; + self.authorize_acl_resource(AclResource::TopicAdmin(topic.id)) + .await?; let addresses = self.swim_sender.list_all_node_addresses().await?; let detail = TopicDetail::from_meta(topic, &addresses); @@ -218,12 +245,10 @@ impl ClientController { name: String, storage_policy: StoragePolicy, ) -> Result { - let group = match self.route(name.as_bytes().to_vec()).await? { - ShardRouting::Local(group) => group, - ShardRouting::Redirect(member) => { - return Err(self.control_plane_redirect(member)); - } - }; + let group = self.route_local(name.as_bytes().to_vec()).await?; + // A new topic has no stable ID yet, so its creator needs the cluster-wide + // grant. Once created, topic-admin/{topic-id} governs its metadata. + self.authorize_acl_resource(AclResource::Cluster).await?; let created_at = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) @@ -241,12 +266,13 @@ impl ClientController { } async fn delete_topic(&self, topic_name: String) -> Result { - let group = match self.route(topic_name.as_bytes().to_vec()).await? { - ShardRouting::Local(group) => group, - ShardRouting::Redirect(member) => { - return Err(self.control_plane_redirect(member)); - } - }; + let group = self.route_local(topic_name.as_bytes().to_vec()).await?; + let topic = self + .raft_sender + .get_topic_metadata(topic_name.clone()) + .await?; + self.authorize_acl_resource(AclResource::TopicAdmin(topic.id)) + .await?; let cmd = DeleteTopic { name: topic_name }; self.propose_topic_write(group.id, cmd).await?; @@ -261,6 +287,23 @@ impl ClientController { .await } + /// Resolves a local metadata shard or returns a redirect for a normal + /// client request. ACL authorization needs the richer route directly. + async fn route_local(&self, key: Vec) -> Result { + match self.route(key).await? { + ShardRouting::Local(group) => Ok(group), + ShardRouting::Redirect(remote) => { + Err(self.control_plane_redirect(remote.and_then(|remote| remote.member))) + } + } + } + + async fn authorize_acl_resource(&self, resource: AclResource) -> Result<(), ServerError> { + self.security + .authorize(self.certificate_principal.as_ref(), resource) + .await + } + /// Structural redirect for a control-plane op that isn't local: to the member if /// one resolves, else a retriable error (no member's address known here yet). fn control_plane_redirect(&self, member: Option) -> ServerError { @@ -304,6 +347,8 @@ impl ClientController { } async fn list_hosted_topics(&self) -> Result { + self.authorize_acl_resource(AclResource::Cluster).await?; + let topics = self .raft_sender .get_topics() @@ -336,6 +381,8 @@ impl ClientController { &self, req: CommitConsumerOffsetRequest, ) -> Result { + self.authorize_acl_resource(req.key.acl()).await?; + let (tx, recv) = tokio::sync::oneshot::channel(); self.data_plane_tx .send_async(CommitConsumerOffset { @@ -364,6 +411,8 @@ impl ClientController { &self, req: FetchConsumerOffsetRequest, ) -> Result { + self.authorize_acl_resource(req.key.acl()).await?; + let (reply, recv) = tokio::sync::oneshot::channel(); self.data_plane_tx .send_async(ReadConsumerOffset { @@ -395,13 +444,17 @@ impl ClientController { let received_at_ms = crate::now_ms(); // Not local (ring unconverged or this node isn't a member) → retriable // redirect; the hint is best-effort, absent until SWIM converges. - if let ShardRouting::Redirect(hint_node) = + if let ShardRouting::Redirect(remote) = self.route(req.topic_name.as_bytes().to_vec()).await? { - return Err(ServerError::ShardNotLocal { hint_node }); + return Err(ServerError::ShardNotLocal { + hint_node: remote.and_then(|remote| remote.member), + }); } let topic = self.raft_sender.get_topic_metadata(req.topic_name).await?; + self.authorize_acl_resource(AclResource::TopicData(topic.id)) + .await?; let producer_identity = req .producer_identity @@ -455,6 +508,8 @@ impl ClientController { /// state machine needs to answer without any further I/O. async fn fetch(&self, req: FetchRequest) -> Result { let topic = self.raft_sender.get_topic_metadata(req.topic_name).await?; + self.authorize_acl_resource(AclResource::TopicData(topic.id)) + .await?; let range = topic.get_range(&req.range_id)?; @@ -487,6 +542,9 @@ impl ClientController { /// serve it. No proxying: a miss returns `SegmentNotLocal` and the client /// retries another replica. async fn fetch_by_id(&self, req: FetchByIdRequest) -> Result { + self.authorize_acl_resource(AclResource::TopicData(req.topic_id)) + .await?; + let (reply_tx, reply_rx) = tokio::sync::oneshot::channel(); let query = Fetch { topic_id: req.topic_id, @@ -510,6 +568,8 @@ impl ClientController { /// for the range's currently-active segment on this node. async fn list_offsets(&self, req: RangeOffsetRequest) -> Result { let topic = self.raft_sender.get_topic_metadata(req.topic_name).await?; + self.authorize_acl_resource(AclResource::TopicData(topic.id)) + .await?; let (reply_tx, reply_rx) = tokio::sync::oneshot::channel(); @@ -529,17 +589,17 @@ impl ClientController { Ok(ClientSuccess::RangeOffset(range_offset)) } - async fn handle_admin(&self, request: AdminRequest) -> ClientResponse { + async fn handle_admin(&self, request: AdminRequest) -> Result { use AdminRequest::*; - let res = match request { + self.authorize_acl_resource(AclResource::Cluster).await?; + + match request { DescribeCluster => self.describe_cluster().await, ListHostedTopicsWithStats => self.list_hosted_topics_with_stats().await, - GetShardInfo { key } => self.get_shard_info(key).await, GetShardLeader { shard_group_id } => self.handle_get_shard_leader(shard_group_id).await, - }; - res.into() + } } async fn describe_cluster(&self) -> Result { @@ -615,15 +675,24 @@ fn keyspace_bound_matches_range(bound: &Option, range: &RangeMeta } pub async fn handle_client_stream( - stream: TcpStream, + stream: TransportTcpStream, node_id: NodeId, swim_sender: SwimSender, raft_sender: MutlRaftSender, data_plane_tx: DataPlaneSender, + security: SecurityHandle, ) { + let certificate_principal = stream.peer_principal(); let (read_half, write_half) = stream.into_split(); let (writer_tx, writer_rx) = mpsc::channel(128); - let handler = ClientController::new(node_id, swim_sender, raft_sender, data_plane_tx); + let handler = ClientController::new( + certificate_principal, + node_id, + swim_sender, + raft_sender, + data_plane_tx, + security, + ); tokio::spawn(run_client_writer( ClientRawWriter::new(write_half), writer_rx, @@ -642,18 +711,29 @@ mod tests { }; use crate::control_plane::consensus::actor::MultiRaftActor; use crate::control_plane::consensus::messages::MultiRaftActorCommand; + use crate::control_plane::consensus::raft::states::security::AclRecord; + use crate::control_plane::consensus::transport::RaftTransportActor; use crate::control_plane::membership::actor::SwimActor; use crate::control_plane::membership::{ QueryCommand, ShardGroup, ShardGroupId, ShardLeaderEntry, SwimActorCommand, }; - use crate::control_plane::metadata::TopicStats as MetadataTopicStats; + use crate::control_plane::metadata::consumer_group::GenerationId; use crate::control_plane::metadata::strategy::{PartitionStrategy, StoragePolicy}; + use crate::control_plane::metadata::{ConsumerGroupResource, TopicStats as MetadataTopicStats}; use crate::control_plane::metadata::{RangeId, TopicId, TopicMeta}; use crate::control_plane::{NodeAddress, NodeId, Replicas, SwimNode, SwimNodeState}; use crate::data_plane::actor::DataPlaneSender; + use crate::data_plane::auxiliary_states::consumer_offsets::state::ConsumerOffsetKey; use crate::data_plane::messages::DataPlaneMessage; use crate::data_plane::messages::command::{DataPlaneCommand, ProduceAck}; + use crate::net::TcpListener; + use crate::security::{NodeTransportSecurity, SecurityActor}; use std::net::SocketAddr; + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::time::Duration; + use tokio::sync::Notify; + use turmoil::Builder; fn addr(port: u16) -> SocketAddr { format!("127.0.0.1:{port}").parse().unwrap() @@ -715,6 +795,61 @@ mod tests { DataPlaneSender(tx) } + fn trusted_controller( + node_id: NodeId, + swim_sender: SwimSender, + raft_sender: MutlRaftSender, + data_plane_tx: DataPlaneSender, + ) -> ClientController { + let security = SecurityActor::spawn( + node_id.clone(), + swim_sender.clone(), + raft_sender.clone(), + NodeTransportSecurity::TrustedDevelopment, + ); + ClientController::new( + None, + node_id, + swim_sender, + raft_sender, + data_plane_tx, + security, + ) + } + + fn authenticated_controller( + principal: &str, + node_id: NodeId, + swim_sender: SwimSender, + raft_sender: MutlRaftSender, + ) -> ClientController { + let security = SecurityActor::spawn( + node_id.clone(), + swim_sender.clone(), + raft_sender.clone(), + NodeTransportSecurity::TrustedDevelopment, + ); + ClientController::new( + Some(CertificatePrincipal::new(principal)), + node_id, + swim_sender, + raft_sender, + dp_stub(), + security, + ) + } + + fn acl_snapshot(resource: AclResource, principals: &[&str]) -> AclRecord { + AclRecord { + resource, + revision: 1, + principals: principals + .iter() + .map(|principal| (*principal).to_owned()) + .collect(), + } + } + fn produce_req() -> ClientRequest { ClientRequest::DataPlane(ClientDataPlaneRequest::Produce(ProduceRequest { topic_name: "t1".into(), @@ -741,6 +876,353 @@ mod tests { ) } + #[tokio::test] + async fn certificate_client_requires_exact_topic_acl() { + let group = ShardGroup { + id: ShardGroupId(42), + replicas: Replicas::new(vec![node_id("self")]), + }; + let swim = swim_sender_with(move |cmd| { + if let SwimActorCommand::Query(QueryCommand::ResolveShardGroup { reply, .. }) = cmd { + let _ = reply.send(Some(group.clone())); + } + }); + let raft = raft_sender_with(|cmd| { + if let MultiRaftActorCommand::GetAclSnapshot(query) = cmd { + assert_eq!(query.shard_group_id, ShardGroupId(42)); + assert_eq!(query.resource, AclResource::TopicData(TopicId(7))); + let _ = query + .reply + .send(Some(acl_snapshot(query.resource, &["orders-service"]))); + } + }); + let controller = authenticated_controller("orders-service", node_id("self"), swim, raft); + + assert_eq!( + controller + .authorize_acl_resource(AclResource::TopicData(TopicId(7))) + .await, + Ok(()) + ); + } + + #[tokio::test] + async fn acl_cache_miss_reads_the_local_snapshot_once() { + let group = test_shard_group(); + let swim = swim_sender_with(move |cmd| { + if let SwimActorCommand::Query(QueryCommand::ResolveShardGroup { reply, .. }) = cmd { + let _ = reply.send(Some(group.clone())); + } + }); + let query_count = Arc::new(AtomicUsize::new(0)); + let observed_query_count = query_count.clone(); + let raft = raft_sender_with(move |cmd| { + if let MultiRaftActorCommand::GetAclSnapshot(query) = cmd { + observed_query_count.fetch_add(1, Ordering::Relaxed); + let _ = query + .reply + .send(Some(acl_snapshot(query.resource, &["orders-service"]))); + } + }); + let controller = authenticated_controller("orders-service", node_id("node-1"), swim, raft); + + for _ in 0..2 { + assert_eq!( + controller + .authorize_acl_resource(AclResource::TopicData(TopicId(7))) + .await, + Ok(()) + ); + } + assert_eq!(query_count.load(Ordering::Relaxed), 1); + } + + #[tokio::test] + async fn topic_metadata_operations_require_topic_admin_acl() { + let group = test_shard_group(); + let swim = swim_sender_with(move |cmd| { + if let SwimActorCommand::Query(QueryCommand::ResolveShardGroup { reply, .. }) = cmd { + let _ = reply.send(Some(group.clone())); + } + }); + let proposal_count = Arc::new(AtomicUsize::new(0)); + let observed_proposals = proposal_count.clone(); + let raft = raft_sender_with(move |cmd| match cmd { + MultiRaftActorCommand::GetTopicMetadata { reply, .. } => { + let _ = reply.send(Some(topic_meta("node-1"))); + } + MultiRaftActorCommand::GetAclSnapshot(query) => { + assert_eq!(query.resource, AclResource::TopicAdmin(TopicId(1))); + let _ = query.reply.send(Some(acl_snapshot(query.resource, &[]))); + } + MultiRaftActorCommand::ClientProposal { .. } => { + observed_proposals.fetch_add(1, Ordering::Relaxed); + } + _ => {} + }); + let controller = authenticated_controller("orders-service", node_id("node-1"), swim, raft); + + assert_eq!( + controller.describe_topic("t1".into()).await, + Err(ServerError::Unauthorized) + ); + assert_eq!( + controller.delete_topic("t1".into()).await, + Err(ServerError::Unauthorized) + ); + assert_eq!(proposal_count.load(Ordering::Relaxed), 0); + } + + #[tokio::test] + async fn cluster_scoped_apis_require_cluster_acl() { + let group = test_shard_group(); + let swim = swim_sender_with(move |cmd| { + if let SwimActorCommand::Query(QueryCommand::ResolveShardGroup { reply, .. }) = cmd { + let _ = reply.send(Some(group.clone())); + } + }); + let proposal_count = Arc::new(AtomicUsize::new(0)); + let observed_proposals = proposal_count.clone(); + let raft = raft_sender_with(move |cmd| match cmd { + MultiRaftActorCommand::GetAclSnapshot(query) => { + assert_eq!(query.resource, AclResource::Cluster); + let _ = query.reply.send(Some(acl_snapshot(query.resource, &[]))); + } + MultiRaftActorCommand::ClientProposal { .. } => { + observed_proposals.fetch_add(1, Ordering::Relaxed); + } + _ => {} + }); + let controller = authenticated_controller("orders-service", node_id("node-1"), swim, raft); + + assert_eq!( + controller + .handle_create_topic( + "t1".into(), + StoragePolicy { + retention_ms: Some(3_600_000), + replication_factor: 1, + partition_strategy: PartitionStrategy::AutoSplit, + }, + ) + .await, + Err(ServerError::Unauthorized) + ); + assert_eq!( + controller.list_hosted_topics().await, + Err(ServerError::Unauthorized) + ); + assert!(matches!( + controller.handle_admin(AdminRequest::DescribeCluster).await, + Err(ServerError::Unauthorized) + )); + assert_eq!(proposal_count.load(Ordering::Relaxed), 0); + } + + #[tokio::test] + async fn consumer_offset_requires_its_group_acl() { + let group = ShardGroup { + id: ShardGroupId(42), + replicas: Replicas::new(vec![node_id("self")]), + }; + let swim = swim_sender_with(move |cmd| { + if let SwimActorCommand::Query(QueryCommand::ResolveShardGroup { reply, .. }) = cmd { + let _ = reply.send(Some(group.clone())); + } + }); + let raft = raft_sender_with(|cmd| { + if let MultiRaftActorCommand::GetAclSnapshot(query) = cmd { + assert_eq!( + query.resource, + AclResource::ConsumerGroup(ConsumerGroupResource { + topic_id: TopicId(7), + group_id: "billing".to_string(), + }) + ); + let _ = query.reply.send(Some(acl_snapshot(query.resource, &[]))); + } + }); + let controller = authenticated_controller("orders-service", node_id("self"), swim, raft); + + let result = controller + .handle_fetch_consumer_offset(FetchConsumerOffsetRequest { + key: ConsumerOffsetKey { + topic_id: TopicId(7), + range_id: RangeId(0), + group_id: "billing".to_string(), + }, + generation: GenerationId(1), + }) + .await; + + assert_eq!(result, Err(ServerError::Unauthorized)); + } + + #[tokio::test] + async fn consumer_group_sync_is_authorized_before_proposal() { + let group = ShardGroup { + id: ShardGroupId(42), + replicas: Replicas::new(vec![node_id("self")]), + }; + let swim = swim_sender_with(move |cmd| { + if let SwimActorCommand::Query(QueryCommand::ResolveShardGroup { reply, .. }) = cmd { + let _ = reply.send(Some(group.clone())); + } + }); + let proposal_count = Arc::new(AtomicUsize::new(0)); + let observed_proposals = proposal_count.clone(); + let raft = raft_sender_with(move |cmd| match cmd { + MultiRaftActorCommand::GetTopicMetadata { reply, .. } => { + let _ = reply.send(Some(topic_meta("self"))); + } + MultiRaftActorCommand::GetAclSnapshot(query) => { + assert_eq!( + query.resource, + AclResource::ConsumerGroup(ConsumerGroupResource { + topic_id: TopicId(1), + group_id: "billing".to_string(), + }) + ); + let _ = query.reply.send(Some(acl_snapshot(query.resource, &[]))); + } + MultiRaftActorCommand::ClientProposal { .. } => { + observed_proposals.fetch_add(1, Ordering::Relaxed); + } + _ => {} + }); + let controller = authenticated_controller("orders-service", node_id("self"), swim, raft); + + let result = controller + .update_consumer_group_member(UpdateConsumerGroupMemberRequest { + topic_name: "t1".to_string(), + group_id: "billing".to_string(), + member_id: uuid::Uuid::new_v4(), + action: ConsumerGroupMemberAction::Heartbeat, + }) + .await; + + assert_eq!(result, Err(ServerError::Unauthorized)); + assert_eq!(proposal_count.load(Ordering::Relaxed), 0); + } + + #[tokio::test] + async fn certificate_client_fails_closed_when_acl_shard_is_remote() { + let group = ShardGroup { + id: ShardGroupId(42), + replicas: Replicas::new(vec![node_id("other")]), + }; + let swim = swim_sender_with(move |cmd| match cmd { + SwimActorCommand::Query(QueryCommand::ResolveShardGroup { reply, .. }) => { + let _ = reply.send(Some(group.clone())); + } + SwimActorCommand::Query(QueryCommand::ResolveAddress { reply, .. }) => { + let _ = reply.send(None); + } + _ => {} + }); + let controller = authenticated_controller( + "orders-service", + node_id("self"), + swim, + raft_sender_with(|_| panic!("remote ACL shard must not be queried locally")), + ); + + assert_eq!( + controller + .authorize_acl_resource(AclResource::TopicData(TopicId(7))) + .await, + Err(ServerError::Unauthorized) + ); + } + + #[test] + fn certificate_client_refreshes_acl_from_remote_shard_host() -> turmoil::Result { + let resource = AclResource::TopicData(TopicId(7)); + let snapshot = acl_snapshot(resource.clone(), &["orders-service"]); + let response_received = Arc::new(Notify::new()); + let mut sim = Builder::new() + .simulation_duration(Duration::from_secs(5)) + .build(); + + let owner_snapshot = snapshot.clone(); + let owner_resource = resource.clone(); + let owner_response_received = response_received.clone(); + sim.host("owner", move || { + let expected_snapshot = owner_snapshot.clone(); + let expected_resource = owner_resource.clone(); + let owner_completion = owner_response_received.clone(); + async move { + let listener = TcpListener::bind("0.0.0.0:9000").await?; + let (raft_tx, mut raft_rx) = MultiRaftActor::channel(8); + let (_transport_tx, transport_rx) = mpsc::channel(1); + let (swim_tx, _swim_rx) = SwimActor::channel(1); + let security = SecurityActor::spawn( + node_id("owner"), + swim_tx.clone(), + raft_tx.clone(), + NodeTransportSecurity::TrustedDevelopment, + ); + tokio::spawn(RaftTransportActor::run( + node_id("owner"), + listener, + raft_tx, + transport_rx, + swim_tx, + security, + )); + + let Some(MultiRaftActorCommand::GetAclSnapshot(query)) = raft_rx.recv().await + else { + panic!("expected remote ACL snapshot query"); + }; + assert_eq!(query.shard_group_id, ShardGroupId(42)); + assert_eq!(query.resource, expected_resource); + let _ = query.reply.send(Some(expected_snapshot)); + owner_completion.notified().await; + Ok(()) + } + }); + + let requester_resource = resource.clone(); + sim.host("requester", move || { + let requested_resource = requester_resource.clone(); + let requester_completion = response_received.clone(); + async move { + let owner_addr = turmoil::lookup("owner"); + let group = ShardGroup { + id: ShardGroupId(42), + replicas: Replicas::new(vec![node_id("owner")]), + }; + let owner = NodeAddress::test( + SocketAddr::new(owner_addr, 9000), + SocketAddr::new(owner_addr, 9001), + ); + let swim = swim_sender_with(move |cmd| match cmd { + SwimActorCommand::Query(QueryCommand::ResolveShardGroup { reply, .. }) => { + let _ = reply.send(Some(group.clone())); + } + SwimActorCommand::Query(QueryCommand::ResolveAddress { reply, .. }) => { + let _ = reply.send(Some(owner)); + } + _ => {} + }); + let controller = authenticated_controller( + "orders-service", + node_id("requester"), + swim, + raft_sender_with(|_| panic!("remote ACL refresh must not query local Raft")), + ); + + let result = controller.authorize_acl_resource(requested_resource).await; + requester_completion.notify_one(); + assert_eq!(result, Ok(())); + Ok(()) + } + }); + + sim.run() + } + /// Ring can't map the key yet (topology not converged) → retriable /// `ShardNotLocal` with no hint, not `TopicNotFound`. #[tokio::test] @@ -750,10 +1232,9 @@ mod tests { let _ = reply.send(None); } }); - let resp = - ClientController::new(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) - .dispatch(produce_req()) - .await; + let resp = trusted_controller(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) + .dispatch(produce_req()) + .await; assert!( matches!( resp, @@ -782,10 +1263,9 @@ mod tests { } _ => {} }); - let resp = - ClientController::new(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) - .dispatch(produce_req()) - .await; + let resp = trusted_controller(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) + .dispatch(produce_req()) + .await; let ClientResponse::Err(ServerError::ShardNotLocal { hint_node }) = resp else { panic!("expected ShardNotLocal, got {resp:?}"); }; @@ -817,7 +1297,7 @@ mod tests { let _ = reply.send(Some(topic_meta("leader"))); } }); - let resp = ClientController::new(me, swim, raft, dp_stub()) + let resp = trusted_controller(me, swim, raft, dp_stub()) .dispatch(produce_req()) .await; let ClientResponse::Err(ServerError::NotWriteLeader { @@ -847,7 +1327,7 @@ mod tests { let _ = reply.send(Some(topic_meta("self"))); } }); - let resp = ClientController::new(me, swim, raft, dp_acking()) + let resp = trusted_controller(me, swim, raft, dp_acking()) .dispatch(produce_req()) .await; let ClientResponse::Ok(ClientSuccess::Produced(entry_id)) = resp else { @@ -875,19 +1355,18 @@ mod tests { } _ => {} }); - let resp = - ClientController::new(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) - .dispatch(ClientRequest::ControlPlane( - ControlPlaneRequest::CreateTopic { - name: "t1".into(), - storage_policy: StoragePolicy { - retention_ms: Some(3_600_000), - replication_factor: 1, - partition_strategy: PartitionStrategy::AutoSplit, - }, + let resp = trusted_controller(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) + .dispatch(ClientRequest::ControlPlane( + ControlPlaneRequest::CreateTopic { + name: "t1".into(), + storage_policy: StoragePolicy { + retention_ms: Some(3_600_000), + replication_factor: 1, + partition_strategy: PartitionStrategy::AutoSplit, }, - )) - .await; + }, + )) + .await; let ClientResponse::Err(ServerError::TopicMetadataRedirect { owner: redirect_owner, }) = resp @@ -910,7 +1389,7 @@ mod tests { let _ = reply.send(Ok(())); } }); - let resp = ClientController::new(node_id("node-1"), swim, raft, dp_stub()) + let resp = trusted_controller(node_id("node-1"), swim, raft, dp_stub()) .dispatch(ClientRequest::ControlPlane( ControlPlaneRequest::CreateTopic { name: "t1".into(), @@ -935,19 +1414,18 @@ mod tests { let _ = reply.send(None); } }); - let resp = - ClientController::new(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) - .dispatch(ClientRequest::ControlPlane( - ControlPlaneRequest::CreateTopic { - name: "t1".into(), - storage_policy: StoragePolicy { - retention_ms: Some(3_600_000), - replication_factor: 1, - partition_strategy: PartitionStrategy::AutoSplit, - }, + let resp = trusted_controller(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) + .dispatch(ClientRequest::ControlPlane( + ControlPlaneRequest::CreateTopic { + name: "t1".into(), + storage_policy: StoragePolicy { + retention_ms: Some(3_600_000), + replication_factor: 1, + partition_strategy: PartitionStrategy::AutoSplit, }, - )) - .await; + }, + )) + .await; assert!( matches!(resp, ClientResponse::Err(ServerError::Internal(_))), "expected InternalError, got {resp:?}" @@ -961,12 +1439,16 @@ mod tests { let _ = reply.send(Some(test_shard_group())); } }); - let raft = raft_sender_with(|cmd| { - if let MultiRaftActorCommand::ClientProposal { reply, .. } = cmd { + let raft = raft_sender_with(|cmd| match cmd { + MultiRaftActorCommand::GetTopicMetadata { reply, .. } => { + let _ = reply.send(Some(topic_meta("node-1"))); + } + MultiRaftActorCommand::ClientProposal { reply, .. } => { let _ = reply.send(Ok(())); } + _ => {} }); - let resp = ClientController::new(node_id("node-1"), swim, raft, dp_stub()) + let resp = trusted_controller(node_id("node-1"), swim, raft, dp_stub()) .dispatch(ClientRequest::ControlPlane( ControlPlaneRequest::DeleteTopic { name: "t1".into() }, )) @@ -984,12 +1466,11 @@ mod tests { let _ = reply.send(Box::new(["alpha".into(), "beta".into()])); } }); - let resp = - ClientController::new(node_id("self"), swim_sender_with(|_| {}), raft, dp_stub()) - .dispatch(ClientRequest::ControlPlane( - ControlPlaneRequest::ListHostedTopics, - )) - .await; + let resp = trusted_controller(node_id("self"), swim_sender_with(|_| {}), raft, dp_stub()) + .dispatch(ClientRequest::ControlPlane( + ControlPlaneRequest::ListHostedTopics, + )) + .await; let ClientResponse::Ok(ClientSuccess::TopicList { topics }) = resp else { panic!("expected TopicList, got {resp:?}"); }; @@ -1022,14 +1503,13 @@ mod tests { _ => {} }) }; - let resp = - ClientController::new(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) - .dispatch(ClientRequest::ControlPlane( - ControlPlaneRequest::DescribeTopic { - name: "elsewhere".into(), - }, - )) - .await; + let resp = trusted_controller(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) + .dispatch(ClientRequest::ControlPlane( + ControlPlaneRequest::DescribeTopic { + name: "elsewhere".into(), + }, + )) + .await; let ClientResponse::Err(ServerError::TopicMetadataRedirect { owner: redirect_owner, }) = resp @@ -1060,7 +1540,7 @@ mod tests { let _ = reply.send(None); } }); - let resp = ClientController::new(me, swim, raft, dp_stub()) + let resp = trusted_controller(me, swim, raft, dp_stub()) .dispatch(ClientRequest::ControlPlane( ControlPlaneRequest::DescribeTopic { name: "missing".into(), @@ -1098,10 +1578,9 @@ mod tests { } }) }; - let resp = - ClientController::new(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) - .dispatch(ClientRequest::Admin(AdminRequest::DescribeCluster)) - .await; + let resp = trusted_controller(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) + .dispatch(ClientRequest::Admin(AdminRequest::DescribeCluster)) + .await; let ClientResponse::Ok(ClientSuccess::ClusterInfo { nodes: info }) = resp else { panic!("expected ClusterInfo, got {resp:?}"); }; @@ -1128,12 +1607,11 @@ mod tests { } }) }; - let resp = - ClientController::new(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) - .dispatch(ClientRequest::Admin(AdminRequest::GetShardInfo { - key: b"any".to_vec(), - })) - .await; + let resp = trusted_controller(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) + .dispatch(ClientRequest::Admin(AdminRequest::GetShardInfo { + key: b"any".to_vec(), + })) + .await; let ClientResponse::Ok(ClientSuccess::ShardInfo { detail: Some(d) }) = resp else { panic!("expected ShardInfo with detail, got {resp:?}"); }; @@ -1151,12 +1629,11 @@ mod tests { let _ = reply.send(None); } }); - let resp = - ClientController::new(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) - .dispatch(ClientRequest::Admin(AdminRequest::GetShardInfo { - key: b"x".to_vec(), - })) - .await; + let resp = trusted_controller(node_id("self"), swim, raft_sender_with(|_| {}), dp_stub()) + .dispatch(ClientRequest::Admin(AdminRequest::GetShardInfo { + key: b"x".to_vec(), + })) + .await; assert!( matches!( resp, @@ -1173,12 +1650,11 @@ mod tests { let _ = reply.send(Some(node_id("n1"))); } }); - let resp = - ClientController::new(node_id("self"), swim_sender_with(|_| {}), raft, dp_stub()) - .dispatch(ClientRequest::Admin(AdminRequest::GetShardLeader { - shard_group_id: ShardGroupId(42), - })) - .await; + let resp = trusted_controller(node_id("self"), swim_sender_with(|_| {}), raft, dp_stub()) + .dispatch(ClientRequest::Admin(AdminRequest::GetShardLeader { + shard_group_id: ShardGroupId(42), + })) + .await; let ClientResponse::Ok(ClientSuccess::ShardLeader { leader }) = resp else { panic!("expected ShardLeader, got {resp:?}"); }; @@ -1196,12 +1672,11 @@ mod tests { }])); } }); - let resp = - ClientController::new(node_id("self"), swim_sender_with(|_| {}), raft, dp_stub()) - .dispatch(ClientRequest::Admin( - AdminRequest::ListHostedTopicsWithStats, - )) - .await; + let resp = trusted_controller(node_id("self"), swim_sender_with(|_| {}), raft, dp_stub()) + .dispatch(ClientRequest::Admin( + AdminRequest::ListHostedTopicsWithStats, + )) + .await; let ClientResponse::Ok(ClientSuccess::TopicStats { topics }) = resp else { panic!("expected TopicStats, got {resp:?}"); }; diff --git a/src/connections/protocol/control_plane.rs b/src/connections/protocol/control_plane.rs index 9392a83a..5936035c 100644 --- a/src/connections/protocol/control_plane.rs +++ b/src/connections/protocol/control_plane.rs @@ -21,8 +21,9 @@ use borsh::{BorshDeserialize, BorshSerialize}; use std::collections::{HashMap, HashSet}; use crate::control_plane::metadata::{ - EntryId, OpenProducerSession, RangeId, RangeMeta, RangeState, SegmentId, SegmentMeta, - SegmentMetaState, SyncConsumerGroupRequest, TopicId, TopicMeta, TopicState, + EntryId, OpenProducerSession, ProducerSessionOwner, RangeId, RangeMeta, RangeState, SegmentId, + SegmentMeta, SegmentMetaState, TopicId, TopicMeta, TopicState, + UpdateConsumerGroupMemberRequest, }; #[derive(Debug, Clone, BorshSerialize, BorshDeserialize)] @@ -38,13 +39,13 @@ pub enum ControlPlaneRequest { DescribeTopic { name: String, }, - SyncConsumerGroup(SyncConsumerGroupRequest), + SyncConsumerGroup(UpdateConsumerGroupMemberRequest), OpenProducerSession(OpenProducerSessionRequest), } impl_from_variant!( ControlPlaneRequest, - SyncConsumerGroup(SyncConsumerGroupRequest), + SyncConsumerGroup(UpdateConsumerGroupMemberRequest), OpenProducerSession(OpenProducerSessionRequest) ); @@ -56,13 +57,14 @@ pub struct OpenProducerSessionRequest { } impl OpenProducerSessionRequest { - pub fn into_command(self) -> OpenProducerSession { + pub fn into_command(self, owner: ProducerSessionOwner) -> OpenProducerSession { const SESSION_TIMEOUT_MS: u64 = 60_000; let observed_at = crate::now_ms(); OpenProducerSession { - topic_name: self.topic_name, + topic_name: self.topic_name.into_boxed_str(), producer_id: self.producer_id, session_nonce: self.session_nonce, + owner, observed_at, session_timeout_ms: SESSION_TIMEOUT_MS, } @@ -70,7 +72,7 @@ impl OpenProducerSessionRequest { } #[derive(Debug, Clone, Copy, PartialEq, Eq, BorshSerialize, BorshDeserialize)] -pub enum ConsumerGroupSyncAction { +pub enum ConsumerGroupMemberAction { Heartbeat, Leave, } diff --git a/src/connections/protocol/error.rs b/src/connections/protocol/error.rs index 11c225fc..252fde36 100644 --- a/src/connections/protocol/error.rs +++ b/src/connections/protocol/error.rs @@ -29,6 +29,9 @@ pub enum ServerError { #[error("topic already exists")] AlreadyExists, + #[error("unauthorized")] + Unauthorized, + #[error("stale range")] StaleRange, @@ -81,6 +84,7 @@ impl From for ServerError TopicNotActive(_) | RangeNotFound | RangeNotActive => ServerError::StaleRange, SegmentNotFound | SegmentNotActive | SegmentNotSealed => ServerError::SegmentNotLocal, InvalidSplitPoint => ServerError::InvalidSplitPoint, + ProducerSessionOwnerMismatch => ServerError::Unauthorized, SplitNotAllowed(_) | RangesNotAdjacent => ServerError::Internal(err.to_string()), } } diff --git a/src/connections/protocol/mod.rs b/src/connections/protocol/mod.rs index 9e8b2c56..6fcdb198 100644 --- a/src/connections/protocol/mod.rs +++ b/src/connections/protocol/mod.rs @@ -30,7 +30,7 @@ pub use error::*; use borsh::{BorshDeserialize, BorshSerialize}; use crate::{ - control_plane::metadata::{EntryId, SyncConsumerGroupRequest}, + control_plane::metadata::{EntryId, UpdateConsumerGroupMemberRequest}, data_plane::{ auxiliary_states::consumer_offsets::state::ConsumerOffsetPosition, messages::query::RangeOffsets, @@ -126,4 +126,8 @@ impl_from_variant_via!( FetchConsumerOffsetRequest ); -impl_from_variant_via!(ClientRequest, ControlPlaneRequest, SyncConsumerGroupRequest); +impl_from_variant_via!( + ClientRequest, + ControlPlaneRequest, + UpdateConsumerGroupMemberRequest +); diff --git a/src/connections/reader.rs b/src/connections/reader.rs index 24d4dd79..8ecc4a6b 100644 --- a/src/connections/reader.rs +++ b/src/connections/reader.rs @@ -16,19 +16,19 @@ use crate::connections::{LEN_PREFIX_SIZE, REQUEST_ID_SIZE}; use std::io::ErrorKind; -use crate::net::OwnedReadHalf; +use crate::net::TransportReadHalf; use bytes::{Buf, BytesMut}; use tokio::io::AsyncReadExt; pub struct ClientStreamReader { - pub(crate) stream: OwnedReadHalf, + pub(crate) stream: TransportReadHalf, buffer: BytesMut, } impl ClientStreamReader { - pub fn new(stream: OwnedReadHalf) -> Self { + pub fn new(stream: impl Into) -> Self { Self { - stream, + stream: stream.into(), buffer: BytesMut::with_capacity(1024), } } diff --git a/src/connections/writer.rs b/src/connections/writer.rs index d9ec3be7..1e28b339 100644 --- a/src/connections/writer.rs +++ b/src/connections/writer.rs @@ -2,17 +2,19 @@ use tokio::io::AsyncWriteExt; use crate::{ connections::{REQUEST_ID_SIZE, protocol::ClientResponse}, - net::OwnedWriteHalf, + net::TransportWriteHalf, }; use tokio::sync::mpsc; pub(crate) struct ClientRawWriter { - stream: OwnedWriteHalf, + stream: TransportWriteHalf, } impl ClientRawWriter { - pub fn new(write_half: OwnedWriteHalf) -> Self { - Self { stream: write_half } + pub fn new(write_half: impl Into) -> Self { + Self { + stream: write_half.into(), + } } pub async fn write( diff --git a/src/control_plane/consensus/actor.rs b/src/control_plane/consensus/actor.rs index 7679972f..e7c87426 100644 --- a/src/control_plane/consensus/actor.rs +++ b/src/control_plane/consensus/actor.rs @@ -8,14 +8,16 @@ use crate::control_plane::NodeId; use crate::control_plane::consensus::messages::*; use crate::control_plane::consensus::multi_raft::MultiRaft; use crate::control_plane::consensus::raft::errors::ProposalError; +use crate::control_plane::consensus::raft::states::security::{AclRecord, AdmissionRecord}; use crate::control_plane::consensus::raft::storage::RaftStorage; use crate::control_plane::membership::actor::SwimSender; use crate::control_plane::membership::{ShardGroupId, SwimCommand, TopologyReader}; use crate::control_plane::metadata::{ - ConsumerGroupAssignment, MetadataCommand, TopicMeta, TopicStats, + AclResource, ConsumerGroupAssignment, MetadataCommand, TopicMeta, TopicStats, }; use crate::data_plane::transport::command::DataTransportCommand; use crate::schedulers::ticker_message::{SchedulerSender, TickerCommand}; +use crate::security::CertificatePrincipal; use tokio::sync::mpsc; use uuid::Uuid; @@ -278,6 +280,45 @@ impl MutlRaftSender { recv.await.unwrap_or_default() } + /// Reads the committed ACL snapshot only when this node hosts the selected + /// metadata shard. A missing ACL is an empty, cacheable denial; `None` + /// means the shard is no longer local or the actor stopped. + pub(crate) async fn get_acl_snapshot( + &self, + shard_group_id: ShardGroupId, + resource: AclResource, + ) -> Option { + let (reply, recv) = tokio::sync::oneshot::channel(); + let _ = self + .send(GetAclSnapshot { + shard_group_id, + resource, + reply, + }) + .await; + recv.await.ok().flatten() + } + + /// Reads the current admission record only when this node hosts the + /// selected metadata shard. Absence is an authoritative `None`; routing or + /// actor failures remain observable errors and must not be cached. + pub(crate) async fn get_admission( + &self, + shard_group_id: ShardGroupId, + node_certificate_principal: CertificatePrincipal, + ) -> Result, ServerError> { + let (reply, recv) = tokio::sync::oneshot::channel(); + self.send(GetAdmission { + shard_group_id, + node_certificate_principal, + reply, + }) + .await + .map_err(|error| ServerError::Internal(error.to_string()))?; + recv.await + .map_err(|error| ServerError::Internal(error.to_string()))? + } + pub(crate) async fn send( &self, cmd: impl Into, diff --git a/src/control_plane/consensus/messages/actor.rs b/src/control_plane/consensus/messages/actor.rs index d123a6de..df40f623 100644 --- a/src/control_plane/consensus/messages/actor.rs +++ b/src/control_plane/consensus/messages/actor.rs @@ -1,13 +1,16 @@ use tokio::sync::oneshot; use uuid::Uuid; +use crate::connections::protocol::ServerError; use crate::control_plane::NodeId; use crate::control_plane::consensus::raft::errors::ProposalError; +use crate::control_plane::consensus::raft::states::security::{AclRecord, AdmissionRecord}; use crate::control_plane::membership::ShardGroupId; -use crate::control_plane::metadata::{ConsumerGroupAssignment, TopicMeta, TopicStats}; +use crate::control_plane::metadata::{AclResource, ConsumerGroupAssignment, TopicMeta, TopicStats}; use crate::data_plane::messages::command::{ DurableSegmentEndReported, SegmentCaughtUp, SegmentPlaced, }; +use crate::security::CertificatePrincipal; use super::command::{ EnsureGroup, InboundRaftRpc, MetadataProposal, ProposeSegmentRoll, RaftProtocolMessage, @@ -52,6 +55,8 @@ pub enum MultiRaftActorCommand { topic_name: String, reply: oneshot::Sender>, }, + GetAclSnapshot(GetAclSnapshot), + GetAdmission(GetAdmission), GetConsumerGroupAssignment(GetConsumerGroupAssignment), /// Data-plane request forwarded to the metadata coordinator for proposal. ProposeSegmentRoll(ProposeSegmentRoll), @@ -74,6 +79,23 @@ pub struct GetConsumerGroupAssignment { pub(crate) reply: oneshot::Sender>, } +/// Returns this node's committed ACL state for a shard it hosts. +/// +/// The caller has already routed the resource to this shard. A missing ACL is +/// returned as an empty record so it can be cached as a bounded denial. +pub struct GetAclSnapshot { + pub(crate) shard_group_id: ShardGroupId, + pub(crate) resource: AclResource, + pub(crate) reply: oneshot::Sender>, +} + +/// Returns one admission record from a metadata shard hosted by this node. +pub struct GetAdmission { + pub(crate) shard_group_id: ShardGroupId, + pub(crate) node_certificate_principal: CertificatePrincipal, + pub(crate) reply: oneshot::Sender, ServerError>>, +} + impl From for MultiRaftActorCommand { fn from(cmd: RaftProtocolMessage) -> Self { MultiRaftActorCommand::ProtocolMessage(cmd) @@ -94,22 +116,33 @@ impl_from_variant_via!( RemoveGroup, ); -impl_from_variant!(MultiRaftActorCommand, GetConsumerGroupAssignment); +impl_from_variant!( + MultiRaftActorCommand, + GetAclSnapshot, + GetAdmission, + GetConsumerGroupAssignment, +); + +/// A synchronous actor result held until the end-of-batch reply flush. +pub(crate) struct DeferredResponse { + pub(crate) reply: oneshot::Sender, + pub(crate) value: T, +} -pub(crate) struct DeferredConsumerGroupAssignment { - pub(crate) reply: oneshot::Sender>, - pub(crate) value: Option, +impl DeferredResponse { + pub(crate) fn send(self) { + let _ = self.reply.send(self.value); + } } pub(crate) enum DeferredReply { - GetLeader(oneshot::Sender>, Option), - GetPeers(oneshot::Sender>, Box<[NodeId]>), - Propose( - oneshot::Sender>, - Result<(), ProposalError>, - ), - GetTopics(oneshot::Sender>, Box<[String]>), - GetTopicStats(oneshot::Sender>, Box<[TopicStats]>), - GetTopicMetadata(oneshot::Sender>, Box>), - GetConsumerGroupAssignment(DeferredConsumerGroupAssignment), + GetLeader(DeferredResponse>), + GetPeers(DeferredResponse>), + Propose(DeferredResponse>), + GetTopics(DeferredResponse>), + GetTopicStats(DeferredResponse>), + GetTopicMetadata(DeferredResponse>), + GetAclSnapshot(DeferredResponse>), + GetAdmission(DeferredResponse, ServerError>>), + GetConsumerGroupAssignment(DeferredResponse>), } diff --git a/src/control_plane/consensus/messages/command.rs b/src/control_plane/consensus/messages/command.rs index dc0c2f02..d68aa117 100644 --- a/src/control_plane/consensus/messages/command.rs +++ b/src/control_plane/consensus/messages/command.rs @@ -8,7 +8,7 @@ use crate::{impl_from_variant, impl_new_struct_wrapper}; pub struct InboundRaftRpc { pub shard_group_id: ShardGroupId, - pub from: NodeId, + pub peer_id: NodeId, pub rpc: RaftRpc, } diff --git a/src/control_plane/consensus/messages/rpc.rs b/src/control_plane/consensus/messages/rpc.rs index eff5194b..0eb7a93f 100644 --- a/src/control_plane/consensus/messages/rpc.rs +++ b/src/control_plane/consensus/messages/rpc.rs @@ -102,6 +102,6 @@ impl OutboundRaftPacket { #[derive(Debug, Clone, BorshSerialize, BorshDeserialize)] pub struct WireRaftMessage { pub shard_group_id: ShardGroupId, - pub sender: NodeId, + pub peer_id: NodeId, pub rpc: RaftRpc, } diff --git a/src/control_plane/consensus/multi_raft.rs b/src/control_plane/consensus/multi_raft.rs index c9a91a5b..f63dcb3b 100644 --- a/src/control_plane/consensus/multi_raft.rs +++ b/src/control_plane/consensus/multi_raft.rs @@ -1,27 +1,31 @@ +use crate::client::ServerError; use crate::control_plane::NodeId; use crate::control_plane::consensus::boundary_recovery::{ BoundaryRecoveryAction, SegmentBoundaryRecovery, }; use crate::control_plane::consensus::messages::{ - DeferredConsumerGroupAssignment, DeferredReply, InboundRaftRpc, LogMutation, MetadataProposal, + DeferredReply, DeferredResponse, InboundRaftRpc, LogMutation, MetadataProposal, MultiRaftActorCommand, ProposeSegmentRoll, RaftEvent, RaftProtocolMessage, RaftTimeoutCallback, }; use crate::control_plane::consensus::raft::errors::ProposalError; use crate::control_plane::consensus::raft::state::{Raft, TimerSeqs}; use crate::control_plane::consensus::raft::states::consensus::LeaderlessSegments; +use crate::control_plane::consensus::raft::states::security::{AclRecord, AdmissionRecord}; use crate::control_plane::consensus::raft::storage::RaftStorage; use crate::control_plane::consensus::raft::{compute_replacement_replica_set, now_ms}; use crate::control_plane::membership::{ShardGroup, ShardGroupId, TopologyReader}; use crate::control_plane::metadata::command::RollSegment; use crate::control_plane::metadata::event::MetadataEvent; use crate::control_plane::metadata::{ - ConsumerGroupAssignment, EntryId, SegmentRollIntent, TopicId, TopicMeta, TopicStats, + AclResource, ConsumerGroupAssignment, EntryId, SegmentRollIntent, TopicId, TopicMeta, + TopicStats, }; use crate::data_plane::SegmentKey; use crate::data_plane::messages::command::{ DurableSegmentEndReported, RequestDurableSegmentEnd, SegmentCaughtUp, SegmentPlaced, }; use crate::data_plane::transport::command::DataTransportCommand; +use crate::security::CertificatePrincipal; use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; use std::hash::{Hash, Hasher}; use tokio::sync::oneshot; @@ -237,11 +241,19 @@ impl MultiRaft { } MultiRaftActorCommand::GetLeader { group_id, reply } => { let result = self.get_leader(group_id); - self.deferred.push(DeferredReply::GetLeader(reply, result)); + self.deferred + .push(DeferredReply::GetLeader(DeferredResponse { + reply, + value: result, + })); } MultiRaftActorCommand::GetPeers { group_id, reply } => { let result = self.get_peers(group_id); - self.deferred.push(DeferredReply::GetPeers(reply, result)); + self.deferred + .push(DeferredReply::GetPeers(DeferredResponse { + reply, + value: result, + })); } MultiRaftActorCommand::ClientProposal { propose, reply } => { self.propose(propose, reply); @@ -249,17 +261,43 @@ impl MultiRaft { MultiRaftActorCommand::GetTopics { reply } => { let topics = self.get_topics(); - self.deferred.push(DeferredReply::GetTopics(reply, topics)); + self.deferred + .push(DeferredReply::GetTopics(DeferredResponse { + reply, + value: topics, + })); } MultiRaftActorCommand::GetTopicStats { reply } => { let stats = self.get_topic_stats(); self.deferred - .push(DeferredReply::GetTopicStats(reply, stats)); + .push(DeferredReply::GetTopicStats(DeferredResponse { + reply, + value: stats, + })); } MultiRaftActorCommand::GetTopicMetadata { topic_name, reply } => { let meta = self.get_topic_metadata(&topic_name); self.deferred - .push(DeferredReply::GetTopicMetadata(reply, Box::new(meta))); + .push(DeferredReply::GetTopicMetadata(DeferredResponse { + reply, + value: meta, + })); + } + MultiRaftActorCommand::GetAclSnapshot(query) => { + let value = self.acl_snapshot(query.shard_group_id, &query.resource); + self.deferred + .push(DeferredReply::GetAclSnapshot(DeferredResponse { + reply: query.reply, + value, + })); + } + MultiRaftActorCommand::GetAdmission(query) => { + let value = self.admission(query.shard_group_id, &query.node_certificate_principal); + self.deferred + .push(DeferredReply::GetAdmission(DeferredResponse { + reply: query.reply, + value, + })); } MultiRaftActorCommand::GetConsumerGroupAssignment(query) => { let value = self.get_consumer_group_assignment( @@ -269,7 +307,7 @@ impl MultiRaft { ); self.deferred .push(DeferredReply::GetConsumerGroupAssignment( - DeferredConsumerGroupAssignment { + DeferredResponse { reply: query.reply, value, }, @@ -293,27 +331,15 @@ impl MultiRaft { pub(crate) fn fire_deferred(&mut self) { for reply in self.deferred.drain(..) { match reply { - DeferredReply::GetLeader(sender, v) => { - let _ = sender.send(v); - } - DeferredReply::GetPeers(sender, v) => { - let _ = sender.send(v); - } - DeferredReply::Propose(sender, v) => { - let _ = sender.send(v); - } - DeferredReply::GetTopics(sender, v) => { - let _ = sender.send(v); - } - DeferredReply::GetTopicStats(sender, v) => { - let _ = sender.send(v); - } - DeferredReply::GetTopicMetadata(sender, v) => { - let _ = sender.send(*v); - } - DeferredReply::GetConsumerGroupAssignment(deferred) => { - let _ = deferred.reply.send(deferred.value); - } + DeferredReply::GetLeader(deferred) => deferred.send(), + DeferredReply::GetPeers(deferred) => deferred.send(), + DeferredReply::Propose(deferred) => deferred.send(), + DeferredReply::GetTopics(deferred) => deferred.send(), + DeferredReply::GetTopicStats(deferred) => deferred.send(), + DeferredReply::GetTopicMetadata(deferred) => deferred.send(), + DeferredReply::GetAclSnapshot(deferred) => deferred.send(), + DeferredReply::GetAdmission(deferred) => deferred.send(), + DeferredReply::GetConsumerGroupAssignment(deferred) => deferred.send(), } } } @@ -407,10 +433,10 @@ impl MultiRaft { tracing::info!("[{}] Removed Raft group {:?}", self.node_id, group_id); } - #[tracing::instrument(level = "trace", skip_all, fields(group = cmd.shard_group_id.0, from = %cmd.from))] + #[tracing::instrument(level = "trace", skip_all, fields(group = cmd.shard_group_id.0, from = %cmd.peer_id))] fn handle_rpc(&mut self, cmd: InboundRaftRpc) { if let Some(raft) = self.groups.get_mut(&cmd.shard_group_id) { - raft.handle_rpc(cmd.from, cmd.rpc); + raft.handle_rpc(cmd.peer_id, cmd.rpc); self.dirty.insert(cmd.shard_group_id); } } @@ -478,6 +504,28 @@ impl MultiRaft { .find_map(|raft| raft.get_topic_by_name(name).cloned()) } + fn acl_snapshot( + &self, + shard_group_id: ShardGroupId, + resource: &AclResource, + ) -> Option { + self.groups + .get(&shard_group_id) + .map(|raft| raft.acl_snapshot(resource)) + } + + fn admission( + &self, + shard_group_id: ShardGroupId, + node_certificate_principal: &CertificatePrincipal, + ) -> Result, ServerError> { + let raft = self + .groups + .get(&shard_group_id) + .ok_or(ServerError::ShardNotLocal { hint_node: None })?; + Ok(raft.admission(node_certificate_principal)) + } + fn get_consumer_group_assignment( &self, topic_name: &str, @@ -747,7 +795,10 @@ impl MultiRaft { self.pending_proposes.insert((gid, index), reply); } Err(e) => { - self.deferred.push(DeferredReply::Propose(reply, Err(e))); + self.deferred.push(DeferredReply::Propose(DeferredResponse { + reply, + value: Err(e), + })); } } } @@ -1025,7 +1076,7 @@ mod tests { } fn new_store(node_id: NodeId, storage: Box) -> MultiRaft { - use crate::control_plane::membership::{Topology, TopologyConfig, topology_channel}; + use crate::control_plane::membership::{Topology, TopologyConfig}; // Tests just need a valid topology reader; empty topology is fine — // these tests don't exercise reconciliation or ring picks. let topology = Topology::new( @@ -1035,7 +1086,7 @@ mod tests { replication_factor: 1, }, ); - let (_pub_handle, reader) = topology_channel(topology); + let (_pub_handle, reader) = topology.channel(); MultiRaft::new(node_id, 0, storage, reader, 10000) } @@ -1348,7 +1399,7 @@ mod tests { // n2 is acting as leader at term 1. Send two entries to n1 (follower). store.handle_consensus(InboundRaftRpc { shard_group_id: TEST_GROUP_ID, - from: n2.clone(), + peer_id: n2.clone(), rpc: RaftRpc::AppendEntries(AppendEntries { term: 1, leader_id: n2.clone(), @@ -1383,7 +1434,7 @@ mod tests { // Raft truncates from index 1 and replaces with the new entry. store.handle_consensus(InboundRaftRpc { shard_group_id: TEST_GROUP_ID, - from: n2.clone(), + peer_id: n2.clone(), rpc: RaftRpc::AppendEntries(AppendEntries { term: 2, leader_id: n2.clone(), @@ -1595,7 +1646,7 @@ mod tests { storage: Box, all_nodes: &[NodeId], ) -> MultiRaft { - use crate::control_plane::membership::{Topology, TopologyConfig, topology_channel}; + use crate::control_plane::membership::{Topology, TopologyConfig}; let topology = Topology::new( all_nodes.iter().cloned(), TopologyConfig { @@ -1603,7 +1654,7 @@ mod tests { replication_factor: 3, }, ); - let (_pub_handle, reader) = topology_channel(topology); + let (_pub_handle, reader) = topology.channel(); MultiRaft::new(node_id, 0, storage, reader, 10000) } @@ -1618,7 +1669,7 @@ mod tests { MultiRaft, std::sync::Arc>, ) { - use crate::control_plane::membership::{Topology, TopologyConfig, topology_channel}; + use crate::control_plane::membership::{Topology, TopologyConfig}; let topology = Topology::new( all_nodes.iter().cloned(), TopologyConfig { @@ -1626,7 +1677,7 @@ mod tests { replication_factor: 3, }, ); - let (pub_handle, reader) = topology_channel(topology); + let (pub_handle, reader) = topology.channel(); ( MultiRaft::new(node_id, 0, storage, reader, 10000), pub_handle, @@ -1849,7 +1900,7 @@ mod tests { let log = store.storage.load_state(gid.0).log; assert!( !log.iter() - .any(|e| e.command == RaftCommand::AddPeer(node("n4"))), + .any(|e| e.command == RaftCommand::EnsurePeer(node("n4"))), "the learner must not be added straight to the quorum (no immediate AddPeer, log: {:?})", log.iter().map(|e| &e.command).collect::>() ); @@ -1926,7 +1977,7 @@ mod tests { ); assert!( !log.iter() - .any(|e| e.command == RaftCommand::AddPeer(node("n9"))), + .any(|e| e.command == RaftCommand::EnsurePeer(node("n9"))), "the eviction must not be paired with an AddPeer" ); } @@ -2122,7 +2173,7 @@ mod tests { // recovery roll, and the leader-gated ring-check won't fire to prune it. store.handle_consensus(InboundRaftRpc { shard_group_id: TEST_GROUP_ID, - from: peer.clone(), + peer_id: peer.clone(), rpc: RaftRpc::AppendEntries(AppendEntries { term: 99, leader_id: peer, diff --git a/src/control_plane/consensus/raft/command.rs b/src/control_plane/consensus/raft/command.rs index 6d67e313..ebb2c1bd 100644 --- a/src/control_plane/consensus/raft/command.rs +++ b/src/control_plane/consensus/raft/command.rs @@ -4,7 +4,7 @@ use crate::control_plane::NodeId; use crate::control_plane::metadata::ReassignSegment; use crate::control_plane::metadata::command::{ CreateTopic, DeleteSegments, DeleteTopic, ExpireProducerSessions, MergeRange, MetadataCommand, - OpenProducerSession, RollSegment, SplitRange, SyncConsumerGroup, + OpenProducerSession, RollSegment, SplitRange, UpdateConsumerGroupMember, }; use crate::{impl_from_variant, impl_from_variant_via}; @@ -16,7 +16,7 @@ use crate::{impl_from_variant, impl_from_variant_via}; pub enum RaftCommand { Noop, Metadata(MetadataCommand), - AddPeer(NodeId), + EnsurePeer(NodeId), RemovePeer(NodeId), } @@ -31,7 +31,7 @@ impl_from_variant_via!( DeleteTopic, ReassignSegment, DeleteSegments, - SyncConsumerGroup, + UpdateConsumerGroupMember, OpenProducerSession, ExpireProducerSessions ); diff --git a/src/control_plane/consensus/raft/state.rs b/src/control_plane/consensus/raft/state.rs index 61420a22..ecf2aa04 100644 --- a/src/control_plane/consensus/raft/state.rs +++ b/src/control_plane/consensus/raft/state.rs @@ -8,6 +8,7 @@ use crate::control_plane::consensus::raft::states::consensus::{ ConsensusState, PeerState, Role, SNAPSHOT_CHUNK_BYTES, SnapshotInstallOutcome, }; use crate::control_plane::consensus::raft::states::metadata_state::MetadataState; +use crate::control_plane::consensus::raft::states::security::{AclRecord, AdmissionRecord}; use crate::control_plane::consensus::raft::storage::{ RaftPersistentState, RaftSnapshot, SnapshotData, }; @@ -16,14 +17,15 @@ use crate::control_plane::membership::{ShardGroupId, TopologyReader}; use crate::control_plane::metadata::command::{DeleteSegments, ExpireProducerSessions}; use crate::control_plane::metadata::event::MetadataEvent; use crate::control_plane::metadata::{ - ConsumerGroupAssignment, ConsumerMemberId, MetadataCommand, ReassignSegment, RollSegment, - SegmentRollIntent, TopicId, TopicMeta, TopicStats, + AclResource, ConsumerGroupAssignment, ConsumerMemberId, MetadataCommand, ReassignSegment, + RollSegment, SegmentRollIntent, TopicId, TopicMeta, TopicStats, }; use crate::control_plane::{NodeId, Replicas}; use crate::data_plane::SegmentKey; use crate::data_plane::messages::command::{PlaceSegment, SegmentCaughtUp, SegmentPlaced}; use crate::data_plane::transport::command::DataTransportCommand; use crate::schedulers::ticker_message::TimerCommand; +use crate::security::CertificatePrincipal; #[cfg(any(test, debug_assertions))] use crate::test_traits::TAssertInvariant; use std::collections::{BTreeSet, HashSet}; @@ -161,6 +163,17 @@ impl Raft { self.metadata.get_topic_by_name(name) } + pub(crate) fn acl_snapshot(&self, resource: &AclResource) -> AclRecord { + self.metadata.acl_snapshot(resource) + } + + pub(crate) fn admission( + &self, + node_certificate_principal: &CertificatePrincipal, + ) -> Option { + self.metadata.admission(node_certificate_principal) + } + pub(crate) fn get_consumer_group_assignment( &self, topic_name: &str, @@ -211,6 +224,15 @@ impl Raft { if *member == self.node_id || !live_set.contains(member) { continue; } + if self.peers.contains(member) { + // Bootstrap group creation can observe different ring snapshots + // on different nodes. Re-applying an existing voter is a "no-op" + // here, but heals followers whose initial voter set omitted it. + changed |= self + .propose(RaftCommand::EnsurePeer(member.clone())) + .is_ok(); + continue; + } // Stage the ring member as a non-voting learner; it's promoted to a // voter once caught up. Never added straight to the quorum — an // un-participating ring member would otherwise freeze commits. @@ -323,7 +345,7 @@ impl Raft { self.consensus.uncommited_log_range().any(|i| { matches!( self.consensus.log_entry(i).map(|e| &e.command), - Some(RaftCommand::AddPeer(_) | RaftCommand::RemovePeer(_)) + Some(RaftCommand::EnsurePeer(_) | RaftCommand::RemovePeer(_)) ) }) } @@ -1162,7 +1184,7 @@ impl Raft { return; } if self.consensus.is_learner_ready_for_promotion(node) { - let _ = self.propose(RaftCommand::AddPeer(node.clone())); + let _ = self.propose(RaftCommand::EnsurePeer(node.clone())); } } @@ -1525,7 +1547,7 @@ impl Raft { match entry.command { RaftCommand::Noop => {} RaftCommand::Metadata(cmd) => self.apply_metadata_entry(cmd, entry.index), - RaftCommand::AddPeer(node_id) => self.apply_add_peer(node_id), + RaftCommand::EnsurePeer(node_id) => self.apply_ensure_peer(node_id), RaftCommand::RemovePeer(node_id) => self.apply_remove_peer(node_id), } } @@ -1569,7 +1591,7 @@ impl Raft { /// Apply-only helper. Invoked from `apply_committed_entries()` when an /// `AddPeer` log entry commits. Never call directly — the peer set is part /// of the replicated state machine and must only mutate through the log. - fn apply_add_peer(&mut self, node_id: NodeId) { + fn apply_ensure_peer(&mut self, node_id: NodeId) { if node_id == self.node_id { return; } @@ -1898,7 +1920,7 @@ impl crate::test_traits::TAssertInvariant for Raft { // Invariant (partial): self is never in peers. The peer set is otherwise // mutated only via apply of committed AddPeer/RemovePeer entries — the - // discipline itself is enforced by keeping `apply_add_peer`/`apply_remove_peer` + // discipline itself is enforced by keeping `apply_ensure_peer`/`apply_remove_peer` // as the sole callers of `peers.insert`/`peers.remove` (callers checked at // compile time by their private visibility). assert!( @@ -3073,7 +3095,7 @@ mod tests { // Direct mutation is gone — the peer set is part of the replicated state. #[test] - fn add_peer_log_entry_inserts_into_peers_on_apply() { + fn ensure_peer_log_entry_inserts_into_peers_on_apply() { let mut raft = single_node_raft(); raft.handle_timeout(RaftTimeoutCallback::ElectionTimeout { shard_group_id: TEST_SHARD, @@ -3083,7 +3105,8 @@ mod tests { assert!(raft.is_leader()); assert_eq!(raft.peers_count(), 0); - raft.propose(RaftCommand::AddPeer(node("node-2"))).unwrap(); + raft.propose(RaftCommand::EnsurePeer(node("node-2"))) + .unwrap(); raft.simulate_flush(); assert!(raft.has_peer(&node("node-2"))); @@ -3091,7 +3114,7 @@ mod tests { } #[test] - fn add_peer_log_entry_skips_self() { + fn ensure_peer_log_entry_skips_self() { let mut raft = single_node_raft(); raft.handle_timeout(RaftTimeoutCallback::ElectionTimeout { shard_group_id: TEST_SHARD, @@ -3099,7 +3122,8 @@ mod tests { }); drain(&mut raft); - raft.propose(RaftCommand::AddPeer(node("node-1"))).unwrap(); + raft.propose(RaftCommand::EnsurePeer(node("node-1"))) + .unwrap(); raft.simulate_flush(); assert!(!raft.has_peer(&node("node-1"))); @@ -3107,7 +3131,7 @@ mod tests { } #[test] - fn add_peer_log_entry_leader_initializes_peer_state() { + fn ensure_peer_log_entry_leader_initializes_peer_state() { let mut raft = single_node_raft(); raft.handle_timeout(RaftTimeoutCallback::ElectionTimeout { shard_group_id: TEST_SHARD, @@ -3116,7 +3140,8 @@ mod tests { drain(&mut raft); assert!(raft.is_leader()); - raft.propose(RaftCommand::AddPeer(node("node-2"))).unwrap(); + raft.propose(RaftCommand::EnsurePeer(node("node-2"))) + .unwrap(); raft.simulate_flush(); drain(&mut raft); @@ -3597,9 +3622,7 @@ mod tests { // verify the proposals appear in the log. // ------------------------------------------------------------------- - use crate::control_plane::membership::{ - Topology, TopologyConfig, TopologyReader, topology_channel, - }; + use crate::control_plane::membership::{Topology, TopologyConfig, TopologyReader}; /// Build a `TopologyReader` seeded with `nodes` as live members. The /// publisher half is dropped on return — the reader's own Arc keeps the @@ -3612,7 +3635,7 @@ mod tests { replication_factor: 3, }, ); - let (_pub_handle, reader) = topology_channel(topology); + let (_pub_handle, reader) = topology.channel(); reader } @@ -3832,6 +3855,28 @@ mod tests { (raft, reader, members) } + #[test] + fn takeover_reasserts_bootstrap_voters_through_the_log() { + let (mut raft, reader, members) = ring_raft_with_stale(&[]); + let expected: HashSet = members + .iter() + .filter(|member| **member != node("node-1")) + .cloned() + .collect(); + + assert!(raft.reconcile(&reader, Some(members.0))); + + let asserted: HashSet = raft + .consensus + .uncommited_log_range() + .filter_map(|index| match &raft.consensus.log_entry(index)?.command { + RaftCommand::EnsurePeer(node_id) => Some(node_id.clone()), + RaftCommand::Noop | RaftCommand::RemovePeer(_) | RaftCommand::Metadata(_) => None, + }) + .collect(); + assert_eq!(asserted, expected); + } + /// Simulate `peer` confirming replication up to the leader's last index. fn ack_to_last(raft: &mut Raft, peer: &NodeId) { let term = raft.current_term(); @@ -3897,7 +3942,8 @@ mod tests { // A membership entry is in flight: the window is complete, but the // one-config-change-at-a-time gate must park the eviction. - raft.propose(RaftCommand::AddPeer(node("node-7"))).unwrap(); + raft.propose(RaftCommand::EnsurePeer(node("node-7"))) + .unwrap(); assert!( raft.reconcile_stale_live_peers(&topology, &live).is_err(), "uncommitted AddPeer must park the eviction" @@ -3914,7 +3960,7 @@ mod tests { assert_eq!( proposals_after_become_leader(&raft), vec![ - RaftCommand::AddPeer(node("node-7")), + RaftCommand::EnsurePeer(node("node-7")), RaftCommand::RemovePeer(node("node-9")), ], ); diff --git a/src/control_plane/consensus/raft/states/metadata_state.rs b/src/control_plane/consensus/raft/states/metadata_state.rs index ec4b19fa..252d1ee7 100644 --- a/src/control_plane/consensus/raft/states/metadata_state.rs +++ b/src/control_plane/consensus/raft/states/metadata_state.rs @@ -1,3 +1,6 @@ +use crate::control_plane::consensus::raft::states::security::{ + AclRecord, AdmissionRecord, SecurityState, +}; use crate::control_plane::metadata::SegmentMeta; use crate::control_plane::metadata::command::*; use crate::control_plane::metadata::event::*; @@ -6,9 +9,13 @@ use crate::control_plane::NodeId; use crate::control_plane::Replicas; use crate::control_plane::membership::ShardGroupId; use crate::control_plane::metadata::ConsumerGroupAssignment; + use crate::control_plane::metadata::topic::{TopicMeta, TopicState, TopicStats}; -use crate::control_plane::metadata::{EntryId, RangeId, SegmentId, TopicId, error::MetadataError}; +use crate::control_plane::metadata::{ + AclResource, EntryId, RangeId, SegmentId, TopicId, error::MetadataError, +}; use crate::data_plane::SegmentKey; +use crate::security::CertificatePrincipal; #[cfg(any(test, debug_assertions))] use crate::test_traits::TAssertInvariant; use MetadataError::*; @@ -19,11 +26,13 @@ use uuid::Uuid; #[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] pub(crate) struct MetadataStateSnapshot { topics: HashMap, + security: Box, next_topic_id: u64, } pub struct MetadataState { pub(crate) topics: HashMap, + security: SecurityState, pub(crate) last_applied_index: u64, topic_name_index: HashMap, next_topic_id: u64, @@ -35,6 +44,7 @@ impl MetadataState { pub(crate) fn new(shard_group_id: ShardGroupId) -> Self { MetadataState { topics: HashMap::new(), + security: SecurityState::default(), last_applied_index: 0, topic_name_index: HashMap::new(), next_topic_id: shard_group_id.0 << 32, @@ -46,6 +56,7 @@ impl MetadataState { pub(crate) fn snapshot(&self) -> MetadataStateSnapshot { MetadataStateSnapshot { topics: self.topics.clone(), + security: Box::new(self.security.clone()), next_topic_id: self.next_topic_id, } } @@ -58,6 +69,7 @@ impl MetadataState { .collect(); Self { topics, + security: *snapshot.security, last_applied_index, topic_name_index, next_topic_id: snapshot.next_topic_id, @@ -82,6 +94,17 @@ impl MetadataState { range.segments.get(&key.segment_id) } + pub(crate) fn acl_snapshot(&self, resource: &AclResource) -> AclRecord { + self.security.acl_snapshot(resource) + } + + pub(crate) fn admission( + &self, + node_certificate_principal: &CertificatePrincipal, + ) -> Option { + self.security.admission(node_certificate_principal) + } + pub(crate) fn get_consumer_group_assignment( &self, topic_name: &str, @@ -206,9 +229,11 @@ impl MetadataState { DeleteTopic(cmd) => self.delete_topic(cmd)?, ReassignSegment(cmd) => self.reassign_segment(cmd)?, DeleteSegments(cmd) => self.delete_segments(cmd)?, - SyncConsumerGroup(cmd) => self.sync_consumer_group(cmd)?, + UpdateConsumerGroupMember(cmd) => self.sync_consumer_group(cmd)?, OpenProducerSession(cmd) => self.open_producer_session(cmd)?, ExpireProducerSessions(cmd) => self.expire_producer_sessions(cmd)?, + GrantAcl(cmd) => self.security.grant(cmd.resource, cmd.principal), + RevokeAcl(cmd) => self.security.revoke(cmd.resource, &cmd.principal), } #[cfg(any(test, debug_assertions))] self.assert_invariants(); @@ -218,20 +243,14 @@ impl MetadataState { fn open_producer_session(&mut self, cmd: OpenProducerSession) -> Result<(), MetadataError> { let topic_id = self .topic_name_index - .get(&cmd.topic_name) + .get(cmd.topic_name.as_ref()) .copied() - .ok_or_else(|| MetadataError::TopicNameNotFound(cmd.topic_name.clone()))?; + .ok_or_else(|| MetadataError::TopicNameNotFound(cmd.topic_name.to_string()))?; let topic = self .topics .get_mut(&topic_id) .ok_or(MetadataError::TopicNotFound(topic_id))?; - topic.producer_sessions.open_producer_session( - cmd.producer_id, - cmd.session_nonce, - cmd.observed_at, - cmd.session_timeout_ms, - ); - Ok(()) + topic.producer_sessions.open_producer_session(cmd) } fn expire_producer_sessions( @@ -494,7 +513,7 @@ impl MetadataState { Ok(()) } - fn sync_consumer_group(&mut self, cmd: SyncConsumerGroup) -> Result<(), MetadataError> { + fn sync_consumer_group(&mut self, cmd: UpdateConsumerGroupMember) -> Result<(), MetadataError> { let group_id = cmd.group_id.clone(); let topic = self @@ -582,6 +601,8 @@ impl crate::test_traits::TAssertInvariant for MetadataState { assert!(id.0 < self.next_topic_id, "topic ID >= next_topic_id"); assert_eq!(*id, topic.id, "topic map key does not match topic identity"); } + + self.security.assert_invariants(); for topic in self.topics.values() { topic.assert_invariants(); } @@ -591,7 +612,10 @@ impl crate::test_traits::TAssertInvariant for MetadataState { #[cfg(test)] mod tests { use super::*; - use crate::connections::protocol::ConsumerGroupSyncAction; + use crate::connections::protocol::ConsumerGroupMemberAction; + use crate::control_plane::consensus::raft::states::security::{ + AclRecord, AdmissionRecord, RevocationRecord, + }; use crate::control_plane::membership::ShardGroupId; use crate::control_plane::metadata::constants::*; use crate::control_plane::metadata::range::*; @@ -619,6 +643,115 @@ mod tests { } } + #[test] + fn security_records_survive_snapshot_restore() { + let mut state = MetadataState::new(ShardGroupId(1)); + let admission = AdmissionRecord { + node_certificate_principal: CertificatePrincipal::new("broker-a"), + revision: 3, + epoch: 2, + node_id: NodeId::new("broker-a::process-2"), + process_public_key: vec![1, 2, 3].into_boxed_slice(), + }; + let acl = AclRecord { + resource: AclResource::TopicData(TopicId(42)), + revision: 4, + principals: vec!["operator".to_string()].into_boxed_slice(), + }; + let revocation = RevocationRecord { + issuer: "cluster-ca".to_string(), + serial: vec![0x12, 0x34].into_boxed_slice(), + revision: 5, + revoked_at: 100, + }; + + state.security.admissions.insert( + admission.node_certificate_principal.clone(), + admission.clone(), + ); + state + .security + .acls + .insert(acl.resource.clone(), acl.clone()); + state.security.revocations.insert( + (revocation.issuer.clone(), revocation.serial.clone()), + revocation.clone(), + ); + + let bytes = borsh::to_vec(&state.snapshot()).unwrap(); + let snapshot = borsh::from_slice(&bytes).unwrap(); + let restored = MetadataState::from_snapshot(snapshot, 9); + + assert_eq!( + restored + .security + .admissions + .get(&CertificatePrincipal::new("broker-a")), + Some(&admission) + ); + assert_eq!(restored.security.acls.get(&acl.resource), Some(&acl)); + assert_eq!( + restored.security.revocations.get(&( + "cluster-ca".to_string(), + vec![0x12, 0x34].into_boxed_slice() + )), + Some(&revocation) + ); + assert_eq!(restored.last_applied_index, 9); + restored.assert_invariants(); + } + + #[test] + fn metadata_acl_snapshot_returns_the_exact_record_or_an_empty_denial() { + let mut state = MetadataState::new(ShardGroupId(1)); + let resource = AclResource::TopicData(TopicId(42)); + state.security.acls.insert( + resource.clone(), + AclRecord { + resource: resource.clone(), + revision: 1, + principals: vec!["orders-service".to_string()].into_boxed_slice(), + }, + ); + + assert_eq!( + state.acl_snapshot(&resource).principals, + vec!["orders-service".to_string()].into_boxed_slice() + ); + assert_eq!( + state + .acl_snapshot(&AclResource::TopicData(TopicId(43))) + .revision, + 0 + ); + } + + #[test] + fn acl_grant_and_revoke_are_idempotent() { + let mut state = MetadataState::new(ShardGroupId(1)); + let grant = GrantAcl { + resource: AclResource::TopicData(TopicId(42)), + principal: "orders-service".to_string(), + }; + let revoke = RevokeAcl { + resource: grant.resource.clone(), + principal: grant.principal.clone(), + }; + + state.apply(grant.clone().into()).unwrap(); + state.apply(grant.into()).unwrap(); + assert_eq!( + state.acl_snapshot(&revoke.resource).principals, + vec!["orders-service".to_string()].into_boxed_slice() + ); + assert_eq!(state.security.acls[&revoke.resource].revision, 1); + + state.apply(revoke.clone().into()).unwrap(); + state.apply(revoke.clone().into()).unwrap(); + assert!(state.acl_snapshot(&revoke.resource).principals.is_empty()); + assert_eq!(state.security.acls[&revoke.resource].revision, 2); + } + fn replica_set() -> Replicas { Replicas::new(vec![ NodeId::new("node-1"), @@ -670,10 +803,12 @@ mod tests { let a = MetadataStateSnapshot { topics: HashMap::from([(first.id, first.clone()), (second.id, second.clone())]), next_topic_id: 3, + security: Box::default(), }; let b = MetadataStateSnapshot { topics: HashMap::from([(second.id, second), (first.id, first)]), next_topic_id: 3, + security: Box::default(), }; assert_eq!(borsh::to_vec(&a).unwrap(), borsh::to_vec(&b).unwrap()); @@ -684,12 +819,12 @@ mod tests { let mut sm = MetadataState::new(ShardGroupId(1)); create_topic(&mut sm, "orders"); let member = uuid::Uuid::new_v4(); - let command = SyncConsumerGroup { - req: SyncConsumerGroupRequest { + let command = UpdateConsumerGroupMember { + req: UpdateConsumerGroupMemberRequest { topic_name: "orders".into(), group_id: "workers".into(), member_id: member, - action: ConsumerGroupSyncAction::Heartbeat, + action: ConsumerGroupMemberAction::Heartbeat, }, observed_at: 100, session_timeout_ms: 10_000, diff --git a/src/control_plane/consensus/raft/states/mod.rs b/src/control_plane/consensus/raft/states/mod.rs index fd907485..5fe6f6c8 100644 --- a/src/control_plane/consensus/raft/states/mod.rs +++ b/src/control_plane/consensus/raft/states/mod.rs @@ -43,3 +43,5 @@ boundaries describe the control-plane Raft metadata log. */ pub(crate) mod consensus; pub(crate) mod metadata_state; + +pub(crate) mod security; diff --git a/src/control_plane/consensus/raft/states/security.rs b/src/control_plane/consensus/raft/states/security.rs new file mode 100644 index 00000000..e2fbfd47 --- /dev/null +++ b/src/control_plane/consensus/raft/states/security.rs @@ -0,0 +1,240 @@ +use std::collections::HashMap; + +use borsh::{BorshDeserialize, BorshSerialize}; + +use crate::control_plane::NodeId; +use crate::control_plane::metadata::AclResource; +use crate::security::CertificatePrincipal; + +/// Security records replicated by one metadata shard. +/// +/// The live metadata state holds this directly. Snapshots box it so security +/// indexes do not enlarge every variant of the Raft snapshot state. +#[derive(Debug, Clone, Default, PartialEq, Eq, BorshSerialize, BorshDeserialize)] +pub(crate) struct SecurityState { + pub(super) admissions: HashMap, + pub(super) acls: HashMap, + pub(super) revocations: HashMap<(String, Box<[u8]>), RevocationRecord>, +} + +/// Current process admitted for `security/node/{node_certificate_principal}`. +/// +/// A restart replaces this record through its metadata shard. Admission checks +/// accept SWIM facts only when the epoch, node ID, and process key match it. +#[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] +pub(crate) struct AdmissionRecord { + pub node_certificate_principal: CertificatePrincipal, + pub revision: u64, + pub epoch: u64, + pub node_id: NodeId, + pub process_public_key: Box<[u8]>, +} + +/// Principals granted the permissions of one exact +/// `security/acl/{resource}` entry. +/// +/// Authorization caches use the revision to reject stale copies. Missing +/// principals and missing records deny access. +#[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] +pub(crate) struct AclRecord { + pub resource: AclResource, + pub revision: u64, + pub principals: Box<[String]>, +} + +/// Certificate blocked by `security/revocation/{issuer}/{serial}`. +/// +/// Brokers cache these records and terminate or reject matching authenticated +/// connections within the cache enforcement window. +#[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] +pub(crate) struct RevocationRecord { + pub issuer: String, + pub serial: Box<[u8]>, + pub revision: u64, + pub revoked_at: u64, +} + +impl SecurityState { + pub(crate) fn admission( + &self, + node_certificate_principal: &CertificatePrincipal, + ) -> Option { + self.admissions.get(node_certificate_principal).cloned() + } + + /// Returns the current ACL record, or an empty revision-zero record when + /// the resource has never been granted to any principal. Both forms deny + /// by default; representing absence explicitly lets callers cache that + /// denial and avoid repeatedly querying the owning shard. + pub(crate) fn acl_snapshot(&self, resource: &AclResource) -> AclRecord { + self.acls + .get(resource) + .cloned() + .unwrap_or_else(|| AclRecord { + resource: resource.clone(), + revision: 0, + principals: Box::new([]), + }) + } + + pub(super) fn grant(&mut self, resource: AclResource, principal: String) { + let acl = self + .acls + .entry(resource.clone()) + .or_insert_with(|| AclRecord { + resource, + revision: 0, + principals: Box::new([]), + }); + if acl.principals.contains(&principal) { + return; + } + + let mut principals = std::mem::take(&mut acl.principals).into_vec(); + principals.push(principal); + acl.principals = principals.into_boxed_slice(); + acl.revision += 1; + } + + pub(super) fn revoke(&mut self, resource: AclResource, principal: &str) { + let Some(acl) = self.acls.get_mut(&resource) else { + return; + }; + let Some(index) = acl.principals.iter().position(|entry| entry == principal) else { + return; + }; + + let mut principals = std::mem::take(&mut acl.principals).into_vec(); + principals.remove(index); + acl.principals = principals.into_boxed_slice(); + acl.revision += 1; + } +} + +#[cfg(any(test, debug_assertions))] +impl crate::test_traits::TAssertInvariant for SecurityState { + fn assert_invariants(&self) { + for (node_certificate_principal, admission) in &self.admissions { + assert_eq!( + node_certificate_principal, &admission.node_certificate_principal, + "admission map key does not match Node Certificate Principal" + ); + } + for (resource, acl) in &self.acls { + assert_eq!( + resource, &acl.resource, + "ACL map key does not match resource" + ); + } + for ((issuer, serial), revocation) in &self.revocations { + assert_eq!( + issuer, &revocation.issuer, + "revocation map key does not match issuer" + ); + assert_eq!( + serial, &revocation.serial, + "revocation map key does not match serial" + ); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::control_plane::metadata::TopicId; + + fn round_trip(value: &T) + where + T: BorshSerialize + BorshDeserialize + PartialEq + std::fmt::Debug, + { + let bytes = borsh::to_vec(value).unwrap(); + assert_eq!(&borsh::from_slice::(&bytes).unwrap(), value); + } + + #[test] + fn security_records_round_trip() { + round_trip(&AdmissionRecord { + node_certificate_principal: CertificatePrincipal::new("broker-a"), + revision: 3, + epoch: 2, + node_id: NodeId::new("broker-a::process-2"), + process_public_key: vec![1, 2, 3].into_boxed_slice(), + }); + round_trip(&AclRecord { + resource: AclResource::TopicData(TopicId(42)), + revision: 4, + principals: vec!["operator".to_string()].into_boxed_slice(), + }); + round_trip(&RevocationRecord { + issuer: "cluster-ca".to_string(), + serial: vec![0x12, 0x34].into_boxed_slice(), + revision: 5, + revoked_at: 100, + }); + } + + #[test] + fn admission_lookup_uses_the_certificate_principal() { + let mut security = SecurityState::default(); + let admission = AdmissionRecord { + node_certificate_principal: CertificatePrincipal::new("broker-a"), + revision: 3, + epoch: 2, + node_id: NodeId::new("broker-a::process-2"), + process_public_key: vec![1, 2, 3].into_boxed_slice(), + }; + security.admissions.insert( + admission.node_certificate_principal.clone(), + admission.clone(), + ); + + assert_eq!( + security.admission(&CertificatePrincipal::new("broker-a")), + Some(admission) + ); + assert_eq!( + security.admission(&CertificatePrincipal::new("broker-b")), + None + ); + } + + #[test] + fn acl_snapshot_returns_the_exact_record_or_an_empty_denial() { + let mut security = SecurityState::default(); + let resource = AclResource::TopicData(TopicId(42)); + security.acls.insert( + resource.clone(), + AclRecord { + resource: resource.clone(), + revision: 1, + principals: vec!["orders-service".to_string()].into_boxed_slice(), + }, + ); + + assert_eq!( + security.acl_snapshot(&resource).principals, + vec!["orders-service".to_string()].into_boxed_slice() + ); + assert_eq!( + security + .acl_snapshot(&AclResource::TopicData(TopicId(43))) + .revision, + 0 + ); + } + + #[test] + fn missing_acl_snapshot_is_an_empty_revision_zero_record() { + let resource = AclResource::TopicData(TopicId(42)); + + assert_eq!( + SecurityState::default().acl_snapshot(&resource), + AclRecord { + resource, + revision: 0, + principals: Box::new([]), + } + ); + } +} diff --git a/src/control_plane/consensus/transport/inbound.rs b/src/control_plane/consensus/transport/inbound.rs index 9df4b3ad..0f1bd3f9 100644 --- a/src/control_plane/consensus/transport/inbound.rs +++ b/src/control_plane/consensus/transport/inbound.rs @@ -2,50 +2,74 @@ use crate::control_plane::NodeId; use crate::control_plane::consensus::actor::MutlRaftSender; use crate::control_plane::consensus::messages::InboundRaftRpc; use crate::control_plane::consensus::messages::WireRaftMessage; -use crate::net::OwnedReadHalf; -use tokio::io::AsyncReadExt; +use crate::net::{TcpStream, TransportReadHalf, TransportTcpStream, TransportWriteHalf}; +use crate::security::{ + AdmissionProof, CertificatePrincipal, SecurityHandle, node_certificate_principal, +}; +use borsh::BorshDeserialize; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; -pub(super) struct RaftRpcListener(pub(super) OwnedReadHalf); +use super::protocol::{ + AclSnapshotResponse, AdmissionLookupResponse, ClusterRequest, InitialClusterMessage, + encode_frame, +}; -impl RaftRpcListener { - pub(super) async fn read_node_id(&mut self) -> anyhow::Result { - let len = self.0.read_u32().await? as usize; - anyhow::ensure!(len <= 1024, "NodeId frame too large: {len} bytes"); - let mut buf = vec![0u8; len]; - self.0.read_exact(&mut buf).await?; - let id = borsh::from_slice::(&buf)?; - Ok(id) +pub(crate) struct ClusterMessageReader { + read_half: TransportReadHalf, + certificate_principal: Option, +} + +impl ClusterMessageReader { + pub(crate) fn new( + read_half: impl Into, + certificate_principal: Option, + ) -> Self { + Self { + read_half: read_half.into(), + certificate_principal, + } } - pub(super) async fn read_message(&mut self) -> anyhow::Result { - let len = self.0.read_u32().await? as usize; + pub(crate) async fn read_frame( + &mut self, + maximum_size: usize, + frame_name: &str, + ) -> anyhow::Result { + let len = self.read_half.read_u32().await? as usize; anyhow::ensure!( - len <= 4 * 1024 * 1024, - "Raft message frame too large: {len} bytes" + len <= maximum_size, + "{frame_name} frame too large: {len} bytes" ); let mut buf = vec![0u8; len]; - self.0.read_exact(&mut buf).await?; - let msg = borsh::from_slice::(&buf)?; - Ok(msg) + self.read_half.read_exact(&mut buf).await?; + Ok(borsh::from_slice(&buf)?) } + #[tracing::instrument( + level = "trace", + skip_all, + fields(peer = %peer, certificate_principal = ?self.certificate_principal) + )] pub(super) async fn run(mut self, tx: MutlRaftSender, peer: NodeId) { loop { - match self.read_message().await { - Ok(msg) => { - if msg.sender != peer { + match self + .read_frame::(4 * 1024 * 1024, "Raft message") + .await + { + Ok(message) => { + if message.peer_id != peer { tracing::warn!( transport_peer = %peer, - claimed_sender = %msg.sender, + claimed_sender = %message.peer_id, "rejected Raft message whose sender differs from the connection peer", ); break; } let _ = tx .send(InboundRaftRpc { - shard_group_id: msg.shard_group_id, - from: peer.clone(), - rpc: msg.rpc, + shard_group_id: message.shard_group_id, + peer_id: peer.clone(), + rpc: message.rpc, }) .await; } @@ -57,3 +81,140 @@ impl RaftRpcListener { } } } + +/// A Raft connection ready to enter the dispatcher writer map. +pub(super) struct AcceptedRaftConnection { + pub(super) peer_id: NodeId, + pub(super) initial_message: WireRaftMessage, + pub(super) reader: ClusterMessageReader, + pub(super) writer: TransportWriteHalf, +} + +impl AcceptedRaftConnection { + fn new( + initial_message: WireRaftMessage, + reader: ClusterMessageReader, + writer: TransportWriteHalf, + ) -> Self { + Self { + peer_id: initial_message.peer_id.clone(), + initial_message, + reader, + writer, + } + } +} + +/// Authenticates a cluster stream and handles its first request. +/// +/// Admission lookups and ACL reads finish here. Only a verified Raft stream is +/// returned to the persistent connection dispatcher. +pub(super) async fn accept_cluster_connection( + stream: TcpStream, + security: SecurityHandle, +) -> anyhow::Result> { + let stream = TransportTcpStream::accept( + stream, + security.node_transport(), + node_certificate_principal, + ) + .await?; + + let certificate_principal = stream.peer_principal(); + let tls_session_binding = if certificate_principal.is_some() { + Some(stream.admission_binding()?) + } else { + None + }; + let (read_half, mut write_half) = stream.into_split(); + let mut reader = ClusterMessageReader::new(read_half, certificate_principal.clone()); + let InitialClusterMessage { + admission_proof, + request, + } = reader + .read_frame::(4 * 1024 * 1024, "initial cluster message") + .await?; + + let tls_session_binding = tls_session_binding + .as_ref() + .map(|binding| binding.as_slice()); + + match request { + ClusterRequest::AdmissionLookup(request) => { + anyhow::ensure!( + admission_proof.is_none(), + "admission lookup carried a process proof" + ); + let admission = security.read_admission(request).await?; + write_half + .write_all(&encode_frame(&AdmissionLookupResponse { admission })?) + .await?; + Ok(None) + } + ClusterRequest::Raft(message) => { + verify_requester( + certificate_principal.as_ref(), + admission_proof.as_ref(), + &message.peer_id, + tls_session_binding, + &security, + &mut write_half, + ) + .await?; + Ok(Some(AcceptedRaftConnection::new( + message, reader, write_half, + ))) + } + ClusterRequest::AclSnapshot(snapshot_req) => { + verify_requester( + certificate_principal.as_ref(), + admission_proof.as_ref(), + &snapshot_req.requester_node_id, + tls_session_binding, + &security, + &mut write_half, + ) + .await?; + let snapshot = security.read_acl(snapshot_req).await?; + write_half + .write_all(&encode_frame(&AclSnapshotResponse { snapshot })?) + .await?; + Ok(None) + } + } +} + +async fn verify_requester( + certificate_principal: Option<&CertificatePrincipal>, + admission_proof: Option<&AdmissionProof>, + requester_node_id: &NodeId, + tls_session_binding: Option<&[u8]>, + security: &SecurityHandle, + write_half: &mut TransportWriteHalf, +) -> anyhow::Result<()> { + let Some(principal) = certificate_principal else { + anyhow::ensure!( + admission_proof.is_none(), + "trusted-development connection carried an admission proof" + ); + return Ok(()); + }; + let Some(proof) = admission_proof else { + anyhow::bail!("secure cluster connection omitted process admission"); + }; + let Some(tls_session_binding) = tls_session_binding else { + anyhow::bail!("secure connection has no TLS session binding"); + }; + let admission = security.lookup_admission(principal).await?; + let peer_id = proof.verify_admission(&admission, principal, tls_session_binding)?; + anyhow::ensure!( + requester_node_id == &peer_id, + "cluster requester differs from admitted process" + ); + write_half + .write_all(&encode_frame( + &security.create_admission_proof(tls_session_binding)?, + )?) + .await?; + Ok(()) +} diff --git a/src/control_plane/consensus/transport/mod.rs b/src/control_plane/consensus/transport/mod.rs index 072d0bf5..2a8c9a7d 100644 --- a/src/control_plane/consensus/transport/mod.rs +++ b/src/control_plane/consensus/transport/mod.rs @@ -1,9 +1,13 @@ #![allow(dead_code)] +mod acl; +pub(crate) use acl::{AclSnapshotActor, AclSnapshotSender}; +mod admission; mod inbound; use inbound::*; mod outbound; use outbound::*; +mod protocol; use tokio::sync::mpsc; @@ -13,6 +17,10 @@ use crate::control_plane::NodeId; use crate::control_plane::consensus::messages::RaftTransportCommand; use crate::control_plane::membership::actor::SwimSender; use crate::net::TcpListener; +use crate::net::TransportTcpStream; +use crate::security::NodeTransportSecurity; +#[cfg(test)] +use crate::security::TransportIdentity; const CONNECT_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2); @@ -26,16 +34,20 @@ impl RaftTransportActor { raft_tx: MutlRaftSender, mut from_actor: mpsc::Receiver>, swim_tx: SwimSender, + security: NodeTransportSecurity, ) { let (dial_tx, mut dial_rx) = mpsc::channel(256); - let mut dispatcher = RaftRpcDispatcher::new(node_id, dial_tx); + let mut dispatcher = RaftRpcDispatcher::new(node_id, dial_tx, security.clone()); let mut cleanup_interval = tokio::time::interval(std::time::Duration::from_secs(300)); cleanup_interval.tick().await; // consume immediate first tick loop { tokio::select! { Ok((stream, _)) = listener.accept() => { - dispatcher.accept(stream, &raft_tx).await; + match TransportTcpStream::accept_node(stream, &security).await { + Ok(stream) => dispatcher.accept(stream, &raft_tx).await, + Err(error) => tracing::debug!("Raft TLS accept rejected: {error}"), + } } Some(batch) = from_actor.recv() => { // Disconnects are applied first so same-batch sends already skip removed peers, then @@ -66,16 +78,22 @@ impl RaftTransportActor { #[cfg(test)] mod tests { + use super::protocol::InitialClusterMessage; use super::*; use crate::control_plane::consensus::actor::MultiRaftActor; use crate::control_plane::consensus::messages::{ MultiRaftActorCommand, RaftProtocolMessage, RaftRpc, RequestVote, WireRaftMessage, }; + use crate::control_plane::consensus::raft::states::security::{AclRecord, AdmissionRecord}; use crate::control_plane::membership::ShardGroupId; + use crate::control_plane::membership::actor::{RemoteShard, ShardRouting}; + use crate::control_plane::metadata::{AclResource, TopicId}; + use crate::control_plane::{NodeAddress, NodeAddressInfo}; use crate::net::OwnedWriteHalf; use crate::net::TcpStream; use std::time::Duration; use tokio::io::AsyncWriteExt; + use tokio::sync::Notify; use turmoil::Builder; /// Write a length-prefixed borsh-encoded value to a raw write half. @@ -91,8 +109,21 @@ mod tests { Ok(()) } + fn request_vote_message(shard_group_id: u64, sender: &str) -> WireRaftMessage { + WireRaftMessage { + shard_group_id: ShardGroupId(shard_group_id), + sender: NodeId::new(sender), + rpc: RaftRpc::RequestVote(RequestVote { + term: 1, + candidate_id: NodeId::new(sender), + last_log_index: 0, + last_log_term: 0, + }), + } + } + #[test] - fn handshake_write_then_read_node_id() -> turmoil::Result { + fn initial_raft_message_identifies_the_peer() -> turmoil::Result { let mut sim = Builder::new() .simulation_duration(Duration::from_secs(5)) .build(); @@ -101,10 +132,14 @@ mod tests { let listener = TcpListener::bind("0.0.0.0:9000").await?; let (stream, _) = listener.accept().await?; let (read_half, _) = stream.into_split(); - let mut reader = RaftRpcListener(read_half); + let mut reader = + ClusterMessageReader::new(read_half, TransportIdentity::TrustedDevelopment); - let peer_id = reader.read_node_id().await.unwrap(); - assert_eq!(peer_id, NodeId::new("node-abc")); + let InitialClusterMessage::Raft(message) = reader.read_initial_message().await.unwrap() + else { + panic!("expected initial Raft message"); + }; + assert_eq!(message.sender, NodeId::new("node-abc")); Ok(()) }); @@ -113,7 +148,11 @@ mod tests { let stream = TcpStream::connect((addr, 9000)).await?; let (_, mut write_half) = stream.into_split(); - write_frame(&mut write_half, &NodeId::new("node-abc")).await?; + write_frame( + &mut write_half, + &InitialClusterMessage::Raft(request_vote_message(42, "node-abc")), + ) + .await?; Ok(()) }); @@ -130,9 +169,10 @@ mod tests { let listener = TcpListener::bind("0.0.0.0:9000").await?; let (stream, _) = listener.accept().await?; let (read_half, _) = stream.into_split(); - let mut reader = RaftRpcListener(read_half); + let mut reader = + ClusterMessageReader::new(read_half, TransportIdentity::TrustedDevelopment); - let msg = reader.read_message().await.unwrap(); + let msg = reader.read_raft_message().await.unwrap(); assert_eq!(msg.shard_group_id, ShardGroupId(42)); assert_eq!(msg.sender, NodeId::new("sender-1")); match msg.rpc { @@ -180,8 +220,9 @@ mod tests { let listener = TcpListener::bind("0.0.0.0:9000").await?; let (stream, _) = listener.accept().await?; let (read_half, _) = stream.into_split(); - let mut reader = RaftRpcListener(read_half); - let peer = reader.read_node_id().await?; + let reader = + ClusterMessageReader::new(read_half, TransportIdentity::TrustedDevelopment); + let peer = NodeId::new("node-a"); let (raft_tx, mut raft_rx) = MultiRaftActor::channel(8); reader.run(raft_tx, peer.clone()).await; @@ -192,7 +233,7 @@ mod tests { else { panic!("expected one inbound Raft RPC") }; - assert_eq!(cmd.from, peer); + assert_eq!(cmd.peer_id, peer); assert_eq!(cmd.shard_group_id, ShardGroupId(1)); assert!(raft_rx.try_recv().is_err()); Ok(()) @@ -202,24 +243,8 @@ mod tests { let addr = turmoil::lookup("server"); let stream = TcpStream::connect((addr, 9000)).await?; let (_, mut writer) = stream.into_split(); - write_frame(&mut writer, &NodeId::new("node-a")).await?; - - for (group, sender) in [(1, "node-a"), (2, "node-b")] { - write_frame( - &mut writer, - &WireRaftMessage { - shard_group_id: ShardGroupId(group), - sender: NodeId::new(sender), - rpc: RaftRpc::RequestVote(RequestVote { - term: 1, - candidate_id: NodeId::new(sender), - last_log_index: 0, - last_log_term: 0, - }), - }, - ) - .await?; - } + write_frame(&mut writer, &request_vote_message(1, "node-a")).await?; + write_frame(&mut writer, &request_vote_message(2, "node-b")).await?; Ok(()) }); @@ -227,24 +252,37 @@ mod tests { } #[test] - fn accepted_connection_registers_writer_after_handshake() -> turmoil::Result { + fn accepted_connection_registers_writer_after_initial_raft_message() -> turmoil::Result { let mut sim = Builder::new() .simulation_duration(Duration::from_secs(5)) .build(); sim.host("acceptor", || async { - let (raft_tx, _raft_rx) = MultiRaftActor::channel(16); + let (raft_tx, mut raft_rx) = MultiRaftActor::channel(16); let listener = TcpListener::bind("0.0.0.0:9000").await?; let (dial_tx, _dial_rx) = tokio::sync::mpsc::channel(8); - let mut state = RaftRpcDispatcher::new(NodeId::new("node-b"), dial_tx); + let mut state = RaftRpcDispatcher::new( + NodeId::new("node-b"), + dial_tx, + NodeTransportSecurity::TrustedDevelopment, + ); let (stream, _) = listener.accept().await?; - state.accept(stream, &raft_tx).await; + state + .accept(TransportTcpStream::TrustedDevelopment(stream), &raft_tx) + .await; assert!( state.contains(&NodeId::new("node-a")), - "writer should be registered after handshake" + "writer should be registered after the initial Raft message" ); + let Some(MultiRaftActorCommand::ProtocolMessage(RaftProtocolMessage::InboundRaftRpc( + rpc, + ))) = raft_rx.recv().await + else { + panic!("expected the initial Raft RPC") + }; + assert_eq!(rpc.peer_id, NodeId::new("node-a")); Ok(()) }); @@ -252,7 +290,11 @@ mod tests { let addr = turmoil::lookup("acceptor"); let stream = TcpStream::connect((addr, 9000)).await?; let (_, mut write_half) = stream.into_split(); - write_frame(&mut write_half, &NodeId::new("node-a")).await?; + write_frame( + &mut write_half, + &InitialClusterMessage::Raft(request_vote_message(1, "node-a")), + ) + .await?; Ok(()) }); @@ -274,18 +316,30 @@ mod tests { let listener = TcpListener::bind("0.0.0.0:9000").await?; let dummy_listener = TcpListener::bind("0.0.0.0:9001").await?; let (dial_tx, _dial_rx) = tokio::sync::mpsc::channel(8); - let mut state = RaftRpcDispatcher::new(NodeId::new("node-b"), dial_tx); + let mut state = RaftRpcDispatcher::new( + NodeId::new("node-b"), + dial_tx, + NodeTransportSecurity::TrustedDevelopment, + ); // First connection from node-a let (stream, _) = listener.accept().await?; - state.accept(stream, &raft_tx).await; + state + .accept(TransportTcpStream::TrustedDevelopment(stream), &raft_tx) + .await; assert!(state.contains(&NodeId::new("node-a"))); // Second connection from node-a (simulating simultaneous connect) let (stream2, _) = dummy_listener.accept().await?; let (read_half, _write_half) = stream2.into_split(); - let mut reader = RaftRpcListener(read_half); - let peer_id = reader.read_node_id().await.unwrap(); + let mut reader = + ClusterMessageReader::new(read_half, TransportIdentity::TrustedDevelopment); + let InitialClusterMessage::Raft(initial_raft_message) = + reader.read_initial_message().await.unwrap() + else { + panic!("expected initial Raft message"); + }; + let peer_id = initial_raft_message.sender; assert_eq!(peer_id, NodeId::new("node-a")); // Conflict: node-a < node-b → incoming wins, replace @@ -303,15 +357,204 @@ mod tests { let stream1 = TcpStream::connect((addr, 9000)).await?; let (_, mut write_half) = stream1.into_split(); - write_frame(&mut write_half, &NodeId::new("node-a")).await?; + write_frame( + &mut write_half, + &InitialClusterMessage::Raft(request_vote_message(1, "node-a")), + ) + .await?; let stream2 = TcpStream::connect((addr, 9001)).await?; let (_, mut write_half2) = stream2.into_split(); - write_frame(&mut write_half2, &NodeId::new("node-a")).await?; + write_frame( + &mut write_half2, + &InitialClusterMessage::Raft(request_vote_message(1, "node-a")), + ) + .await?; Ok(()) }); sim.run() } + + #[test] + fn acl_snapshot_actor_coalesces_concurrent_refreshes() -> turmoil::Result { + let resource = AclResource::TopicData(TopicId(7)); + let snapshot = AclRecord { + resource: resource.clone(), + revision: 3, + principals: vec!["orders-service".to_owned()].into(), + }; + let owner = NodeAddressInfo::new( + NodeId::new("owner"), + NodeAddress::test( + "127.0.0.1:9000".parse().unwrap(), + "127.0.0.1:9001".parse().unwrap(), + ), + ); + let response_received = std::sync::Arc::new(Notify::new()); + let mut sim = Builder::new() + .simulation_duration(Duration::from_secs(5)) + .build(); + + let server_snapshot = snapshot.clone(); + let server_resource = resource.clone(); + let server_response_received = response_received.clone(); + sim.host("owner", move || { + let expected_snapshot = server_snapshot.clone(); + let expected_resource = server_resource.clone(); + let owner_completion = server_response_received.clone(); + async move { + let listener = TcpListener::bind("0.0.0.0:9000").await?; + let (raft_tx, mut raft_rx) = MultiRaftActor::channel(8); + let (dial_tx, _dial_rx) = tokio::sync::mpsc::channel(8); + let mut dispatcher = RaftRpcDispatcher::new( + NodeId::new("owner"), + dial_tx, + NodeTransportSecurity::TrustedDevelopment, + ); + + let (stream, _) = listener.accept().await?; + dispatcher + .accept(TransportTcpStream::TrustedDevelopment(stream), &raft_tx) + .await; + + let Some(MultiRaftActorCommand::GetAclSnapshot(query)) = raft_rx.recv().await + else { + panic!("expected ACL snapshot query"); + }; + assert_eq!(query.shard_group_id, ShardGroupId(42)); + assert_eq!(query.resource, expected_resource); + let _ = query.reply.send(Some(expected_snapshot)); + owner_completion.notified().await; + Ok(()) + } + }); + + let client_resource = resource.clone(); + let client_snapshot = snapshot.clone(); + let client_owner = owner.clone(); + sim.host("requester", move || { + let requested_resource = client_resource.clone(); + let expected_snapshot = client_snapshot.clone(); + let remote_owner = client_owner.clone(); + let requester_completion = response_received.clone(); + async move { + let client = AclSnapshotActor::spawn(NodeTransportSecurity::TrustedDevelopment); + let first_owner = remote_owner.clone(); + let first_resource = requested_resource.clone(); + let (first, second) = tokio::join!( + client.fetch( + NodeId::new("requester"), + first_owner, + ShardGroupId(42), + first_resource, + ), + client.fetch( + NodeId::new("requester"), + remote_owner, + ShardGroupId(42), + requested_resource, + ), + ); + assert_eq!(first, Some(expected_snapshot.clone())); + assert_eq!(second, Some(expected_snapshot)); + requester_completion.notify_one(); + Ok(()) + } + }); + + sim.run() + } + + #[test] + fn admission_lookup_actor_coalesces_remote_reads() -> turmoil::Result { + let admission = AdmissionRecord { + node_certificate_principal: "broker-a".to_string(), + revision: 3, + epoch: 8, + node_id: NodeId::new("broker-a::process-2"), + process_public_key: vec![1, 2, 3].into_boxed_slice(), + }; + let owner = NodeAddressInfo::new( + NodeId::new("owner"), + NodeAddress::test( + "127.0.0.1:9000".parse().unwrap(), + "127.0.0.1:9001".parse().unwrap(), + ), + ); + let response_received = std::sync::Arc::new(Notify::new()); + let mut sim = Builder::new() + .simulation_duration(Duration::from_secs(5)) + .build(); + + let server_admission = admission.clone(); + let server_response_received = response_received.clone(); + sim.host("owner", move || { + let expected_admission = server_admission.clone(); + let owner_completion = server_response_received.clone(); + async move { + let listener = TcpListener::bind("0.0.0.0:9000").await?; + let (raft_tx, mut raft_rx) = MultiRaftActor::channel(8); + let (dial_tx, _dial_rx) = tokio::sync::mpsc::channel(8); + let mut dispatcher = RaftRpcDispatcher::new( + NodeId::new("owner"), + dial_tx, + NodeTransportSecurity::TrustedDevelopment, + ); + + let (stream, _) = listener.accept().await?; + dispatcher + .accept(TransportTcpStream::TrustedDevelopment(stream), &raft_tx) + .await; + + let Some(MultiRaftActorCommand::GetAdmission(query)) = raft_rx.recv().await else { + panic!("expected admission query"); + }; + assert_eq!(query.shard_group_id, ShardGroupId(42)); + assert_eq!(query.node_certificate_principal.as_ref(), "broker-a"); + let _ = query.reply.send(Ok(Some(expected_admission))); + owner_completion.notified().await; + Ok(()) + } + }); + + let client_admission = admission.clone(); + let client_owner = owner.clone(); + sim.host("requester", move || { + let expected_admission = client_admission.clone(); + let remote_owner = client_owner.clone(); + let requester_completion = response_received.clone(); + async move { + let (raft_tx, _raft_rx) = MultiRaftActor::channel(8); + let lookup = admission::AdmissionLookupActor::spawn( + raft_tx, + NodeTransportSecurity::TrustedDevelopment, + ); + let first_owner = remote_owner.clone(); + let (first, second) = tokio::join!( + lookup.lookup( + ShardRouting::Redirect(Some(RemoteShard { + group_id: ShardGroupId(42), + member: Some(first_owner), + })), + "broker-a".into(), + ), + lookup.lookup( + ShardRouting::Redirect(Some(RemoteShard { + group_id: ShardGroupId(42), + member: Some(remote_owner), + })), + "broker-a".into(), + ), + ); + assert_eq!(first, Ok(Some(expected_admission.clone()))); + assert_eq!(second, Ok(Some(expected_admission))); + requester_completion.notify_one(); + Ok(()) + } + }); + + sim.run() + } } diff --git a/src/control_plane/consensus/transport/outbound.rs b/src/control_plane/consensus/transport/outbound.rs index 35530f5f..d24a19e9 100644 --- a/src/control_plane/consensus/transport/outbound.rs +++ b/src/control_plane/consensus/transport/outbound.rs @@ -6,29 +6,36 @@ use tokio::time::Instant; use crate::control_plane::consensus::actor::MutlRaftSender; -use crate::control_plane::consensus::messages::{OutboundRaftPacket, WireRaftMessage}; +use crate::control_plane::consensus::messages::{ + InboundRaftRpc, OutboundRaftPacket, WireRaftMessage, +}; use crate::control_plane::NodeId; -use crate::control_plane::consensus::transport::RaftRpcListener; +use crate::control_plane::consensus::transport::{AcceptedRaftConnection, ClusterMessageReader}; use crate::control_plane::membership::actor::SwimSender; -use crate::net::{OwnedWriteHalf, TcpStream}; +use crate::net::{TransportTcpStream, TransportWriteHalf}; +use crate::security::{AdmissionProof, CertificatePrincipal}; + +use super::protocol::{ClusterRequest, InitialClusterMessage, encode_frame}; +use crate::security::SecurityHandle; const CONNECT_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2); -/// Upper bound on messages buffered per peer while its dial is in flight; +const ADMISSION_HANDSHAKE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(8); +/// Upper bound on messages buffered per peer while its connection is in flight; /// overflow is dropped (raft retries by timer). -const PENDING_DIAL_BUFFER_CAP: usize = 256; +const CONNECTING_PEER_BUFFER_CAP: usize = 256; /// Manages peer connections, address resolution, and dead-peer tracking. /// /// On simultaneous connect, the connection initiated by the **lower `NodeId`** /// wins; the other is dropped. /// -/// Handshake: after connecting, the initiator sends its `NodeId`. The acceptor -/// reads it, and if a connection to that peer already exists (from our own -/// outbound connect), the tie is broken by NodeId ordering. +/// A connection begins with its first Raft message, which identifies the peer. +/// The acceptor uses that sender to key the writer slot and resolve simultaneous +/// connection conflicts. pub(super) struct RaftRpcDispatcher { node_id: NodeId, - writers: HashMap, + writers: HashMap, /// Peers explicitly disconnected via DisconnectPeer. Outbound RPCs /// to these peers are silently dropped until a new connection is /// accepted (peer restart with new UUID won't hit this — different NodeId). @@ -38,49 +45,59 @@ pub(super) struct RaftRpcDispatcher { /// the transport's select loop and stall flush_events in MultiRaftActor. connect_backoffs: HashMap, /// Messages buffered for peers whose connection is being established on a - /// background task; flushed (or dropped on failure) in `on_dial_result`. - pending_dials: HashMap>, - dial_tx: mpsc::Sender, + /// background task; flushed (or dropped on failure) in `on_connection_result`. + connecting_peers: HashMap>, + connection_result_tx: mpsc::Sender, + security: SecurityHandle, } -/// Result of a background dial attempt, delivered back to the transport loop. -pub(super) struct DialOutcome { +/// Result of a background connection attempt, delivered to the transport loop. +pub(super) struct ConnectionAttemptResult { target: NodeId, - outcome: anyhow::Result<(RaftRpcListener, OwnedWriteHalf)>, + result: anyhow::Result<(ClusterMessageReader, TransportWriteHalf)>, } impl RaftRpcDispatcher { - pub(super) fn new(node_id: NodeId, dial_tx: mpsc::Sender) -> Self { + pub(super) fn new( + node_id: NodeId, + connection_result_tx: mpsc::Sender, + security: SecurityHandle, + ) -> Self { Self { node_id, writers: HashMap::new(), dead_peers: HashSet::new(), connect_backoffs: HashMap::new(), - pending_dials: HashMap::new(), - dial_tx, + connecting_peers: HashMap::new(), + connection_result_tx, + security, } } - pub(super) async fn accept(&mut self, stream: TcpStream, raft_tx: &MutlRaftSender) { - let (read_half, write_half) = stream.into_split(); - let mut reader = RaftRpcListener(read_half); + pub(super) fn accept(&mut self, connection: AcceptedRaftConnection, raft_tx: &MutlRaftSender) { + let AcceptedRaftConnection { + peer_id: connection_peer_id, + initial_message, + reader, + writer, + } = connection; - let Ok(peer_id) = reader.read_node_id().await else { - tracing::error!("Failed to read peer NodeId during accept"); - return; + let initial_rpc = InboundRaftRpc { + shard_group_id: initial_message.shard_group_id, + peer_id: connection_peer_id, + rpc: initial_message.rpc, }; - - if self.writers.contains_key(&peer_id) && peer_id > self.node_id { - tracing::debug!( - peer = %peer_id, - "simultaneous connect: dropping accepted connection, \ - our outbound dial wins the tie-break (lower NodeId)", - ); + if self.writers.contains_key(&initial_rpc.peer_id) && initial_rpc.peer_id > self.node_id { + // simultaneous connect: dropping accepted connection return; } - - self.writers.insert(peer_id.clone(), write_half); - tokio::spawn(reader.run(raft_tx.clone(), peer_id)); + self.writers.insert(initial_rpc.peer_id.clone(), writer); + let raft_tx = raft_tx.clone(); + tokio::spawn(async move { + let reader_peer_id = initial_rpc.peer_id.clone(); + let _ = raft_tx.send(initial_rpc).await; + reader.run(raft_tx, reader_peer_id).await; + }); } pub(super) async fn send(&mut self, packets: Vec, swim_tx: &SwimSender) { @@ -103,7 +120,7 @@ impl RaftRpcDispatcher { .or_default() .push(WireRaftMessage { shard_group_id: pkt.shard_group_id, - sender: self.node_id.clone(), + peer_id: self.node_id.clone(), rpc: pkt.rpc, }); } @@ -113,7 +130,7 @@ impl RaftRpcDispatcher { async fn send_to_target( &mut self, target_id: NodeId, - msgs: Vec, + mut msgs: Vec, swim_tx: &SwimSender, ) { if let Some(&failed_at) = self.connect_backoffs.get(&target_id) { @@ -127,39 +144,40 @@ impl RaftRpcDispatcher { { return; } - // No usable writer: hand the messages to the in-flight dial (if any) - // or start one on a background task. Dials must never run inline — a - // hung connect (crashed peer; acceptor starved because *its* loop is - // mid-dial) blocks this select loop for the full connect timeout, - // stalling every queued batch and the accept arm with it (#133). - if let Some(buffered) = self.pending_dials.get_mut(&target_id) { - if buffered.len() + msgs.len() <= PENDING_DIAL_BUFFER_CAP { + // No usable writer: hand the messages to the in-flight connection (if + // any) or start one on a background task. Connection attempts must + // never run inline: a hung connect blocks this select loop for the full + // timeout, stalling every queued batch and the accept arm with it (#133). + if let Some(buffered) = self.connecting_peers.get_mut(&target_id) { + if buffered.len() + msgs.len() <= CONNECTING_PEER_BUFFER_CAP { buffered.extend(msgs); } return; } - self.pending_dials.insert(target_id.clone(), msgs); - let dial_task = dial(self.node_id.clone(), target_id.clone(), swim_tx.clone()); - let dial_tx = self.dial_tx.clone(); - tokio::spawn(async move { - let outcome = dial_task.await; - let _ = dial_tx - .send(DialOutcome { - target: target_id, - outcome, - }) - .await; - }); + let initial_raft_message = msgs.remove(0); + self.connecting_peers.insert(target_id.clone(), msgs); + + tokio::spawn(connect_peer( + target_id, + swim_tx.clone(), + self.security.clone(), + initial_raft_message, + self.connection_result_tx.clone(), + )); } - /// Installs (or discards, per the NodeId tie-break) a completed dial and - /// flushes any messages buffered while it was in flight. - pub(super) async fn on_dial_result(&mut self, result: DialOutcome, raft_tx: &MutlRaftSender) { - let DialOutcome { target, outcome } = result; - let buffered = self.pending_dials.remove(&target).unwrap_or_default(); + /// Installs (or discards, per the NodeId tie-break) a completed connection + /// and flushes any messages buffered while it was in flight. + pub(super) async fn on_connection_result( + &mut self, + attempt: ConnectionAttemptResult, + raft_tx: &MutlRaftSender, + ) { + let ConnectionAttemptResult { target, result } = attempt; + let buffered = self.connecting_peers.remove(&target).unwrap_or_default(); - let Ok((reader, write_half)) = outcome.inspect_err(|err| { - tracing::warn!(peer = %target, "dial failed: {err}"); + let Ok((reader, write_half)) = result.inspect_err(|err| { + tracing::warn!(peer = %target, "connection attempt failed: {err}"); }) else { self.connect_backoffs.insert(target, Instant::now()); return; @@ -176,7 +194,7 @@ impl RaftRpcDispatcher { tracing::debug!( peer = %target, buffered = buffered.len(), - "simultaneous connect: discarding our dial, peer's connection \ + "simultaneous connect: discarding our connection, peer's connection \ wins the tie-break (lower NodeId); rerouting buffered messages", ); if !buffered.is_empty() { @@ -194,7 +212,7 @@ impl RaftRpcDispatcher { pub(super) fn disconnect(&mut self, peer_id: NodeId) { self.writers.remove(&peer_id); - self.pending_dials.remove(&peer_id); + self.connecting_peers.remove(&peer_id); tracing::info!("[{}] Disconnected dead peer {:?}", self.node_id, peer_id); self.dead_peers.insert(peer_id); } @@ -217,10 +235,8 @@ impl RaftRpcDispatcher { })?; let mut buf = Vec::new(); for msg in msgs { - let bytes = borsh::to_vec(msg)?; - let len = bytes.len() as u32; - buf.extend_from_slice(&len.to_be_bytes()); - buf.extend_from_slice(&bytes); + let frame = encode_frame(msg).map_err(std::io::Error::other)?; + buf.extend_from_slice(&frame); } let result = writer.write_all(&buf).await; if result.is_err() { @@ -235,31 +251,112 @@ impl RaftRpcDispatcher { } } -/// Resolve, connect (3s cap), and handshake — on a spawned task, so a hung -/// connect can never block the transport select loop. The loop -/// installs the writer and flushes buffered messages in `on_dial_result`. -// ! never inline this. Actor Model should onkly do work whose duration it controls. -// ! Anything whose latency the outside actor controls must not be awaited in the handler. -async fn dial( - node_id: NodeId, +/// Connects, performs the opening Raft exchange, and reports the result. +async fn connect_peer( target_id: NodeId, swim_tx: SwimSender, -) -> anyhow::Result<(RaftRpcListener, OwnedWriteHalf)> { - let Some(addr) = swim_tx.resolve_address(target_id.clone()).await? else { - anyhow::bail!("[{}] Cannot resolve address for {:?}", node_id, target_id); - }; - - let stream = tokio::time::timeout( - std::time::Duration::from_secs(3), - TcpStream::connect(addr.cluster_addr()), - ) - .await??; - - let (read_half, mut write_half) = stream.into_split(); - let bytes = borsh::to_vec(&node_id) - .map_err(|e| anyhow::anyhow!("[{}] Handshake encode failed: {e}", node_id))?; - let len = bytes.len() as u32; - write_half.write_all(&len.to_be_bytes()).await?; - write_half.write_all(&bytes).await?; - Ok((RaftRpcListener(read_half), write_half)) + security: SecurityHandle, + initial_raft_message: WireRaftMessage, + connection_result_tx: mpsc::Sender, +) { + let result = async { + let Some(addr) = swim_tx.resolve_address(target_id.clone()).await? else { + anyhow::bail!("cannot resolve address for {target_id}"); + }; + + let stream = tokio::time::timeout( + std::time::Duration::from_secs(3), + TransportTcpStream::connect_node(addr.cluster_addr(), security.node_transport()), + ) + .await??; + + let mut connection = OutboundClusterConnection::new(stream, target_id.clone())?; + tokio::time::timeout( + ADMISSION_HANDSHAKE_TIMEOUT, + connection.send_initial_request(&security, ClusterRequest::Raft(initial_raft_message)), + ) + .await??; + Ok(connection.into_parts()) + } + .await; + + let _ = connection_result_tx + .send(ConnectionAttemptResult { + target: target_id, + result, + }) + .await; +} + +/// One outbound cluster stream before its first Raft or ACL message. +/// +/// It owns the expected peer identity and both stream halves so callers cannot +/// accidentally perform only one side of the mutual admission exchange. +pub(crate) struct OutboundClusterConnection { + pub(crate) reader: ClusterMessageReader, + writer: TransportWriteHalf, + expected_peer_id: NodeId, + /// Present only for a TLS stream; trusted-development streams skip + /// admission entirely. + tls_peer: Option<(CertificatePrincipal, [u8; 32])>, +} + +impl OutboundClusterConnection { + pub(crate) fn new( + stream: TransportTcpStream, + expected_peer_id: NodeId, + ) -> anyhow::Result { + let certificate_principal = stream.peer_principal(); + let tls_peer = match &certificate_principal { + Some(principal) => Some((principal.clone(), stream.admission_binding()?)), + None => None, + }; + + let (read_half, writer) = stream.into_split(); + Ok(Self { + reader: ClusterMessageReader::new(read_half, certificate_principal), + writer, + expected_peer_id, + tls_peer, + }) + } + + pub(crate) async fn send_initial_request( + &mut self, + security: &SecurityHandle, + request: ClusterRequest, + ) -> anyhow::Result<()> { + let admission_proof = match self.tls_peer.as_ref() { + Some((_, tls_session_binding)) => { + Some(security.create_admission_proof(tls_session_binding)?) + } + None => None, + }; + self.writer + .write_all(&encode_frame(&InitialClusterMessage { + admission_proof, + request, + })?) + .await?; + + let Some((peer_principal, tls_session_binding)) = self.tls_peer.as_ref() else { + return Ok(()); + }; + let peer_admission = security.lookup_admission(peer_principal).await?; + let peer_proof = self + .reader + .read_frame::(4 * 1024, "admission proof") + .await?; + let admitted_peer = + peer_proof.verify_admission(&peer_admission, peer_principal, tls_session_binding)?; + anyhow::ensure!( + admitted_peer == self.expected_peer_id, + "connected broker differs from the expected admitted process" + ); + Ok(()) + } + + fn into_parts(self) -> (ClusterMessageReader, TransportWriteHalf) { + (self.reader, self.writer) + } } diff --git a/src/control_plane/consensus/transport/protocol.rs b/src/control_plane/consensus/transport/protocol.rs new file mode 100644 index 00000000..04f666c6 --- /dev/null +++ b/src/control_plane/consensus/transport/protocol.rs @@ -0,0 +1,67 @@ +use anyhow::Result; +use borsh::{BorshDeserialize, BorshSerialize}; + +use crate::control_plane::NodeId; +use crate::control_plane::consensus::messages::WireRaftMessage; +use crate::control_plane::consensus::raft::states::security::{AclRecord, AdmissionRecord}; +use crate::control_plane::membership::ShardGroupId; +use crate::control_plane::metadata::AclResource; + +use crate::security::{AdmissionProof, CertificatePrincipal}; + +/// The first frame on a cluster TCP connection. +/// +/// Secure Raft and ACL requests include a process proof bound to their TLS +/// session. Admission lookups and trusted-development requests omit it. +#[derive(Debug, Clone, BorshSerialize, BorshDeserialize)] +pub(crate) struct InitialClusterMessage { + pub(crate) admission_proof: Option, + pub(crate) request: ClusterRequest, +} + +/// Requests accepted as the first frame on a cluster connection. +#[derive(Debug, Clone, BorshSerialize, BorshDeserialize)] +pub(crate) enum ClusterRequest { + AdmissionLookup(AdmissionRecordKey), + Raft(WireRaftMessage), + AclSnapshot(AclSnapshotRequest), +} + +/// One read of a committed ACL record from a shard host. +#[derive(Debug, Clone, BorshSerialize, BorshDeserialize)] +pub(crate) struct AclSnapshotRequest { + pub(crate) requester_node_id: NodeId, + pub(crate) shard_group_id: ShardGroupId, + pub(crate) resource: AclResource, +} + +/// The response to one ACL snapshot request on its dedicated connection. +#[derive(Debug, Clone, BorshSerialize, BorshDeserialize)] +pub(crate) struct AclSnapshotResponse { + pub(crate) snapshot: Option, +} + +/// Identifies one admission record in its metadata shard. +/// +/// The admission lookup actor uses the same value for routing, cache +/// coalescing, and the limited wire request. +#[derive(Debug, Clone, PartialEq, Eq, Hash, BorshSerialize, BorshDeserialize)] +pub(crate) struct AdmissionRecordKey { + pub(crate) shard_group_id: ShardGroupId, + pub(crate) node_certificate_principal: CertificatePrincipal, +} + +/// Response to one limited admission lookup, after which the connection closes. +#[derive(Debug, Clone, BorshSerialize, BorshDeserialize)] +pub(crate) struct AdmissionLookupResponse { + pub(crate) admission: Option, +} + +pub(crate) fn encode_frame(value: &impl BorshSerialize) -> Result> { + let bytes = borsh::to_vec(value)?; + let len = u32::try_from(bytes.len())?; + let mut frame = Vec::with_capacity(std::mem::size_of::() + bytes.len()); + frame.extend_from_slice(&len.to_be_bytes()); + frame.extend_from_slice(&bytes); + Ok(frame) +} diff --git a/src/control_plane/membership/actor.rs b/src/control_plane/membership/actor.rs index ff3fda0b..6e4fe9c2 100644 --- a/src/control_plane/membership/actor.rs +++ b/src/control_plane/membership/actor.rs @@ -144,9 +144,18 @@ pub(crate) enum ShardRouting { /// This node hosts the key's shard group — proceed locally. Carries the group /// for callers that need its members/id (control-plane writes). Local(ShardGroup), - /// Not local; redirect the client to a member. `None` until the ring/addresses - /// converge here — still retriable. - Redirect(Option), + /// A client should retry elsewhere. `None` means the ring cannot map the + /// key yet; a remote shard can still lack an address while SWIM converges. + Redirect(Option), +} + +/// A shard this node does not host, with an optional reachable member. +/// +/// The shard ID lets internal callers validate cached records against the +/// current owner even when the client-facing redirect has no usable address. +pub(crate) struct RemoteShard { + pub(crate) group_id: ShardGroupId, + pub(crate) member: Option, } #[derive(Clone, Debug)] @@ -229,8 +238,9 @@ impl SwimSender { Ok(None) } - /// Route a key relative to `node_id`: `Local` if it hosts the key's shard group, - /// else a `Redirect` to a resolvable member (no hint until the ring converges). + /// Route a key relative to `node_id`: `Local` if it hosts the key's shard + /// group, otherwise a redirect. A resolved remote shard retains its ID for + /// internal ACL refreshes; an unresolved ring has no remote shard yet. pub(crate) async fn resolve_shard_routing( &self, key: Vec, @@ -243,7 +253,10 @@ impl SwimSender { return Ok(ShardRouting::Local(group)); } let member = self.resolve_any(&group.replicas).await?; - Ok(ShardRouting::Redirect(member)) + Ok(ShardRouting::Redirect(Some(RemoteShard { + group_id: group.id, + member, + }))) } pub(crate) async fn list_all_node_addresses( diff --git a/src/control_plane/membership/tests.rs b/src/control_plane/membership/tests.rs index 5681c43f..8c4273c8 100644 --- a/src/control_plane/membership/tests.rs +++ b/src/control_plane/membership/tests.rs @@ -7,7 +7,7 @@ use crate::control_plane::membership::actor::SwimActor; use crate::control_plane::membership::peer_discovery::JoinConfig; use crate::control_plane::membership::swim::Swim; use crate::control_plane::membership::{ - self, DIRECT_ACK_TIMEOUT_TICKS, OutboundPacket, QueryCommand, SwimActorCommand, SwimCommand, + DIRECT_ACK_TIMEOUT_TICKS, OutboundPacket, QueryCommand, SwimActorCommand, SwimCommand, SwimHeader, SwimPacket, SwimTimer, Topology, TopologyConfig, }; use crate::control_plane::{NodeAddress, NodeId, SwimNode, SwimNodeState}; @@ -162,7 +162,7 @@ async fn setup_with_config(port: u32, join_config: JoinConfig) -> TestHarness { // Topology publish/read channel — this harness has no reader-side // consumer, but SwimActor still needs the writer half to publish into. - let (topology_pub, _topology_reader) = membership::topology_channel(swim.topology.clone()); + let (topology_pub, _topology_reader) = swim.topology.clone().channel(); let ticker_tx = spawn_scheduling_actor( tx_in.clone(), diff --git a/src/control_plane/membership/topology.rs b/src/control_plane/membership/topology.rs index 1ccba559..07f2f753 100644 --- a/src/control_plane/membership/topology.rs +++ b/src/control_plane/membership/topology.rs @@ -149,17 +149,6 @@ impl TopologyReader { } } -/// Construct a (publisher, reader) pair sharing one underlying `ArcSwap`. -/// -/// The publisher half stays with `SwimActor` (single writer); the reader can -/// be cloned freely to any number of consumers. Both see the same atomic slot; -/// what differs is the API surface — readers can only `load()`. -pub(crate) fn topology_channel(initial: Topology) -> (Arc>, TopologyReader) { - let arc = Arc::new(ArcSwap::from_pointee(initial)); - let reader = TopologyReader(arc.clone()); - (arc, reader) -} - impl Topology { pub fn new(nodes: impl IntoIterator, config: TopologyConfig) -> Self { let mut topology = Self { @@ -181,6 +170,17 @@ impl Topology { topology } + /// Construct a (publisher, reader) pair sharing one underlying `ArcSwap`. + /// + /// The publisher half stays with `SwimActor` (single writer); the reader can + /// be cloned freely to any number of consumers. Both see the same atomic slot; + /// what differs is the API surface — readers can only `load()`. + pub(crate) fn channel(self) -> (Arc>, TopologyReader) { + let arc = Arc::new(ArcSwap::from_pointee(self)); + let reader = TopologyReader(arc.clone()); + (arc, reader) + } + /// Consume the dirty flag. Returns true if the topology has been mutated /// since the last `take_dirty()` call (or since construction). The owning /// SwimActor calls this at the end of each event-loop iteration to decide @@ -838,7 +838,7 @@ mod tests { replication_factor: 2, }, ); - let (_pub_handle, reader) = topology_channel(topology); + let (_pub_handle, reader) = topology.channel(); assert_eq!(reader.live_nodes().len(), 2); } @@ -852,7 +852,7 @@ mod tests { replication_factor: 1, }, ); - let (pub_handle, reader) = topology_channel(topology); + let (pub_handle, reader) = topology.channel(); assert_eq!(reader.live_nodes().len(), 1); // Simulate what SwimActor does at the end of an iteration: build a fresh @@ -878,7 +878,7 @@ mod tests { replication_factor: 1, }, ); - let (pub_handle, reader1) = topology_channel(topology); + let (pub_handle, reader1) = topology.channel(); let reader2 = reader1.clone(); pub_handle.store(Arc::new(topology_from( diff --git a/src/control_plane/membership/transport.rs b/src/control_plane/membership/transport.rs index 7b10cdd4..a021fe60 100644 --- a/src/control_plane/membership/transport.rs +++ b/src/control_plane/membership/transport.rs @@ -1,10 +1,7 @@ -use crate::control_plane::membership::{OutboundPacket, SwimCommand, actor::SwimSender}; +use tokio::sync::mpsc; -// ========================================== -// TRANSPORT LAYER (Presentation) -// ========================================== +use crate::control_plane::membership::{OutboundPacket, SwimCommand, actor::SwimSender}; use crate::net::UdpSocket; -use tokio::sync::mpsc; pub struct SwimTransportActor; @@ -15,7 +12,7 @@ impl SwimTransportActor { mut from_actor: mpsc::Receiver>, ) { tracing::info!( - "Transport Layer listening on {}", + "SWIM trusted-development transport listening on {}", socket.local_addr().unwrap() ); diff --git a/src/control_plane/metadata/command.rs b/src/control_plane/metadata/command.rs index de8395b4..53fa5cba 100644 --- a/src/control_plane/metadata/command.rs +++ b/src/control_plane/metadata/command.rs @@ -4,10 +4,13 @@ use borsh::{BorshDeserialize, BorshSerialize}; use uuid::Uuid; use crate::{ - connections::protocol::ConsumerGroupSyncAction, + connections::protocol::ConsumerGroupMemberAction, control_plane::{ Replicas, - metadata::{EntryId, RangeId, SegmentId, TopicId, strategy::StoragePolicy}, + metadata::{ + AclResource, EntryId, ProducerSessionOwner, RangeId, SegmentId, TopicId, + strategy::StoragePolicy, + }, }, data_plane::SegmentKey, impl_from_variant, @@ -84,8 +87,8 @@ pub struct DeleteSegments { } #[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] -pub struct SyncConsumerGroup { - pub req: SyncConsumerGroupRequest, +pub struct UpdateConsumerGroupMember { + pub req: UpdateConsumerGroupMemberRequest, // TODO consider using logical clock pub observed_at: u64, pub session_timeout_ms: u64, @@ -93,9 +96,10 @@ pub struct SyncConsumerGroup { #[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] pub struct OpenProducerSession { - pub topic_name: String, + pub topic_name: Box, pub producer_id: Uuid, pub session_nonce: Uuid, + pub owner: ProducerSessionOwner, pub observed_at: u64, pub session_timeout_ms: u64, } @@ -107,26 +111,38 @@ pub struct ExpireProducerSessions { } #[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] -pub struct SyncConsumerGroupRequest { +pub struct GrantAcl { + pub resource: AclResource, + pub principal: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] +pub struct RevokeAcl { + pub resource: AclResource, + pub principal: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] +pub struct UpdateConsumerGroupMemberRequest { pub topic_name: String, pub group_id: String, pub member_id: Uuid, - pub action: ConsumerGroupSyncAction, + pub action: ConsumerGroupMemberAction, } -impl Deref for SyncConsumerGroup { - type Target = SyncConsumerGroupRequest; +impl Deref for UpdateConsumerGroupMember { + type Target = UpdateConsumerGroupMemberRequest; fn deref(&self) -> &Self::Target { &self.req } } -impl SyncConsumerGroup { - pub(crate) fn new(req: SyncConsumerGroupRequest) -> Self { +impl UpdateConsumerGroupMember { + pub(crate) fn new(req: UpdateConsumerGroupMemberRequest) -> Self { const SESSION_TIMEOUT_MS: u64 = 10_000; let observed_at = crate::now_ms(); - SyncConsumerGroup { + UpdateConsumerGroupMember { req, observed_at, session_timeout_ms: SESSION_TIMEOUT_MS, @@ -143,9 +159,11 @@ pub enum MetadataCommand { DeleteTopic(DeleteTopic), ReassignSegment(ReassignSegment), DeleteSegments(DeleteSegments), - SyncConsumerGroup(SyncConsumerGroup), + UpdateConsumerGroupMember(UpdateConsumerGroupMember), OpenProducerSession(OpenProducerSession), ExpireProducerSessions(ExpireProducerSessions), + GrantAcl(GrantAcl), + RevokeAcl(RevokeAcl), } impl_from_variant!( @@ -157,7 +175,9 @@ impl_from_variant!( DeleteTopic, ReassignSegment, DeleteSegments, - SyncConsumerGroup, + UpdateConsumerGroupMember, OpenProducerSession, - ExpireProducerSessions + ExpireProducerSessions, + GrantAcl, + RevokeAcl ); diff --git a/src/control_plane/metadata/consumer_group.rs b/src/control_plane/metadata/consumer_group.rs index d4b1b365..8780073d 100644 --- a/src/control_plane/metadata/consumer_group.rs +++ b/src/control_plane/metadata/consumer_group.rs @@ -3,7 +3,7 @@ use std::collections::BTreeMap; use borsh::{BorshDeserialize, BorshSerialize}; use uuid::Uuid; -use crate::{connections::protocol::ConsumerGroupSyncAction, impl_new_struct_wrapper}; +use crate::{connections::protocol::ConsumerGroupMemberAction, impl_new_struct_wrapper}; use super::RangeId; @@ -38,7 +38,7 @@ impl ConsumerGroupMeta { pub(crate) fn sync_member( &mut self, member_id: ConsumerMemberId, - action: ConsumerGroupSyncAction, + action: ConsumerGroupMemberAction, observed_at: u64, session_timeout_ms: u64, active_ranges: &[RangeId], @@ -48,10 +48,10 @@ impl ConsumerGroupMeta { .retain(|_, last_seen| observed_at.saturating_sub(*last_seen) <= session_timeout_ms); match action { - ConsumerGroupSyncAction::Heartbeat => { + ConsumerGroupMemberAction::Heartbeat => { self.members.insert(member_id, observed_at); } - ConsumerGroupSyncAction::Leave => { + ConsumerGroupMemberAction::Leave => { self.members.remove(&member_id); } } @@ -150,14 +150,14 @@ mod tests { let a = uuid::Uuid::new_v4(); let b = uuid::Uuid::new_v4(); - assert!(group.sync_member(a, ConsumerGroupSyncAction::Heartbeat, 1, 10, &ranges)); + assert!(group.sync_member(a, ConsumerGroupMemberAction::Heartbeat, 1, 10, &ranges)); assert_eq!(*group.generation, 1); assert_eq!(group.ranges_for(a).as_ref(), &ranges); - assert!(!group.sync_member(a, ConsumerGroupSyncAction::Heartbeat, 2, 10, &ranges)); + assert!(!group.sync_member(a, ConsumerGroupMemberAction::Heartbeat, 2, 10, &ranges)); assert_eq!(*group.generation, 1); - assert!(group.sync_member(b, ConsumerGroupSyncAction::Heartbeat, 3, 10, &ranges)); + assert!(group.sync_member(b, ConsumerGroupMemberAction::Heartbeat, 3, 10, &ranges)); assert_eq!(*group.generation, 2); group.assert_assignments(&ranges); } @@ -168,10 +168,10 @@ mod tests { let ranges = [RangeId(1), RangeId(2)]; let stale = uuid::Uuid::new_v4(); let live = uuid::Uuid::new_v4(); - group.sync_member(stale, ConsumerGroupSyncAction::Heartbeat, 1, 10, &ranges); - group.sync_member(live, ConsumerGroupSyncAction::Heartbeat, 2, 10, &ranges); + group.sync_member(stale, ConsumerGroupMemberAction::Heartbeat, 1, 10, &ranges); + group.sync_member(live, ConsumerGroupMemberAction::Heartbeat, 2, 10, &ranges); - assert!(group.sync_member(live, ConsumerGroupSyncAction::Heartbeat, 20, 10, &ranges)); + assert!(group.sync_member(live, ConsumerGroupMemberAction::Heartbeat, 20, 10, &ranges)); assert_eq!(*group.generation, 3); assert!(group.ranges_for(stale).is_empty()); assert_eq!(group.ranges_for(live).as_ref(), &ranges); diff --git a/src/control_plane/metadata/error.rs b/src/control_plane/metadata/error.rs index f6e63147..92f7e561 100644 --- a/src/control_plane/metadata/error.rs +++ b/src/control_plane/metadata/error.rs @@ -26,4 +26,6 @@ pub enum MetadataError { RangesNotAdjacent, #[error("invalid split point")] InvalidSplitPoint, + #[error("produce session belongs to another principal")] + ProducerSessionOwnerMismatch, } diff --git a/src/control_plane/metadata/mod.rs b/src/control_plane/metadata/mod.rs index 3b5c1693..fb92a4c5 100644 --- a/src/control_plane/metadata/mod.rs +++ b/src/control_plane/metadata/mod.rs @@ -7,6 +7,7 @@ pub(crate) mod event; pub(crate) mod range; mod producer_sessions; + pub mod strategy; pub(crate) mod topic; @@ -16,19 +17,141 @@ pub(crate) use topic::{TopicMeta, TopicState, TopicStats}; pub(crate) mod segment; use borsh::{BorshDeserialize as Deser, BorshSerialize as Ser}; +use uuid::Uuid; pub(crate) use command::*; pub(crate) use consumer_group::{ConsumerGroupAssignment, ConsumerGroupMeta, ConsumerMemberId}; pub(crate) use segment::*; -use crate::impl_new_struct_wrapper; +use crate::{impl_new_struct_wrapper, security::CertificatePrincipal}; #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Ser, Deser)] pub struct TopicId(pub(crate) u64); impl_new_struct_wrapper!(TopicId, u64); +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Ser, Deser)] +pub enum AclResource { + Cluster, + TopicAdmin(TopicId), + TopicData(TopicId), + ConsumerGroup(ConsumerGroupResource), + ProducerSession(ProducerSessionResource), + SecurityCluster, +} + +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Ser, Deser)] +pub struct ConsumerGroupResource { + pub topic_id: TopicId, + pub group_id: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Ser, Deser)] +pub struct ProducerSessionResource { + pub topic_id: TopicId, + pub producer_id: Uuid, +} + +/// Durable owner of one producer session. +/// +/// This is deliberately separate from the connection's optional certificate +/// principal: Raft snapshots retain ownership after the TLS connection +/// disappears, and trusted-development ownership must remain explicit in +/// persisted metadata rather than being encoded as an absent value. +#[derive(Debug, Clone, PartialEq, Eq, Hash, Ser, Deser)] +pub enum ProducerSessionOwner { + CertificatePrincipal(Box), + TrustedDevelopment, +} + +impl From> for ProducerSessionOwner { + fn from(certificate_principal: Option<&CertificatePrincipal>) -> Self { + match certificate_principal { + Some(principal) => Self::CertificatePrincipal(principal.as_ref().into()), + None => Self::TrustedDevelopment, + } + } +} + +impl AclResource { + pub(crate) fn routing_key(&self) -> Vec { + self.to_string().into_bytes() + } +} + +impl std::fmt::Display for AclResource { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Cluster => formatter.write_str("cluster"), + Self::TopicAdmin(topic_id) => write!(formatter, "topic-admin/{}", topic_id.0), + Self::TopicData(topic_id) => write!(formatter, "topic-data/{}", topic_id.0), + Self::ConsumerGroup(resource) => write!( + formatter, + "consumer-group/{}/{}", + resource.topic_id.0, resource.group_id + ), + Self::ProducerSession(resource) => write!( + formatter, + "producer-session/{}/{}", + resource.topic_id.0, resource.producer_id + ), + Self::SecurityCluster => formatter.write_str("security/cluster"), + } + } +} + +impl std::str::FromStr for AclResource { + type Err = String; + + fn from_str(input: &str) -> Result { + let topic_id = |candidate: &str| { + candidate + .parse::() + .map(TopicId) + .map_err(|_| format!("invalid ACL topic ID: {candidate}")) + }; + + if input == "cluster" { + return Ok(Self::Cluster); + } + if input == "security/cluster" { + return Ok(Self::SecurityCluster); + } + if let Some(topic) = input.strip_prefix("topic-admin/") { + return topic_id(topic).map(Self::TopicAdmin); + } + if let Some(topic) = input.strip_prefix("topic-data/") { + return topic_id(topic).map(Self::TopicData); + } + if let Some(consumer_group) = input.strip_prefix("consumer-group/") { + let (topic, group_id) = consumer_group + .split_once('/') + .ok_or_else(|| "consumer-group ACL requires a group ID".to_string())?; + if group_id.is_empty() { + return Err("consumer-group ACL requires a non-empty group ID".to_string()); + } + return Ok(Self::ConsumerGroup(ConsumerGroupResource { + topic_id: topic_id(topic)?, + group_id: group_id.to_string(), + })); + } + if let Some(producer_session) = input.strip_prefix("producer-session/") { + let (topic, producer_id) = producer_session + .split_once('/') + .ok_or_else(|| "producer-session ACL requires a producer ID".to_string())?; + let producer_id = Uuid::parse_str(producer_id) + .map_err(|_| format!("invalid ACL producer ID: {producer_id}"))?; + return Ok(Self::ProducerSession(ProducerSessionResource { + topic_id: topic_id(topic)?, + producer_id, + })); + } + + Err(format!("unknown ACL resource: {input}")) + } +} + #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Ser, Deser, PartialOrd, Ord)] pub struct RangeId(pub(crate) u64); @@ -156,3 +279,49 @@ impl std::ops::Sub for EntryId { } } } + +#[cfg(test)] +mod acl_resource_tests { + use super::*; + + #[test] + fn resources_round_trip_through_canonical_routing_keys() { + let producer_id = Uuid::parse_str("6ba7b810-9dad-11d1-80b4-00c04fd430c8").unwrap(); + let resources = [ + AclResource::Cluster, + AclResource::TopicAdmin(TopicId(42)), + AclResource::TopicData(TopicId(42)), + AclResource::ConsumerGroup(ConsumerGroupResource { + topic_id: TopicId(42), + group_id: "billing/readers".to_string(), + }), + AclResource::ProducerSession(ProducerSessionResource { + topic_id: TopicId(42), + producer_id, + }), + AclResource::SecurityCluster, + ]; + + for resource in resources { + let key = resource.to_string(); + assert_eq!(key.parse::(), Ok(resource.clone())); + assert_eq!(resource.routing_key(), key.as_bytes()); + } + } + + #[test] + fn malformed_resource_keys_are_rejected() { + for key in [ + "", + "topic-data/", + "topic-data/name", + "topic-data/42/extra", + "consumer-group/42", + "consumer-group/42/", + "producer-session/42/not-a-uuid", + "unknown/42", + ] { + assert!(key.parse::().is_err(), "{key} was accepted"); + } + } +} diff --git a/src/control_plane/metadata/producer_sessions.rs b/src/control_plane/metadata/producer_sessions.rs index 866480fc..0f8fe63b 100644 --- a/src/control_plane/metadata/producer_sessions.rs +++ b/src/control_plane/metadata/producer_sessions.rs @@ -2,12 +2,16 @@ use std::collections::HashMap; use borsh::{BorshDeserialize, BorshSerialize}; +use crate::control_plane::metadata::ProducerSessionOwner; +use crate::control_plane::metadata::command::OpenProducerSession; +use crate::control_plane::metadata::error::MetadataError; use crate::impl_new_struct_wrapper; -#[derive(Debug, Clone, Copy, PartialEq, Eq, BorshSerialize, BorshDeserialize)] +#[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] pub(crate) struct ProducerSessionMeta { pub(crate) incarnation: u32, pub(crate) expires_at: u64, + pub(crate) owner: ProducerSessionOwner, session_nonce: uuid::Uuid, } @@ -16,28 +20,41 @@ pub(crate) struct ProducerSessions(HashMap); impl_new_struct_wrapper!(ProducerSessions,HashMap); impl ProducerSessions { + pub(crate) fn get_for_owner( + &self, + producer_id: &uuid::Uuid, + owner: &ProducerSessionOwner, + ) -> Result, MetadataError> { + match self.get(producer_id) { + Some(session) if &session.owner != owner => { + Err(MetadataError::ProducerSessionOwnerMismatch) + } + session => Ok(session), + } + } + pub(crate) fn open_producer_session( &mut self, - producer_id: uuid::Uuid, - session_nonce: uuid::Uuid, - observed_at: u64, - session_timeout_ms: u64, - ) -> ProducerSessionMeta { - self.expire_producer_sessions(observed_at); + command: OpenProducerSession, + ) -> Result<(), MetadataError> { + self.expire_producer_sessions(command.observed_at); - let incarnation = match self.get(&producer_id) { - Some(session) if session.session_nonce == session_nonce => session.incarnation, + let incarnation = match self.get_for_owner(&command.producer_id, &command.owner)? { + Some(session) if session.session_nonce == command.session_nonce => session.incarnation, Some(session) => session.incarnation.saturating_add(1), None => 0, }; - let expires_at = observed_at.saturating_add(session_timeout_ms); + let expires_at = command + .observed_at + .saturating_add(command.session_timeout_ms); let session = ProducerSessionMeta { incarnation, expires_at, - session_nonce, + owner: command.owner, + session_nonce: command.session_nonce, }; - self.insert(producer_id, session); - session + self.insert(command.producer_id, session); + Ok(()) } pub(crate) fn has_expired_producer_sessions(&self, observed_at: u64) -> bool { @@ -55,14 +72,43 @@ fn producer_session_recovery_bumps_incarnation_and_expiry_removes_it() { let mut producer_sessions = ProducerSessions::default(); let producer_id = uuid::Uuid::new_v4(); let first_nonce = uuid::Uuid::new_v4(); + let owner = ProducerSessionOwner::CertificatePrincipal("orders-service".into()); + + let command = |session_nonce, session_owner, observed_at| OpenProducerSession { + topic_name: "orders".into(), + producer_id, + session_nonce, + owner: session_owner, + observed_at, + session_timeout_ms: 100, + }; - let session = producer_sessions.open_producer_session(producer_id, first_nonce, 10, 100); + producer_sessions + .open_producer_session(command(first_nonce, owner.clone(), 10)) + .unwrap(); + let session = &producer_sessions[&producer_id]; assert_eq!((session.incarnation, session.expires_at), (0, 110)); - let session2 = producer_sessions.open_producer_session(producer_id, first_nonce, 20, 100); + + producer_sessions + .open_producer_session(command(first_nonce, owner.clone(), 20)) + .unwrap(); + let session2 = &producer_sessions[&producer_id]; assert_eq!((session2.incarnation, session2.expires_at), (0, 120)); - let session3 = - producer_sessions.open_producer_session(producer_id, uuid::Uuid::new_v4(), 20, 100); + + producer_sessions + .open_producer_session(command(uuid::Uuid::new_v4(), owner.clone(), 20)) + .unwrap(); + let session3 = &producer_sessions[&producer_id]; assert_eq!((session3.incarnation, session3.expires_at), (1, 120)); + assert_eq!( + producer_sessions.open_producer_session(command( + uuid::Uuid::new_v4(), + ProducerSessionOwner::CertificatePrincipal("other-service".into()), + 20, + )), + Err(MetadataError::ProducerSessionOwnerMismatch) + ); + assert_eq!(producer_sessions[&producer_id].owner, owner); assert!(producer_sessions.has_expired_producer_sessions(121)); producer_sessions.expire_producer_sessions(121); diff --git a/src/control_plane/metadata/topic.rs b/src/control_plane/metadata/topic.rs index 7b30dce2..7cf76139 100644 --- a/src/control_plane/metadata/topic.rs +++ b/src/control_plane/metadata/topic.rs @@ -449,7 +449,7 @@ impl TopicMeta { Ok(merged_id) } - pub(crate) fn sync_consumer_group(&mut self, cmd: SyncConsumerGroup) -> bool { + pub(crate) fn sync_consumer_group(&mut self, cmd: UpdateConsumerGroupMember) -> bool { let consumer_group_meta = self .consumer_groups .entry(cmd.group_id.clone()) diff --git a/src/data_plane/auxiliary_states/consumer_offsets/state.rs b/src/data_plane/auxiliary_states/consumer_offsets/state.rs index 5f019576..779648d5 100644 --- a/src/data_plane/auxiliary_states/consumer_offsets/state.rs +++ b/src/data_plane/auxiliary_states/consumer_offsets/state.rs @@ -5,7 +5,7 @@ use borsh::{BorshDeserialize, BorshSerialize}; use crate::client::RangeId; use crate::control_plane::metadata::consumer_group::GenerationId; -use crate::control_plane::metadata::{EntryId, TopicId}; +use crate::control_plane::metadata::{AclResource, ConsumerGroupResource, EntryId, TopicId}; use crate::data_plane::SegmentKey; #[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, BorshSerialize, BorshDeserialize)] @@ -19,6 +19,12 @@ impl ConsumerOffsetKey { pub(crate) fn placement_key(&self) -> (TopicId, RangeId) { (self.topic_id, self.range_id) } + pub(crate) fn acl(&self) -> AclResource { + AclResource::ConsumerGroup(ConsumerGroupResource { + topic_id: self.topic_id, + group_id: self.group_id.clone(), + }) + } } #[derive(Default, Debug, Clone, Copy, PartialEq, Eq, BorshSerialize, BorshDeserialize)] diff --git a/src/data_plane/state.rs b/src/data_plane/state.rs index c3f7e3c3..bf8facfd 100644 --- a/src/data_plane/state.rs +++ b/src/data_plane/state.rs @@ -3682,8 +3682,8 @@ mod tests { /// `WalRecord` format agreement with `checkpoint.rs`, and `record_count` /// round-tripping through the file. Bypasses the coordinator seal round-trip /// (driven directly) so it's deterministic. - #[test] - fn cold_fetch_serves_sealed_segment_from_disk() { + #[tokio::test] + async fn cold_fetch_serves_sealed_segment_from_disk() { use crate::connections::protocol::RangeProgressSignal; use crate::data_plane::cold_read::ColdReadPool; use crate::data_plane::messages::query::{DataPlaneQuery, Fetch}; @@ -3777,9 +3777,7 @@ mod tests { reply, }))); - let result = reply_rx - .blocking_recv() - .expect("cold-read pool dropped the reply"); + let result = reply_rx.await.expect("cold-read pool dropped the reply"); let FetchedRecords { entries, next_entry_id, diff --git a/src/data_plane/transport/mod.rs b/src/data_plane/transport/mod.rs index 480277ec..5f87367c 100644 --- a/src/data_plane/transport/mod.rs +++ b/src/data_plane/transport/mod.rs @@ -9,7 +9,8 @@ use crate::control_plane::membership::TopologyReader; use crate::control_plane::membership::actor::SwimSender; use crate::data_plane::actor::DataPlaneSender; -use crate::net::TcpListener; +use crate::net::{TcpListener, TransportTcpStream}; +use crate::security::{NodeTransportSecurity, node_certificate_principal}; use command::DataTransportCommand; use writers::TransportState; @@ -24,8 +25,9 @@ impl DataTransportActor { mut from_actor: mpsc::Receiver>, swim_tx: SwimSender, topology: TopologyReader, + security: NodeTransportSecurity, ) { - let mut state = TransportState::new(node_id); + let mut state = TransportState::new(node_id, security.clone()); let mut cleanup_interval = tokio::time::interval(std::time::Duration::from_secs(300)); cleanup_interval.tick().await; @@ -67,6 +69,17 @@ impl DataTransportActor { } Ok((stream, _)) = listener.accept() => { + let stream = match TransportTcpStream::accept( + stream, + &security, + node_certificate_principal, + ).await { + Ok(stream) => stream, + Err(error) => { + tracing::debug!("Data TLS accept rejected: {error}"); + continue; + } + }; match state.accept(stream).await { Ok((peer, reader)) => { tokio::spawn(reader.run(data_plane_tx.clone(), peer, disconnect_tx.clone())); diff --git a/src/data_plane/transport/reader.rs b/src/data_plane/transport/reader.rs index b6a62a58..608bf18f 100644 --- a/src/data_plane/transport/reader.rs +++ b/src/data_plane/transport/reader.rs @@ -4,19 +4,33 @@ use tokio::sync::mpsc; use crate::control_plane::NodeId; use crate::data_plane::actor::DataPlaneSender; use crate::data_plane::messages::command::{DataPlaneCommand, ReceivePeerMessage}; -use crate::net::OwnedReadHalf; +use crate::net::TransportReadHalf; +use crate::security::CertificatePrincipal; const NODE_ID_FRAME_MAX: usize = 1024; const DATA_FRAME_MAX: usize = 64 * 1024 * 1024; -pub(super) struct DataReader(pub OwnedReadHalf); +pub(super) struct DataReader { + read_half: TransportReadHalf, + certificate_principal: Option, +} impl DataReader { + pub(super) fn new( + read_half: TransportReadHalf, + certificate_principal: Option, + ) -> Self { + Self { + read_half, + certificate_principal, + } + } + async fn read_frame(&mut self, max: usize) -> anyhow::Result { - let len = self.0.read_u32().await? as usize; + let len = self.read_half.read_u32().await? as usize; anyhow::ensure!(len <= max, "frame too large: {len} bytes (max {max})"); let mut buf = vec![0u8; len]; - self.0.read_exact(&mut buf).await?; + self.read_half.read_exact(&mut buf).await?; let val = borsh::from_slice::(&buf)?; Ok(val) } @@ -25,6 +39,11 @@ impl DataReader { self.read_frame(NODE_ID_FRAME_MAX).await } + #[tracing::instrument( + level = "trace", + skip_all, + fields(peer = %peer, certificate_principal = ?self.certificate_principal) + )] pub(crate) async fn run( mut self, data_plane_tx: DataPlaneSender, diff --git a/src/data_plane/transport/writers.rs b/src/data_plane/transport/writers.rs index 22725ae1..558863f9 100644 --- a/src/data_plane/transport/writers.rs +++ b/src/data_plane/transport/writers.rs @@ -11,7 +11,8 @@ use crate::data_plane::actor::DataPlaneSender; use crate::data_plane::messages::command::{ DataPlaneCommand, DataPlanePeerMessage, ReceivePeerMessage, }; -use crate::net::{OwnedWriteHalf, TcpStream}; +use crate::net::{TransportTcpStream, TransportWriteHalf}; +use crate::security::NodeTransportSecurity; use super::reader::DataReader; @@ -19,30 +20,33 @@ const CONNECT_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2); pub(super) struct TransportState { node_id: NodeId, - writers: HashMap, + writers: HashMap, dead_peers: HashSet, /// Tracks when the last connect attempt to a peer failed. Skips retry /// for CONNECT_BACKOFF (2s) to avoid blocking the select loop on repeated /// 3s TCP timeouts to unreachable peers. Cleared by periodic cleanup (300s). connect_backoffs: HashMap, + security: NodeTransportSecurity, } impl TransportState { - pub fn new(node_id: NodeId) -> Self { + pub fn new(node_id: NodeId, security: NodeTransportSecurity) -> Self { Self { node_id, writers: HashMap::new(), dead_peers: HashSet::new(), connect_backoffs: HashMap::new(), + security, } } pub async fn accept( &mut self, - stream: crate::net::TcpStream, + stream: TransportTcpStream, ) -> anyhow::Result<(NodeId, DataReader)> { + let certificate_principal = stream.peer_principal(); let (read_half, write_half) = stream.into_split(); - let mut reader = DataReader(read_half); + let mut reader = DataReader::new(read_half, certificate_principal); let peer_id = reader.read_node_id().await?; @@ -132,12 +136,13 @@ impl TransportState { let stream = tokio::time::timeout( std::time::Duration::from_secs(3), - TcpStream::connect(node_addr.data_addr()), + TransportTcpStream::connect_node(node_addr.data_addr(), &self.security), ) .await .context("connect timed out")? .context("TCP connect failed")?; + let certificate_principal = stream.peer_principal(); let (read_half, write_half) = stream.into_split(); self.writers.insert(target_id.clone(), write_half); @@ -151,7 +156,7 @@ impl TransportState { return Err(e).context("initial write failed"); } - Ok(DataReader(read_half)) + Ok(DataReader::new(read_half, certificate_principal)) } pub fn disconnect(&mut self, peer_id: NodeId) { diff --git a/src/it/helpers.rs b/src/it/helpers.rs index 9ad2cc3c..4b1d7d84 100644 --- a/src/it/helpers.rs +++ b/src/it/helpers.rs @@ -1,5 +1,5 @@ use crate::client::ClientSuccess; -use crate::config::Environment; +use crate::config::{Environment, SecurityMode}; use crate::connections::protocol::{AdminRequest, ClientRequest, ClientResponse, NodeState}; use crate::connections::reader::ClientStreamReader; use crate::connections::writer::ClientRawWriter; @@ -9,6 +9,10 @@ use crate::net::TcpStream; pub fn default_env(idx: u32, node_id: String, client_port: u16, cluster_port: u16) -> Environment { Environment { + security_mode: SecurityMode::TrustedDevelopment, + certificate_chain_path: None, + private_key_path: None, + trust_root_path: None, config_dir: std::env::temp_dir() .join(format!("eastguard-config-{}-{}", idx, uuid::Uuid::new_v4())) .to_string_lossy() diff --git a/src/it/raft/election.rs b/src/it/raft/election.rs index ae901097..e83d1156 100644 --- a/src/it/raft/election.rs +++ b/src/it/raft/election.rs @@ -15,6 +15,7 @@ use crate::control_plane::membership::{ShardGroup, ShardGroupId}; use crate::control_plane::{NodeId, Replicas}; use crate::impls::metadata_storage::MetadataStorage; use crate::net::{TcpListener, TcpStream}; +use crate::security::{NodeTransportSecurity, SecurityActor}; use crate::schedulers::actor::spawn_scheduling_actor; use crate::schedulers::ticker::{PROBE_INTERVAL_TICKS, TICK_PERIOD_100_MS}; @@ -103,12 +104,19 @@ async fn run_raft_node( TICK_PERIOD_100_MS, Some(PROBE_INTERVAL_TICKS), ); + let security = SecurityActor::spawn( + node_id.clone(), + swim_tx.clone(), + raft_tx.clone(), + NodeTransportSecurity::TrustedDevelopment, + ); tokio::spawn(RaftTransportActor::run( node_id.clone(), listener, raft_tx.clone(), transport_rx, swim_tx.clone(), + security, )); let db = MetadataStorage::open(std::env::temp_dir().join(uuid::Uuid::new_v4().to_string())); let election_jitter_seed = { diff --git a/src/it/raft/leader_event.rs b/src/it/raft/leader_event.rs index bc71ce25..192ed053 100644 --- a/src/it/raft/leader_event.rs +++ b/src/it/raft/leader_event.rs @@ -19,6 +19,7 @@ use crate::impls::metadata_storage::MetadataStorage; use crate::net::{TcpListener, TcpStream}; use crate::schedulers::actor::spawn_scheduling_actor; use crate::schedulers::ticker::{PROBE_INTERVAL_TICKS, TICK_PERIOD_100_MS}; +use crate::security::{NodeTransportSecurity, SecurityActor}; use super::CLUSTER_PORT; @@ -114,12 +115,19 @@ fn leader_election_emits_leader_change_event() -> turmoil::Result { TICK_PERIOD_100_MS, Some(PROBE_INTERVAL_TICKS), ); + let security = SecurityActor::spawn( + node_id.clone(), + swim_tx.clone(), + raft_tx.clone(), + NodeTransportSecurity::TrustedDevelopment, + ); tokio::spawn(RaftTransportActor::run( node_id.clone(), listener, raft_tx.clone(), transport_rx, swim_tx.clone(), + security, )); let db = MetadataStorage::open( std::env::temp_dir().join(uuid::Uuid::new_v4().to_string()), diff --git a/src/it/raft/membership_change.rs b/src/it/raft/membership_change.rs index 0e968c96..8371bd0a 100644 --- a/src/it/raft/membership_change.rs +++ b/src/it/raft/membership_change.rs @@ -20,6 +20,7 @@ use crate::net::{TcpListener, TcpStream}; use crate::schedulers::actor::spawn_scheduling_actor; use crate::schedulers::ticker::{PROBE_INTERVAL_TICKS, TICK_PERIOD_100_MS}; use crate::schedulers::ticker_message::{SchedulerSender, TickerCommand}; +use crate::security::{NodeTransportSecurity, SecurityActor}; use super::{CLUSTER_PORT, QUERY_PORT, mock_swim_handler}; @@ -68,12 +69,19 @@ async fn start_raft_node( Some(PROBE_INTERVAL_TICKS), ); + let security = SecurityActor::spawn( + node_id.clone(), + swim_tx.clone(), + raft_tx.clone(), + NodeTransportSecurity::TrustedDevelopment, + ); tokio::spawn(RaftTransportActor::run( node_id.clone(), listener, raft_tx.clone(), transport_rx, swim_tx.clone(), + security, )); let db = MetadataStorage::open(std::env::temp_dir().join(uuid::Uuid::new_v4().to_string())); let election_jitter_seed = { diff --git a/src/it/raft/mod.rs b/src/it/raft/mod.rs index 1cb1582d..be960e28 100644 --- a/src/it/raft/mod.rs +++ b/src/it/raft/mod.rs @@ -6,7 +6,7 @@ use arc_swap::ArcSwap; use tokio::sync::mpsc; use crate::control_plane::membership::{ - QueryCommand, SwimActorCommand, Topology, TopologyConfig, TopologyReader, topology_channel, + QueryCommand, SwimActorCommand, Topology, TopologyConfig, TopologyReader, }; use crate::control_plane::{NodeAddress, NodeId}; @@ -54,7 +54,7 @@ pub(super) fn stub_topology_channel( replication_factor: 3, }, ); - topology_channel(topology) + topology.channel() } mod election; diff --git a/src/it/raft/snapshot.rs b/src/it/raft/snapshot.rs index 9de420fb..33a8e005 100644 --- a/src/it/raft/snapshot.rs +++ b/src/it/raft/snapshot.rs @@ -19,6 +19,7 @@ use crate::impls::metadata_storage::MetadataStorage; use crate::net::{TcpListener, TcpStream}; use crate::schedulers::actor::spawn_scheduling_actor; use crate::schedulers::ticker::{PROBE_INTERVAL_TICKS, TICK_PERIOD_100_MS}; +use crate::security::{NodeTransportSecurity, SecurityActor}; use super::{CLUSTER_PORT, mock_swim_handler}; @@ -49,12 +50,19 @@ async fn run_node(name: &'static str, ordinal: u16) -> Result<(), Box Result<()> { + let security = NodeTransportSecurity::load(&self.env)?; + if security.is_secure() { + anyhow::bail!("secure transport listeners are not implemented"); + } + // Bind sockets before spawning — fail fast on port conflicts let udp_socket = UdpSocket::bind(self.env.peer_bind_addr()).await?; let tcp_listener = TcpListener::bind(self.env.peer_bind_addr()).await?; @@ -79,12 +88,18 @@ impl StartUp { let (data_transport_tx, data_transport_rx) = mpsc::channel::>(100); - let state = self.env.swim(self.rng_seed); - let node_id = state.node_id.clone(); + let swim = self.env.swim(self.rng_seed); + let node_id = swim.node_id.clone(); + let security_handle = SecurityActor::spawn( + node_id.clone(), + swim_sender.clone(), + raft_tx.clone(), + security.clone(), + ); // Topology snapshot channel: SwimActor publishes, all other actors read. // Single-writer / many-readers via ArcSwap — no locks, no contention. - let (topology_pub, topology_reader) = topology_channel(state.topology.clone()); + let (topology_pub, topology_reader) = swim.topology.clone().channel(); // Recover local durable state before this node serves or joins the // cluster: scan + replay the WAL into the segment files, then clear the @@ -106,13 +121,14 @@ impl StartUp { raft_tx.clone(), raft_transport_rx, swim_sender.clone(), + security_handle.clone(), )); // Protocol actors (each spawns its own scheduler internally) SwimActor::spawn( swim_sender.clone(), swim_mailbox, - state, + swim, tx_outbound, raft_tx.clone().into(), topology_pub, @@ -137,6 +153,7 @@ impl StartUp { data_transport_rx, swim_sender.clone(), topology_reader.clone(), + security.clone(), )); MultiRaftActor::spawn( @@ -159,7 +176,14 @@ impl StartUp { // Client handler let _ = self - .receive_client_streams(node_id, swim_sender, raft_tx, data_plane_tx) + .receive_client_streams( + node_id, + swim_sender, + raft_tx, + data_plane_tx, + security, + security_handle, + ) .await; Ok(()) } @@ -170,6 +194,8 @@ impl StartUp { swim_sender: SwimSender, raft_tx: MutlRaftSender, data_plane_tx: DataPlaneSender, + security: NodeTransportSecurity, + security_handle: SecurityHandle, ) { let addr = self.env.bind_addr(); let listener = TcpListener::bind(&addr).await.unwrap(); @@ -180,12 +206,30 @@ impl StartUp { ); while let Ok((stream, _)) = listener.accept().await { + let stream = + match TransportTcpStream::accept(stream, &security, client_certificate_principal) + .await + { + Ok(stream) => stream, + Err(error) => { + tracing::debug!("client authentication failed: {error}"); + continue; + } + }; let node_id = node_id.clone(); let swim_tx = swim_sender.clone(); let raft = raft_tx.clone(); let dp = data_plane_tx.clone(); - - tokio::spawn(handle_client_stream(stream, node_id, swim_tx, raft, dp)); + let security_handle = security_handle.clone(); + + tokio::spawn(handle_client_stream( + stream, + node_id, + swim_tx, + raft, + dp, + security_handle, + )); } } } diff --git a/src/macros/mod.rs b/src/macros/mod.rs index 08d0b4c3..b9e400d1 100644 --- a/src/macros/mod.rs +++ b/src/macros/mod.rs @@ -32,7 +32,6 @@ macro_rules! impl_new_struct_wrapper { $wrapper_name(val) } } - $crate::smart_pointer!($wrapper_name, $type); }; } diff --git a/src/net.rs b/src/net.rs deleted file mode 100644 index be50afb9..00000000 --- a/src/net.rs +++ /dev/null @@ -1,133 +0,0 @@ -#![allow(clippy::disallowed_types)] -use std::ops::{Deref, DerefMut}; -use std::pin::Pin; -use std::task::{Context, Poll}; -use tokio::io::{AsyncRead, AsyncWrite, ReadBuf}; - -#[cfg(not(test))] -pub mod inner { - pub use tokio::net::{ - TcpListener, TcpStream, ToSocketAddrs, UdpSocket, tcp::OwnedReadHalf, tcp::OwnedWriteHalf, - }; -} - -#[cfg(test)] -pub mod inner { - pub use turmoil::ToSocketAddrs; - pub use turmoil::net::{ - TcpListener, TcpStream, UdpSocket, tcp::OwnedReadHalf, tcp::OwnedWriteHalf, - }; -} -// 2. A single macro handles ALL types now! -macro_rules! wrap_type { - ($name:ident) => { - pub struct $name(inner::$name); - - impl Deref for $name { - type Target = inner::$name; - fn deref(&self) -> &Self::Target { - &self.0 - } - } - impl DerefMut for $name { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.0 - } - } - }; -} - -// 3. Generate the wrappers -wrap_type!(TcpListener); -wrap_type!(UdpSocket); -wrap_type!(TcpStream); -wrap_type!(OwnedReadHalf); -wrap_type!(OwnedWriteHalf); - -// 4. Implement AsyncRead / AsyncWrite safely using `Unpin` - -impl AsyncRead for TcpStream { - fn poll_read( - mut self: Pin<&mut Self>, - cx: &mut Context<'_>, - buf: &mut ReadBuf<'_>, - ) -> Poll> { - Pin::new(&mut self.0).poll_read(cx, buf) - } -} - -impl AsyncWrite for TcpStream { - fn poll_write( - mut self: Pin<&mut Self>, - cx: &mut Context<'_>, - buf: &[u8], - ) -> Poll> { - Pin::new(&mut self.0).poll_write(cx, buf) - } - fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - Pin::new(&mut self.0).poll_flush(cx) - } - fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - Pin::new(&mut self.0).poll_shutdown(cx) - } -} - -impl AsyncRead for OwnedReadHalf { - fn poll_read( - mut self: Pin<&mut Self>, - cx: &mut Context<'_>, - buf: &mut ReadBuf<'_>, - ) -> Poll> { - Pin::new(&mut self.0).poll_read(cx, buf) - } -} - -impl AsyncWrite for OwnedWriteHalf { - fn poll_write( - mut self: Pin<&mut Self>, - cx: &mut Context<'_>, - buf: &[u8], - ) -> Poll> { - Pin::new(&mut self.0).poll_write(cx, buf) - } - fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - Pin::new(&mut self.0).poll_flush(cx) - } - fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - Pin::new(&mut self.0).poll_shutdown(cx) - } -} - -impl TcpListener { - pub async fn bind(addr: A) -> std::io::Result { - let inner_listener = inner::TcpListener::bind(addr).await?; - Ok(Self(inner_listener)) - } - - pub async fn accept(&self) -> std::io::Result<(TcpStream, std::net::SocketAddr)> { - // We intercept accept() so it returns our wrapper TcpStream instead of the inner one - let (inner_stream, addr) = self.0.accept().await?; - Ok((TcpStream(inner_stream), addr)) - } -} - -impl TcpStream { - #[allow(dead_code)] - pub async fn connect(addr: A) -> std::io::Result { - let inner_stream = inner::TcpStream::connect(addr).await?; - Ok(Self(inner_stream)) - } - - pub fn into_split(self) -> (OwnedReadHalf, OwnedWriteHalf) { - // We intercept into_split() so it returns our wrapped halves - let (read_half, write_half) = self.0.into_split(); - (OwnedReadHalf(read_half), OwnedWriteHalf(write_half)) - } -} - -impl UdpSocket { - pub async fn bind(addr: A) -> std::io::Result { - let inner_socket = inner::UdpSocket::bind(addr).await?; - Ok(Self(inner_socket)) - } -} diff --git a/src/net/mod.rs b/src/net/mod.rs new file mode 100644 index 00000000..70dc9afc --- /dev/null +++ b/src/net/mod.rs @@ -0,0 +1,24 @@ +#![allow(clippy::disallowed_types)] + +mod tcp; +pub(crate) mod udp; + +#[cfg(test)] +pub use tcp::OwnedWriteHalf; +pub use tcp::{TcpListener, TcpStream, TransportReadHalf, TransportTcpStream, TransportWriteHalf}; +pub use udp::UdpSocket; + +#[cfg(not(test))] +pub(super) mod inner { + pub use tokio::net::{ + TcpListener, TcpStream, ToSocketAddrs, UdpSocket, tcp::OwnedReadHalf, tcp::OwnedWriteHalf, + }; +} + +#[cfg(test)] +pub(super) mod inner { + pub use turmoil::ToSocketAddrs; + pub use turmoil::net::{ + TcpListener, TcpStream, UdpSocket, tcp::OwnedReadHalf, tcp::OwnedWriteHalf, + }; +} diff --git a/src/net/tcp.rs b/src/net/tcp.rs new file mode 100644 index 00000000..bdddbab4 --- /dev/null +++ b/src/net/tcp.rs @@ -0,0 +1,580 @@ +use std::ops::{Deref, DerefMut}; +use std::pin::Pin; +use std::sync::Arc; +use std::task::{Context, Poll}; + +use anyhow::{Context as _, Result}; +use rustls::pki_types::ServerName; +use tokio::io::{AsyncRead, AsyncWrite, ReadBuf}; +use tokio_rustls::{TlsAcceptor, TlsConnector, TlsStream}; + +use super::inner; +use crate::security::{CertificatePrincipal, NodeTransportSecurity, node_certificate_principal}; + +const NODE_ADMISSION_EXPORTER_LABEL: &[u8] = b"EXPORTER-EastGuard-node-admission-v1"; +const NODE_ADMISSION_BINDING_BYTES: usize = 32; + +macro_rules! tcp_wrapper { + ($name:ident) => { + pub struct $name(pub(super) inner::$name); + + impl Deref for $name { + type Target = inner::$name; + + fn deref(&self) -> &Self::Target { + &self.0 + } + } + + impl DerefMut for $name { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.0 + } + } + }; +} + +tcp_wrapper!(TcpListener); +tcp_wrapper!(TcpStream); +tcp_wrapper!(OwnedReadHalf); +tcp_wrapper!(OwnedWriteHalf); + +/// Mutually authenticated node connection used by the Raft and data transports. +/// +/// TLS authenticates the certificate chain before this stream exposes the peer's +/// Node Certificate Principal. Admission later binds that stable principal to +/// the process-specific `NodeId` carried by the transport handshake. +pub struct AuthenticatedTcpStream { + peer_principal: CertificatePrincipal, + stream: TlsStream, +} + +impl AuthenticatedTcpStream { + pub async fn connect( + addr: A, + config: Arc, + ) -> Result { + let stream = TcpStream::connect(addr).await?; + // NodeServerCertVerifier authenticates the certificate chain and node + // principal. It intentionally does not use DNS-name matching. + let server_name = ServerName::try_from("unused.eastguard")?; + let stream = TlsConnector::from(config) + .connect(server_name, stream) + .await?; + Self::from_tls_stream(stream.into(), node_certificate_principal) + } + + fn from_tls_stream( + stream: TlsStream, + read_principal: fn(&rustls::pki_types::CertificateDer<'_>) -> Result, + ) -> Result { + let certificate = stream + .get_ref() + .1 + .peer_certificates() + .and_then(|certificates| certificates.first()) + .context("authenticated TLS peer supplied no certificate")?; + let peer_principal = read_principal(certificate)?; + Ok(Self { + peer_principal, + stream, + }) + } + + pub fn peer_principal(&self) -> &CertificatePrincipal { + &self.peer_principal + } + + /// Derives a value unique to this completed TLS session. + /// + /// Both peers derive the same bytes. Signing them binds a process-admission + /// proof to this connection, so a captured proof cannot be replayed. + fn admission_binding(&self) -> Result<[u8; NODE_ADMISSION_BINDING_BYTES]> { + let output = [0; NODE_ADMISSION_BINDING_BYTES]; + match &self.stream { + TlsStream::Client(stream) => stream.get_ref().1.export_keying_material( + output, + NODE_ADMISSION_EXPORTER_LABEL, + None, + ), + TlsStream::Server(stream) => stream.get_ref().1.export_keying_material( + output, + NODE_ADMISSION_EXPORTER_LABEL, + None, + ), + } + .context("failed to derive node-admission TLS session binding") + } + + fn into_split( + self, + ) -> ( + tokio::io::ReadHalf, + tokio::io::WriteHalf, + ) { + tokio::io::split(self) + } +} + +/// TCP transport connection. Secure mode carries an authenticated certificate +/// principal; trusted-development mode preserves the existing plaintext path. +pub enum TransportTcpStream { + TrustedDevelopment(TcpStream), + Secure(Box), +} + +pub enum TransportReadHalf { + TrustedDevelopment(OwnedReadHalf), + Secure(tokio::io::ReadHalf), +} + +pub enum TransportWriteHalf { + TrustedDevelopment(OwnedWriteHalf), + Secure(tokio::io::WriteHalf), +} + +impl From for TransportReadHalf { + fn from(value: OwnedReadHalf) -> Self { + Self::TrustedDevelopment(value) + } +} + +impl From for TransportWriteHalf { + fn from(value: OwnedWriteHalf) -> Self { + Self::TrustedDevelopment(value) + } +} + +impl TransportTcpStream { + pub async fn accept( + stream: TcpStream, + security: &NodeTransportSecurity, + read_principal: fn(&rustls::pki_types::CertificateDer<'_>) -> Result, + ) -> Result { + match security { + NodeTransportSecurity::Secure(security) => { + let stream = TlsAcceptor::from(security.server_config()) + .accept(stream) + .await?; + AuthenticatedTcpStream::from_tls_stream(stream.into(), read_principal) + .map(Box::new) + .map(Self::Secure) + } + NodeTransportSecurity::TrustedDevelopment => Ok(Self::TrustedDevelopment(stream)), + } + } + + pub async fn connect_node( + addr: A, + security: &NodeTransportSecurity, + ) -> Result { + match security { + NodeTransportSecurity::Secure(security) => { + AuthenticatedTcpStream::connect(addr, security.client_config()) + .await + .map(Box::new) + .map(Self::Secure) + } + NodeTransportSecurity::TrustedDevelopment => TcpStream::connect(addr) + .await + .map(Self::TrustedDevelopment) + .map_err(Into::into), + } + } + + pub fn peer_principal(&self) -> Option { + match self { + Self::Secure(stream) => Some(stream.peer_principal().clone()), + Self::TrustedDevelopment(_) => None, + } + } + + pub(crate) fn admission_binding(&self) -> Result<[u8; NODE_ADMISSION_BINDING_BYTES]> { + match self { + Self::Secure(stream) => stream.admission_binding(), + Self::TrustedDevelopment(_) => { + anyhow::bail!("trusted-development connections have no TLS session binding") + } + } + } + + pub fn into_split(self) -> (TransportReadHalf, TransportWriteHalf) { + match self { + Self::TrustedDevelopment(stream) => { + let (read, write) = TcpStream::into_split(stream); + ( + TransportReadHalf::TrustedDevelopment(read), + TransportWriteHalf::TrustedDevelopment(write), + ) + } + Self::Secure(stream) => { + let (read, write) = AuthenticatedTcpStream::into_split(*stream); + ( + TransportReadHalf::Secure(read), + TransportWriteHalf::Secure(write), + ) + } + } + } +} + +macro_rules! impl_transport_io { + ($type:ty) => { + impl AsyncRead for $type { + fn poll_read( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + match &mut *self { + Self::TrustedDevelopment(stream) => Pin::new(stream).poll_read(cx, buf), + Self::Secure(stream) => Pin::new(stream).poll_read(cx, buf), + } + } + } + + impl AsyncWrite for $type { + fn poll_write( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + match &mut *self { + Self::TrustedDevelopment(stream) => Pin::new(stream).poll_write(cx, buf), + Self::Secure(stream) => Pin::new(stream).poll_write(cx, buf), + } + } + + fn poll_flush( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> Poll> { + match &mut *self { + Self::TrustedDevelopment(stream) => Pin::new(stream).poll_flush(cx), + Self::Secure(stream) => Pin::new(stream).poll_flush(cx), + } + } + + fn poll_shutdown( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> Poll> { + match &mut *self { + Self::TrustedDevelopment(stream) => Pin::new(stream).poll_shutdown(cx), + Self::Secure(stream) => Pin::new(stream).poll_shutdown(cx), + } + } + } + }; +} + +impl_transport_io!(TransportTcpStream); + +impl AsyncRead for TransportReadHalf { + fn poll_read( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + match &mut *self { + Self::TrustedDevelopment(stream) => Pin::new(stream).poll_read(cx, buf), + Self::Secure(stream) => Pin::new(stream).poll_read(cx, buf), + } + } +} + +impl AsyncWrite for TransportWriteHalf { + fn poll_write( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + match &mut *self { + Self::TrustedDevelopment(stream) => Pin::new(stream).poll_write(cx, buf), + Self::Secure(stream) => Pin::new(stream).poll_write(cx, buf), + } + } + + fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + match &mut *self { + Self::TrustedDevelopment(stream) => Pin::new(stream).poll_flush(cx), + Self::Secure(stream) => Pin::new(stream).poll_flush(cx), + } + } + + fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + match &mut *self { + Self::TrustedDevelopment(stream) => Pin::new(stream).poll_shutdown(cx), + Self::Secure(stream) => Pin::new(stream).poll_shutdown(cx), + } + } +} + +impl AsyncRead for AuthenticatedTcpStream { + fn poll_read( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + Pin::new(&mut self.stream).poll_read(cx, buf) + } +} + +impl AsyncWrite for AuthenticatedTcpStream { + fn poll_write( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + Pin::new(&mut self.stream).poll_write(cx, buf) + } + + fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.stream).poll_flush(cx) + } + + fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.stream).poll_shutdown(cx) + } +} + +impl AsyncRead for TcpStream { + fn poll_read( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + Pin::new(&mut self.0).poll_read(cx, buf) + } +} + +impl AsyncWrite for TcpStream { + fn poll_write( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + Pin::new(&mut self.0).poll_write(cx, buf) + } + + fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.0).poll_flush(cx) + } + + fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.0).poll_shutdown(cx) + } +} + +impl AsyncRead for OwnedReadHalf { + fn poll_read( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + Pin::new(&mut self.0).poll_read(cx, buf) + } +} + +impl AsyncWrite for OwnedWriteHalf { + fn poll_write( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &[u8], + ) -> Poll> { + Pin::new(&mut self.0).poll_write(cx, buf) + } + + fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.0).poll_flush(cx) + } + + fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.0).poll_shutdown(cx) + } +} + +impl TcpListener { + pub async fn bind(addr: A) -> std::io::Result { + let listener = inner::TcpListener::bind(addr).await?; + Ok(Self(listener)) + } + + pub async fn accept(&self) -> std::io::Result<(TcpStream, std::net::SocketAddr)> { + let (stream, addr) = self.0.accept().await?; + Ok((TcpStream(stream), addr)) + } +} + +impl TcpStream { + #[allow(dead_code)] + pub async fn connect(addr: A) -> std::io::Result { + let stream = inner::TcpStream::connect(addr).await?; + Ok(Self(stream)) + } + + pub fn into_split(self) -> (OwnedReadHalf, OwnedWriteHalf) { + let (read_half, write_half) = self.0.into_split(); + (OwnedReadHalf(read_half), OwnedWriteHalf(write_half)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::security::client_certificate_principal; + use rcgen::string::Ia5String; + use rcgen::{CertificateParams, KeyPair, SanType}; + use rustls::pki_types::{PrivateKeyDer, PrivatePkcs8KeyDer}; + use rustls::server::WebPkiClientVerifier; + use rustls::{ClientConfig, RootCertStore, ServerConfig}; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + use turmoil::Builder; + + fn certificate( + principal_kind: &str, + principal: &str, + dns_name: Option<&str>, + ) -> ( + rustls::pki_types::CertificateDer<'static>, + PrivateKeyDer<'static>, + ) { + let mut params = CertificateParams::default(); + params.subject_alt_names.push(SanType::URI( + Ia5String::try_from(format!("urn:eastguard:{principal_kind}:{principal}")).unwrap(), + )); + if let Some(dns_name) = dns_name { + params + .subject_alt_names + .push(SanType::DnsName(Ia5String::try_from(dns_name).unwrap())); + } + let key = KeyPair::generate().unwrap(); + let certificate = params.self_signed(&key).unwrap().der().clone(); + let key = PrivateKeyDer::Pkcs8(PrivatePkcs8KeyDer::from(key.serialize_der())); + (certificate, key) + } + + fn tls_configs() -> (Arc, Arc) { + let (server_certificate, server_key) = + certificate("node", "broker-server", Some("unused.eastguard")); + let (client_certificate, client_key) = certificate("node", "broker-client", None); + + let mut client_roots = RootCertStore::empty(); + client_roots.add(client_certificate.clone()).unwrap(); + let client_verifier = WebPkiClientVerifier::builder(Arc::new(client_roots)) + .build() + .unwrap(); + let server = ServerConfig::builder_with_protocol_versions(&[&rustls::version::TLS13]) + .with_client_cert_verifier(client_verifier) + .with_single_cert(vec![server_certificate.clone()], server_key) + .unwrap(); + + let mut server_roots = RootCertStore::empty(); + server_roots.add(server_certificate).unwrap(); + let client = ClientConfig::builder_with_protocol_versions(&[&rustls::version::TLS13]) + .with_root_certificates(server_roots) + .with_client_auth_cert(vec![client_certificate], client_key) + .unwrap(); + (Arc::new(server), Arc::new(client)) + } + + #[test] + fn mutual_tls_shares_session_binding_under_turmoil() -> turmoil::Result { + let (server_config, client_config) = tls_configs(); + let mut sim = Builder::new().build(); + + sim.host("server", move || { + let server_config = server_config.clone(); + async move { + let listener = TcpListener::bind("0.0.0.0:9000").await?; + let (stream, _) = listener.accept().await?; + let stream = TlsAcceptor::from(server_config).accept(stream).await?; + let mut stream = AuthenticatedTcpStream::from_tls_stream( + stream.into(), + node_certificate_principal, + )?; + assert_eq!(stream.peer_principal().as_ref(), "broker-client"); + let session_binding = stream.admission_binding().unwrap(); + let mut message = [0; 4]; + stream.read_exact(&mut message).await?; + assert_eq!(&message, b"ping"); + stream.write_all(b"pong").await?; + stream.write_all(&session_binding).await?; + Ok(()) + } + }); + + sim.client("client", async move { + let mut stream = + AuthenticatedTcpStream::connect((turmoil::lookup("server"), 9000), client_config) + .await + .unwrap(); + assert_eq!(stream.peer_principal().as_ref(), "broker-server"); + let session_binding = stream.admission_binding().unwrap(); + stream.write_all(b"ping").await?; + let mut message = [0; 4]; + stream.read_exact(&mut message).await?; + assert_eq!(&message, b"pong"); + let mut peer_session_binding = [0; NODE_ADMISSION_BINDING_BYTES]; + stream.read_exact(&mut peer_session_binding).await?; + assert_eq!(peer_session_binding, session_binding); + Ok(()) + }); + + sim.run() + } + + #[test] + fn mutual_tls_exposes_client_principal_under_turmoil() -> turmoil::Result { + let (server_certificate, server_key) = + certificate("node", "broker-server", Some("unused.eastguard")); + let (client_certificate, client_key) = certificate("client", "producer-a", None); + + let mut client_roots = RootCertStore::empty(); + client_roots.add(client_certificate.clone()).unwrap(); + let client_verifier = WebPkiClientVerifier::builder(Arc::new(client_roots)) + .build() + .unwrap(); + let server_config = Arc::new( + ServerConfig::builder_with_protocol_versions(&[&rustls::version::TLS13]) + .with_client_cert_verifier(client_verifier) + .with_single_cert(vec![server_certificate.clone()], server_key) + .unwrap(), + ); + + let mut server_roots = RootCertStore::empty(); + server_roots.add(server_certificate).unwrap(); + let client_config = Arc::new( + ClientConfig::builder_with_protocol_versions(&[&rustls::version::TLS13]) + .with_root_certificates(server_roots) + .with_client_auth_cert(vec![client_certificate], client_key) + .unwrap(), + ); + + let mut sim = Builder::new().build(); + sim.host("server", move || { + let server_config = server_config.clone(); + async move { + let listener = TcpListener::bind("0.0.0.0:9000").await?; + let (stream, _) = listener.accept().await?; + let stream = TlsAcceptor::from(server_config).accept(stream).await?; + let stream = AuthenticatedTcpStream::from_tls_stream( + stream.into(), + client_certificate_principal, + )?; + assert_eq!(stream.peer_principal().as_ref(), "producer-a"); + Ok(()) + } + }); + sim.client("client", async move { + let stream = + AuthenticatedTcpStream::connect((turmoil::lookup("server"), 9000), client_config) + .await + .unwrap(); + assert_eq!(stream.peer_principal().as_ref(), "broker-server"); + Ok(()) + }); + + sim.run() + } +} diff --git a/src/net/udp.rs b/src/net/udp.rs new file mode 100644 index 00000000..d067ef19 --- /dev/null +++ b/src/net/udp.rs @@ -0,0 +1,14 @@ +use crate::impl_new_struct_wrapper; + +use super::inner; + +pub struct UdpSocket(inner::UdpSocket); + +impl_new_struct_wrapper!(UdpSocket, inner::UdpSocket); + +impl UdpSocket { + pub async fn bind(addr: A) -> std::io::Result { + let socket = inner::UdpSocket::bind(addr).await?; + Ok(Self(socket)) + } +} diff --git a/src/security/admission_proof.rs b/src/security/admission_proof.rs new file mode 100644 index 00000000..687a9dbc --- /dev/null +++ b/src/security/admission_proof.rs @@ -0,0 +1,176 @@ +use anyhow::{Context, Result}; +use borsh::{BorshDeserialize, BorshSerialize}; +#[cfg(test)] +use ring::signature::KeyPair; +use ring::{ + rand::SystemRandom, + signature::{ED25519, Ed25519KeyPair, UnparsedPublicKey}, +}; + +use crate::control_plane::consensus::raft::states::security::AdmissionRecord; +use crate::{control_plane::NodeId, security::CertificatePrincipal}; + +const ADMISSION_PROOF_DOMAIN: &str = "eastguard-node-admission-v1"; +const ED25519_SIGNATURE_BYTES: usize = 64; + +/// Private signing key generated for one broker process. +/// +/// Only its public key is committed in the admission record. The private key +/// remains in this process and proves that a connection belongs to the +/// currently admitted process. +pub(crate) struct ProcessSigningKey(Ed25519KeyPair); + +impl ProcessSigningKey { + pub(crate) fn generate() -> Result { + let random = SystemRandom::new(); + let encoded = Ed25519KeyPair::generate_pkcs8(&random) + .map_err(|_| anyhow::anyhow!("failed to generate process signing key"))?; + let key = Ed25519KeyPair::from_pkcs8(encoded.as_ref()) + .map_err(|_| anyhow::anyhow!("failed to load generated process signing key"))?; + Ok(Self(key)) + } + + #[cfg(test)] + fn public_key(&self) -> Box<[u8]> { + self.0.public_key().as_ref().into() + } + + pub(crate) fn sign( + &self, + node_certificate_principal: &CertificatePrincipal, + node_id: &NodeId, + tls_session_binding: &[u8], + ) -> Result { + let message = AdmissionProof::signing_message( + node_certificate_principal, + node_id, + tls_session_binding, + )?; + let signature = self.0.sign(&message); + let signature = signature + .as_ref() + .try_into() + .expect("Ed25519 signatures are always 64 bytes"); + Ok(AdmissionProof { + node_id: node_id.clone(), + signature, + }) + } +} + +/// Proof that this TLS connection belongs to an admitted broker process. +/// +/// The receiver checks the node ID and signature against the current admission +/// record. Binding the signature to this TLS session prevents replay. +#[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)] +pub(crate) struct AdmissionProof { + pub(crate) node_id: NodeId, + signature: [u8; ED25519_SIGNATURE_BYTES], +} + +impl AdmissionProof { + pub(crate) fn verify( + &self, + process_public_key: &[u8], + node_certificate_principal: &CertificatePrincipal, + tls_session_binding: &[u8], + ) -> bool { + let Ok(message) = Self::signing_message( + node_certificate_principal, + &self.node_id, + tls_session_binding, + ) else { + return false; + }; + UnparsedPublicKey::new(&ED25519, process_public_key) + .verify(&message, &self.signature) + .is_ok() + } + + pub(crate) fn verify_admission( + &self, + admission: &AdmissionRecord, + node_certificate_principal: &CertificatePrincipal, + tls_session_binding: &[u8], + ) -> Result { + anyhow::ensure!( + admission.node_certificate_principal == *node_certificate_principal + && admission.node_id == self.node_id + && self.verify( + &admission.process_public_key, + node_certificate_principal, + tls_session_binding, + ), + "node admission proof does not match the current record" + ); + Ok(self.node_id.clone()) + } + + fn signing_message( + node_certificate_principal: &CertificatePrincipal, + node_id: &NodeId, + tls_session_binding: &[u8], + ) -> Result> { + borsh::to_vec(&( + ADMISSION_PROOF_DOMAIN, + node_certificate_principal.as_ref(), + node_id, + tls_session_binding, + )) + .context("failed to encode admission proof") + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn current_process_proves_its_admitted_identity() { + let key = ProcessSigningKey::generate().unwrap(); + let node_id = NodeId::new("broker-a::process-2"); + let binding = [7; 32]; + let principal = CertificatePrincipal::new("broker-a"); + let proof = key.sign(&principal, &node_id, &binding).unwrap(); + + assert!(proof.verify(&key.public_key(), &principal, &binding)); + } + + #[test] + fn proof_is_bound_to_the_certificate_principal_and_tls_session() { + let key = ProcessSigningKey::generate().unwrap(); + let node_id = NodeId::new("broker-a::process-2"); + let binding = [7; 32]; + let other_binding = [8; 32]; + let broker_a = CertificatePrincipal::new("broker-a"); + let broker_b = CertificatePrincipal::new("broker-b"); + let proof = key.sign(&broker_a, &node_id, &binding).unwrap(); + + assert!(!proof.verify(&key.public_key(), &broker_b, &binding)); + assert!(!proof.verify(&key.public_key(), &broker_a, &other_binding)); + } + + #[test] + fn old_process_key_cannot_prove_the_current_admission() { + let current_key = ProcessSigningKey::generate().unwrap(); + let old_key = ProcessSigningKey::generate().unwrap(); + let node_id = NodeId::new("broker-a::process-2"); + let binding = [7; 32]; + let principal = CertificatePrincipal::new("broker-a"); + let proof = old_key.sign(&principal, &node_id, &binding).unwrap(); + + assert!(!proof.verify(¤t_key.public_key(), &principal, &binding)); + } + + #[test] + fn changing_the_claimed_identity_invalidates_the_signature() { + let key = ProcessSigningKey::generate().unwrap(); + let node_id = NodeId::new("broker-a::process-2"); + let binding = [7; 32]; + let principal = CertificatePrincipal::new("broker-a"); + let mut proof = key.sign(&principal, &node_id, &binding).unwrap(); + + proof.node_id = NodeId::new("broker-a::process-3"); + assert!(!proof.verify(&key.public_key(), &principal, &binding)); + } +} diff --git a/src/security/certificates.rs b/src/security/certificates.rs new file mode 100644 index 00000000..0f57cb48 --- /dev/null +++ b/src/security/certificates.rs @@ -0,0 +1,156 @@ +use anyhow::{Context, Result}; +use rustls::pki_types::CertificateDer; +use x509_parser::extensions::GeneralName; +use x509_parser::prelude::{FromDer, X509Certificate}; + +use crate::security::CertificatePrincipal; + +/// Reads the stable node principal from a leaf certificate's URI Subject +/// Alternative Name. +/// +/// The certificate must contain exactly one URI beginning with +/// `urn:eastguard:node:`. The text after that prefix is the principal used as +/// the admission-record key. This function only parses the certificate; callers +/// must use it after rustls has authenticated the peer's certificate chain. +pub(crate) fn node_certificate_principal( + certificate: &CertificateDer<'_>, +) -> Result { + certificate_principal( + certificate, + "urn:eastguard:node:", + "node", + "Node Certificate Principal", + ) +} + +/// Reads the client principal from exactly one +/// `urn:eastguard:client:` URI Subject Alternative Name. +/// +/// TLS authentication must succeed before callers use this parsed identity for +/// authorization. +pub(crate) fn client_certificate_principal( + certificate: &CertificateDer<'_>, +) -> Result { + certificate_principal( + certificate, + "urn:eastguard:client:", + "client", + "Client Certificate Principal", + ) +} + +fn certificate_principal( + certificate: &CertificateDer<'_>, + uri_prefix: &str, + certificate_kind: &str, + principal_name: &str, +) -> Result { + let (_, certificate) = + X509Certificate::from_der(certificate.as_ref()).context("invalid X.509 certificate")?; + let subject_alt_name = certificate + .subject_alternative_name() + .context("invalid X.509 subject alternative name")? + .with_context(|| { + format!("{certificate_kind} certificate has no subject alternative name") + })?; + + let mut principals = + subject_alt_name + .value + .general_names + .iter() + .filter_map(|name| match name { + GeneralName::URI(uri) => uri.strip_prefix(uri_prefix), + _ => None, + }); + let principal = principals + .next() + .filter(|principal| !principal.is_empty()) + .with_context(|| format!("{certificate_kind} certificate has no {principal_name}"))?; + anyhow::ensure!( + principals.next().is_none(), + "{certificate_kind} certificate has multiple {principal_name}s" + ); + Ok(CertificatePrincipal::new(principal)) +} + +#[cfg(test)] +mod tests { + use super::*; + use rcgen::string::Ia5String; + use rcgen::{CertificateParams, KeyPair, SanType}; + + fn certificate_with_uris(uris: &[&str]) -> CertificateDer<'static> { + let mut params = CertificateParams::default(); + params.subject_alt_names = uris + .iter() + .map(|uri| SanType::URI(Ia5String::try_from(*uri).unwrap())) + .collect(); + let key = KeyPair::generate().unwrap(); + params.self_signed(&key).unwrap().der().clone() + } + + #[test] + fn reads_node_principal_from_uri_subject_alternative_name() { + let certificate = + certificate_with_uris(&["urn:example:unrelated", "urn:eastguard:node:broker-a"]); + + assert_eq!( + node_certificate_principal(&certificate).unwrap().as_ref(), + "broker-a" + ); + } + + #[test] + fn requires_exactly_one_node_principal() { + let missing = certificate_with_uris(&["urn:example:unrelated"]); + let ambiguous = + certificate_with_uris(&["urn:eastguard:node:broker-a", "urn:eastguard:node:broker-b"]); + + assert_eq!( + node_certificate_principal(&missing) + .unwrap_err() + .to_string(), + "node certificate has no Node Certificate Principal" + ); + assert_eq!( + node_certificate_principal(&ambiguous) + .unwrap_err() + .to_string(), + "node certificate has multiple Node Certificate Principals" + ); + } + + #[test] + fn reads_client_principal_from_uri_subject_alternative_name() { + let certificate = + certificate_with_uris(&["urn:example:unrelated", "urn:eastguard:client:producer-a"]); + + assert_eq!( + client_certificate_principal(&certificate).unwrap().as_ref(), + "producer-a" + ); + } + + #[test] + fn requires_exactly_one_client_principal() { + let missing = certificate_with_uris(&["urn:example:unrelated"]); + let ambiguous = certificate_with_uris(&[ + "urn:eastguard:client:producer-a", + "urn:eastguard:client:producer-b", + ]); + + assert_eq!( + client_certificate_principal(&missing) + .unwrap_err() + .to_string(), + "client certificate has no Client Certificate Principal" + ); + assert_eq!( + client_certificate_principal(&ambiguous) + .unwrap_err() + .to_string(), + "client certificate has multiple Client Certificate Principals" + ); + } +} diff --git a/src/security/mod.rs b/src/security/mod.rs new file mode 100644 index 00000000..1c84ec37 --- /dev/null +++ b/src/security/mod.rs @@ -0,0 +1,11 @@ +mod actor; +mod admission_proof; +mod certificates; +mod message; +mod state; +mod transport; + +pub(crate) use actor::{SecurityActor, SecurityHandle}; +pub(crate) use admission_proof::AdmissionProof; +pub(crate) use certificates::{client_certificate_principal, node_certificate_principal}; +pub(crate) use transport::{CertificatePrincipal, NodeTransportSecurity}; diff --git a/src/security/transport.rs b/src/security/transport.rs new file mode 100644 index 00000000..42edc352 --- /dev/null +++ b/src/security/transport.rs @@ -0,0 +1,335 @@ +use std::fmt; +use std::fs::File; +use std::io::BufReader; +use std::path::Path; +use std::sync::Arc; + +use super::admission_proof::{AdmissionProof, ProcessSigningKey}; +use super::certificates::node_certificate_principal; +use crate::config::{Environment, SecurityMode}; +use crate::control_plane::NodeId; +use anyhow::{Context, Result}; +use borsh::{BorshDeserialize, BorshSerialize}; +use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier}; +use rustls::client::verify_server_cert_signed_by_trust_anchor; +use rustls::crypto::{WebPkiSupportedAlgorithms, verify_tls12_signature, verify_tls13_signature}; +use rustls::pki_types::{CertificateDer, PrivateKeyDer, ServerName, UnixTime}; +use rustls::server::ParsedCertificate; +use rustls::server::WebPkiClientVerifier; +use rustls::{ + CertificateError, ClientConfig, DigitallySignedStruct, Error as RustlsError, OtherError, + RootCertStore, ServerConfig, SignatureScheme, +}; + +/// Verifies certificates presented to EastGuard's outbound node connections. +/// +/// The shared client config uses this verifier when Raft or data transport +/// connects to another broker. It retains certificate-chain, validity, +/// server-usage, and TLS handshake-signature verification. It does not compare +/// the certificate with a DNS name because brokers are identified by the Node +/// Certificate Principal carried in the certificate; admission later binds +/// that stable principal to the process-specific `NodeId`. +/// +/// Peer certificate +// ├── trusted CA chain? ── no → reject +// ├── valid lifetime and server usage? ── no → reject +// ├── valid TLS handshake signature? ── no → reject +// └── exactly one Node Certificate Principal? ── no → reject +struct NodeServerCertVerifier { + roots: Arc, + supported: WebPkiSupportedAlgorithms, +} + +impl NodeServerCertVerifier { + fn new(roots: Arc) -> Self { + Self { + roots, + supported: rustls::crypto::ring::default_provider().signature_verification_algorithms, + } + } +} + +impl fmt::Debug for NodeServerCertVerifier { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("NodeServerCertVerifier") + .finish_non_exhaustive() + } +} + +impl ServerCertVerifier for NodeServerCertVerifier { + fn verify_server_cert( + &self, + end_entity: &CertificateDer<'_>, + intermediates: &[CertificateDer<'_>], + _server_name: &ServerName<'_>, + _ocsp_response: &[u8], + now: UnixTime, + ) -> std::result::Result { + let certificate = ParsedCertificate::try_from(end_entity)?; + verify_server_cert_signed_by_trust_anchor( + &certificate, + &self.roots, + intermediates, + now, + self.supported.all, + )?; + node_certificate_principal(end_entity).map_err(|error| { + CertificateError::Other(OtherError(Arc::new(std::io::Error::other( + error.to_string(), + )))) + })?; + Ok(ServerCertVerified::assertion()) + } + + fn verify_tls12_signature( + &self, + message: &[u8], + certificate: &CertificateDer<'_>, + signature: &DigitallySignedStruct, + ) -> std::result::Result { + verify_tls12_signature(message, certificate, signature, &self.supported) + } + + fn verify_tls13_signature( + &self, + message: &[u8], + certificate: &CertificateDer<'_>, + signature: &DigitallySignedStruct, + ) -> std::result::Result { + verify_tls13_signature(message, certificate, signature, &self.supported) + } + + fn supported_verify_schemes(&self) -> Vec { + self.supported.supported_schemes() + } +} + +#[derive(Clone)] +pub(crate) enum NodeTransportSecurity { + Secure(NodeCredentials), + TrustedDevelopment, +} + +impl NodeTransportSecurity { + pub(crate) fn load(env: &Environment) -> Result { + match env.security_mode { + SecurityMode::Secure => { + let certificate_chain = env + .certificate_chain_path + .as_deref() + .context("certificate_chain_path is required in secure mode")?; + let private_key_path = env + .private_key_path + .as_deref() + .context("private_key_path is required in secure mode")?; + let trust_roots = env + .trust_root_path + .as_deref() + .context("trust_root_path is required in secure mode")?; + Ok(Self::Secure(Self::load_from_paths( + certificate_chain, + private_key_path, + trust_roots, + )?)) + } + SecurityMode::TrustedDevelopment => Ok(Self::TrustedDevelopment), + } + } + + fn load_from_paths( + certificate_chain_path: &Path, + private_key_path: &Path, + trust_root_path: &Path, + ) -> Result { + let certificate_chain = + Self::load_certificates(certificate_chain_path, "certificate chain")?; + let node_certificate_principal = node_certificate_principal(&certificate_chain[0])?; + let private_key = Self::load_private_key(private_key_path)?; + let trust_roots = Arc::new(Self::load_trust_roots(trust_root_path)?); + let client_verifier = WebPkiClientVerifier::builder(trust_roots.clone()).build()?; + let server = ServerConfig::builder_with_protocol_versions(&[&rustls::version::TLS13]) + .with_client_cert_verifier(client_verifier) + .with_single_cert(certificate_chain.clone(), private_key.clone_key())?; + let client = ClientConfig::builder_with_protocol_versions(&[&rustls::version::TLS13]) + .dangerous() + .with_custom_certificate_verifier(Arc::new(NodeServerCertVerifier::new(trust_roots))) + .with_client_auth_cert(certificate_chain, private_key)?; + + Ok(NodeCredentials { + server: Arc::new(server), + client: Arc::new(client), + node_certificate_principal, + process_signing_key: Arc::new(ProcessSigningKey::generate()?), + }) + } + + pub(crate) fn is_secure(&self) -> bool { + matches!(self, Self::Secure(_)) + } + + fn load_certificates(path: &Path, kind: &'static str) -> Result>> { + let file = + File::open(path).context(format!("failed to open {kind} file {}", path.display()))?; + + let certificates = rustls_pemfile::certs(&mut BufReader::new(file)) + .collect::, _>>() + .context(format!("failed to parse {kind} file {}", path.display()))?; + + anyhow::ensure!( + !certificates.is_empty(), + "{kind} file {} contains no certificates", + path.display() + ); + Ok(certificates) + } + + fn load_private_key(path: &Path) -> Result> { + let file = File::open(path).context(format!( + "failed to open private key file {}", + path.display() + ))?; + + rustls_pemfile::private_key(&mut BufReader::new(file)) + .context(format!( + "failed to parse private key file {}", + path.display() + ))? + .context(format!( + "private key file {} contains no supported key", + path.display() + )) + } + + fn load_trust_roots(path: &Path) -> Result { + let certificates = Self::load_certificates(path, "trust root")?; + let mut roots = RootCertStore::empty(); + for certificate in certificates { + roots + .add(certificate) + .with_context(|| format!("invalid trust root in {}", path.display()))?; + } + Ok(roots) + } +} + +/// TLS credentials and process identity loaded for secure mode. +#[derive(Clone)] +pub(crate) struct NodeCredentials { + server: Arc, + client: Arc, + node_certificate_principal: CertificatePrincipal, + process_signing_key: Arc, +} + +#[derive(Debug, Clone, PartialEq, Eq, Ord, PartialOrd, Hash, BorshSerialize, BorshDeserialize)] +pub(crate) struct CertificatePrincipal(Box); + +impl CertificatePrincipal { + pub(crate) fn new(principal: impl Into>) -> Self { + Self(principal.into()) + } +} + +impl AsRef for CertificatePrincipal { + fn as_ref(&self) -> &str { + &self.0 + } +} + +impl NodeCredentials { + pub(crate) fn server_config(&self) -> Arc { + self.server.clone() + } + + pub(crate) fn client_config(&self) -> Arc { + self.client.clone() + } + + /// Creates proof that a node connection belongs to this exact process. + /// + /// The TLS session binding makes the proof unique to one connection. Only + /// the process public key is stored in metadata Raft. + pub(crate) fn create_admission_proof( + &self, + node_id: &NodeId, + tls_session_binding: &[u8], + ) -> Result { + self.process_signing_key.sign( + &self.node_certificate_principal, + node_id, + tls_session_binding, + ) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use clap::Parser; + use rcgen::string::Ia5String; + use rcgen::{CertificateParams, KeyPair, SanType}; + + fn certificate_with_uris(uris: &[&str]) -> CertificateDer<'static> { + let mut params = CertificateParams::default(); + params.subject_alt_names = uris + .iter() + .map(|uri| SanType::URI(Ia5String::try_from(*uri).unwrap())) + .collect(); + let key = KeyPair::generate().unwrap(); + params.self_signed(&key).unwrap().der().clone() + } + + #[test] + fn secure_mode_requires_every_credential_path() { + let env = Environment::try_parse_from(["eastguard"]).unwrap(); + + let error = NodeTransportSecurity::load(&env) + .err() + .expect("secure mode without credential paths must fail"); + + assert_eq!( + error.to_string(), + "certificate_chain_path is required in secure mode" + ); + } + + #[test] + fn trusted_development_does_not_load_credentials() { + let env = + Environment::try_parse_from(["eastguard", "--security-mode", "trusted-development"]) + .unwrap(); + + assert!(matches!( + NodeTransportSecurity::load(&env).unwrap(), + NodeTransportSecurity::TrustedDevelopment + )); + } + + #[test] + fn server_verifier_requires_trust_and_node_principal() { + let trusted = certificate_with_uris(&["urn:eastguard:node:broker-a"]); + let missing_principal = certificate_with_uris(&["urn:example:unrelated"]); + let mut roots = RootCertStore::empty(); + roots.add(trusted.clone()).unwrap(); + roots.add(missing_principal.clone()).unwrap(); + let verifier = NodeServerCertVerifier::new(Arc::new(roots)); + let server_name = ServerName::try_from("unused.eastguard").unwrap(); + + verifier + .verify_server_cert(&trusted, &[], &server_name, &[], UnixTime::now()) + .unwrap(); + assert!( + verifier + .verify_server_cert(&missing_principal, &[], &server_name, &[], UnixTime::now(),) + .is_err() + ); + + let untrusted = certificate_with_uris(&["urn:eastguard:node:broker-b"]); + assert!( + verifier + .verify_server_cert(&untrusted, &[], &server_name, &[], UnixTime::now()) + .is_err() + ); + } +}