Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
a625f32
feat: config added for secure mode
Migorithm Jul 27, 2026
dd5b0a4
security config
Migorithm Jul 27, 2026
ab388ab
add quinn, update lock
Migorithm Jul 27, 2026
c629309
infra: quinn udp socket
Migorithm Jul 27, 2026
6344070
temporary assertion
Migorithm Jul 27, 2026
b05979e
fix: address validation
Migorithm Jul 27, 2026
9421d7d
text: turmoil with quinn
Migorithm Jul 27, 2026
a256149
simplify smart p
Migorithm Jul 28, 2026
6f8f8d9
clarify ambiguity
Migorithm Jul 28, 2026
57ed875
security record
Migorithm Jul 28, 2026
e21f369
embed security state into metadata state
Migorithm Jul 28, 2026
23f5c14
rename Cert Node Id -> node cert principal
Migorithm Jul 28, 2026
e492f26
principal extraction from certificate -> SAN -> uri
Migorithm Jul 28, 2026
d7f9362
server cert verifier
Migorithm Jul 28, 2026
8d6f617
feat: authenticate TCP prims
Migorithm Jul 28, 2026
02e784f
feat: NodeTcpStream
Migorithm Jul 28, 2026
0024867
data-transport adopt NodeTransportSecurity, NodeTcpStream
Migorithm Jul 28, 2026
6b071e9
d8 doc update
Migorithm Jul 28, 2026
a55db57
defer udp /quic
Migorithm Jul 28, 2026
de673ca
fix: asymmetric bootstrap voter sets
Migorithm Jul 28, 2026
8dc736c
rename : add peer -> ensure peer
Migorithm Jul 28, 2026
d6db30c
convert cold-read integration test to tokio
Migorithm Jul 28, 2026
5f38f90
client principal
Migorithm Jul 28, 2026
74329a7
client support for tls
Migorithm Jul 28, 2026
1c97a29
feat: client controller
Migorithm Jul 28, 2026
6537e66
roadmap update
Migorithm Jul 29, 2026
ee5d0db
authorize at metadata level
Migorithm Jul 29, 2026
e551cce
feat: authorize_data_access
Migorithm Jul 29, 2026
6e00e16
feat: grant/revoke acl
Migorithm Jul 29, 2026
0ef80b4
feat: AclResource
Migorithm Jul 29, 2026
3dcc0bc
fix: correct ACL scope in client controller
Migorithm Jul 29, 2026
db33728
rename + consumer group coordination
Migorithm Jul 29, 2026
f9b7376
feat: produer-session ownership checkpoint
Migorithm Jul 29, 2026
b995c33
feat: admin acl
Migorithm Jul 29, 2026
d3730d1
feat: client -> data node -> cache flow for acl verification
Migorithm Jul 29, 2026
63d9190
refactor: acl cache
Migorithm Jul 29, 2026
8a02bb6
feat(security): refresh ACLs from shard hosts
Migorithm Jul 29, 2026
8da30c2
feat: ProcessSigningKey
Migorithm Jul 29, 2026
25fd3d5
feat: Admission cache
Migorithm Jul 29, 2026
a098bfa
update raft-transport rule
Migorithm Jul 30, 2026
38efa2c
roadmap updated
Migorithm Jul 30, 2026
046675a
admission query, completion, param
Migorithm Jul 31, 2026
7456be0
refactor: consolidate AdmissionLookupKey and AdmissionLookupRequest i…
Migorithm Jul 31, 2026
3dd0782
clean up for security broker
Migorithm Jul 31, 2026
940712f
channel method
Migorithm Jul 31, 2026
d52f99a
security actor injection, replace TransportIdentity with CertificateP…
Migorithm Jul 31, 2026
31c0c63
Box<str> -> CertificatePrincipal
Migorithm Jul 31, 2026
d31e265
accept node/client with principal read
Migorithm Aug 1, 2026
f8c21ce
signing and verifying admission proof
Migorithm Aug 1, 2026
1309866
pass client certificate principal read
Migorithm Aug 1, 2026
eb21fd5
transport protocol done
Migorithm Aug 1, 2026
4659d03
simplify dispatcher accept, rename dial to connection
Migorithm Aug 1, 2026
9ae9306
in
Migorithm Aug 1, 2026
e641cb8
sender -> peer id
Migorithm Aug 2, 2026
2877097
rm AdmissionRequest, ProcessAdmission
Migorithm Aug 2, 2026
3f499aa
feat: accept_cluster_connection
Migorithm Aug 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/rules/code-convention.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Functions taking `&self`, `&mut self`, or a reference to a struct as their first

## Enum struct pattern

Enum variants that carry data must use the tuple-variant + named-struct pattern. Never use inline fields on enum variants. See `/.claude/skills/enum-struct-pattern/SKILL.md` for the full pattern including `impl_from_variant!` usage.
Enum variants that carry data must use the tuple-variant + named-struct pattern. Prefer not using inline fields on enum variants. See `/.claude/skills/enum-struct-pattern/SKILL.md` for the full pattern including `impl_from_variant!` usage.

## Result over Option for observability

Expand Down
8 changes: 7 additions & 1 deletion .claude/rules/metadata-state-machine.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,11 @@ MetadataStateMachine (one per shard group)
│ SegmentMeta (nested inside RangeMeta)
└── topic_name_index: HashMap<String, TopicId>
├── topic_name_index: HashMap<String, TopicId>
└── security
├── admissions: Node Certificate Principal → Admission Record
├── ACLs: Resource → ACL Record
└── revocations: (Issuer, Serial) → Revocation Record
```

## Commands
Expand Down Expand Up @@ -80,3 +84,5 @@ MetadataStateMachine (one per shard group)
21. **`ReassignSegment` only re-points a sealed segment.** `apply_reassign_segment()` accepts only a `Sealed` segment, swaps `replica_set`, and changes nothing else — state stays `Sealed`; data, offsets, lineage, and timestamps stay frozen (invariant 3). An active, deleting, or unknown segment is rejected (`SegmentNotSealed` / `SegmentNotFound`), logged but not fatal (invariant 11). Re-applying with the same `replica_set` succeeds without raising a metadata event, tolerating duplicate death detection and no-leader re-proposals (cf. invariant 19). The swap runs through `apply`, so the umbrella `assert_invariants` re-checks every other invariant afterward — a reassignment cannot leave the machine inconsistent.

22. **A committed consumer-group generation assigns each active range exactly once.** When a group has members, its assignment keys exactly equal the topic's active ranges and every assignment names a current member. When it has no members, it has no assignments. Membership or range-topology changes advance the generation and recompute the full desired assignment through the Raft log; heartbeat refreshes that do not change membership leave the generation unchanged.

23. **Security map keys match their records.** Admission keys equal the record's Node Certificate Principal, ACL keys equal the record's resource, and revocation keys equal the record's issuer and serial. A map therefore contains at most one current admission per Node Certificate Principal and snapshot restore cannot silently associate a security record with a different lookup key.
78 changes: 66 additions & 12 deletions .claude/rules/raft-transport.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,92 @@
# Raft Transport (Invariants)

`RaftTransportActor` — async TCP transport for Raft RPCs. Manages persistent bidirectional connections between nodes. Each connection split into a reader task and a writer half held in the per-node `writers` map.
`RaftTransportActor` — async TCP transport for Raft RPCs. It manages persistent
bidirectional Raft connections between nodes. Each connection splits into a
reader task and a writer half held in the per-node `writers` map.
The same authenticated cluster listener also serves a one-shot ACL snapshot read used to
refresh a broker's local authorization cache and a limited admission-record read used to authenticate a connecting process.

Separate from SWIM's UDP transport. Raft uses TCP for reliable, ordered delivery.

## Architecture (brief)

```
RaftTransportActor
├── listener (raft_port, TCP)
├── writers: HashMap<NodeId, OwnedWriteHalf> (one write half per peer)
└── reader tasks (one per accepted/established connection)
cluster listener (TCP)
├── limited admission read ──► security actor → Raft → reply → close
└── request with process proof
├── both sides verify a TLS-session-bound process proof
├── Raft ──► persistent reader + one writer per peer
└── ACL ───► security actor → Raft → reply → close
```

## Wire Protocol

Length-prefixed bincode frames:
1. **Handshake** (first frame on every connection): `[len: u32][NodeId: bincode]`
2. **Messages**: `[len: u32][WireRaftMessage: bincode]`, where `WireRaftMessage` carries `shard_group_id` so transport can dispatch to the correct Raft group.
Length-prefixed Borsh frames:

1. **Initial message:** `InitialClusterMessage` carries one `ClusterRequest` and
an optional process-admission proof. Secure Raft and ACL requests include the
proof; the limited admission lookup omits it.
2. **Mutual admission:** the acceptor verifies the connecting node, then replies
with its own `AdmissionProof`. Both proofs sign the same TLS exporter value and are
checked against the peer's current admission record. The exporter lets both
ends derive identical connection-specific bytes without sending those bytes;
another TLS connection derives a different value.
3. **After mutual admission:**
- The first Raft message carries its sender. Later frames are raw
`WireRaftMessage` values until close. Each carries `shard_group_id`.
- An ACL snapshot request carries its requesting node, shard, and resource.
Its response is one `AclSnapshotResponse`, then the connection closes.
4. **Trusted-development initial message:** no cryptographic admission exchange;
the Raft or ACL request carries no admission proof.

## Invariants

1. **Connection identity is established by handshake before any RPCs.** First frame on every connection is the initiator's `NodeId`. The acceptor uses this to key the writer slot and to detect the simultaneous-connect race. Without the handshake, the acceptor cannot route inbound messages to a peer-identified slot.
1. **Secure connection identity comes from mTLS plus mutual process admission.**
TLS supplies each stable Node Certificate Principal. A signature over the TLS
exporter value proves possession of the current process key and cannot be
replayed on another TLS session. Mutual proof is required because the Raft
connection carries traffic in both directions. The Raft sender or ACL requester
must equal its admitted `NodeId`. The first Raft sender then keys the writer slot
and detects the simultaneous-connect race.

2. **At most one writer per peer.** `writers` is keyed by `NodeId`. Coexisting writers would split messages to the same peer across two TCP connections; per-connection ordering would let later messages overtake earlier ones in unpredictable patterns, causing the leader to chase its own retries.

3. **Lower NodeId wins on simultaneous connect.** When both sides connect concurrently, the acceptor drops the incoming connection if a writer for the peer already exists AND `peer_id > self.node_id`. Without this rule, both sides retain both connections (each thinks it won), violating invariant 2.

4. **Address resolution is always live.** Every connect attempt queries SWIM for the peer's current address; the transport keeps no local address cache. A stale local cache would connect to the wrong host after a peer moves or restarts on a different address.

5. **Frame sizes are bounded.** Handshake frames capped at 1KB; message frames at 4MB. Without bounds, a malicious or buggy peer can exhaust memory by sending a giant length prefix before any payload.
5. **Handshake work is bounded.** The listener acquires a permit before
spawning a handshake task, applies a total handshake deadline, and uses a
bounded queue to return verified Raft streams to the dispatcher. A slow TLS,
admission lookup, or proof exchange never blocks the transport select loop.

6. **Frame sizes are bounded.** Initial, process-admission, Raft, proof, and response
frames are capped before allocating their payload.

6. **Transport validates envelope identity but never interprets the RPC.** The
connection peer must match the envelope `sender`; a mismatch closes that
7. **Transport validates message identity but never interprets the RPC.** The
connection peer must match the message `sender`; a mismatch closes that
connection. The transport routes by `shard_group_id` and passes the authenticated
peer onward, but the RPC remains opaque. Voter, learner, leader, term, and log
checks belong to the target Raft state machine.

## Limited Admission Lookup Rule

Admission records are sharded, so the acceptor may need another broker to read
the record required for its proof check. Requiring process admission for that
read would recurse. `AdmissionLookup(AdmissionRecordKey)` is therefore accepted
after mTLS but before process admission. It can read one named admission record
from one shard, returns one `AdmissionLookupResponse`, and closes. It cannot
carry Raft, ACL, client, or admission-write traffic.

## ACL Snapshot Rule

An ACL snapshot request is not a Raft RPC and never enters a Raft state machine.
In secure mode it is served only after the requester completes process
admission. It asks the local multi-Raft actor for the selected shard's committed
ACL record through the broker security actor, returns that record on the same
connection, then closes. It carries no client data request and cannot proxy one.
A connection admitted for Raft carries only raw Raft frames after its first
message; an invalid frame closes the connection.
Loading
Loading