feat: secure transport foundation - #205
Open
Migorithm wants to merge 56 commits into
Open
Conversation
- Secure is the default. - Trusted development requires --security-mode trusted-development or SECURITY_MODE=trusted-development. - Integration environments opt in explicitly. - The example YAML documents the mode.
QuinnUdpSocket and its turmoil test
The deterministic turmoil test now proves:
client Initial
│
▼
server Retry token
│
▼
client retries
│
▼
server confirms address validated
│
▼
mutual TLS handshake + QUIC datagram
- Maximum 128 pending handshakes. - No bidirectional or unidirectional QUIC streams. - 64 × 1200-byte send and receive datagram buffers. - 1200-byte maximum UDP payload. - TLS early data/0-RTT confirmed disabled. - Required Retry address validation.
turmoil TCP socket
↓
TLS 1.3 mutual authentication
↓
AuthenticatedTcpStream
├── encrypted byte stream
└── authenticated peer principal
- Added tokio-rustls.
- Supports inbound and outbound mutual TLS.
- Extracts both peers’ Node Certificate Principals.
- Deterministic turmoil test exchanges ping/pong.
- Not yet used by production actors; the explicit dead_code allowance will be removed when Raft/data transports adopt it in the next slice.
- Focused turmoil test and all-target/all-feature clippy pass.
- Formatting and diff checks pass.
- RaftRpcListener now has one constructor taking the read half and explicit transport identity. - Raft transport uses shared node-stream halves. - Secure streams retain the certificate principal at the connection boundary. - Trusted-development tests name their mode explicitly. - Large TLS stream variant is boxed. - Mutual-TLS turmoil test and five Raft transport tests pass. - All-target/all-feature clippy, formatting, and diff checks pass.
NodeTransportSecurity
│
├── Raft TCP
└── Data TCP
↓
NodeTcpStream
├── Secure: mutual TLS + certificate principal
└── TrustedDevelopment: plaintext
Changes:
- Data readers/writers now use shared node-stream halves.
- Inbound and outbound data connections explicitly select the configured security mode.
- Connection identity stays attached to each reader for later admission authorization.
- No optional TLS configuration.
- Secure startup remains fail-closed because SWIM and client listeners are not secured yet.
- Fixed asymmetric bootstrap voter sets by reasserting known voters through the Raft log during takeover. - Kept strict committed-voter RPC authorization. - Added a focused regression test. - Exact failing turmoil seed now passes. - Full clippy, formatting, and diff checks pass.
TLS client principal
↓
TransportIdentity
↓
ClientController
↓
future ACL checks
Changes:
- Renamed NodeTransportIdentity to shared TransportIdentity.
- Captures identity before splitting the stream.
- Stores identity on ClientController.
- Trusted-development tests explicitly use TrustedDevelopment.
- No ACL behavior added yet.
Certificate-authenticated clients now require topic-data/{topic-id} authorization before:
- Produce
- Fetch and fetch-by-ID
- List offsets
- Commit consumer offset
- Fetch consumer offset
- Data-locality/write-leader checks
- Any data-plane dispatch
Behavior: - Operates on one exact resource/principal pair. - Duplicate grant and revoke are no-ops. - Revision advances only when authorization actually changes. - Revoking the final principal retains an empty ACL re
Added typed variants for:
- cluster
- topic-admin/{topic-id}
- topic-data/{topic-id}
- consumer-group/{topic-id}/{group-id}
- producer-session/{topic-id}/{producer-id}
- security/cluster
Also added strict string parsing via FromStr:
- Invalid topic IDs are rejected.
- Invalid producer UUIDs are rejected.
- Empty group IDs are rejected.
- Unknown and incomplete resource keys are rejected.
- Group IDs containing / round-trip correctly.
Corrected ACL scope: - Produce, fetch, and list-offsets use TopicData. - Consumer-offset reads and commits use the exact ConsumerGroup(topic ID, group ID) resource. - Authorization still occurs before any data-plane dispatch. - Trusted-development behavior remains unchanged.
Implemented: - TransportIdentity::CertificatePrincipal(Box<str>) for immutable connection identity. - Separate persisted ProducerSessionOwner, with comments explaining why transport evidence must not become snapshot schema. - Producer sessions store their owner. - Different principals cannot renew or recover an active producer ID. - Topic-data authorization occurs before opening a session. - Ownership is rechecked after commit to cover concurrent-open races. - Session expiry still removes metadata and data-plane deduplication state. - Roadmap wording now says ownership lasts for the session lifetime.
ACL records and data replicas are usually on different shard groups.
Client → data replica
│
├─ serves topic-data/42
└─ may not host ACL topic-data/42
Today, the data replica can only check ACL state if it hosts that ACL’s metadata shard. Otherwise it denies the request—even when the client is allowed.
The intended fix is:
Client → data replica → local ACL cache
│
fresh → allow / deny
miss → fetch ACL record from owner shard
│
└─ cache ≤60s → allow / deny
ClientController
└─ cache.authorize_or_refresh(resource, shard, principal, lazy_fetch)
├─ fresh cached grant → Ok(())
├─ fresh cached denial → Err(Unauthorized)
└─ cache miss
├─ snapshot fetched → store → recheck → Ok / Unauthorized
└─ no snapshot → Err(Unauthorized)
- Removed CachedAuthorization; cache internals no longer leak Authorized / Denied / Miss.
- SharedAclCache now returns Result<(), ServerError> and owns the fail-closed policy.
- ClientController no longer contains cache-miss, snapshot-to-cache, or recheck logic—it simply awaits the cache result.
- The snapshot fetch is lazy: on a fresh cache entry, no Raft query is made and the resource is not cloned for the query.
- CachedAcl and conversion from a replicated ACL record are private implementation details.
- Tests now assert success/failure results instead of cache-state variants.
- Read ACLs locally or from the current remote shard host without proxying client data. - Bound and coalesce remote reads; deny on failure or saturation. - Carry the initial Raft RPC or ACL request in the opening frame.
- Ed25519 process signing key using rustls’s existing ring backend. - Fresh 32-byte admission challenges. - Serializable admission proofs. - Tests covering valid proof, replay with another challenge, wrong principal, changed identity/epoch, and old process key.
Admission cache
│
├── fresh ───────────────► return record/denial
│
└── missing or expired
│
├── local shard ──► MultiRaft committed read
└── remote shard ─► limited admission lookup
│
└── reply once, then close
Implemented:
- Local committed admission-record queries.
- Limited bootstrap endpoint, accepted before process admission but restricted to one admission read.
- Local/remote lookup actor with:
- 16 active lookups
- 128 queued lookups
- 256 waiters per record
- concurrent identical requests combined
- Revision-aware 60-second cache.
- Lookup failures are not cached.
- Older revisions cannot renew expired authority.
- Records returned for another certificate principal are rejected.
…nto AdmissionRecordKey
Shard routing
│
▼
AdmissionRecordKey
├── cache/coalescing key
└── remote wire lookup
ProcessAdmissionRequest
│ verify process proof
▼
AcceptedRaftConnection
Changes:
- Consolidated AdmissionLookupKey and AdmissionLookupRequest into AdmissionRecordKey.
- The same key now flows directly from routing onto the wire.
- Renamed:
- InitialClusterMessage::Admitted → ProcessAdmission
- AdmittedClusterMessage → ProcessAdmissionRequest
- Updated the Raft transport rule.
- Removed all old-name references.
- Borsh variant positions and field order remain unchanged, so the wire layout is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Establish the S0 security foundation:
Quinn feasibility
The turmoil test verifies:
Quinn's Bloom-backed Retry-token log provides replay protection.
Current behavior