fix(runtime): bind peer credentials to claimed node identities#2665
Merged
Conversation
Introduce the per-node PeerAuthSnapshot / public-staging PeerAuthConfig
authority that binds an authenticated peer credential (Noise static key
on TCP, certificate SPKI on quic-mesh) to the NodeId it may claim.
- PeerCredential / PeerBindings map NodeId -> {credential}.
- authorize() returns a structured PeerAuthz verdict (never a bare bool):
a different credential claiming a bound NodeId is CredentialMismatch,
a credential bound elsewhere is reported with its actual NodeId.
- posture_for() decides per-connection posture from the remote endpoint
class: an unconfigured node is strict on non-loopback/Unknown and
Unverified only on loopback (never a blanket unverified pass).
- Owner-scoped ConfigState (Building/Starting/Running) governs only the
public Node::* API; identity material lives behind Arc for cheap,
secret-preserving clones.
Native-only and feature-agnostic (opaque byte identity material) so it
compiles under every native feature combination. Types only; no wiring
yet. 16 unit tests.
Introduce the per-node PeerAuthSnapshot as the authority for a node's NodeId in strict mode, replacing the process-global config as the source of truth for who a node claims to be. - HewConnMgr carries a frozen PeerAuthSnapshot installed at construction. Split hew_connmgr_new into the unchanged C ABI shim (which installs an unconfigured snapshot, so existing callers are untouched) and an internal connmgr_new that the production node-start path calls with the node's real snapshot. Concurrent managers hold independent snapshots; there is no shared admission authority across nodes. - HewNode carries its installed snapshot. hew_node_set_auth_snapshot installs it while STOPPED and is refused once the node is running. hew_node_start validates the snapshot and, before any allocation or listen, reconciles node_id against the snapshot's strict binding identity: a zero id adopts the pinned NodeId, a conflicting explicit id is rejected fail-closed with a diagnostic, and a self-inconsistent snapshot refuses to start. - The singleton public Node::* API stages configuration in a separate, owner-scoped ConfigState (Building -> Starting -> Running) guarded by PEER_AUTH_STATE. hew_node_api_start merges HEW_NODE_ID / HEW_DIST_ UNVERIFIED posture, validates before allocation, freezes the snapshot onto the node, runs the low-level start WITHOUT holding the staging lock, then transitions to Running or restores the held config on failure — all gated on a generation+owner guard so a concurrent stage is never clobbered. hew_node_api_shutdown resets staging to Building only when the shutting node owns it, mirroring the CURRENT_NODE owner check; a secondary low-level stop leaves staging intact. Default configuration (no HEW_NODE_ID, empty bindings) keeps the prior PID-derived id and unconfigured snapshot, so existing distributed behaviour is unchanged. Transport-selection-from-snapshot read is deferred to the transport slice (env remains authoritative; default TCP matches). Adds unit coverage for the NodeId authority, malformed-snapshot refusal, the setter's running-node guard, and the public owner-scoped staging lifecycle. Refs #2652
…ture
Add the typed remote-endpoint accessor and wire per-connection posture into
the admission path so a connection's authority is decided from the actual
endpoint against the node's own snapshot — never a process-global authority.
- transport.rs: hew_transport_conn_remote_ip_class dispatches by ops-identity
(no vtable change): TCP reads the stored socket's peer_addr; quic-mesh reads
the live PeerConn::remote_address; plain-quic / stub / null yield Unknown
(strict, fail-closed). classify_remote_socket_addr maps loopback vs routable.
- quic_mesh.rs: hew_quic_mesh_transport_conn_remote_ip_class ops-verifies the
transport before any impl cast and classifies the resolved conn.
- connection.rs: hew_connmgr_add computes posture = mgr.auth.posture_for(class)
after the numeric identity check and stores it on the ConnectionActor.
Credential-free posture rejects fire before credential resolution: an
unconfigured node (no bindings, no HEW_NODE_ID, not opt-out) on a
non-loopback/Unknown endpoint is rejected ("requires HEW_NODE_ID"), and any
strict connection over an Unknown-class transport (plain quic / stub) is
rejected ("strict binding unsupported on plain quic transport"). Loopback
endpoints on an unconfigured node stay Unverified (delivery-only), preserving
existing loopback-dev behaviour.
The manager's `auth` field is now read (drops the transitional dead-code
allow). Credential extraction + structured authorize wire in on the mesh/Noise
slices; the claim state machine consumes the stored posture next.
Adds accessor unit coverage (loopback/routable classification, null+stub
Unknown, real TCP loopback peer_addr). Full hew-runtime lib suite green under
default and quic; builds clean under --no-default-features.
Refs #2652
Introduce the reserve -> publish -> retire claim machine that binds an authenticated peer credential to the NodeId it is admitted under, so at most one connection may own a NodeId's route + cluster token at a time (issue #2652, plan decision D3). - HewConnMgr gains a per-manager `claims: (Mutex<HashMap<u16, LiveClaim>>, Condvar)` table; the mutex is the single serializing guard for the reserve/publish/retire window and the condvar coordinates the handoff. - reserve_claim rejects a different-credential takeover of a live Published claim fail-closed, supersedes a same-credential reconnect, waits (bounded) on an in-flight Reserved claim, and inserts a fresh reservation otherwise. - Admission (hew_connmgr_add) resolves the peer credential (Noise static key under encryption; mesh SPKI deferred), authorizes the claimed NodeId against the frozen per-node authority, and reserves before spawning the reader. Reader-spawn and install failures abort the reservation, restoring any superseded claim. - publish_connection_established transitions Reserved -> Published under the same lock as the route + cluster token, aborts if superseded meanwhile, and demotes a superseded same-credential connection by closing its transport. - retire_connection_publication removes the claim only for the exact owner and drives route removal + MonitorLost fan-out off that removal, so a superseded connection no longer fabricates a lost event for a peer still live under a newer claim. Adds direct claim-machine unit tests (exact-owner lifecycle, different- credential reject, same-credential supersede+abort-restore, fresh-abort, non-owner retire, publish-aborts-when-superseded, Reserved-wait, concurrent exactly-one-owner) and threads a reservation through the existing publish/ retire race tests. Also stabilises the S3 tcp peer-addr classification test by giving accept a real timeout instead of racing the client dial.
…nding Bind a peer's authenticated credential to the NodeId it is permitted to claim (issue #2652). allow_peer takes (node_id: u16, credential_hex: string) and stages a PeerCredential into the pre-start Building config instead of a global allowlist: the credential is interpreted by the node's pinned transport (TCP => 32-byte Noise pubkey; quic-mesh => cert SPKI; plain quic => rejected, no peer-credential channel). Reserved id 0 and this node's own HEW_NODE_ID are rejected, as is a credential already bound to a different NodeId (one credential binds to exactly one NodeId; rotation overlap binds multiple credentials to one NodeId). Node::start pins the resolved transport into the frozen snapshot (so admission interprets credentials consistently with the listener) and bridges the snapshot's mesh SPKI allowlist into the quic-mesh listener before it binds, so a bound peer is admitted and an unbound one rejected (fail-closed) end-to-end. ABI migrated atomically across catalog (U16_STRING), types registration, runtime signature, and both native+wasm32 r4_runtime_abi goldens. The env-merge extraction (merge_start_env_into_config) keeps start under the line budget. E2e fixtures move to the two-arg form; the peer-auth surface test sets HEW_NODE_ID (strict binding requires this node's own id). Tests: hew-runtime lib green across default/quic/encryption/no-default (2099 each; one pre-existing parallel-exec flake passes in isolation); run_e2e peer-auth + bad-hex green; clippy -D warnings + fmt clean.
Move the QUIC-mesh listener identity, peer-SPKI allowlist, and fail-closed setup-error poison off the four process-global statics (ACTIVE_MESH_SPKI_ ALLOWLIST, ACTIVE_MESH_IDENTITY, MESH_AUTH_SETUP_ERROR, and the shared test lock) and onto a per-instance InstalledMeshAuth on each QuicMeshTransport. Two concurrent mesh nodes now hold independent identities, allowlists, and setup state with no cross-contamination (issue #2652, D14). The install seam is hew_quic_mesh_transport_install_auth(transport, snapshot): hew_node_start builds the transport's auth material from the node's frozen PeerAuthSnapshot before listen, so the mTLS handshake admits exactly the bound peers and a poisoned setup refuses to bind. mesh_identity_load_or_create now returns typed MeshIdentityMaterial that Node::load_keys stages into the Building config instead of writing a global; node_peer_auth_setup_failed and the Node::start pre-flight read the per-node config setup_error rather than a process-global failure record. Removed the now-dead extern-C global allowlist wrappers (hew_quic_mesh_peer_spki_add/remove/clear) and their symbol-classification entries. Migrated the mesh unit tests and the quic_mesh_spki_allowlist integration test to independent per-transport install, and added a mesh_transport_auth_isolation test proving two transports stay isolated.
…S6) Expose Node::identity_key() -> string returning this node's own stable public credential for the pinned transport as lowercase hex (cert SPKI on quic-mesh, Noise public key on tcp-noise), or "" before an identity is loaded. Peers pin it via their own allow_peer, completing the credential round-trip. - runtime: hew_node_api_identity_key three-state read of the staging config (Building cfg / held Starting.config / retained Running.identity_export) via malloc_cstring; empty-not-null, never derefs the owner pointer. - load_keys: early transport pin in the Building config so identity_key can export the pinned mesh credential before Node::start freezes the snapshot (idempotent with merge_start_env_into_config). - ABI registration across layers: HIR catalog (RuntimeFfiShim), types registration, MIR FreshOwnedString ownership contract + CONTRACT_SYMBOLS (count 170->171), jit-symbol classification. - goldens: surgical 'declare ptr @hew_node_api_identity_key()' after allow_peer in native + wasm32 r4_runtime_abi.ll, byte-verified against real codegen output (local toolchain cannot run make ll-diff; see handoff). - tests: runtime three-state unit test (no owner deref); e2e peer-auth fixture asserts identity_key exports non-empty lowercase hex pre-start. Refs #2652
…S6) Complete the tcp-noise half of the peer-identity binding: a node now presents a stable Noise static identity that peers can pin, and the Noise admission pre-gate consults this node's per-node snapshot instead of a process-global allowlist. - encryption: noise_identity_load_or_create(path) loads (or mints and persists owner-only) a stable X25519 Noise keypair; an existing file is loaded verbatim (never re-minted), a wrong-size file rejected fail-closed. - load_keys: transport-selected — mesh TLS identity on quic-mesh, stable Noise keypair on tcp-noise; plain quic rejected (no pinnable identity). Both stage into the Building config and pin the transport early. - connmgr: hew_connmgr_add presents mgr.auth.noise_identity() (stable) on every connection instead of a fresh per-connection generate_keypair(), which had made a node's Noise public key unbindable. The strict Noise pre-gate now rejects a peer key absent from THIS node's snapshot allowlist (mgr.auth.noise_pubkey_allowlisted) rather than the global ACTIVE_ALLOWLIST; authorize() then binds the key to the claimed NodeId. Unverified (loopback dev / opt-out) stays delivery-only. The standalone hew_allowlist_* C API and its EncryptedTransport keep the global allowlist unchanged (separate user-facing surface, not the distributed-node path). - tests: noise_identity_load_or_create persistence/stability + wrong-size reject. Downgrade/missing-credential, mismatch, unpinned-reject, pinned- accept and allowlist decisions remain covered by the authorize/snapshot unit tests (peer_binding). Full runtime lib matrix green (default / quic / encryption / no-default, 2100 each). Refs #2652
…/D6) Complete the peer-identity admission unit so an authenticated key can only claim the NodeId it is bound to, and an Unverified connection is delivery-only across both the control and data planes. Control + data-plane gating (D9/D12): - authenticated_peer_node_id_for_conn resolves control authority only for a Strict posture + Published claim + exact conn-id owner (supersede-safe); peer_node_id_for_conn stays the posture-agnostic delivery id (D9 single Unverified capability). - Gate registry-gossip apply/flush, SWIM, and cluster publication behind the authenticated peer; inbound asks from a non-owner/Unverified peer are rejected with a diagnostic (no reply, no route, no membership side effect). - PendingReply completion validates the originating (conn_mgr, conn_id); a reply arriving on a different connection is dropped (complete/fail_remote_reply now take the origin connection). Mesh SPKI credential extraction (D6): - hew_transport_quic_mesh_peer_spki recovers the peer pinned leaf SPKI so a quic-mesh Strict connection resolves PeerCredential::Spki and binds the claimed NodeId to the authenticated key (previously left credential-free, which made mesh Strict admission fail closed). Credentialed test harnesses (no test-only posture promotion): - start_authorized_quic_mesh_pair installs cross-bound SPKI->NodeId snapshots on in-process mesh nodes; the 12 two_node ask/reply/worker/SWIM mechanism tests now exercise a real Authorized connection. - start_authorized_tcp_node + broker_two_process_noise_keys give the three cross-process tests a real bidirectional TCP-Noise key exchange, binding each peer Noise static key before connect. - Unverified delivery/rejection tests retained as policy controls; added unverified_gating and reply_binding coverage.
…S8) Close out the peer-identity admission unit with the wire-identity proof, documentation, and the strengthened durable lesson. Wire-identity E2E (BLOCK-5 point 1): - wire_identity_peer_observes_configured_node_id asserts that after an authorized quic-mesh handshake the responder observes the initiator under its exact operator-pinned HEW_NODE_ID (not a PID-derived value), and that the observed id is credential-bound (authenticated), not merely declared. Non-vacuous: an unbound/PID-derived id would fail admission and never route. Docs: - HEW-DIST-SPEC.md gains 12.2 (NodeId <-> authenticated-credential binding): an admitted key may claim only its bound NodeId; the handshake NodeId is the operator-pinned id; Unverified is delivery-only across control AND data planes; per-node auth isolation. - CHANGELOG.md documents the breaking two-arg Node::allow_peer signature, the new Node::identity_key export, and the admission-binding security model. Lesson: - Strengthen ctrl-frame-binds-to-authenticated-peer: "authenticated peer" is the exact current owner of a Published, credential-bound claim; the handshake NodeId must be credential-bound AND equal to the pinned HEW_NODE_ID; Unverified is delivery-only across both planes; reply completion validates the originating connection. Composes with the numeric-pin work (#2646). Composition with #2646 (numeric pin) is automatic: every CTRL_* handler already consumes authenticated_peer_node_id, which now delegates to the stricter exact-owner anchor, so Unverified peers are rejected across all control frames.
CTRL_LINK_DOWN was authorised via the self-declared delivery NodeId (peer_node_id_for_conn), which stays nonzero even for an Unverified (delivery-only) or superseded connection. A peer whose declared id matched a stored link's remote_node_id could therefore drive a cross-node link-exit and crash an actor linked to the genuine, still -alive owner, because deliver_link_down_to_ref only rejects an authenticated peer of 0. Route the handler through the exact-owner gate (authenticated_peer_node_id) so an Unverified/superseded frame resolves to no identity and is dropped with a diagnostic. Symmetrically, outbound registry gossip and SWIM membership traffic gated only on ACTIVE + gossip-capable, so an Unverified peer could still receive cluster state and be probed. Gate all four outbound paths (active_gossip_connection_ids, flush_registry_gossip_to_connection, hew_connmgr_active_swim_peers, hew_connmgr_send_swim) on the exact authenticated claim owner, snapshotting connection ids before the claim-lock check so it never nests inside the connections read guard. Fire-and-forget user-message delivery to Unverified peers is unchanged. peer_node_id_for_conn had exactly one production caller (the link_down handler); inbound delivery routes via the admission-populated routing table, so it becomes a test-only accessor for the wire-identity check. Adds negative tests proving a gated link-down leaves a diagnostic for both an Unverified and a superseded owner, and that an Unverified peer receives no outbound gossip or SWIM while delivery stays allowed; the existing gossip-broadcast test now seeds published claims.
The peer-authentication guide named HEW_NODE_ID but never showed how to launch a node with it, and gave no migration path off the old one-argument allow_peer. Add a runnable HEW_NODE_ID launch example, describe the identity_key() credential-exchange workflow, and document replacing every one-argument allow_peer(node_id) call with the two-argument allow_peer(node_id, credential_hex) form (the one-argument form no longer compiles; a malformed credential fails closed at Node::start). The distributed_hello example now prints identity_key() and shows the HEW_NODE_ID launch line so the credential-pinning workflow is demonstrable end to end. Corrects the changelog to state that Unverified peers neither inject nor receive registry gossip or SWIM membership.
An outbound ask is control-bearing: its reply deposits into the local pending table and its dispatch runs a handler on the peer. It must go only to the exact authenticated owner (Strict posture + published claim) of the target NodeId. Previously setup_remote_ask serialized and sent the ask regardless of the target connection's posture; an unverified or superseded target silently dropped the inbound ask, so the initiator saw only a Timeout, with reply-completion validation the sole backstop. Gate at initiation: reject with AskError::Unauthorized when authenticated_peer_node_id_for_conn != target_node_id, before any bytes are serialized or a pending reply is registered. This is the outbound symmetric of inbound_ask_denied_unverified. Migrate the five pending-ask tests that drove asks over unverified loopback-TCP connections onto a genuinely credentialed pair (start_authorized_tcp_pair, cross-pinned Noise identities). Add explicit coverage: an unverified outbound ask returns Unauthorized, and the outbound gate authorizes only the exact owner (unverified/superseded rejected).
Node::set_transport only wrote the process-global HEW_TRANSPORT env. It could flip the transport after Node::allow_peer / Node::load_keys had already staged credentials — which allow_peer/load_keys interpret under the pinned transport (a 32-byte Noise pubkey on TCP vs a cert SPKI on quic-mesh). A later flip silently reinterpreted the staged material, and merge_start_env_into_config then unconditionally overrode the config transport from env at start, so the started node could bind a transport inconsistent with its staged identity. Pin the selection in the Building config and reject a post-staging change: - add PeerAuthConfig::has_staged_credentials (bindings or a stable Noise/mesh identity present); - set_transport now acquires the staging lock and, when credentials are already staged, refuses fail-closed (no env mutation) any change to a different transport; an idempotent same-transport re-selection is allowed; on success it pins cfg.transport as well as the env; - allow_peer pins cfg.transport after staging (idempotent with load_keys / set_transport); - merge_start_env_into_config treats a pinned selection as authoritative (env wins only when unpinned) and re-asserts the effective selection onto HEW_TRANSPORT so the low-level transport construction builds the stored selection. Tests: a post-staging set_transport is rejected and changes neither the pinned config nor HEW_TRANSPORT; the exported identity_key and the pinned selection start will use are stable across a rejected flip.
The peer credential/NodeId binding work changed the ABI-classified symbol
set in scripts/jit-symbol-classification.toml, so the count of classified-
but-uncontracted symbols the verifier tracks moved from 987 to 985.
Audit of the two classification changes (net -2):
+1 hew_node_api_identity_key — new #[no_mangle] runtime export
(Node::identity_key exporting this node's stable public credential).
It is classified `stable` and carries no ownership contract, exactly
like every sibling hew_node_api_* symbol and hew_uint_to_string: the
owned-string allocation contract (allocate via malloc_cstring / free
via hew_string_drop) is proven by the runtime FFI round-trip tests
(node_identity_key_reads_three_state_config_without_owner_deref,
identity_key_stable_across_attempted_transport_flip), not by a
jit-symbol ownership contract. So it adds 1 uncontracted symbol.
-3 hew_quic_mesh_peer_spki_add / _remove / _clear — retired exports.
The D14 per-node transport-auth isolation moved the mesh SPKI
allowlist off the process-global ACTIVE_* surface onto per-node
snapshots, so these three #[no_mangle] exports no longer exist in
hew-runtime. They were classified `stable` and uncontracted, so
removing them drops 3 uncontracted symbols.
987 + 1 - 3 = 985. The verifier confirms 985 with no "unclassified
runtime exports" or "not ABI-classified" errors, i.e. the classification
still exactly matches the live runtime export set.
Under strict peer binding, plain-TCP loopback connections run Unverified (delivery-only): they cannot resolve the registry, ask, or gossip. The distributed two-process E2E suite still stood up plain-TCP nodes, so every cross-process scenario that needs a lookup now correctly fails closed with `lookup-unresolved` (ask, node identity/location, stale ref, remote monitor clean-exit/crash/connection-drop/setup-after-drop, watcher-node death, link cascade clean-exit/crash/partition/non-crashlinked, cross-node monitor-close reclaim, wire CBOR round-trip, partition ask). Migrate the whole shared harness to real authorized TCP-Noise: - Fixture (`dist_node.hew`): each node mints a stable Noise identity via `Node::load_keys`, publishes its `Node::identity_key()` to a shared key-exchange dir (atomic temp-then-rename), waits for the peer's, and binds it with `allow_peer(peer_node_id, key)` BEFORE `Node::start` — a real mutual authenticated pin, never an injected posture. The server signals listener liveness via a `server.ready` file (the client cannot observe the harness READY line, and a failed one-shot connect is not retried); the client connects to `1@127.0.0.1:<port>` so admission binds the server's claimed NodeId to its authenticated key. - Harness: a `SecureScenario` context owns the key-exchange dir and pins each role's `HEW_NODE_ID` (server 1, client 2, required by strict mode). All five scenario runners launch BOTH processes concurrently — the server cannot reach READY until the client has published its credential (each side stages `allow_peer` before start), so the old launch-then-wait-then-launch ordering would deadlock. Strict mode is never relaxed and no test-only posture is injected; the separate Unverified delivery-only runtime tests are untouched. Refs #2652
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Peer credentials are now bound to the specific node identity that claims them, closing a remote-impersonation gap where an unauthenticated peer could satisfy admission by presenting any accepted key regardless of which node it claimed to be. Node identity is now a stable, per-node Noise key (persisted across restarts), and
allow_peerbecomes a two-argument call —allow_peer(node_id, key)— that binds a credential to one claimedNodeIdrather than to a shared allowlist. Binding is enforced consistently across the TCP-Noise and QUIC mesh transports, on both control-plane and data-plane operations (admission, inbound/outbound ask, gossip/SWIM), with a claim state machine that admits first-claim, allows same-credential reclaim, and rejects a different credential claiming an already-bound identity. Unverified/no-binding mode remains available but is now strict by default: it is loopback-scoped and delivery-only, with no non-loopback peer able to operate under an unverified identity unless explicitly configured. This is a breaking change toallow_peer's signature; existing callers must migrate to the two-argument form.Verification
make ci-preflightat the gated head (fee2b45ac0): 14/14 lanes greenOut of scope
EncryptedTransportpath and its associated allowlist globals — dead code with zero production callers, left for a follow-up cleanupFixes #2652