Skip to content

refactor: implement observed capability providers - #45

Merged
rcyw merged 11 commits into
mainfrom
refactor/observed-capability-providers
Aug 9, 2026
Merged

refactor: implement observed capability providers#45
rcyw merged 11 commits into
mainfrom
refactor/observed-capability-providers

Conversation

@rcyw

@rcyw rcyw commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Introduce the Observation → Policy → Enforcement capability architecture:
    • replace string-based node capabilities with typed observations and atomic snapshots;
    • centralize platform capability derivation, evidence validation, and loss policy;
    • keep extension capabilities typed, namespaced, and separate from platform capabilities.
  • Enforce capability correctness across the allocation lifecycle:
    • re-evaluate typed requirements under the PostgreSQL node-row lock;
    • persist placement and create-time evidence with allocation dependencies;
    • reconcile capability loss with allocation-specific verification and fail-stop handling.
  • Add observed capability providers and durable recovery:
    • verify cgroup, runtime, filestore, EROFS, and network facts;
    • run bounded runc/runsc conformance self-tests;
    • preserve fresh node heartbeats and summaries while capability providers are warming;
    • reconcile runtime state, rootfs storage, and incomplete teardown after restart.
  • Remove obsolete contracts:
    • delete dynamic capability callbacks, boot markers, and arbitrary platform strings;
    • remove compatibility aliases and writable-layer terminology;
    • require coordinated controld, axnoded, proto, SDK, database, and node-state rollout.

Validation

  • Continuous integration:
    • Open Source, Repo Assets, Go, Rust, TypeScript, Python, SDK Artifacts, Documentation, Release Contracts, Axrun Gates, Managed Rollout Gate, and DCO passed.
  • HK functional acceptance:
    • release axern-hk-20260809104110 completed with 6/6 nodes ready;
    • managed Rollout completed with healthy diagnosis and complete evidence;
    • OCI and Nydus cold/warm 36-service matrices completed with balanced six-node placement;
    • Nydus cold Ready P95 was 1.957s against the 2.5s gate;
    • crash-like axnoded replacement, recovery, cleanup, and post-fault verification passed.
  • Runtime cleanup verification:
    • runc/runsc runtime inventories were empty after validation;
    • capability self-test bundles, projections, and writable artifacts were fully removed;
    • no forced-teardown deadlock or cleanup failure remained.

Compatibility

  • This is an intentional coordinated breaking change:
    • mixed old/new controld, axnoded, proto, or SDK versions are unsupported;
    • development databases and node-local state must be rebuilt during rollout;
    • legacy capability strings and fallback behavior are not retained.

Closes #42

rcyw added 11 commits August 9, 2026 14:41
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
Signed-off-by: wayne <rcywjo@gmail.com>
@rcyw
rcyw force-pushed the refactor/observed-capability-providers branch from 391d070 to c0d5912 Compare August 9, 2026 10:40
@rcyw
rcyw marked this pull request as ready for review August 9, 2026 11:02
@rcyw
rcyw merged commit 1edfa1c into main Aug 9, 2026
12 checks passed
@rcyw
rcyw deleted the refactor/observed-capability-providers branch August 9, 2026 11:03
@rcyw rcyw changed the title Refactor node capabilities into observed providers refactor: implement observed capability providers Aug 9, 2026
rcyw added a commit that referenced this pull request Aug 10, 2026
- Redesign the observed capability contract.
  - Separate evidence identity from freshness and model config, boot, mount, runtime, and derived identities explicitly.
  - Centralize catalog ownership, audience, dependency, loss-policy, transition, and requirement derivation semantics.
  - Remove legacy capability strings, boot markers, compatibility fields, and denormalized JSON contracts.
- Enforce capabilities independently on every node.
  - Schedule provider observations independently and expire failed, malformed, or stale observations to UNKNOWN.
  - Derive allocation requirements locally before side effects and again after resolving actual rootfs backing.
  - Verify cgroup, process membership, project quota, OverlayFS, runsc envelope, runtime identity, and filestore enforcement.
  - Serialize durable capability-loss reconciliation and fail-stop cleanup per allocation.
- Make control-plane admission and reporting atomic.
  - Re-evaluate typed eligibility under PostgreSQL node locks and persist complete placement and create proofs.
  - Normalize dependencies, conditions, transitions, and reconcile generations into indexed tables.
  - Keep capability conditions separate from allocation lifecycle and avoid full-node allocation JSON scans.
- Complete operator and recovery contracts.
  - Add typed CLI diagnostics for snapshots, transitions, backlog, conditions, and allocation evidence.
  - Keep runtime inventory recovery fail-closed until every configured runtime handler is available.
  - Validate with generated contracts, race tests, PostgreSQL tests, Linux runtime tests, full CN verification, CI, and HK lifecycle fault recovery.

Closes #46.
Relates to #42, #43, and #45.

Signed-off-by: wayne <rcywjo@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

proposal: replace dynamic node capabilities with observed capability providers

1 participant