proposal: define a runtime-neutral memory overhead admission contract

[rcyw]

Problem

Controld currently reserves a fixed amount of additional node memory for each runsc sandbox and persists it as memory_overhead_bytes. The default is intended to account for host-side Sentry and gofer memory that is not represented by the workload-declared memory request.

Runsc writable rootfs now uses a mandatory file-backed overlay (root:dir) instead of root:memory. That removes an unbounded memory-backed writable layer, but it does not make runtime memory overhead zero. Sentry, gofer, runtime metadata, kernel objects, and the file-backed overlay page cache can still consume memory and must be included in node admission or cgroup attribution.

The present policy is directionally safe, but its long-term contract is underspecified:

• the configuration and policy are named specifically for runsc rather than modeling runtime-class overhead;
• the fixed 256 MiB default is conservative and is not yet backed by production attribution measurements;
• it is not explicit whether public memory_bytes means workload-usable memory or the total sandbox cgroup budget;
• depending on the cgroup hierarchy and hard-limit boundary, adding overhead to placement while enforcing the declared limit on the whole sandbox can either preserve workload headroom or accidentally double count memory;
• runc and future runtime classes have no general way to declare measured host-side overhead;
• page-cache attribution for EROFS lower reads and file-backed writable-layer copy-up/writeback has not been qualified.

This is a non-blocking follow-up. Do not remove the existing runsc admission reserve merely because writable storage is file-backed; retain the current fail-safe behavior until the resource semantics and attribution are verified.

Proposed outcome

Define a runtime-neutral internal admission contract such as:

type RuntimeResourceOverhead struct {
    MemoryBytes int64
}

Each runtime class supplies an internally configured and/or measured overhead profile. The value remains an operator/internal scheduling policy and is not exposed as a user-settable ResourceQuantity field.

Lock the public memory semantics explicitly. The preferred model is:

workload-declared memory request
+ internal runtime memory overhead
= node scheduling commitment

Under that model, namespace quota charges the workload-declared request, while node admission reserves both values. The cgroup hierarchy and hard-limit behavior must ensure this does not double count or silently reduce the workload-usable budget. If Axern instead defines memory_bytes as the total sandbox cgroup budget, the separate placement overhead must be reconsidered.

Replace runsc-specific policy naming with runtime-neutral naming, for example RuntimeMemoryOverheadBytes in internal reservation types and a per-runtime profile in admission policy. Database storage may remain a generic memory_overhead_bytes column because it records the resolved reservation rather than a runsc API.

Required investigation

Measure and document memory attribution for representative runc and runsc sandboxes, including:

• Sentry and gofer anonymous/file memory;
• guest workload processes;
• shared memory;
• EROFS lower page-cache reads;
• file-backed writable-overlay reads, copy-up, dirty pages, and writeback;
• idle baseline, sandbox startup peak, steady-state overhead, and concurrent sandbox scaling;
• cgroup v2 memory.current, memory.stat, memory.events, OOM, reclaim, and writeback behavior;
• host runtime PIDs and workload PIDs actually attributed to the intended sandbox boundary.

The investigation must determine whether the configured overhead represents a fixed reserve, a runtime-specific estimate, or a node-level system reserve, and whether any part is already included in node allocatable memory.

Ownership and boundaries

• Runtime implementations own PID classification and runtime-specific attribution evidence.
• Axnoded owns cgroup enforcement, PID membership verification, and memory telemetry.
• Controld resource policy owns the resolved runtime overhead used for placement and reservation.
• The reservation ledger persists declared memory and resolved runtime overhead separately for diagnostics and reconciliation.
• Namespace quota continues to operate on the public workload request unless the public memory contract is deliberately changed.

The public API must not expose a memory_overhead_bytes resource knob. Users choose workload resources and runtime class; operators own runtime implementation overhead.

Alternatives considered

1. Remove the overhead because writable storage is file-backed. Rejected: file backing still consumes page cache, and Sentry/gofer remain host memory consumers.
2. Charge a fixed overhead only for runsc forever. Safe as a temporary policy, but it hard-codes one runtime and lacks evidence-based calibration.
3. Subtract a global worst-case runtime reserve from node allocatable. Simpler, but wastes capacity and does not scale with the active runtime mix.
4. Include all runtime overhead inside the user memory request. Possible only if the public contract clearly defines memory as a total sandbox budget and accepts reduced workload-usable memory.

Acceptance criteria

• Public memory_bytes semantics are documented unambiguously.
• Internal admission policy represents overhead per runtime class without a runsc-specific data model.
• Placement, transactional reservation, namespace quota, and cgroup enforcement are checked for double counting.
• Sentry, gofer, workload, page cache, dirty/writeback, and EROFS/file-backed-overlay attribution are tested on Linux.
• The default overhead values are justified by measurements with documented safety margins.
• Capability publication for memory hard limits remains fail-closed and requires PID attribution.
• Diagnostics expose declared memory, runtime overhead, total node commitment, actual cgroup usage, and enforcement status separately.
• runc and runsc concurrency tests verify node admission behavior near memory saturation.