[RFC]: Recipe capability layer — separate model-type capabilities and eval serving requirements from hardware profiles

[ethan-scitix]

Motivation.

The inference recipe system organizes serving params as a single flat bag per profiles[hardware][precision][framework], matched by family + size only (no base-vs-instruct awareness). Two problems follow:

1. Capability params live in the wrong place. tool_call_parser, enable_auto_tool_choice, reasoning_parser are baked into every hardware×precision profile. They are meaningless for a base-model logprob eval, but a base checkpoint of the same family/size resolves to the same profile and inherits them anyway.

2. There is no channel for an eval to state what it needs from serving. Two live instances already exist:

   • Input logprobs / prefix cache. PPL tasks that read the input prompt's logprobs (echo=True) require the serving framework's prefix cache to be off. Verified backend behavior:
     • sglang (radix cache, on by default): on a cache hit it truncates input_token_logprobs to prompt_tokens − cached_tokens and does not recompute cached positions — silently returning a partial, non-reproducible token set. Needs --disable-radix-cache.
     • vLLM V1: prompt_logprobs/echo is mutually exclusive with automatic prefix caching — the request errors rather than truncating. Needs --no-enable-prefix-caching.
     • Output-only logprob tasks (e.g. CMMLU, echo=False, first-output-token top-k) are unaffected and benefit from the cache. So the trigger is "reads input logprobs" (echo=True), finer than eval_mode.
   • Top-k breadth. CMMLU needs the server launched with --max-logprobs 100 on vLLM (default 20; sglang serves 100 out of the box) so all option tokens appear in the top-k. Today this relies on an operator reading the task docstring.

Neither problem is served by the current recipe structure; both point at the same missing dimension — model capability.

Proposed Change.

Reorganize recipes around two independent param sources, kept separate by design (different owners, lifetimes, vocabularies).

Mechanism A — model-type capability profiles (recipe-authored). Split the flat profiles bag into a hardware layer (hw × precision × framework: dtype, memory, parallelism, context) and a capabilities layer (model-type × framework: parsers, tool-choice, and model-intrinsic serving recommendations such as gpt-oss's prefix-cache-off). A recipe entry serves both the base and instruct checkpoint of a family/size; capabilities holds both variants and resolution picks one.

qwen2.5-7b:
  size_range: [6, 8]
  hardware:
    H100-80G:
      bf16:
        vllm:   { dtype: bfloat16, gpu_memory_utilization: 0.90, max_model_len: 32768 }
        sglang: { dtype: bfloat16, mem_fraction_static: 0.85, context_length: 32768 }
  capabilities:
    instruct:
      vllm:   { enable_auto_tool_choice: true, tool_call_parser: hermes }
      sglang: { tool_call_parser: qwen }
    base:
      vllm: {}
      sglang: {}

Model type comes from the model config's existing type field (chat → instruct, gen → base; default chat), threaded into resolution — no checkpoint sniffing (unreliable: modern base checkpoints often ship a chat template).

Mechanism B — eval serving requirements (eval-authored, backend-abstract). A task declares needs_input_logprobs: bool and min_top_logprobs: int | None on its metadata. sieval run unions these across the tasks resolving to each served base model (following the derived→base chain), stamps a typed ServingRequirements onto the DeploymentPlan, and the translator maps it to backend flags:

• needs_input_logprobs → sglang --disable-radix-cache / vLLM --no-enable-prefix-caching
• min_top_logprobs=n → vLLM --max-logprobs n (sglang default suffices)

The translator suppresses a flag when engine_params already expresses that knob in either direction, so a recipe capability param (gpt-oss quirk) or a user override wins — no contradictory flag pair. "Trust the user": overrides beat the eval requirement; the runtime guard (below) is the backstop.

Loader is strict: the legacy flat profiles key is rejected with a migration error (recipe files are in-repo; user customization is via overrides, unchanged).

Reproducibility. The recipe schema never reaches a persisted artifact (effective_config.yaml references the recipe by name; infer_plans.yaml holds the resolved plan). Two guarantees, enforced by tests: (1) instruct-path engine_params stay byte-identical to today (golden characterization test); (2) serving_requirements is serialized into infer_plans.yaml only when non-empty, so no-requirement plans keep byte-identical and --resume matching is unaffected.

Layer boundaries are preserved: infer never imports tasks; ServingRequirements is plain data built by the CLI.

Implementation is planned in two phases: (1) the recipe schema split + model-type resolution + migration + golden test; (2) the ServingRequirements channel + translator mapping + CMMLU wiring. Each is independently shippable and testable.

Feedback Period.

One week.

Any Other Things.

• Runtime backstop. sglang's silent truncation is caught by a SglangGenModel echo-guard (fails loud when the returned input-logprob count doesn't match the prompt) — tracked as a follow-up to feat(models): add SglangGenModel for echoed-input logprobs via sglang /generate #21, and the reason this RFC can "trust the user" on overrides.
• Relationship to open RFCs. This is orthogonal to [RFC]: Resume version-compatibility gate + per-record version provenance #24 (resume version gate) and [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25 (capability-based Model IR): [RFC]: Resume version-compatibility gate + per-record version provenance #24's version gate is what makes cross-version infer_plans.yaml drift explicit; [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25 is the model-access decoupling, whereas this RFC is about serving-launch configuration.
• Out of scope. New echo=True benchmarks (HellaSwag/ARC); auto-deriving requirements inside standalone infer start (no task context — operator sets flags or the guard fires); runtime multi-server partitioning (base vs instruct are already separate served models).

Before submitting a new issue...

• Make sure you already searched for relevant issues and documentation.