[RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol

[ethan-scitix]

Motivation

Model's access interface is overfit to the OpenAI /v1/completions protocol, and
that assumption is about to break in three directions at once: RL/logprobs
consumers, the "big three" latest APIs (OpenAI Responses, Anthropic Messages,
Google genai / AI Studio), and richer eval logic (reasoning, multi-turn, continuation).

Evidence the current abstraction leaks:

• Model.alogprobs(prompt, *, max_tokens, logprobs: int, echo, temperature) is
  completions vocabulary. ChatModel._alogprobs_impl accepts echo and silently
  ignores it — a phantom param on the universal interface.
• Every non-completions backend must translate: echo -> logprob_start_len,
  logprobs: int -> top_logprobs_num, max_tokens -> max_new_tokens.
• echo-based prompt scoring is itself an OpenAI workaround for the lack of a
  native scoring endpoint; sglang /generate has return_logprob and no echo
  concept. The base models a workaround as if it were the universal primitive.
• The output shape (top_logprobs: list[dict[str, float]]) cannot carry token ids
  or a reasoning channel — both required by real consumers (RL, reasoning eval).

The interface was only ever validated by one protocol. The incoming sglang native
backend, plus the big-three protocols, are the "second implementors" that expose the
boundary. Rather than keep bolting workarounds on, define a provider-agnostic
IR for model capabilities and make every provider an adapter onto it.

Proposed Change

Core idea — one Model IR + transports as frontends

• One Model IR, two message shapes: Request and Response (the existing
  ModelOutput is the Response side; rename or alias).
• Transport = provider frontend (strategy, composed into Model, not a base
  class): it lowers a Request into its native wire and lifts the native
  response back into a Response. OpenAI completions / OpenAI chat / OpenAI
  Responses / Anthropic Messages / Google genai / sglang native / vLLM are all just
  frontends.
• Capabilities = which IR features a frontend can honor, declared per transport
  as optional Protocols. The catalog is open: adding a capability later =
  add a Protocol + optional Request/Response fields (additive, not a redesign).
• Validation = a Request using a feature the transport lacks is rejected at
  setup, never silently ignored (kills the echo-on-chat class of bug).

Composition (Model)

Model = { resource pool / limiter (explicit collaborator) } + { Transport (frontend) }
  async def arun(req: Request) -> Response          # the one primitive (acquires limiters)
  # capability-gated sugar (thin wrappers over arun, present only when supported):
  #   agenerate(...)      [any]
  #   achat(...)          [Chat]
  #   ascore_input(...)   [InputScoring]

• Drop as_type (cross-kind base<->inst conversion): its only use case
  (mixing base and inst on one endpoint in a single eval) has no real consumer.
• with_args decomposes: per-task infer_args become plain Request fields
  (no derived model object); concurrency_limit sub-quota becomes an explicit
  resource-pool collaborator. No "conversion capability" axis survives.

Consumer -> capability matrix

consumer	capabilities
1. RL / agentic	Chat + Tools + SampledLogprobs(+token_ids) + (Reasoning)
2. reasoning eval	Chat + Reasoning + big-three adapters (Responses/Messages/genai)
3. multi-turn	Chat
4. base ppl / bpb	Completion + InputScoring(+byte/char accounting) + TopKLogprobs
5. special (budget/truncate/continuation)	Reasoning(controls) + Prefill / FIM

Carved OUT of the model interface (task layer): fallback-extract, LLM-judge
orchestration, thinking-truncation post-processing. (judge = a Chat call + task scoring.)

Provider -> capability matrix

transport	capabilities
OpenAI completions	Completion, InputScoring(echo), TopK
OpenAI chat	Chat, Tools, TopK, Structured
OpenAI Responses	Chat, Tools, Reasoning, Structured
Anthropic Messages	Chat, Tools, Reasoning(+budget), Prefill
Google genai / AI Studio	Chat, Tools, Reasoning
sglang /generate	Completion, InputScoring, SampledLogprobs, TopK, Prefill
vLLM (oai-compat)	~OpenAI + prompt_logprobs

IR shapes + capability catalog

Request:  input: str | list[Message]            # modality (rides with transport)
          sampling (temp/top_p/max_tokens/stop/seed)
          tools? ; reasoning?(budget/effort) ; response_format?
          prefix? / suffix?                      # constrained-context corner (Prefill/FIM)
          return_logprobs? / score_input? / top_k?

Response: texts
          reasoning?: { text, opaque_roundtrip } # opaque needed for AnthropIc/Google multi-turn signatures
          tool_calls?
          logprobs?(+token_ids) ; top_logprobs?
          input_scoring?(byte/char)              # for BPB/CLP
          usage

Capabilities: Completion | Chat | Tools | Reasoning | Prefill | FIM |
              InputScoring | SampledLogprobs | TopKLogprobs | StructuredOutput

• Core-now (has consumers today: feat(theoremqa): add dataset and k-shot base-model task #3/feat(session): tolerate throughput-only config diffs on --resume #4 + sglang track):
  Completion, Chat, InputScoring, TopKLogprobs.
• Staged (design the seam now, implement on demand):
  Reasoning, Tools, Prefill, FIM, SampledLogprobs, StructuredOutput.
• token_ids field name/shape stays deferred until the RL consumer pins it
  (aligned-with-logprobs vs standalone output_ids; prompt ids or not).

Feedback Period

~1-2 weeks (design-heavy).

Any Other Things

• Lock with RFC [RFC]: Resume version-compatibility gate + per-record version provenance #24. The Response IR is the persisted record schema
  (@sieval_record). So IR evolution is exactly what [RFC]: Resume version-compatibility gate + per-record version provenance #24's resume version gate
  protects: a breaking IR change ships as a >= minor bump and the gate refuses
  cross-series resume. The two RFCs are coupled by design.
• Non-goals.
  • Environment / sandbox execution (e2b-style RPC) is a sibling abstraction,
    not a model transport. The Model IR only needs to represent tool_call /
    tool_result; who executes them (e2b, local) is a separate Executor IR /
    future RFC. Folding sandbox RPC into the model transport repeats infra-layer
    role confusion.
  • Task-layer eval logic (fallback-extract, judge orchestration, truncation).
  • Structured-FIM wiring until a FIM benchmark exists (OSS FIM is usually
    Completion + a task-built template, not a transport feature).
  • as_type / cross-kind model conversion.
• Sequencing. The sglang native backend (feat(models): add SglangGenModel for echoed-input logprobs via sglang /generate #21 + follow-up: SglangGenModel
  onto Model[str] + native /generate) lands first under the current interface,
  translating params internally. This RFC is the eventual clean target and does not
  block that work.
• Dependencies. Big-three adapters mean multiple SDKs -> optional-dependency
  groups + lazy transport loading.
• Capability requirement declaration. Extend @sieval_task to declare required
  capabilities (today it only declares model_type); assert at setup. Replaces the
  current type: chat | gen with a richer capability requirement.

Before submitting a new issue

• Make sure you already searched for relevant issues and documentation.