Skip to content

Add cross-provider protocol conformance canaries #71

Description

@0xzr

Problem

A model returning a short chat completion does not prove that it supports streaming, tool calls, structured JSON, multimodal input, the Responses API, or the Anthropic Messages bridge.

Scope

  • Define a small provider/model conformance matrix for supported protocol features.
  • Add bounded canaries for chat, streaming, tool calls, JSON output, vision where advertised, Responses API translation, and Anthropic Messages translation.
  • Record feature-level verification separately from basic model health.
  • Route feature-specific requests only to targets that have passed that feature or are explicitly pinned by the user.
  • Preserve per-model failures and serving provenance in reports.
  • Re-run canaries after model-ID or adapter changes.

Acceptance criteria

  • Every advertised feature has a deterministic minimal canary and normalized expected result.
  • Unsupported capability errors are not counted as provider availability failures.
  • Streaming tests detect mid-stream resets and truncated clean-looking responses.
  • Tool/JSON tests reject syntactically successful but semantically invalid output.
  • Capability results are machine-readable and visible in model/status output.
  • Canaries are quota-bounded and never send repository or user content.

Related: #1 and #6.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestprovider-catalogProvider catalog, keys, limits, or model metadatatestsTests and fixtures

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions