API Model Auditor

Try: https://api.tool.modeldetect.txzy.net/

API Model Auditor is a local, black-box audit tool for LLM APIs.

It is built for one practical question:

"What model tier is this API actually delivering, how good is it, and how confident can I be that it matches what it claims?"

The auditor works with:

• OpenAI
• Anthropic
• Google Gemini
• OpenAI-compatible gateways, relays, and proxies

It produces a single, easy-to-read outcome:

• an overall quality score from 1 to 100
• a likely model range such as frontier / strong / mid / weak
• a heuristic claim/observation match indicator from 0 to 100 (not a calibrated probability)
• supporting probe evidence and ranked candidate hints

The old research prototype is archived under backup/.

Basic Introduction

This project is not a benchmark suite and not a chatbot wrapper. It is a black-box auditor that sends a small set of probes to an API, analyzes the replies, and turns those replies into an operational assessment.

Use it when you want to answer questions like:

• Is this API really a frontier-class model?
• Is it closer to a mini / flash / haiku tier?
• Does the returned behavior match the declared provider?
• Is the service consistent enough to trust for real work?

The main interface is the audit command. The local dashboard and history store are built around the same audit result.

Usage

1. Install

uv sync

2. Configure

You can provide settings via CLI flags or environment variables.

Supported environment variables:

• LLM_AUDIT_PROVIDER
• LLM_AUDIT_API_KEY
• LLM_AUDIT_BASE_URL
• LLM_AUDIT_MODEL
• LLM_AUDIT_DB_PATH
• LLM_AUDIT_DASHBOARD_API_BASE_URL
• LLM_AUDIT_CORS_ALLOW_ORIGIN

Provider-specific fallbacks are also supported:

• OPENAI_API_KEY, OPENAI_BASE_URL, OPENAI_MODEL
• OPENAI_API_MODEL
• ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, ANTHROPIC_MODEL
• ANTHROPIC_AUTH_TOKEN, ANTHROPIC_API_MODEL
• GOOGLE_API_KEY, GOOGLE_BASE_URL, GOOGLE_MODEL
• GEMINI_API_KEY, GEMINI_BASE_URL, GEMINI_MODEL

Provider styles:

• openai
• anthropic
• google
• openai-compatible

If --model is omitted, the tool falls back to:

• openai / openai-compatible -> gpt-5.3-codex
• anthropic -> claude-sonnet-4.5
• google -> gemini-2.5-pro

3. Run an audit

uv run api-model-auditor audit \
  --provider openai \
  --model gpt-5.4 \
  --api-key sk-... \
  --base-url https://api.openai.com/v1

If your values are already in the environment, you can simply run:

uv run api-model-auditor audit

4. View history

uv run api-model-auditor history

5. Open the dashboard

uv run api-model-auditor dashboard

The dashboard is a minimal local web UI for:

• launching a new audit
• browsing saved runs
• inspecting the overall score, model range, and evidence

If you deploy the frontend separately from the Python backend, point the dashboard to the real backend API origin:

uv run api-model-auditor dashboard \
  --api-base-url https://api.tool.modeldetect.txzy.net \
  --cors-allow-origin https://your-frontend.vercel.app

Equivalent environment variables:

LLM_AUDIT_DASHBOARD_API_BASE_URL=https://api.tool.modeldetect.txzy.net
LLM_AUDIT_CORS_ALLOW_ORIGIN=https://your-frontend.vercel.app

Use the full origin, including https://, and do not add a trailing slash.

6. Optional database location

By default, audits are stored in SQLite at:

.llm_audit/audits.sqlite3

Set LLM_AUDIT_DB_PATH if you want to store history elsewhere.

Output

Every audit is summarized into a compact report with:

• Quality score: 1 to 100
• Quality tier: frontier, strong, mid, or weak
• Likely model range: a coarse range derived from behavior
• Match indicator: 0-100 heuristic score comparing the API's claimed model family to the observed behavior (not a calibrated probability)
• Ranked candidates: likely backend family/variant hints
• Probe evidence: the raw behavioral signals behind the score

The score is designed to answer the practical question first:

"Is this API delivering high-quality model behavior, or not?"

The model-family guess is secondary and should be treated as a supported hypothesis, not a proof. The raw model field returned by the API is treated as a claim being audited, not as identification evidence.

Principle

The auditor follows a layered black-box workflow:

1. Probe generation

   • Send a small set of behavior probes that test formatting, reasoning, instruction following, stability, and structured output.

2. Transport normalization

   • Speak native OpenAI, Anthropic, Google, or OpenAI-compatible API shapes.
   • Normalize the returned payload into a common internal format.

3. Feature extraction

   • Convert each response into measurable features such as length, line count, fence usage, repetition, latency, and answer correctness (where the probe has a verifiable answer).

4. Candidate scoring

   • Rank candidates using behavior features only. The API's self-reported model field is intentionally excluded from candidate scoring so it cannot be used to identify itself.

5. Quality scoring

   • Combine probe results into a 1-100 quality score and a coarse tier. Probes with verifiable answers (math, JSON keys, code structure) drive the score primarily by correctness, not formatting.

6. Claim/observation comparison

   • Compare the claimed model family (from the request) with the family hinted by the API's response. Emit a heuristic 0-100 match indicator and flag mismatches in the report notes. This number is NOT a calibrated probability.

7. Persistence and dashboard

   • Save every run to SQLite.
   • Reuse the same history for CLI output and the local dashboard.

The important design choice is this:

Quality comes first. Model identity comes second.

That keeps the tool useful even when the backend is a relay, a proxy, or a partially hidden gateway.

Reference

This project is informed by a few related directions:

• Model substitution auditing: black-box methods for checking whether an API is really serving the model it claims.
• Behavioral fingerprinting: probe-based family and variant inference from outputs.
• Proxy / relay auditing: checking whether a gateway is modifying requests or responses in unexpected ways.

The archived backup/ folder contains an earlier prototype kept only for historical reference. The current auditor does not depend on it.

Development

uv run pytest -q
uv run python -m compileall -q .

Project Layout

• cli.py - command line entry point
• detector/ - audit logic, scoring, storage, and dashboard server
• backup/ - archived research prototype

License

No license has been declared yet. Add one before distributing the project.