Skip to content

Add end-to-end integration test with real LLM provider #3

Description

@riccardomerenda

Problem

All tests mock LiteLLM responses. This gives confidence in internal logic but doesn't verify that the tool works against real APIs — response format changes, unexpected error codes, or LiteLLM breaking changes would go undetected.

Proposal

Add an integration test (in tests/test_integration.py or similar) that:

  1. Uses a cheap model (e.g., openai/gpt-4o-mini or anthropic/claude-haiku)
  2. Runs a minimal corpus (3-5 prompts, 1 batch)
  3. Verifies the full pipeline: executor → quality → decision → report output
  4. Gated behind a marker: @pytest.mark.integration (skipped in CI by default, runnable with pytest -m integration)

Acceptance criteria

  • Test passes with a live API key
  • Skipped cleanly when no key is set
  • Documented in CLAUDE.md / README

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions