Skip to content

test: production-harden the suite (error contract, tasks, async, real URLs, coverage gate)#18

Merged
Abhinavexist merged 2 commits into
mainfrom
test/harden-suite
Jul 22, 2026
Merged

test: production-harden the suite (error contract, tasks, async, real URLs, coverage gate)#18
Abhinavexist merged 2 commits into
mainfrom
test/harden-suite

Conversation

@Abhinavexist

Copy link
Copy Markdown
Collaborator

Turns the happy-path suite (24 tests, ~81% line coverage) into a production-grade one: 119 tests, 100% line+branch coverage, gate floor at 90%.

What was missing (and is now covered)

  • Error contract (test_errors.py) — status→exception mapping 400/401/403/404/429/500/503 → the right openai type (asserting status_code and the error body reaches the caller); retries fire on 429/5xx (respx side_effect + monkeypatched sleep, assert call_count); timeoutsAPITimeoutError. Sync + async. (Previously zero error tests.)
  • Task helpers (test_tasks.py) — all 8 (ocr/object_detection/gui_detection/transcribe/web_search/scrape/translate/forecast), sync + async: assert the <task> tag on the wire and result extraction; forecast precontext / fallback / skip-unrelated-entries paths; _extract edge cases. (Was ~48% covered.)
  • Async parity (test_async.py) — real async coverage, not one smoke test.
  • Streaming failure modes — empty stream, streamed tool-call accumulation, double-consume, malformed SSE, mid-iteration get_final_completion.
  • Inputsfrom_path (real temp file), blacklist via from_path, unknown-extension, auto_part routing.
  • Guard / schema (test_guard.py, test_schema.py) edge cases.

Real URLs

Added tests/assets.py with the canonical Interfaze/JigsawStack doc asset URLs (OCR receipt, STT wav, forecast CSV, YOLO bus.jpg, arXiv PDF, HN, …) and replaced the fake https://x.com/... placeholders.

Coverage gate (self-contained)

pytest-cov in the dev extra + [tool.coverage.run] branch=true + addopts=--cov --cov-fail-under=90 in pyproject.toml. Travels with the tests, so any CI running pytest enforces it. Tests only — no src/ changes.

Findings the hardened tests surfaced (documented in the tests; flagging for follow-up)

  1. gui_detection silently mis-routes extensionless image URLs. A CDN URL with no .ext (e.g. Unsplash) makes auto_part fall through to a generic file part instead of image_url (mime detection is extension-only), and tasks.gui_detection() has no filename/format hint to fix it. Real silent-failure gap for a common URL shape — worth an SDK follow-up.
  2. Malformed SSE data: lines raise a raw json.JSONDecodeError from openai's SSE decoder (before interfaze code runs), rather than a typed error — a stream robustness gap.
  3. All 5xx map to InternalServerError (a subclass of APIStatusError), not a distinct type per code — tests assert actual behavior.
  4. Pre-existing ruff format drift in 6 src/ files (untouched here; deferred format-check from the CI PR).

@Khurdhula-Harshavardhan Khurdhula-Harshavardhan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please help resolve the following.

@Abhinavexist

Copy link
Copy Markdown
Collaborator Author

⚠️ Semantic overlap with #20 — needs a design call, not just a rebase. #20 changes .stream() to yield OpenAI events instead of raw chunks (a deliberate breaking change). This PR hardens the chunk-based contract (e.g. list(stream) == chunk count, tool-call accumulation by iterating, role-from-first-delta). Each PR is green on its own, but merging both breaks this PR's stream tests against #20's event contract. Decide first whether .stream() is chunk- or event-based going forward; #20 obsoletes part of this PR's stream hardening, so those tests must be rewritten to the event API (or #20 deferred).

@Khurdhula-Harshavardhan Khurdhula-Harshavardhan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please help resolve the following.

Comment thread pyproject.toml
- Error contract: status->exception mapping (400-503), retry behavior, and
  timeouts, sync + async (were completely untested).
- All 8 task helpers unit-tested (sync + async): asserts the <task> tag on the
  wire AND result extraction; forecast precontext/fallback/skip-unrelated paths.
- Async parity, streaming failure modes (empty stream, streamed tool calls,
  double-consume, malformed SSE), input edge cases (from_path), guard/schema.
- Real Interfaze/JigsawStack doc asset URLs (tests/assets.py); dropped the fake
  x.com placeholders.
- Coverage gate: pytest-cov + branch coverage + --cov-fail-under=90 (suite hits
  100% line+branch).

24 -> 119 tests.
@Abhinavexist
Abhinavexist merged commit d41d0d2 into main Jul 22, 2026
6 checks passed
@Abhinavexist
Abhinavexist deleted the test/harden-suite branch July 22, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants