feat: live production integration suite gates the release (macOS + Windows) - #176
Merged
Merged
Conversation
…ndows) Closes #175. tests/integration/ drives the CLI as real subprocesses against production — piped-stdin API-key login, invalid-key rejection, parse, cached re-run dedup, find, crop, extract, logout — over a checked-in 714-byte fixture PDF with fixed text, so assertions are exact. The suite skips itself unless ADE_INTEGRATION_API_KEY is set, keeping plain pytest hermetic and free. integration.yml runs it on macos-latest and windows-latest, manually (workflow_dispatch) or as the release gate (workflow_call, secret VISION_AGENT_API_KEY); never on push/PR — every run bills real parse + extract credits. release.yml's check job now needs it, placed ahead of check on purpose: under manual dispatch check pushes the release tag, and a failed gate must leave no tag behind. Contract-tested in tests/test_release_pipeline.py; contributor docs in CONTRIBUTING.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a production integration gate for releases, covering real CLI workflows on macOS and Windows.
Changes:
- Adds eight live production integration tests and a deterministic PDF fixture.
- Adds reusable/manual integration workflow and release dependency.
- Documents usage and adds workflow contract tests.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/integration.yml |
Runs the live suite across macOS and Windows. |
.github/workflows/release.yml |
Gates release checks on integration. |
tests/integration/test_production.py |
Tests production auth, parse, find, crop, and extract. |
tests/integration/fixtures/invoice.pdf |
Provides deterministic integration input. |
tests/test_release_pipeline.py |
Adds release and integration workflow guards. |
CONTRIBUTING.md |
Documents integration testing and release gating. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…KEY secret One name end to end: the repo secret, the workflow_call contract, and the env var the tests gate on are all ADE_INTEGRATION_API_KEY (was going to reuse the VISION_AGENT_API_KEY secret of unknown freshness; a dedicated secret is clearer to audit and rotate). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…empty Copilot review catch: workflow_call enforces the secret via required: true, but a direct workflow_dispatch does not — a missing secret expands to "", every test skips itself, and the manual run reports green without touching production. A preflight step now fails the job on an empty ADE_INTEGRATION_API_KEY, contract-tested in tests/test_release_pipeline.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
yzld2002
approved these changes
Aug 6, 2026
A merged PR that breaks production integration now surfaces immediately instead of at the next release. pull_request stays excluded — every run bills real parse + extract credits, and fork PRs must never reach the secret; PR CI (ci.yml) keeps running the offline suite only. The contract test now pins push to [main] and rejects any pull_request trigger key (prose mentions allowed, pull_request_target included). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two clean check rows ("Integration / macOS", "Integration / Windows")
instead of "integration (macos, macos-latest)"; fail-fast was already
off, so the two platforms were independent results all along.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
yuanwen-tian
added a commit
that referenced
this pull request
Aug 6, 2026
Bumps `pyproject.toml` and `uv.lock` to 1.0.4 (minimal two-line diff; the lockfile edit is validated by `uv sync --locked`, full suite green, `ade version` reports 1.0.4). ##⚠️ Merge order **Merge #177 first, then this PR** — v1.0.4 is intended to include: - #177 — fix: auto-download URL-sourced documents on view/crop with notice and progress - #176 — feat: live production integration suite gates the release (already on main) Once both are in, cut the release via **Actions → Release → "Run workflow"** on `main`. The pipeline will run the live integration suite (macOS + Windows, against production) before tagging `v1.0.4` — first exercised green on run [31072130505](https://github.com/landing-ai/ade-cli/actions/runs/31072130505). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #175.
What
A live integration suite that drives the real CLI against production, run on macOS and Windows, that the release pipeline must pass before anything is tagged or published — and that can also be triggered manually.
tests/integration/test_production.pyUnlike the offline suite (in-process runner, faked transport), these 8 tests run the CLI as real subprocesses out of a temp
ADE_HOME, with every ambientADE_*variable stripped:login --api-key -via piped stdin (the headless path that broke on Windows in v1.0.2), verified-before-stored (ADR-0007);auth status; an invalid key must come back as the canonical 401 error with nothing stored;logoutclears the credential.run_id(nothing resubmitted).findlocates the invoice line with a sane normalized box;crop --allwrites real PNGs.invoice_numberverbatim, referencing the parse job item.Assertions are exact, not model-lenient: the checked-in fixture is a hand-assembled 714-byte one-page PDF with fixed text, verified renderable by pypdfium2 (the same renderer
cropuses).The suite skips itself entirely unless
ADE_INTEGRATION_API_KEYis set, so plainpyteststays hermetic and free..github/workflows/integration.ymlMatrix over
macos-latest+windows-latest. CI at a glance, after this PR:ci.yml(unchanged)main(e.g. PR merged)integration.ymlrelease.yml→integration.ymlintegration.ymlDeliberately never on
pull_request: every run bills real parse + extract credits (one of each per OS), and fork PRs must not reach the secret. No double-run within a release: a dispatch release isn't a push tomain, and a tag push isn't a branch push, so only theworkflow_callgate fires.A preflight step fails the job loudly if the
ADE_INTEGRATION_API_KEYsecret is empty or unset (a missing secret would otherwise skip every test and report a false green). Authenticates with theADE_INTEGRATION_API_KEYrepo secret — the same name as the env var the tests gate on.release.ymlchecknowneeds: integration. The gate sits ahead ofcheckdeliberately: under manual dispatchcheckpushes the release tag, so a failed integration run must leave no tag behind (otherwise a re-dispatch would hit "tag already exists").Guard rails
Three new string-level contract tests in
tests/test_release_pipeline.pykeep this from drifting: the release workflow gates on integration ahead ofcheck; the integration workflow covers both OSes, runs on pushes tomain, stays dispatchable, and never triggers onpull_request; every file intests/integration/hangs off the env-var skip gate. Contributor docs added toCONTRIBUTING.md.Verification
creditsis "0 on a cached hit"; the payload actually echoes the original run's bill) — left out of scope here, tracked separately.Note for reviewers
The
ADE_INTEGRATION_API_KEYrepo secret must be created (Settings → Secrets and variables → Actions, orgh secret set ADE_INTEGRATION_API_KEY) with a valid production ADE key before the first run. Prefer a dedicated low-quota integration account's key.🤖 Generated with Claude Code