feat: add V1 production e2e tests and release gate - #142
Conversation
Add a `production`-marked live suite (tests/contract/test_v1_production.py) covering the full V1 surface: parse, extract, build-schema, classify, section, split, and both job types. It runs via a manual/reusable workflow (.github/workflows/e2e-production.yml) and gates release.yml before any tag exists. Excluded from the default `pytest` run and the staging contract gate by the pytest markers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds production V1 end-to-end validation as a pre-release gate.
Changes:
- Covers synchronous V1 endpoints and asynchronous job flows.
- Adds a reusable/manual production workflow.
- Excludes production tests from default and staging runs.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
tests/contract/test_v1_production.py |
Adds production V1 E2E tests. |
pyproject.toml |
Registers and excludes the production marker by default. |
.github/workflows/e2e-production.yml |
Runs production tests manually or as a reusable workflow. |
.github/workflows/release.yml |
Gates releases on production E2E success. |
Suppressed comments (1)
tests/contract/test_v1_production.py:256
- This has the same vacuous-pass problem as the parse-job list check: an empty response exercises the request but verifies neither listing nor filtering. Require a non-empty page (with a short retry if necessary) before iterating over statuses.
assert isinstance(listed.jobs, list)
/v1/ade/extract/jobs rejects a bare string markdown (sent as an unnamed file part) with 422; wrap it as a named file tuple. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
.github/workflows/e2e-production.yml:22
- Remove this temporary push trigger before merge. It runs the feature branch's own checkout and tests with the production environment secret, so anyone able to push to this branch can execute arbitrary code with
LANDINGAI_ADE_PRODUCTION_APIKEY; it also contradicts the documented manual/release-only trigger policy and spends credits on every push.
# TEMPORARY (remove before merge): lets us exercise this workflow from the feature
# branch, since workflow_dispatch only works once the file is on the default branch.
push:
branches: [feat/v1-production-e2e]
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
.github/workflows/e2e-production.yml:37
workflow_dispatchcan run this workflow on a selected branch or tag, and the default checkout then executes that ref's test code with the production key. Because the required setup explicitly omits reviewers and does not restrict deployment branches, unmerged branch code can receive the production credential. Restrict theproduction-e2eenvironment's deployment branches tomain(and document that requirement here); a job-levelgithub.refcheck is useful defense in depth but is not a substitute for the environment rule because the workflow itself can differ on the selected ref.
# `production-e2e` and add LANDINGAI_ADE_PRODUCTION_APIKEY as an environment secret.
# Do NOT add required reviewers unless you want every release to pause for a manual
# approval — release.yml blocks on this job. Until the environment exists the job
# falls back to a repo-level secret of the same name.
environment: production-e2e
Address Copilot review on #142: - e2e-production.yml: require the `production-e2e` environment to be restricted to `main` (the control that actually protects the key), drop the unsafe repo-level-secret fallback recommendation, and add a defense-in-depth `if: github.ref == 'refs/heads/main'` job gate. - test_v1_production.py: the parse/extract job-list checks passed vacuously on an empty page. Require a non-empty completed list (short retry for list-indexing lag) before asserting the status filter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
.github/workflows/e2e-production.yml:62
- This error directs maintainers to add the production credential at repository scope, contradicting the environment-only requirement above. A repository secret can be read by a workflow dispatched from a modified branch that removes the protected environment, bypassing the main-only deployment rule. Keep the setup guidance environment-only.
"\`production-e2e\` environment (or at the repo level). Refusing to report a green" \
Adds a live V1 end-to-end suite against production and wires it into the release flow.
tests/contract/test_v1_production.py(productionmarker) — full V1 surface:parse,extract,extract_build_schema,classify,section,split, and both job types (create + poll + list). Parses once and shares the markdown to hold per-run spend to ~8 inference calls..github/workflows/e2e-production.yml— runs on manual dispatch and viaworkflow_call; fails fast if the production key is missing (so a green run can't mean zero tests).release.yml— gated on the e2e workflow vianeeds, before any commit/tag/Release exists.pytestrun and the staging contract gate via theproductionmarker.Setup required: create a
production-e2eenvironment with secretLANDINGAI_ADE_PRODUCTION_APIKEY(no required reviewers, or every release pauses for approval).Verified locally: ruff clean, pyright strict clean, marker gating both directions, suite skips cleanly with no key. Not yet run against live production (needs the key + spends credits).