Skip to content

feat: add V1 production e2e tests and release gate - #142

Merged
tian-lan-landing merged 5 commits into
mainfrom
feat/v1-production-e2e
Aug 5, 2026
Merged

feat: add V1 production e2e tests and release gate#142
tian-lan-landing merged 5 commits into
mainfrom
feat/v1-production-e2e

Conversation

@tian-lan-landing

Copy link
Copy Markdown
Collaborator

Adds a live V1 end-to-end suite against production and wires it into the release flow.

  • tests/contract/test_v1_production.py (production marker) — full V1 surface: parse, extract, extract_build_schema, classify, section, split, and both job types (create + poll + list). Parses once and shares the markdown to hold per-run spend to ~8 inference calls.
  • .github/workflows/e2e-production.yml — runs on manual dispatch and via workflow_call; fails fast if the production key is missing (so a green run can't mean zero tests).
  • release.yml — gated on the e2e workflow via needs, before any commit/tag/Release exists.
  • Excluded from the default pytest run and the staging contract gate via the production marker.

Setup required: create a production-e2e environment with secret LANDINGAI_ADE_PRODUCTION_APIKEY (no required reviewers, or every release pauses for approval).

Verified locally: ruff clean, pyright strict clean, marker gating both directions, suite skips cleanly with no key. Not yet run against live production (needs the key + spends credits).

Add a `production`-marked live suite (tests/contract/test_v1_production.py)
covering the full V1 surface: parse, extract, build-schema, classify,
section, split, and both job types. It runs via a manual/reusable workflow
(.github/workflows/e2e-production.yml) and gates release.yml before any
tag exists. Excluded from the default `pytest` run and the staging contract
gate by the pytest markers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings August 5, 2026 06:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds production V1 end-to-end validation as a pre-release gate.

Changes:

  • Covers synchronous V1 endpoints and asynchronous job flows.
  • Adds a reusable/manual production workflow.
  • Excludes production tests from default and staging runs.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
tests/contract/test_v1_production.py Adds production V1 E2E tests.
pyproject.toml Registers and excludes the production marker by default.
.github/workflows/e2e-production.yml Runs production tests manually or as a reusable workflow.
.github/workflows/release.yml Gates releases on production E2E success.
Suppressed comments (1)

tests/contract/test_v1_production.py:256

  • This has the same vacuous-pass problem as the parse-job list check: an empty response exercises the request but verifies neither listing nor filtering. Require a non-empty page (with a short retry if necessary) before iterating over statuses.
    assert isinstance(listed.jobs, list)

Comment thread tests/contract/test_v1_production.py Outdated
Comment thread .github/workflows/e2e-production.yml
Copilot AI review requested due to automatic review settings August 5, 2026 07:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread .github/workflows/e2e-production.yml Outdated
/v1/ade/extract/jobs rejects a bare string markdown (sent as an
unnamed file part) with 422; wrap it as a named file tuple.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 5, 2026 08:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/e2e-production.yml:22

  • Remove this temporary push trigger before merge. It runs the feature branch's own checkout and tests with the production environment secret, so anyone able to push to this branch can execute arbitrary code with LANDINGAI_ADE_PRODUCTION_APIKEY; it also contradicts the documented manual/release-only trigger policy and spends credits on every push.
  # TEMPORARY (remove before merge): lets us exercise this workflow from the feature
  # branch, since workflow_dispatch only works once the file is on the default branch.
  push:
    branches: [feat/v1-production-e2e]

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 5, 2026 08:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/e2e-production.yml:37

  • workflow_dispatch can run this workflow on a selected branch or tag, and the default checkout then executes that ref's test code with the production key. Because the required setup explicitly omits reviewers and does not restrict deployment branches, unmerged branch code can receive the production credential. Restrict the production-e2e environment's deployment branches to main (and document that requirement here); a job-level github.ref check is useful defense in depth but is not a substitute for the environment rule because the workflow itself can differ on the selected ref.
    # `production-e2e` and add LANDINGAI_ADE_PRODUCTION_APIKEY as an environment secret.
    # Do NOT add required reviewers unless you want every release to pause for a manual
    # approval — release.yml blocks on this job. Until the environment exists the job
    # falls back to a repo-level secret of the same name.
    environment: production-e2e

Address Copilot review on #142:
- e2e-production.yml: require the `production-e2e` environment to be
  restricted to `main` (the control that actually protects the key), drop
  the unsafe repo-level-secret fallback recommendation, and add a
  defense-in-depth `if: github.ref == 'refs/heads/main'` job gate.
- test_v1_production.py: the parse/extract job-list checks passed
  vacuously on an empty page. Require a non-empty completed list (short
  retry for list-indexing lag) before asserting the status filter.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 5, 2026 09:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/e2e-production.yml:62

  • This error directs maintainers to add the production credential at repository scope, contradicting the environment-only requirement above. A repository secret can be read by a workflow dispatched from a modified branch that removes the protected environment, bypassing the main-only deployment rule. Keep the setup guidance environment-only.
                 "\`production-e2e\` environment (or at the repo level). Refusing to report a green" \

@tian-lan-landing
tian-lan-landing merged commit bda1f1d into main Aug 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants