Add agent-friendly e2e failure report hosted next to the HTML report - #1853
Open
pedrobonamin wants to merge 1 commit into
Open
Add agent-friendly e2e failure report hosted next to the HTML report#1853pedrobonamin wants to merge 1 commit into
pedrobonamin wants to merge 1 commit into
Conversation
Generate a plain-markdown failure digest during the e2e report merge: error messages, code snippets, and Playwright's error-context page snapshots. Host it on the Vercel report deployment at <report-url>/agent-report.md, link it from the failed PR comment as a 'Share with an AI agent' blockquote, and append it to the job summary.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
|
✅ E2E Tests🟢 26 passed • view full report • view run Studio: https://plugins-e2e-test-studio-f0vq6xxf8.sanity.dev Datasets: |
Coverage Report
File CoverageNo changed files found. |
pedrobonamin
marked this pull request as ready for review
August 6, 2026 10:21
pedrobonamin
requested review from
Copilot and
stipsan
and removed request for
a team and
Copilot
August 6, 2026 10:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The merged Playwright HTML report deployed to Vercel is a JS single-page app — great for humans, but an AI agent fetching the report URL only gets an empty HTML shell. The only machine-readable output today is
test-summary.json(counts + failed file names), which lacks everything an agent needs to debug a failure.Solution
Playwright (≥1.53, we're on 1.62) already attaches
error-context.md— an ARIA/YAML snapshot of the page at the moment of failure, designed for AI consumption — to every failed test. That data travels inside the blob reports CI already merges; it was just never surfaced anywhere fetchable.This PR extracts it into a single plain-markdown digest hosted on the existing Vercel report deployment:
e2e/reporters/summary.ts(the merge-time reporter) now also writesplaywright-report/agent-report.mdcontaining, per failed/flaky test: title, spec location, browser, attempt count, the full ANSI-stripped error + stack + code snippet, the inlinederror-contextpage snapshot, other attachment names, plus run metadata and the exact env-complete local repro command. When everything passes it writes a one-line "all passed" file so the URL is always valid.test-summary.jsongains ahasFailuresflag.cp -r e2e/playwright-report/* .vercel/output/static/step, so it's served at<report-url>/agent-report.md— one URL an agent can fetch in a single request.e2e/README.mddocuments the new output.No changeset: only the private
e2epackage and CI workflow are touched.Report example:
https://plugins-e2e-test-studio-gmw0f2cne.sanity.dev/agent-report.md
Verification
Verified end-to-end locally against a real ephemeral dataset: ran a deliberately failing spec with the CI reporter setup (blob), merged with
playwright merge-reports --reporter html,./reporters/summary.ts, and confirmedagent-report.mdcontains the error, code snippet, page snapshot, and repro command (see run artifacts in the agent session).pnpm format,pnpm lint,pnpm knip,pnpm build, andpnpm testall pass.