Skip to content

Improve verification diff UX (field summary, changed-fields-only, truncation/full mode) #30

Description

@rsanheim

Summary

Polish verification failure output to make mismatches faster to diagnose, especially for large stdout/stderr snapshots.

Problem

Current failures can be noisy for large outputs and make it harder to identify the highest-signal differences quickly.

Proposal

Improve VerificationError messaging and diff presentation:

  1. Field-level summary first

    • e.g. stdout changed, stderr unchanged, status changed
  2. Show diffs only for changed fields

    • skip unchanged fields entirely
  3. Use compact unified diff with context limit

    • configurable context lines (default small, e.g. 3)
  4. Truncate very large diffs with clear marker

    • include "truncated" indicator and hint for full output option
  5. Optional full diff toggle

    • env var or config option to force complete diff output

Example Failure Shape

Backspin verification failed!
Record: fixtures/backspin/example.yml

Summary:
- stdout: changed
- stderr: unchanged
- status: changed (expected 0, actual 1)

[stdout diff]
@@ ... @@
- old line
+ new line
...
(diff truncated, set BACKSPIN_FULL_DIFF=1 for full output)

Acceptance Criteria

  • changed fields are clearly enumerated before diff blocks
  • unchanged fields are omitted from diff section
  • large diffs are truncated deterministically with explicit notice
  • docs include guidance for full diff mode

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions