Skip to content

test: migrate golden specs to Backspin.compare - #107

Merged
rsanheim merged 7 commits into
mainfrom
rjs/backspin-compare-migration
Jul 26, 2026
Merged

test: migrate golden specs to Backspin.compare#107
rsanheim merged 7 commits into
mainfrom
rjs/backspin-compare-migration

Conversation

@rsanheim

Copy link
Copy Markdown
Owner

Migrates the rspec-vs-plur golden specs to Backspin.compare (new in backspin 0.13.0) and deletes the workarounds that only existed because there was no differential-testing API.

57 insertions, 448 deletions. Specs alone: -272/+35.

The pattern being removed

Each of these specs called Backspin.run twice with the same record name: — once for rspec, once for plur — so a recorded snapshot acted as a hand-rolled differential test:

chdir(fixture) { Backspin.run(rspec_cmd, name: "x", filter: normalize) }
result = chdir(fixture) { Backspin.run(plur_cmd, name: "x", filter: normalize) }
expect(result.verified?).to be(true)

Backspin.compare does this directly: one call, one chdir, one filter, nothing persisted to disk.

Three workarounds fall out with it:

  • The "args" => ["[SOMETHING_COMMAND]"] placeholder in every filter. compare never compares argv, so normalizing it was dead code.
  • expect(result.verified?).to be(true). A mismatch raises Backspin::VerificationError with the diff before the expectation is reached, so the assertion could never fail — only fire after the raise already had.
  • Three examples in single_failure_golden_spec.rb that differed only by record name and were otherwise byte-identical.

Six snapshot YAMLs are deleted. The remaining six in fixtures/backspin/ are ordinary single-command snapshot tests using Backspin.run correctly and are untouched.

Order-dependence fixed

minitest_integration_spec.rb recorded in one example and verified in a different one, so it passed only under RSpec's defined order and broke under --order random. One compare call makes that failure mode structurally impossible.

single_failure_spec.rb removed

It was a hand-rolled Backspin.compare: same two commands, same fixture, its own copy of the timing normalizer, asserting stdout_lines == rspec_stdout_lines. Strictly weaker than compare, which also checks exit status and stderr.

Its absolute assertions — exit status, failure count, plur's stderr banner — moved onto result.actual in the compare-based example, so they cost no extra subprocess pair. --color=never coverage remains in colorized_output_spec.rb and tty_output_spec.rb.

A pending spec dropped rather than migrated

The plur -n1 vs raw-grouped-minitest comparison could never pass: the RSpec-style duration line and consolidated progress dots are deliberate decisions pinned by passing specs in that same file, and the dot/puts interleaving position isn't stable because plur's worker command passes no seed. The real defect it pointed at — minitest dropping test stdout — is fixed separately in #106.

Verification

Full suite green in both defined and random order. Each migrated spec was confirmed to still fail when it should: pointing plur at --color=never against rspec's --force-color raises VerificationError with a real diff in all three golden specs.

Worth noting the migration made the suite faster, not slower. Backspin.run shells out in verify mode too, so the old code was already re-running rspec on every test run and comparing it against a snapshot of itself. Collapsing the duplicate examples removed 4 subprocess spawns per run.

Also bumps backspin to ~> 0.13.0 and regenerates the Appraisal gemfiles and lockfiles. Appraisal2 emits hashrocket syntax, so there is a follow-up standard:fix commit restoring the 1.9 hash syntax those files already used.

rsanheim added 7 commits July 26, 2026 00:02
0.13.0 adds Backspin.compare for differential testing. Regenerates the
Appraisal gemfiles and lockfiles along with the root lockfile.
Each of these ran the same record name through Backspin.run twice - once
for rspec, once for plur - so the recorded snapshot acted as a hand-rolled
differential test. Backspin.compare does that directly: one call, one
chdir, one filter, nothing persisted to disk.

Drops the workarounds that existed only for the two-call pattern:

- the "args" => ["[SOMETHING_COMMAND]"] placeholder in every filter.
  compare never compares argv, so normalizing it is dead code.
- expect(result.verified?).to be(true). A mismatch raises
  VerificationError with the diff before the expectation is reached.
- the three single-failure examples, which differed only by record name
  and were otherwise byte-identical. Collapsed to one, keeping the union
  of the color, backtrace, and description assertions.

Deletes the five snapshot YAMLs those specs recorded into. The remaining
fixtures/backspin records are ordinary single-command snapshot tests and
are unaffected.
This pair recorded in one example and verified in a different one, so it
passed only under RSpec's defined order and broke under --order random.
One compare call makes that failure mode structurally impossible.

The comparison is still pending - plur's output does not match raw
minitest output yet - and compare's raise on mismatch registers as
pending-as-expected rather than an error.
Appraisal2 emits hashrocket syntax; standard:fix restores the 1.9 hash
syntax the committed gemfiles already used.
Follow-up cleanup to the Backspin.compare migration. No behavior change.

- Delete the `fixture_path` aliases, which just forwarded to the
  suite-wide `project_fixture` helper.
- Inline the single-use command builder methods. Collapsing the
  duplicate examples left each one with exactly one call site, and
  `reference:`/`actual:` already name what they are.
- Collapse the two-layer normalizers. `make_summary_line_consistent`,
  `normalize_timing`, and `normalize_minitest_output` each had a single
  caller in the snapshot filter directly below them.
- Drop the dead plur-banner regexes from the pending-output filter. They
  scrubbed stdout, but plur writes that banner to stderr - and the
  worker regex required a plural "workers" that never matched the
  single-worker runs these specs produce.
- Inline the minitest seed and drop the `seed:` keyword; it existed to
  make a recording deterministic and nothing is recorded now.
- Note why each filter blanks stderr, so the next reader doesn't have to
  rediscover that plur's version/worker banner has no rspec counterpart.

348 -> 294 lines.
single_failure_spec.rb was a hand-rolled Backspin.compare: it ran the
same two commands against the same fixture, kept its own copy of the
timing normalizer, and asserted stdout_lines == rspec_stdout_lines. The
comparison is strictly weaker than compare's - no exit status, no stderr.

compare's result carries the real captured output, so the absolute
assertions that comparison alone can't make - exit status, the failure
count, plur's stderr banner - hang off result.actual in the same example.
No extra subprocess pair to make them.

Removes 4 subprocess spawns per suite run and the third copy of
make_summary_line_consistent. --color=never runs stay covered by
colorized_output_spec.rb and tty_output_spec.rb.
It compared plur's full minitest output against raw grouped minitest
output and could never pass. Two of the differences are deliberate
decisions pinned by passing specs in this same file - the RSpec-style
duration line, and consolidating progress dots onto the first line - and
the dot/puts interleaving position is not stable anyway, since plur's
worker command passes no seed.

The one real defect it pointed at is that minitest mode drops a test's
own stdout. That is tracked in plur-internal
(docs/reviews/2026-07-26-minitest-stdout-dropped.md); once fixed it wants
a direct assertion, not a full-output diff.
@rsanheim
rsanheim merged commit b70b053 into main Jul 26, 2026
4 of 5 checks passed
@rsanheim
rsanheim deleted the rjs/backspin-compare-migration branch July 26, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant