test(minitest): canonical minitest-outcomes fixture and consolidated minitest specs - #115
Merged
Conversation
…y integration specs Two fixtures with identical test files, pinned to minitest ~> 5.25 and ~> 6.0, plus outside-in integration specs that run plur against both majors. A backspin compare runs the same suite under raw minitest and under plur and verifies the invariants that must not depend on the runner: every test counted as progress, and the summary counts. Also characterizes the current limitation that test-written stdout is dropped on passing runs (see #106) so any change to output handling must flip that expectation deliberately.
… alphabet 12 tests covering every outcome the progress line can show - pass, failure, error, skip - plus the stdout styles that stress line handling: interleaved puts, unterminated print, multi-line output. Every case carries a unique greppable token, and tokens written mid-run avoid the characters .FES so progress characters can be counted exactly in raw minitest output. The integration spec asserts exact progress-alphabet counts on the order-independent subset, plur's aggregated summary and failure attribution serially and across parallel workers, and a backspin compare against raw minitest on outcome counts and failing test IDs. Characterized (not fixed) along the way, deterministic under any seed: progress characters glued after an unterminated print are dropped by the leading-chars parser heuristic, so the progress line under-counts.
This was referenced Aug 7, 2026
The version axis (minitest 5 vs 6 pins and the parity spec) expands scope beyond the fixture/spec cleanup and will return in a follow-up built on the minitest-outcomes fixture. History retains the work at 00c58b7.
Retire fixtures/projects/minitest-success. It was a happy-path-only near-duplicate of minitest-outcomes, which already covers every progress outcome (pass/fail/error/skip) plus the stdout styles that stress line handling. Everything that pointed at it now points at minitest-outcomes, targeting test/passing_test.rb where a green exit is required. Its unused .rux.toml goes with it. Collapse all minitest integration coverage into a single file, spec/integration/spec/minitest_integration_spec.rb, organized by aspect: outcomes and progress, failure reporting, stdout visibility, parity with raw minitest, discovery and detection. minitest_outcomes_spec.rb is merged in and deleted. Folded in from elsewhere: - stdout_streaming_spec's "Minitest stdout behavior" example, rewritten to actually assert non-duplication (each mid-run token appears exactly once in a failed worker's surfaced stdout). stdout_streaming_spec is now purely RSpec-focused. - fixture_projects_spec's two minitest examples are retired; the consolidated spec's Backspin.compare already raw-runs minitest-outcomes against plur, which is strictly stronger. test-unit examples unchanged. Retired as redundant: the old "runs successfully" / "duration formatting" / "dot count" examples are now one green-run example; the minitest-failures "reports test failures" example (matched only /\d+ failures?/) is subsumed by the outcomes fixture's precise failure attribution. Kept fixtures/projects/minitest-failures: it is the only fixture with failures in more than one file, so it is the only way to exercise a parallel run with two *failing* workers. Added one example covering that (summed counts across both, both detail blocks surviving, no cross-worker renumbering) so the fixture stays justified and covered. Determinism rules from the outcomes spec are preserved: assertions hold for any minitest order, exact progress counts only on newline-terminated files, and the unterminated-print undercount (#116) stays characterized as a range.
Restore all-green multi-worker coverage lost in the consolidation (two passing workers summed into one passing summary), exercise framework auto-detection on a real run instead of only via --dry-run, assert the per-worker failure renumbering the comment claimed, pin the progress line to progress characters in the under-count characterization, and tighten two comments.
rsanheim
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixtures and specs only — no Go or runtime changes.
New canonical fixture:
fixtures/projects/minitest-outcomes. 12 tests covering every outcome the progress line can show — 8 pass, 2 fail (one after writing stdout), 1 error, 1 skip — plus the stdout styles that stress line handling: interleavedputs,STDOUTconstant, unterminatedprint, multi-line output. Every case carries a unique greppable token, and tokens written mid-run avoid the characters.FESso progress characters can be counted exactly in raw minitest output.Consolidated minitest specs. All minitest run behavior now lives in one file,
spec/integration/spec/minitest_integration_spec.rb(10 examples, grouped by aspect): green runs (serial and multi-worker, including real framework auto-detection), exact progress-alphabet counts, the unterminated-print under-count characterization (#116), failure/error attribution by token, two-failing-worker aggregation, failed-worker-only stdout visibility, a Backspin compare against raw minitest on outcome counts + failing test IDs, and discovery.Retired:
fixtures/projects/minitest-success(happy-path-only; all five consumers repointed) and the minitest examples instdout_streaming_spec/fixture_projects_spec(folded or subsumed). Kept:minitest-failures— the only fixture with failures in two files, i.e. the only way to get two failing workers in one parallel run.Determinism: minitest randomizes order and plur can't forward
--seedon multi-file runs (#117), so every expectation holds for any order. Exact counts are asserted only where all mid-run stdout writes are newline-terminated; the unterminated-print case is characterized with an order-independent bound.Out of scope (deferred)
Minitest 5-vs-6 version-pinned fixtures and the parity spec were developed and then deliberately removed (history at 00c58b7) — they'll return as a follow-up built on this fixture.
Verification
Backspin::VerificationError).bin/rakegreen at the fixture commit. Later local full runs show 6 failures in untouched files (stdout_streaming,general_integration) caused by a machine-level mise ruby-pin change mid-session — reproduced identically on the pre-change commit, unrelated to this branch.Filed along the way: #116 (glued progress chars dropped after unterminated print), #117 (extra args broken on multi-file minitest runs).