Skip to content

fix(minitest): stop discarding stdout written by the tests themselves - #105

Closed
rsanheim wants to merge 2 commits into
mainfrom
rjs/minitest-stdout-passthrough
Closed

fix(minitest): stop discarding stdout written by the tests themselves#105
rsanheim wants to merge 2 commits into
mainfrom
rjs/minitest-stdout-passthrough

Conversation

@rsanheim

@rsanheim rsanheim commented Jul 26, 2026

Copy link
Copy Markdown
Owner

In minitest mode, plur silently discarded stdout written by the tests themselves. A puts inside a test never reached the user on a passing run.

Reproducing

fixtures/projects/minitest-success already contains puts "in test_addition" and puts "in test_titleize". Before this change:

........

Finished in 0.29917 seconds.
8 runs, 23 assertions, 0 failures, 0 errors, 0 skips

Neither string appears anywhere in stdout or stderr. After:

........
in test_addition
in test_titleize

Finished in 0.25710 seconds.
8 runs, 23 assertions, 0 failures, 0 errors, 0 skips

Root cause

Minitest writes progress characters with no trailing newline, so a puts from inside a test lands as a tail on the current progress line — ..in test_foo.

The parser pulled the leading progress characters off that line but returned consumed = false, so the whole line went to the collector's rawOutput buffer. That buffer is only ever printed for failed or errored workers (result.go), gated further upstream by streamTestOutput's streamStdout parameter, which runner.go passes as Framework.Name != "minitest". On a green run there was no code path that printed it.

Worth noting the text was never lost inside the parser — on a failing run the raw dump has always contained it. The defect was purely that nothing printed it when the run passed.

The fix

Split the two streams in the parser: leading progress characters feed the aggregated progress line, and the remaining text becomes a TestStdout notification the collector buffers separately. PrintResults flushes it once the progress line is complete, so the consolidated dot line stays intact across parallel workers. This mirrors how the RSpec path already separates its JSON protocol lines from arbitrary test stdout.

The split is deliberately the last check in ParseLine, so minitest's own output — failure headers, the timing line, the summary — is always claimed first. Placing it earlier causes the summary line to be swallowed as test output and SuiteFinished never fires; two existing unit tests cover that ordering.

Behavior notes

  • For tests that use puts, the consolidated progress line is unchanged: the first stdout line is still exactly the run's progress characters at both -n 1 and -n 2.
  • Test output is relocated. Minitest writes it inline with the dots; plur buffers it and prints it after the progress line. That is inherent to consolidating the dots and cannot be avoided while doing so.
  • Nothing appends a newline the test did not write. The line reader is the last place that knows whether the source line ended in one, so the terminator is carried from there rather than re-added downstream.
  • On a failing run, puts output now appears once rather than being embedded in the raw failure dump twice.
  • Run options: --seed N still only shows on failing runs. Left as-is here; making the seed visible for reproducing order-dependent failures is a reasonable follow-up.

Verification

  • Full Ruby suite: 378 examples, 0 failures, 4 pending (all pre-existing).
  • go test ./...: all packages ok. PLUR_RACE=1 bin/rake test:go clean, with the race banner confirmed present rather than assumed.
  • spec/integration/spec/minitest_integration_spec.rb: 10 examples, 0 failures. No existing example was modified or weakened; the 2-worker example gained assertions.
  • The new integration spec was confirmed to fail against a pre-fix binary and pass against this one.

Known limitation: print without a trailing newline

countLeadingProgressChars only claims progress characters at the start of a line. Minitest packs dots and test output onto one physical line, so with print (no trailing newline) every dot appearing after the first chunk of text is treated as literal text.

Given tests doing print "AAA"; print "BBB" and print "CCC", raw minitest emits one line with all 8 dots interleaved:

.CCC.....AAABBB..

plur renders a 1-character progress line and carries the remaining 7 dots inside the text:

.
CCC......AAABBB.

This is pre-existing — a pre-fix binary also shows a 1-dot progress line — and this PR does not make the dot count any worse. It does make the mangling visible, where before the text was silently discarded. puts, which is the overwhelmingly common case, is unaffected because each write is newline-terminated.

Properly fixing this needs the structured reporter described below, not more heuristics: stripping all progress characters from a line would eat a test that legitimately prints ....

Known limitation, pre-existing

countLeadingProgressChars treats a leading F/E/S in test output as a progress character, so puts "FIRST_LINE" prints IRST_LINE and injects a spurious F into the progress line. This predates the change — a pre-fix binary already emits the bogus F — and this PR only makes the truncated remainder visible. Distinguishing "F for a failed test" from "F starting a line of test output" is ambiguous in a line-based parse; the principled fix is a structured minitest reporter mirroring internal/framework/rspec/formatter.rb. Tracked separately.

rsanheim added 2 commits July 26, 2026 01:17
Minitest writes progress characters without a trailing newline, so a `puts`
from inside a test lands as a tail on the current progress line ("..in
test_foo"). The parser extracted the leading progress characters and left the
whole line unconsumed, which sent it to the collector's rawOutput buffer - and
that buffer is only ever printed for failed or errored workers. On a green run
every bit of test output was dropped on the floor.

Split the two streams in the parser instead: leading progress characters feed
the aggregated progress line, the remaining text becomes a TestStdout
notification the collector buffers separately. PrintResults flushes it after
the progress line is complete, so the consolidated dot line stays intact across
parallel workers. This mirrors how RSpec already separates its JSON protocol
lines from arbitrary test stdout.

The split runs as the last check in ParseLine so minitest's own output -
failure headers, the timing line, the summary - is always claimed first.
…ing one

The collector appended "\n" to every captured fragment. For a `puts` that
happens to be right - the reader stripped a real newline and this put it
back - but it is the wrong rule: a fragment from a line with no trailing
newline got one it never had.

ReadString only returns a nil error when it actually found the delimiter,
so stream_helper is the last place that knows. Carry the terminator from
there and write the content verbatim downstream.

This does not change how a `puts` renders. It stops plur from
manufacturing output the test did not write.
@rsanheim

Copy link
Copy Markdown
Owner Author

Superseded by #106.

This fixed the dropped output, but with heuristics layered on the existing guess: it kept countLeadingProgressChars and only made the already-truncated text visible. It did not fix the underlying ambiguity — with print (no trailing newline) an 8-test run still rendered a one-character progress line, and puts "FIRST_LINE" still printed IRST_LINE with a bogus F in the progress stream.

#106 removes the ambiguity instead of coping with it, by injecting a minitest plugin that tags test-written stdout — the same mechanism plur already uses for RSpec's formatter, with the marking inverted so bare progress characters stay free. That let the heuristics be deleted outright.

@rsanheim rsanheim closed this Jul 26, 2026
@rsanheim
rsanheim deleted the rjs/minitest-stdout-passthrough branch July 27, 2026 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant