fix(minitest): tag test-written stdout instead of guessing - #106
Draft
rsanheim wants to merge 1 commit into
Draft
Conversation
Minitest writes progress characters with no trailing newline, so a test's `puts` lands on the same physical line as the dots. The parser used to guess: it took the leading run of [.FES] and called the rest text. That miscounted whenever a test printed after the dots, stole the F out of `puts "FIRST_LINE"`, and dropped the text entirely on passing runs. Invert the marking. Progress stays bare - it is the common case and costs nothing - and plur injects a minitest plugin that prefixes everything the tests write with PLUR_OUT:. Minitest's reporters capture the real $stdout when they are constructed, which happens before plugin init, so swapping $stdout only redirects test output. Parsing is then exact: before the marker is progress, after it is the test's own bytes. The plugin delegates formatting to StringIO so puts/print/write/printf keep Ruby's semantics, guards its line buffer with a mutex for parallelize_me!, and overrides dup/reopen so capture_subprocess_io still works (duping the wrapper would have closed the real stdout and killed the run). Test output is collected per worker and printed after the progress line rather than streamed, so the run of dots stays intact. Needs minitest 5.0+ for the plugin system; below that, and for bytes written straight to fd 1 by a subprocess, output falls back to the previous behaviour of appearing only for failed workers.
This was referenced Jul 26, 2026
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Minitest mode could not tell plur's own progress characters apart from stdout written by the tests, because minitest writes progress with no trailing newline — so a test's
putslands on the same physical line as the dots (..in test_foo). The parser guessed, taking only the leading run of[.FES]and treating the rest as text.That guess was wrong in three ways, and on a passing run the text was dropped entirely (it only reached
rawOutput, which is printed for failed workers).Instead of guessing, make it unambiguous
plur already does this for RSpec:
framework.goinjects-r <embedded formatter.rb> --format Plur::JsonRowsFormatter, and the formatter prefixes its output withPLUR_JSON:. Minitest never got the equivalent.The marking is inverted relative to RSpec, deliberately. Progress characters are the common case and stay bare — you'd otherwise pay a marker on every test. Instead the tests' own writes get tagged, which costs a prefix only when a test actually prints something.
An embedded
plur_plugin.rbis written to$PLUR_HOME/formatter/rubylib/minitest/and loaded via-I. Minitest's reporters capture the real$stdoutwhen constructed, andMinitest.plugin_*_initruns after that, so swapping$stdoutinside the plugin leaves minitest's own progress untouched and redirects only what tests write.Parsing collapses to: the first marker on a line splits progress from test bytes.
countLeadingProgressChars, the mixed-line splitting, and the entireinRunningPhasestate machine are deleted.Before / after
plur --use minitest -n 1onfixtures/projects/minitest-success:With
print(no trailing newline) the old parser produced a one-character progress line for an 8-test run, stranding 7 dots inside the text. It now counts all 8. A test doingputs "FIRST_LINE"used to printIRST_LINEand inject a bogusFinto the progress line; the text now survives intact and theFin the progress stream is only ever a real failure.Compatibility
--no-plugins/MT_NO_PLUGINS, or for non-minitest runners, the-Iis inert and behavior falls back to today's. Verified byte-identical output forfixtures/projects/testunit-success.minitest-reporterscoexists.DelegateReporter#init_all_reportersassignsoptions[:io], captured inMinitest.process_argsbefore any plugin init, so load order cannot leak the marker. Confirmed against theminitest-reportersfixture.dup/reopenare overridden. Without that,capture_subprocess_io— which does$stdout.dupthen closes the original — closed the real stdout and aborted the run.reopensuspends tagging while stdout points elsewhere, so captured output is byte-identical to a no-plugin baseline.puts "PLUR_OUT:x"renders verbatim.Known limitations, documented in
docs/usage.mdsystem/backtick subprocess) cannot be tagged. They still reachrawOutput(failed workers only), and progress characters sharing that physical line are not counted — the parser refuses to guess rather than miscount.printwith no trailing newline followed byexit!loses the partial. Plain Ruby behaves the same (ruby -e 'print "x"; exit!' | catprints nothing).$stdout.flushdoes not force a partial line out; it stays buffered until a newline or exit.Tests
New
fixtures/projects/minitest-stdoutcovering print-without-newline, mixed write styles, writes from parallel threads, binary bytes, and lines larger than the read buffer; anoutput_on_failure_testforminitest-failures, which previously had noputsat all; and a Ruby unit spec for the plugin.All six new integration examples were confirmed to fail against a binary built from
mainand pass here — including the old bug verbatim, a 14-character progress line for an 8-test run.Full Ruby suite 407 examples / 0 failures / 4 pre-existing pending; all Go packages ok;
PLUR_RACE=1clean with the race detector confirmed genuinely live via a deliberate probe. The three guarded examples inminitest_integration_spec.rbare unmodified.One caveat worth stating: the thread-safety spec is probabilistic. Removing the plugin's mutex and adding a
Thread.passin the window reproduces the corruption, but removing the mutex alone passed 12/12 runs, since MRI's GVL rarely preempts there. The race is real; the detector is not deterministic.Supersedes #105, which fixed the dropped output with more heuristics rather than removing the ambiguity.