Skip to content

fix(minitest): tag test-written stdout instead of guessing - #106

Draft
rsanheim wants to merge 1 commit into
mainfrom
rjs/minitest-tagged-stdout
Draft

fix(minitest): tag test-written stdout instead of guessing#106
rsanheim wants to merge 1 commit into
mainfrom
rjs/minitest-tagged-stdout

Conversation

@rsanheim

Copy link
Copy Markdown
Owner

Minitest mode could not tell plur's own progress characters apart from stdout written by the tests, because minitest writes progress with no trailing newline — so a test's puts lands on the same physical line as the dots (..in test_foo). The parser guessed, taking only the leading run of [.FES] and treating the rest as text.

That guess was wrong in three ways, and on a passing run the text was dropped entirely (it only reached rawOutput, which is printed for failed workers).

Instead of guessing, make it unambiguous

plur already does this for RSpec: framework.go injects -r <embedded formatter.rb> --format Plur::JsonRowsFormatter, and the formatter prefixes its output with PLUR_JSON:. Minitest never got the equivalent.

The marking is inverted relative to RSpec, deliberately. Progress characters are the common case and stay bare — you'd otherwise pay a marker on every test. Instead the tests' own writes get tagged, which costs a prefix only when a test actually prints something.

An embedded plur_plugin.rb is written to $PLUR_HOME/formatter/rubylib/minitest/ and loaded via -I. Minitest's reporters capture the real $stdout when constructed, and Minitest.plugin_*_init runs after that, so swapping $stdout inside the plugin leaves minitest's own progress untouched and redirects only what tests write.

Parsing collapses to: the first marker on a line splits progress from test bytes. countLeadingProgressChars, the mixed-line splitting, and the entire inRunningPhase state machine are deleted.

Before / after

plur --use minitest -n 1 on fixtures/projects/minitest-success:

 before                          after
 ........                        ........
                                 in test_addition
 Finished in 0.26779 seconds.    in test_titleize
 8 runs, 23 assertions, …
                                 Finished in 0.27881 seconds.
                                 8 runs, 23 assertions, …

With print (no trailing newline) the old parser produced a one-character progress line for an 8-test run, stranding 7 dots inside the text. It now counts all 8. A test doing puts "FIRST_LINE" used to print IRST_LINE and inject a bogus F into the progress line; the text now survives intact and the F in the progress stream is only ever a real failure.

Compatibility

  • minitest >= 5.0 (plugin system, 2013-05-10). Below that, or under --no-plugins / MT_NO_PLUGINS, or for non-minitest runners, the -I is inert and behavior falls back to today's. Verified byte-identical output for fixtures/projects/testunit-success.
  • minitest-reporters coexists. DelegateReporter#init_all_reporters assigns options[:io], captured in Minitest.process_args before any plugin init, so load order cannot leak the marker. Confirmed against the minitest-reporters fixture.
  • dup/reopen are overridden. Without that, capture_subprocess_io — which does $stdout.dup then closes the original — closed the real stdout and aborted the run. reopen suspends tagging while stdout points elsewhere, so captured output is byte-identical to a no-plugin baseline.
  • A test printing the marker itself is safe: the plugin tags first, so plur's marker is always the first on the line and puts "PLUR_OUT:x" renders verbatim.

Known limitations, documented in docs/usage.md

  • Bytes written straight to fd 1 (a system/backtick subprocess) cannot be tagged. They still reach rawOutput (failed workers only), and progress characters sharing that physical line are not counted — the parser refuses to guess rather than miscount.
  • print with no trailing newline followed by exit! loses the partial. Plain Ruby behaves the same (ruby -e 'print "x"; exit!' | cat prints nothing).
  • $stdout.flush does not force a partial line out; it stays buffered until a newline or exit.

Tests

New fixtures/projects/minitest-stdout covering print-without-newline, mixed write styles, writes from parallel threads, binary bytes, and lines larger than the read buffer; an output_on_failure_test for minitest-failures, which previously had no puts at all; and a Ruby unit spec for the plugin.

All six new integration examples were confirmed to fail against a binary built from main and pass here — including the old bug verbatim, a 14-character progress line for an 8-test run.

Full Ruby suite 407 examples / 0 failures / 4 pre-existing pending; all Go packages ok; PLUR_RACE=1 clean with the race detector confirmed genuinely live via a deliberate probe. The three guarded examples in minitest_integration_spec.rb are unmodified.

One caveat worth stating: the thread-safety spec is probabilistic. Removing the plugin's mutex and adding a Thread.pass in the window reproduces the corruption, but removing the mutex alone passed 12/12 runs, since MRI's GVL rarely preempts there. The race is real; the detector is not deterministic.

Supersedes #105, which fixed the dropped output with more heuristics rather than removing the ambiguity.

Minitest writes progress characters with no trailing newline, so a test's
`puts` lands on the same physical line as the dots. The parser used to guess:
it took the leading run of [.FES] and called the rest text. That miscounted
whenever a test printed after the dots, stole the F out of `puts "FIRST_LINE"`,
and dropped the text entirely on passing runs.

Invert the marking. Progress stays bare - it is the common case and costs
nothing - and plur injects a minitest plugin that prefixes everything the tests
write with PLUR_OUT:. Minitest's reporters capture the real $stdout when they
are constructed, which happens before plugin init, so swapping $stdout only
redirects test output. Parsing is then exact: before the marker is progress,
after it is the test's own bytes.

The plugin delegates formatting to StringIO so puts/print/write/printf keep
Ruby's semantics, guards its line buffer with a mutex for parallelize_me!, and
overrides dup/reopen so capture_subprocess_io still works (duping the wrapper
would have closed the real stdout and killed the run).

Test output is collected per worker and printed after the progress line rather
than streamed, so the run of dots stays intact.

Needs minitest 5.0+ for the plugin system; below that, and for bytes written
straight to fd 1 by a subprocess, output falls back to the previous behaviour
of appearing only for failed workers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant