Add --rerun-show-tracebacks to surface retried failures#329
Merged
Conversation
Only the final attempt of a flaky test currently produces a traceback, so failures from earlier attempts -- including those of tests that eventually pass -- are silently discarded. That makes triaging intermittent failures much harder than it needs to be: you can see that something flaked, but not what flaked, unless you happen to be lucky on the last attempt. This adds a --rerun-show-tracebacks flag that appends each retried attempt's traceback to the "rerun test summary info" section. The section is emitted automatically when the flag is set, so -rR is not required alongside it. Fixes pytest-dev#156.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently
pytest-rerunfailuresonly emits the traceback for the final attempt of a flaky test. If the last attempt passes, the failures that triggered the reruns are silently discarded; if the last attempt fails, the earlier tracebacks are gone even when they hit a different exception. That makes triaging flaky tests harder than it needs to be — you can see that a test was retried, but not what actually went wrong on each attempt.This adds
--rerun-show-tracebacks, which appends each retried attempt's traceback to thererun test summary infosection. The section is emitted automatically when the flag is set, so-rRis no longer required.For example, a test that hits a transient assertion twice and passes on the third attempt:
As a side effect, the per-char loop in
pytest_terminal_summaryis gone. The loop calledshow_rerunonce per matching char inreportchars, so passing-r rRdoubled every RERUN line in the summary; the new code callsshow_rerunonce. Happy to revert that part if the duplication is intentional.Fixes #156.