Stop skipping "test exception" test cases, they are not synthetic#11427
Open
bric3 wants to merge 1 commit into
Open
Stop skipping "test exception" test cases, they are not synthetic#11427bric3 wants to merge 1 commit into
"test exception" test cases, they are not synthetic#11427bric3 wants to merge 1 commit into
Conversation
bric3
changed the title
"test exception" testc ases as synthetic skip
bric3
added
type: bug
bric3
changed the title
"test exception" testc ases as synthetic skip"test exception" test cases as synthetic skip
bric3
changed the title
"test exception" test cases as synthetic skip"test exception" test cases, they are not synthetic
Contributor
|
bric3
force-pushed
the
bdu/revert-test-exception-synthetic-skip
branch
from
6e334c2 to
8a557eb
Compare
"test exception" is not a synthetic placeholder emitted by any test framework — JUnit 4, JUnit Platform, Gradle's adapters, Spock and Kotest all reserve "initializationError"/"executionError" for that purpose, not generic English names. In dd-trace-java the entries observed in CI came from a real Spock feature method (`def "test exception"()` in HttpServerTest.groovy), not a framework synthetic. Tagging that name unconditionally silently masks real pass/fail outcomes for any Spock/JUnit 5 @DisplayName/Kotest test that happens to use it. Drop "test exception" from JUnitReport.tagSyntheticFailures and document the criteria for future synthetic entries (must be framework-emitted, must link to the source pinned to a release tag). Annotate the two remaining entries — initializationError (JUnit 4 + Gradle) and executionError (Gradle) — with the canonical source locations.
bric3
force-pushed
the
bdu/revert-test-exception-synthetic-skip
branch
from
8a557eb to
f60311a
Compare
1 task
mhdatie
approved these changes
May 20, 2026
Contributor
mhdatie
left a comment
mhdatie
left a comment
There was a problem hiding this comment.
LGTM! Was there an example in CI where "test exception" should have not been skipped? And thanks for adding the guardrails as comments too, I have another related ticket which can leverage them 👍 🚀
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Removes
"test exception"from being skipped skipped in the JUnit collect result script. It was added in #11259.They are legitimate synthetic names (
initializationError,executionError), because they are actually coming from framework or libraries."test exception"is not.Also, the changes documents the criteria for any future entry, i.e. this should be motivated by a proof with a source link where that synthetic test name is added.
Motivation
Tagging
"test exception"name unconditionally will silently skip real tests named that way.In fact
"test exception"isn't a synthetic placeholder emitted by any test framework or runner used with this project:JUnit 4 emits
"initializationError"fromErrorReportingRunner(r4.13.2 line 83) when a test class cannot be loaded, has no runnable methods, or its@BeforeClassfails.Gradle's JUnit Platform listener emits both
"initializationError"and"executionError"fromJUnitPlatformTestExecutionListener(v8.14.5) (also in later version of Gradle) for synthetic failures.JUnit 5, Spock, Kotest, TestNG, Maven Surefire/Failsafe — none use the literal
"test exception"as a synthetic.A code search across
junit-team/junit4,junit-team/junit5,gradle/gradle,spockframework/spock, andjetbrains/kotlinconfirms this:the only Gradle hits are unrelated Javadoc on
TestExceptionFormatplususer-named retry tests.
The
test exceptionentries observed for this repo came from the actual Spock test,HttpServerTest.test exception"().In addition to dropping this case, this PR documents in a code comment that
future synthetic entries must:
Additional Notes
def "test exception"()todef "Instrumentation test exception"()) is left in place as it makes the test more self-describing.