Skip to content

[PS-2066] Warn when skip is unsupported for the runner#532

Draft
gchan wants to merge 1 commit into
mainfrom
gordon/ps-2066-warn-skip-unsupported
Draft

[PS-2066] Warn when skip is unsupported for the runner#532
gchan wants to merge 1 commit into
mainfrom
gordon/ps-2066-warn-skip-unsupported

Conversation

@gchan
Copy link
Copy Markdown
Contributor

@gchan gchan commented Jun 4, 2026

Demonstration PR, not for merge. Opened as a draft to show this approach as an option for PS-2066 / PS-2061.

For runners that cannot skip tests (e.g. Jest), bktec now warns loudly when the suite has tests marked for skipping in files the runner will run, instead of failing the build silently.

Test Engine honours skips by omitting tests from a node's task list, which only works for runners that split by example. File-splitting runners like Jest receive the whole file and run every test in it, so a "skipped" test runs anyway and its failure fails the build with no indication that skip is unsupported — the behaviour a customer hit in PS-2061.

Detection uses the filter_tests endpoint rather than testPlan.SkippedTests. The server tags each returned file with a reason ("slow file" or "file contains 1 or more skipped tests"); skipped files are always returned, slow files only when split_by_example is true. The client previously decoded only path and discarded reason. This change decodes reason and, for non-skip runners, warns when any file carries the skip reason.

Decisions Made

filter_tests, not the Quarantine REST API. GET .../tests/skipped returns example-level data but is suite-wide, so it over-reports tests from files this build doesn't run. filter_tests is build-scoped (filters against the POSTed files).

Advisory only. The warning never mutes, never alters the task list, and never changes the exit code. Errors from filter_tests (including the endpoint's retryable "still filtering" responses) are logged to debug and ignored, since this is off the critical path.

File-level, not test-level. filter_tests returns files, not examples, so the warning names the affected files but not the individual skipped tests. This is the same client-side limitation that made the PS-2065 mute-fallback unworkable.

Context

Linear: PS-2066, related to PS-2061. Alternative to PS-2065 (#531, closed).

Verification

go test ./internal/command/ ./internal/api/ — backed by specs. New tests cover: a skip-reason file produces a warning (and a slow-reason file in the same response does not), no warning when only slow files are returned, and skip-capable runners (RSpec) skip the check entirely (no filter_tests call). Server contract (reason values, skip-vs-slow gating) fact-checked against the buildkite Rails app. AI assisted.

Rollback

Yes. Additive, advisory-only change behind a runner-capability check; reverting the commit removes the warning with no behavioural change.

Test Engine honours skips by omitting tests from a node's task list,
which only works for runners that split by example. File-splitting
runners like Jest receive the whole file and run every test in it, so a
test marked for skipping runs anyway and its failure fails the build,
silently.

Detect this via the filter_tests endpoint, which tags each returned file
with a reason. Decode the previously discarded reason field and, for
runners that cannot skip, warn loudly when any file carries the reason
'file contains 1 or more skipped tests'. Advisory only: nothing about
what runs changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant