Problem
functional-tests-complete is the branch-protection required check for functional tests. It uses if: always() and only fails when a dependency reports failure or cancelled:
if [ "$TESTS_RESULT" = "failure" ] || [ "$TESTS_RESULT" = "cancelled" ]; then
When gate is skipped (e.g. a pull_request_target run where ok-to-test has not been applied yet), detect and functional-tests are also skipped. The roll-up job sees skipped for all three dependencies, which does not match failure or cancelled, so it exits 0 and reports SUCCESS.
This means a PR can satisfy the functional-tests-complete branch protection rule without any functional test ever running.
Observed on PR #381
| Timestamp (UTC) |
Job |
Conclusion |
| 2026-07-23 01:03 |
gate |
SKIPPED |
| 2026-07-23 01:03 |
detect |
SKIPPED |
| 2026-07-23 01:03 |
functional-tests |
SKIPPED |
| 2026-07-23 01:03 |
functional-tests-complete |
SUCCESS |
Tests only started running after the ok-to-test label was applied (07:37 run).
Expected behavior
functional-tests-complete should not report SUCCESS when gate or functional-tests was skipped due to missing authorization. Either:
- Treat
skipped as a non-passing state when gate was skipped (distinguish "skipped because no agents matched" from "skipped because not authorized").
- Have
gate fail instead of skip when authorization is denied, so the roll-up catches it via the existing failure check.
Problem
functional-tests-completeis the branch-protection required check for functional tests. It usesif: always()and only fails when a dependency reportsfailureorcancelled:if [ "$TESTS_RESULT" = "failure" ] || [ "$TESTS_RESULT" = "cancelled" ]; thenWhen
gateis skipped (e.g. apull_request_targetrun whereok-to-testhas not been applied yet),detectandfunctional-testsare also skipped. The roll-up job seesskippedfor all three dependencies, which does not matchfailureorcancelled, so it exits 0 and reports SUCCESS.This means a PR can satisfy the
functional-tests-completebranch protection rule without any functional test ever running.Observed on PR #381
Tests only started running after the
ok-to-testlabel was applied (07:37 run).Expected behavior
functional-tests-completeshould not report SUCCESS whengateorfunctional-testswas skipped due to missing authorization. Either:skippedas a non-passing state whengatewas skipped (distinguish "skipped because no agents matched" from "skipped because not authorized").gatefail instead of skip when authorization is denied, so the roll-up catches it via the existingfailurecheck.