Fail fast when no tests are selected after collection

[p-datadog]

What does this PR do?

When pytest collects zero tests for a scenario, the run currently either silently exits with code 0 (a "green" CI build that tested nothing) or hangs waiting for test setup that never arrives. This makes it easy to miss misconfigured scenarios or broken test selectors — the build looks healthy when it isn't.

This PR adds a check in pytest_collection_finish that calls pytest.exit() with return code 1 when no test items survive collection filtering, giving a clear error message instead of a silent pass.

Why this approach?

session.testscollected doesn't work here

One thing that caught my eye during investigation: session.testscollected is always 0 inside pytest_collection_finish. This isn't a bug in system-tests — it's a pytest 7.1.3 hook ordering issue. In Session.perform_collect, the assignment happens after the hook fires:

# _pytest/main.py, Session.perform_collect (pytest 7.1.3)
hook.pytest_collection_modifyitems(...)   # line 57 — filters items
hook.pytest_collection_finish(session=self) # line 61 — our hook runs here  
self.testscollected = len(items)            # line 63 — set AFTER the hook!

len(session.items) is the correct check — it reflects the already-filtered list.

Graceful exit, not a hard kill

An earlier attempt at this used os.kill(os.getpid(), 9) (SIGKILL) to force an immediate exit. That works, but it skips all cleanup — container teardown, log flushing, pytest plugins, atexit handlers. If containers were started during session setup, they'd be orphaned. pytest.exit() runs the normal shutdown path.

Respecting legitimate zero-test modes

The check is placed after the --collect-only and --declaration-report guards (which return early and don't need tests), and explicitly skips two modes where zero tests are intentional:

• --sleep deselects all items on purpose (it keeps the environment running for manual exploration)
• --skip-empty-scenario is used when all tests in a scenario are xfail/skip — pytest_sessionfinish already converts NO_TESTS_COLLECTED to exit code 0 for this case

The change

conftest.py — 10 lines added in pytest_collection_finish, between the existing early-return guards and the sleep mode handler:

if (
    len(session.items) == 0
    and not session.config.option.sleep
    and not session.config.option.skip_empty_scenario
):
    pytest.exit("No tests were selected — check scenario name and test filters", returncode=1)

What it looks like in practice

Before (wrong scenario name):

$ ./run.sh --scenario DOES_NOT_EXIST
...
===== no tests ran =====      # exit code 0 — CI reports success

After:

$ ./run.sh --scenario DOES_NOT_EXIST
...
!!! No tests were selected — check scenario name and test filters
                              # exit code 1 — CI reports failure

Modes unaffected

Mode	Zero tests expected?	Behavior
--collect-only	N/A	Returns before the check
--declaration-report	N/A	Returns before the check
--sleep	Yes	Excluded from check
--skip-empty-scenario	Yes	Excluded from check
Normal run	No	Fails with exit code 1

Test plan

• Run a valid scenario — tests execute normally
• Run with a bogus scenario name — exits with code 1 and clear message
• Run with --collect-only — works as before
• Run with --skip-empty-scenario on a scenario with only xfail tests — exits 0
• Run with --sleep — enters sleep mode as before

🤖 Generated with Claude Code