fix(scripts): skip hidden output-transaction dirs when resolving fit targets

[frankbuckley]

Note

Drafted by a LLM-based AI tool (Claude Code/Opus 5).

What

The _subdirs() helper shared by four scripts returned every subdirectory of the statistical-models output root, including the hidden staging siblings that StatisticalFitContext.reset_output_dir creates. Each run is staged in .<model-id>-<config>.staging-XXXXXXXX and promoted only after every fit stage succeeds, so a dotted directory is either a run in progress or an abandoned one. Publication also leaves a hidden .<name>.backup-XXXX sibling until it ages out — a previous fit, not the current one.

Each _subdirs now excludes dotted names and carries a docstring giving the caller-specific reason, matching the fix already made in regenerate_psense.py (which is separate work-in-progress and not on this branch, so its own test file covers it there).

Why it matters, per caller

• regenerate_key_findings.py (the script this started from) and regenerate_itt_contrast_figures.py write artefacts that are about to be discarded, or race a live fit. The contrast-figure script gates only on config.json and trace.nc existing, both of which a running fit writes well before it finishes.
• upload.py is the worst case, and was not in the original scope — it turned up in a sweep for the same pattern. Its all / stat branch pushes each resolved directory to blob storage under its own name, so an unfiltered walk publishes the half-written artefacts of a fit that was never accepted, under a .…staging-XXXX label no report or comparison refers to, and records those URLs in uploaded_urls.txt.
• regenerate_mediation_calibration.py is not actually reachable today: every branch of its resolve_targets filters on d.name.startswith(f"{mid}-"), which a leading-dot name fails. The filter is added anyway so the rule is a property of the helper rather than an accident of its caller — flagging it explicitly because it is a behavioural no-op, and it is reasonable to ask for it to be dropped.

Tests

New tests/test_regenerate_key_findings.py pins the exclusion for all four scripts: the staging case end to end through resolve_targets, the .backup- case, the single-model-id form, and _subdirs directly. The direct _subdirs assertion is there because a script whose own id filter happens to exclude dotted names (mediation, above) would otherwise pass without carrying the rule.

Every assertion was verified to fail against the pre-fix code before the fix was restored — 5 failures for the three regenerate scripts, 2 more for upload.py.

Reviewer notes

• The file is named for regenerate_key_findings.py (the script the change started from and the bulk of its cases) but covers the sibling walkers too; the module docstring says so. Happy to split it per script if you would rather match the test_regenerate_psense.py one-file-per-script convention.
• I also swept the rest of scripts/ and src/ for raw iterdir() over an output root. The only other hits are tune_model.py and base_pipeline.py, both operating on their own directories rather than the statistical-models root, so neither is affected. The GB output root has no output transactions at all — base_pipeline clears in place — so the filter there is inert but harmless.
• No estimand, artefact schema or published output changes. The only behavioural difference is which directories these scripts decline to touch.

Checks

• ruff check src/ (also ran over scripts/ and tests/) — pass
• npm run format:check — pass
• npm run spellcheck — pass
• Full pytest suite — 1200 passed

Relates to #394