Problem
scrub-eval-results.sh restricts processing to files matching TEXT_SUFFIXES = {".log", ".txt", ".json", ".jsonl", ".yaml", ".yml", ".md"} via iter_text_files(). The upload step archives all files under eval/runs/ (minus .eval-env).
Files with unlisted extensions (.xml, .csv, .html, .sh, .toml, .cfg, extensionless) bypass both redaction and the fail-closed verification but are still uploaded. The "fail closed" guarantee has a blind spot exactly aligned with the upload path — it cannot detect secrets in files it never reads.
Currently the eval harness produces .json, .yaml, and .log outputs, so the primary leak vector is covered. But as more agents join the eval pipeline, new file types could be introduced silently.
Suggested Fix
Either:
(a) Expand verification to scan all text-decodable files (preferred):
Add a fallback in iter_text_files that yields any file decodable as UTF-8 regardless of extension, at least for the verification pass. Use a binary detection heuristic (null bytes in the first 8KB) to skip true binaries.
(b) Restrict upload glob to match scrub scope:
path: |
eval/runs/**/*.log
eval/runs/**/*.txt
eval/runs/**/*.json
eval/runs/**/*.jsonl
eval/runs/**/*.yaml
eval/runs/**/*.yml
eval/runs/**/*.md
Context
Found during review of #184. The scrub script was introduced in that PR and all prior review findings were addressed, but this gap was not caught by the automated reviewers.
Problem
scrub-eval-results.shrestricts processing to files matchingTEXT_SUFFIXES = {".log", ".txt", ".json", ".jsonl", ".yaml", ".yml", ".md"}viaiter_text_files(). The upload step archives all files undereval/runs/(minus.eval-env).Files with unlisted extensions (
.xml,.csv,.html,.sh,.toml,.cfg, extensionless) bypass both redaction and the fail-closed verification but are still uploaded. The "fail closed" guarantee has a blind spot exactly aligned with the upload path — it cannot detect secrets in files it never reads.Currently the eval harness produces
.json,.yaml, and.logoutputs, so the primary leak vector is covered. But as more agents join the eval pipeline, new file types could be introduced silently.Suggested Fix
Either:
(a) Expand verification to scan all text-decodable files (preferred):
Add a fallback in
iter_text_filesthat yields any file decodable as UTF-8 regardless of extension, at least for the verification pass. Use a binary detection heuristic (null bytes in the first 8KB) to skip true binaries.(b) Restrict upload glob to match scrub scope:
Context
Found during review of #184. The scrub script was introduced in that PR and all prior review findings were addressed, but this gap was not caught by the automated reviewers.