From a full-repo review (line refs from ecb7fe0).
tools/upload_leaderboard_substrate.py — the write path to the public leaderboard dataset has no guards:
- Silent overwrite (
:209-215): api.upload_file unconditionally replaces any existing <track>/<method>.parquet — no existence check, no --force, no PR mode. A name collision silently clobbers an existing entry, including the locf / seasonal_naive baselines every paired skill score depends on.
- Forecasting uploads bypass validation (
:203-204): validate_method_column runs only for ("imputation", "downstream"). Forecasting parquets key on a model column that is never checked — and tools/leaderboard_docs/forecasting/SCHEMA.md:20-26 warns in bold that a substrate left at the evaluate_forecasting default method_name="custom" "collides with every other default submission" — the exact footgun the tool's docstring says the guard exists to prevent. scripts/paper_results/forecasting/stage_leaderboard_substrates.py actively prints upload commands through this unguarded path.
- Wrong-file upload (
:117-124): find_parquet falls back to "the sole parquet in the directory" when no name matches — --method typo_name against a one-parquet dir uploads that file under the wrong identity.
- Minor: sidecar written only if a sidecar flag is provided (
:241) — a plain invocation uploads no meta.json; parquet + sidecar are two non-atomic HF commits; --private defaults True for a dataset that must be public; the --fallback-rate flag silently overrides the measured value from --results-json (honor-system leak in an otherwise measured pipeline).
Suggested fix: check existence and require --force to overwrite (hard-refuse for baseline method names); validate the model column for forecasting; drop the single-parquet fallback (exact match or fail); always write the sidecar (derive from results.json); consider removing the manual --fallback-rate override.
From a full-repo review (line refs from
ecb7fe0).tools/upload_leaderboard_substrate.py— the write path to the public leaderboard dataset has no guards::209-215):api.upload_fileunconditionally replaces any existing<track>/<method>.parquet— no existence check, no--force, no PR mode. A name collision silently clobbers an existing entry, including thelocf/seasonal_naivebaselines every paired skill score depends on.:203-204):validate_method_columnruns only for("imputation", "downstream"). Forecasting parquets key on amodelcolumn that is never checked — andtools/leaderboard_docs/forecasting/SCHEMA.md:20-26warns in bold that a substrate left at theevaluate_forecastingdefaultmethod_name="custom""collides with every other default submission" — the exact footgun the tool's docstring says the guard exists to prevent.scripts/paper_results/forecasting/stage_leaderboard_substrates.pyactively prints upload commands through this unguarded path.:117-124):find_parquetfalls back to "the sole parquet in the directory" when no name matches —--method typo_nameagainst a one-parquet dir uploads that file under the wrong identity.:241) — a plain invocation uploads nometa.json; parquet + sidecar are two non-atomic HF commits;--privatedefaults True for a dataset that must be public; the--fallback-rateflag silently overrides the measured value from--results-json(honor-system leak in an otherwise measured pipeline).Suggested fix: check existence and require
--forceto overwrite (hard-refuse for baseline method names); validate themodelcolumn for forecasting; drop the single-parquet fallback (exact match or fail); always write the sidecar (derive from results.json); consider removing the manual--fallback-rateoverride.