Problem
The compute_skill_scores docstring in src/imputation_evaluation/evaluation/paper_metrics_core.py (currently around lines 419–437) describes an obsolete aggregation and scope layout:
overall as a flat geomean over all 19 per-channel tasks across scenarios
- collapsed scopes named
cat_collapsed:sleep / cat_collapsed:workouts
- an
overall_binary_collapsed scope
That no longer matches the executable implementation. Since the C2/C3 consolidation, T2 overall is the universal three-level B.2 aggregation:
- Within each
(scenario, bucket), mean log(clip(R_task)) over constituent tasks.
- Within each scenario, mean over buckets present.
- Across scenarios, mean the scenario-level log ratios.
- Return
1 - exp(mean_log_ratio).
Bucket sourcing is:
- activity: continuous
ch_0..ch_4
- physiology: continuous
ch_5..ch_6
- sleep:
cat_collapsed:sleep
- workouts:
cat_collapsed:workouts
Per-channel binary rows ch_7..ch_18 do not contribute directly to per-scenario, semantic, or overall scopes. They enter those scopes only through the collapsed binary rows. The emitted headline scope is overall; overall_binary_collapsed no longer exists.
Reimplementing from the stale docstring produces a materially different, channel-count-weighted result (especially overweighting workouts).
The rank aggregation docstring near aggregate_task_ranks_to_scopes is stale in the same way, and several nearby comments still mention overall_binary_collapsed despite no executable code emitting that scope.
Current source of truth
- Three-level skill aggregation:
paper_metrics_core.py around lines 512–530 and 590–597
- Bucket mapping:
b2_bucket_for_channel and _bucket_log_R_per_scenario
- Canonical documentation:
src/imputation_evaluation/METRICS.md, section 5.3.1
- Historical rename/removal: commit
e863802 (overall_binary_collapsed → overall; legacy flat overall deleted)
Acceptance criteria
Problem
The
compute_skill_scoresdocstring insrc/imputation_evaluation/evaluation/paper_metrics_core.py(currently around lines 419–437) describes an obsolete aggregation and scope layout:overallas a flat geomean over all 19 per-channel tasks across scenarioscat_collapsed:sleep/cat_collapsed:workoutsoverall_binary_collapsedscopeThat no longer matches the executable implementation. Since the C2/C3 consolidation, T2
overallis the universal three-level B.2 aggregation:(scenario, bucket), meanlog(clip(R_task))over constituent tasks.1 - exp(mean_log_ratio).Bucket sourcing is:
ch_0..ch_4ch_5..ch_6cat_collapsed:sleepcat_collapsed:workoutsPer-channel binary rows
ch_7..ch_18do not contribute directly to per-scenario, semantic, or overall scopes. They enter those scopes only through the collapsed binary rows. The emitted headline scope isoverall;overall_binary_collapsedno longer exists.Reimplementing from the stale docstring produces a materially different, channel-count-weighted result (especially overweighting workouts).
The rank aggregation docstring near
aggregate_task_ranks_to_scopesis stale in the same way, and several nearby comments still mentionoverall_binary_collapseddespite no executable code emitting that scope.Current source of truth
paper_metrics_core.pyaround lines 512–530 and 590–597b2_bucket_for_channeland_bucket_log_R_per_scenariosrc/imputation_evaluation/METRICS.md, section 5.3.1e863802(overall_binary_collapsed→overall; legacy flatoveralldeleted)Acceptance criteria
compute_skill_scoresscope/aggregation docstring to describe the universal three-level B.2 form.ch_7..ch_18are excluded from headline aggregation and enter viacat_collapsed:*.overall_binary_collapsedis an emitted scope.aggregate_task_ranks_to_scopesdocstring and nearby stale comments consistently.cat:activity,cat:physiology,cat:sleep,cat:workouts,semantic, andoverall.overall_binary_collapsedscope is emitted and thatoverallfollows bucket × scenario balancing.