Skip to content

Fix stale T2 aggregation documentation in paper_metrics_core.py #115

Description

@NarayanSchuetz

Problem

The compute_skill_scores docstring in src/imputation_evaluation/evaluation/paper_metrics_core.py (currently around lines 419–437) describes an obsolete aggregation and scope layout:

  • overall as a flat geomean over all 19 per-channel tasks across scenarios
  • collapsed scopes named cat_collapsed:sleep / cat_collapsed:workouts
  • an overall_binary_collapsed scope

That no longer matches the executable implementation. Since the C2/C3 consolidation, T2 overall is the universal three-level B.2 aggregation:

  1. Within each (scenario, bucket), mean log(clip(R_task)) over constituent tasks.
  2. Within each scenario, mean over buckets present.
  3. Across scenarios, mean the scenario-level log ratios.
  4. Return 1 - exp(mean_log_ratio).

Bucket sourcing is:

  • activity: continuous ch_0..ch_4
  • physiology: continuous ch_5..ch_6
  • sleep: cat_collapsed:sleep
  • workouts: cat_collapsed:workouts

Per-channel binary rows ch_7..ch_18 do not contribute directly to per-scenario, semantic, or overall scopes. They enter those scopes only through the collapsed binary rows. The emitted headline scope is overall; overall_binary_collapsed no longer exists.

Reimplementing from the stale docstring produces a materially different, channel-count-weighted result (especially overweighting workouts).

The rank aggregation docstring near aggregate_task_ranks_to_scopes is stale in the same way, and several nearby comments still mention overall_binary_collapsed despite no executable code emitting that scope.

Current source of truth

  • Three-level skill aggregation: paper_metrics_core.py around lines 512–530 and 590–597
  • Bucket mapping: b2_bucket_for_channel and _bucket_log_R_per_scenario
  • Canonical documentation: src/imputation_evaluation/METRICS.md, section 5.3.1
  • Historical rename/removal: commit e863802 (overall_binary_collapsedoverall; legacy flat overall deleted)

Acceptance criteria

  • Rewrite the compute_skill_scores scope/aggregation docstring to describe the universal three-level B.2 form.
  • Document that ch_7..ch_18 are excluded from headline aggregation and enter via cat_collapsed:*.
  • Remove references implying that overall_binary_collapsed is an emitted scope.
  • Update the aggregate_task_ranks_to_scopes docstring and nearby stale comments consistently.
  • Ensure scope names match current output: cat:activity, cat:physiology, cat:sleep, cat:workouts, semantic, and overall.
  • Add or update a regression test that asserts no overall_binary_collapsed scope is emitted and that overall follows bucket × scenario balancing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions