Skip to content

[Bug] compute() returns 2 values on error but all callers unpack 3 — ValueError crash in BWT/FWT/Matrix metrics #531

Description

@g-k-s-03

Environment

  • File: core/testcasecontroller/metrics/metrics.py (lines 70, 99–132)
  • Affects: All lifelong learning / incremental learning benchmarks using BWT, FWT, or Matrix metrics
  • Discovered: 2026-06-11

Description

compute() returns a 2-tuple when the input matrix is malformed (flag=False),
but every caller in the same file unconditionally unpacks 3 values.
This causes a hard ValueError crash for any benchmark run using BWT/FWT metrics on a non-square task matrix.

Current Behavior

# compute() on bad input — returns only 2 values:
if not flag:
    bwt_score = np.nan
    fwt_score = np.nan
    return bwt_score, fwt_score   # ← 2-tuple

# All three callers unpack 3 values:
_, BWT_score, _ = compute("all", info["all"])    # ValueError
_, _, FWT_score = compute("all", info["all"])    # ValueError
my_matrix, _, _ = compute(key, info[key])        # ValueError

Error

ValueError: not enough values to unpack (expected 3, got 2)

Expected Behavior

compute() should always return a consistent 3-tuple regardless of the flag value.

Proposed Fix

# In compute(), replace the early return:
if not flag:
    return None, np.nan, np.nan

Impact

This is a core framework file — not an example. Every user running lifelong learning
or incremental learning benchmarks with a non-square task matrix hits this crash.
The three affected callers are bwt_func(), fwt_func(), and matrix_func().

Steps to Reproduce

  1. Run any benchmark using bwt, fwt, or matrix as a metric
  2. Provide a non-square task matrix as input
  3. Observe ValueError: not enough values to unpack

Additional Context

I am willing to submit a PR for this fix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions