Environment
- File:
core/testcasecontroller/metrics/metrics.py (lines 70, 99–132)
- Affects: All lifelong learning / incremental learning benchmarks using BWT, FWT, or Matrix metrics
- Discovered: 2026-06-11
Description
compute() returns a 2-tuple when the input matrix is malformed (flag=False),
but every caller in the same file unconditionally unpacks 3 values.
This causes a hard ValueError crash for any benchmark run using BWT/FWT metrics on a non-square task matrix.
Current Behavior
# compute() on bad input — returns only 2 values:
if not flag:
bwt_score = np.nan
fwt_score = np.nan
return bwt_score, fwt_score # ← 2-tuple
# All three callers unpack 3 values:
_, BWT_score, _ = compute("all", info["all"]) # ValueError
_, _, FWT_score = compute("all", info["all"]) # ValueError
my_matrix, _, _ = compute(key, info[key]) # ValueError
Error
ValueError: not enough values to unpack (expected 3, got 2)
Expected Behavior
compute() should always return a consistent 3-tuple regardless of the flag value.
Proposed Fix
# In compute(), replace the early return:
if not flag:
return None, np.nan, np.nan
Impact
This is a core framework file — not an example. Every user running lifelong learning
or incremental learning benchmarks with a non-square task matrix hits this crash.
The three affected callers are bwt_func(), fwt_func(), and matrix_func().
Steps to Reproduce
- Run any benchmark using
bwt, fwt, or matrix as a metric
- Provide a non-square task matrix as input
- Observe
ValueError: not enough values to unpack
Additional Context
I am willing to submit a PR for this fix.
Environment
core/testcasecontroller/metrics/metrics.py(lines 70, 99–132)Description
compute()returns a 2-tuple when the input matrix is malformed (flag=False),but every caller in the same file unconditionally unpacks 3 values.
This causes a hard
ValueErrorcrash for any benchmark run using BWT/FWT metrics on a non-square task matrix.Current Behavior
Error
ValueError: not enough values to unpack (expected 3, got 2)
Expected Behavior
compute()should always return a consistent 3-tuple regardless of theflagvalue.Proposed Fix
Impact
This is a core framework file — not an example. Every user running lifelong learning
or incremental learning benchmarks with a non-square task matrix hits this crash.
The three affected callers are
bwt_func(),fwt_func(), andmatrix_func().Steps to Reproduce
bwt,fwt, ormatrixas a metricValueError: not enough values to unpackAdditional Context
I am willing to submit a PR for this fix.