Description
In lifelong_learning.py, the standard benchmark mode (elif mode != 'multi-inference') starts its loop at r=1, but _train() uses
rounds < 1 to determine whether the initial training has completed.
Since 1 < 1 is always False, HAS_COMPLETED_INITIAL_TRAINING is
set to 'True' on the very first call — before any training has
actually occurred.
Files Involved
core/testcasecontroller/algorithm/paradigm/lifelong_learning/lifelong_learning.py
- Line 266: standard mode loop starts at
r=1
- Line 359: condition
if rounds < 1 is never True when called from standard mode
Root Cause
The no-inference and hard-example-mining modes start their loops at
r=0, so rounds < 1 correctly evaluates to True on the first
call and sets HAS_COMPLETED_INITIAL_TRAINING = 'False'. The standard
mode starts at r=1, making the condition always False, so the env
var is immediately set to 'True' before the first training round.
Impact
This is a silent correctness bug. On the very first training call,
every lifelong learning algorithm receives
HAS_COMPLETED_INITIAL_TRAINING == 'True' and assumes a prior model
state exists. The algorithm either attempts to load a non-existent
checkpoint or skips knowledge-base initialization. No exception is
raised — training proceeds but builds on an incorrect initial state,
silently corrupting all subsequent benchmark metrics (BWT, FWT, Matrix).
Steps to Reproduce
Add the following assertion to any lifelong learning basemodel:
def train(self, train_data):
assert os.environ["HAS_COMPLETED_INITIAL_TRAINING"] == "False", \
f"Got {os.environ['HAS_COMPLETED_INITIAL_TRAINING']} on first call"
Run any lifelong learning benchmark in standard mode. The assertion
fails on the first round, confirming the env var is incorrectly set.
Expected Behavior
HAS_COMPLETED_INITIAL_TRAINING should be 'False' on the first
training call and 'True' on all subsequent calls, regardless of the
loop start index.
Proposed Fix
Change line 359 from:
to:
This ensures the condition is True when r=1 (first call in
standard mode) and False for all subsequent rounds (r=2,3,...).
The no-inference and hard-example-mining modes are unaffected since
they call _train with r=0, which still satisfies r <= 1.
Why This Is Multi-File
The bug is set in lifelong_learning.py but its impact is realized
in user algorithm code (any basemodel.py consuming
HAS_COMPLETED_INITIAL_TRAINING). The broken contract propagates
silently from the paradigm layer into algorithm implementations,
making it invisible unless you trace the env var across both layers.
Description
In
lifelong_learning.py, the standard benchmark mode (elif mode != 'multi-inference') starts its loop atr=1, but_train()usesrounds < 1to determine whether the initial training has completed.Since
1 < 1is alwaysFalse,HAS_COMPLETED_INITIAL_TRAININGisset to
'True'on the very first call — before any training hasactually occurred.
Files Involved
core/testcasecontroller/algorithm/paradigm/lifelong_learning/lifelong_learning.pyr=1if rounds < 1is neverTruewhen called from standard modeRoot Cause
The no-inference and hard-example-mining modes start their loops at
r=0, sorounds < 1correctly evaluates toTrueon the firstcall and sets
HAS_COMPLETED_INITIAL_TRAINING = 'False'. The standardmode starts at
r=1, making the condition alwaysFalse, so the envvar is immediately set to
'True'before the first training round.Impact
This is a silent correctness bug. On the very first training call,
every lifelong learning algorithm receives
HAS_COMPLETED_INITIAL_TRAINING == 'True'and assumes a prior modelstate exists. The algorithm either attempts to load a non-existent
checkpoint or skips knowledge-base initialization. No exception is
raised — training proceeds but builds on an incorrect initial state,
silently corrupting all subsequent benchmark metrics (BWT, FWT, Matrix).
Steps to Reproduce
Add the following assertion to any lifelong learning basemodel:
Run any lifelong learning benchmark in standard mode. The assertion
fails on the first round, confirming the env var is incorrectly set.
Expected Behavior
HAS_COMPLETED_INITIAL_TRAININGshould be'False'on the firsttraining call and
'True'on all subsequent calls, regardless of theloop start index.
Proposed Fix
Change line 359 from:
to:
This ensures the condition is
Truewhenr=1(first call instandard mode) and
Falsefor all subsequent rounds (r=2,3,...).The no-inference and hard-example-mining modes are unaffected since
they call
_trainwithr=0, which still satisfiesr <= 1.Why This Is Multi-File
The bug is set in
lifelong_learning.pybut its impact is realizedin user algorithm code (any
basemodel.pyconsumingHAS_COMPLETED_INITIAL_TRAINING). The broken contract propagatessilently from the paradigm layer into algorithm implementations,
making it invisible unless you trace the env var across both layers.