Skip to content

bug(core/lifelong): HAS_COMPLETED_INITIAL_TRAINING set to True on first training call in standard mode #572

Description

@g-k-s-03

Description

In lifelong_learning.py, the standard benchmark mode (elif mode != 'multi-inference') starts its loop at r=1, but _train() uses
rounds < 1 to determine whether the initial training has completed.
Since 1 < 1 is always False, HAS_COMPLETED_INITIAL_TRAINING is
set to 'True' on the very first call — before any training has
actually occurred.

Files Involved

  • core/testcasecontroller/algorithm/paradigm/lifelong_learning/lifelong_learning.py
    • Line 266: standard mode loop starts at r=1
    • Line 359: condition if rounds < 1 is never True when called from standard mode

Root Cause

The no-inference and hard-example-mining modes start their loops at
r=0, so rounds < 1 correctly evaluates to True on the first
call and sets HAS_COMPLETED_INITIAL_TRAINING = 'False'. The standard
mode starts at r=1, making the condition always False, so the env
var is immediately set to 'True' before the first training round.

Impact

This is a silent correctness bug. On the very first training call,
every lifelong learning algorithm receives
HAS_COMPLETED_INITIAL_TRAINING == 'True' and assumes a prior model
state exists. The algorithm either attempts to load a non-existent
checkpoint or skips knowledge-base initialization. No exception is
raised — training proceeds but builds on an incorrect initial state,
silently corrupting all subsequent benchmark metrics (BWT, FWT, Matrix).

Steps to Reproduce

Add the following assertion to any lifelong learning basemodel:

def train(self, train_data):
    assert os.environ["HAS_COMPLETED_INITIAL_TRAINING"] == "False", \
        f"Got {os.environ['HAS_COMPLETED_INITIAL_TRAINING']} on first call"

Run any lifelong learning benchmark in standard mode. The assertion
fails on the first round, confirming the env var is incorrectly set.

Expected Behavior

HAS_COMPLETED_INITIAL_TRAINING should be 'False' on the first
training call and 'True' on all subsequent calls, regardless of the
loop start index.

Proposed Fix

Change line 359 from:

if rounds < 1:

to:

if rounds <= 1:

This ensures the condition is True when r=1 (first call in
standard mode) and False for all subsequent rounds (r=2,3,...).
The no-inference and hard-example-mining modes are unaffected since
they call _train with r=0, which still satisfies r <= 1.

Why This Is Multi-File

The bug is set in lifelong_learning.py but its impact is realized
in user algorithm code (any basemodel.py consuming
HAS_COMPLETED_INITIAL_TRAINING). The broken contract propagates
silently from the paradigm layer into algorithm implementations,
making it invisible unless you trace the env var across both layers.

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions