Skip to content

fix(core/lifelong): correct rounds condition to fix HAS_COMPLETED_INITIAL_TRAINING in standard mode#573

Open
g-k-s-03 wants to merge 2 commits into
kubeedge:mainfrom
g-k-s-03:fix/lifelong-has-completed-initial-training
Open

fix(core/lifelong): correct rounds condition to fix HAS_COMPLETED_INITIAL_TRAINING in standard mode#573
g-k-s-03 wants to merge 2 commits into
kubeedge:mainfrom
g-k-s-03:fix/lifelong-has-completed-initial-training

Conversation

@g-k-s-03

@g-k-s-03 g-k-s-03 commented Jul 2, 2026

Copy link
Copy Markdown

Description

In standard benchmark mode (elif mode != 'multi-inference'), the
training loop starts at r=1. The _train() method uses rounds < 1
to determine whether initial training has completed. Since 1 < 1 is
always False, HAS_COMPLETED_INITIAL_TRAINING is set to 'True'
on the very first call — before any training has occurred.

This is a silent correctness bug. Every lifelong learning algorithm
receives an incorrect signal on the first training round, causing it
to either attempt loading a non-existent checkpoint or skip
knowledge-base initialization. Benchmark metrics (BWT, FWT, Matrix)
are silently corrupted as a result.

The no-inference and hard-example-mining modes are unaffected — they
pass r=0, which still satisfies rounds <= 1.

Root Cause

The condition rounds < 1 only evaluates to True when r=0. The
standard mode loop starts at r=1, so the condition is never True
on the first call. Changing < to <= makes the condition True
when r=1 (first call in standard mode) and False for all
subsequent rounds (r=2, 3, ...).

Fix

# Before (line 359):
if rounds < 1:

# After:
if rounds <= 1:

Files Changed

  • core/testcasecontroller/algorithm/paradigm/lifelong_learning/lifelong_learning.py

Testing

The fix can be verified by adding the following assertion to any
lifelong learning basemodel and running a standard mode benchmark:

def train(self, train_data):
    assert os.environ["HAS_COMPLETED_INITIAL_TRAINING"] == "False", \
        f"Got {os.environ['HAS_COMPLETED_INITIAL_TRAINING']} on first call"

This assertion passes after the fix and fails on the unfixed code.

Fixes #572

@kubeedge-bot kubeedge-bot requested review from MooreZheng and hsj576 July 2, 2026 07:21
@kubeedge-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: g-k-s-03
To complete the pull request process, please assign jaypume after the PR has been reviewed.
You can assign the PR to them by writing /assign @jaypume in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubeedge-bot kubeedge-bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jul 2, 2026
…TIAL_TRAINING in standard mode, Issue 572

Signed-off-by: g-k-s-03 <govindsingh97704@gmail.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds validation to ensure the aggregation module is present and initialized in federated learning, raising a ValueError if not. It also modifies the condition for checking if initial training has completed in lifelong learning from rounds < 1 to rounds <= 1. However, this change to rounds <= 1 introduces a regression in no-inference and hard-example-mining modes where the first round starts at r=0. A suggestion is provided to conditionally determine the initial training round based on the active mode.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines 359 to 362
if rounds <= 1:
os.environ["HAS_COMPLETED_INITIAL_TRAINING"] = 'False'
else:
os.environ["HAS_COMPLETED_INITIAL_TRAINING"] = 'True'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Changing the condition to rounds <= 1 fixes the issue in standard mode (where the first round starts at r=1), but it introduces a regression in no-inference and hard-example-mining modes. In those modes, the first round starts at r=0 and the second round is r=1. With rounds <= 1, the second round (r=1) will be incorrectly treated as the initial training round (HAS_COMPLETED_INITIAL_TRAINING set to 'False').

To support both loop starting points correctly, we should determine whether it is the initial training round based on the active mode.

Suggested change
if rounds <= 1:
os.environ["HAS_COMPLETED_INITIAL_TRAINING"] = 'False'
else:
os.environ["HAS_COMPLETED_INITIAL_TRAINING"] = 'True'
mode = (self.model_eval_config or {}).get('model_metric', {}).get('mode')
is_initial = rounds < 1 if mode in ('no-inference', 'hard-example-mining') else rounds <= 1
if is_initial:
os.environ['HAS_COMPLETED_INITIAL_TRAINING'] = 'False'
else:
os.environ['HAS_COMPLETED_INITIAL_TRAINING'] = 'True'

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid point — updated the fix to be mode-aware. The condition now uses rounds < 1 for no-inference and hard-example-mining modes (preserving original behavior) and rounds <= 1 for standard mode (the actual fix). This avoids the regression you identified

@g-k-s-03 g-k-s-03 force-pushed the fix/lifelong-has-completed-initial-training branch from 9adcc5f to f042092 Compare July 2, 2026 07:23
@kubeedge-bot kubeedge-bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 2, 2026
…L_TRAINING

Signed-off-by: g-k-s-03 <govindsingh97704@gmail.com>
@g-k-s-03

g-k-s-03 commented Jul 2, 2026

Copy link
Copy Markdown
Author

@jaypume The PR description above reflects the initial approach. The actual fix is mode-aware it uses rounds < 1 for no-inference and hard-example-mining modes (preserving original behaviour) and rounds <= 1 for standard mode. See the latest commit for the final implementation

@g-k-s-03

g-k-s-03 commented Jul 2, 2026

Copy link
Copy Markdown
Author

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(core/lifelong): HAS_COMPLETED_INITIAL_TRAINING set to True on first training call in standard mode

2 participants