[LFX Term 2 2026] Proposal for Example Restoration and CI Example Validation Framework

[ken6078]

What type of PR is this?

/kind bug
/kind documentation
/kind design

What this PR does / why we need it:

This PR adds an Example Restoration CI pipeline that automatically scans and validates all examples, classifies detected issues, and publishes health reports for maintainers. It also includes targeted restoration work for examples/llm_simple_qa, including portable paths, documented dependencies, dataset/JSONL validation, model configuration, device fallback, and metric robustness.

Many examples may become invalid over time due to code, dependency, or documentation changes. Without automated validation, these issues are difficult to track at scale. The proposed CI provides continuous visibility into example status, helps maintain a reliable example ecosystem, and uses llm_simple_qa as the first restored reference example for validating future LLM-related examples.

Which issue(s) this PR fixes:
Fixes #230

[kubeedge-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ken6078
To complete the pull request process, please assign moorezheng after the PR has been reviewed.
You can assign the PR to them by writing /assign @moorezheng in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive proposal for the KubeEdge Ianvs Example Classification CI Validation Framework, which aims to automate example classification, validation, and the restoration of the llm_simple_qa example. The review feedback correctly identifies minor path and spelling inconsistencies in the proposal metadata and documentation, specifically recommending the correction of examples/llm_qa_example/requestment.txt to examples/llm_simple_qa/requirements.txt.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[MooreZheng]

Good to see the new proposal. As discussion,

1. User flow can be added, e.g., the use of github action and robot
2. Architecture design, e.g., modified and submitted modules
3. Directory organization and design, e.g., proposals and CI
4. Demo

[MooreZheng]

Good to see the updates, especally the demo, directories and user flow

1. More thoughts on User flow

• Developer use local CI, Github CI -> robot/CI check PR -> CI for docs, CI for example, CI for core code, take a look at doc guidance to learn how to check results and modification. Need to check out whether all user cases are covered
• Maintainer use Github CI -> integration test. But how to triger CI and where to check the report?
• Priority: Global CI > local CI > Robot.

2. Some Methods

• As for CI for docs, take a look at baoyue's work #105
• Add Document guidance. Might consider adding a statement in KubeEdge robot in the future
• For integration CI test, might use unit test for developer small scale dataset.

3. Architecture design

• Show relationship between new modules and existing modules
• New module: CI in .github/workflow, take "workflow" as a module. might rename as ci-workflow
• Directory: ianvs/tool -> ianvs/resource/tool

[MooreZheng]

1. Put example status table in the example directory
2. Add A-B-C example and justify the reason why we compare BC instead AB.
3. Add definition for Tier 0, Tier 1... etc
4. T2 and T3 output is not the same,and avoid link to the same "classiy failure"
5. Merged pull request triggers T3, instead of CICD triggers T3. In this case, T2 failure report can also update T3 failure report
6. UC numbering and naming update
7. Update module name like Event Trigger

[MooreZheng]

This version is very close to the final one

1. add example status into examples/README.md
2. event trigger and example validator under CICD instead of workflow
3. put resources/tools into example validator
4. add other functions in validator into validator/tools
5. alpha-beta-gramma rename

[ken6078]

Dear @MooreZheng,

Thank you for your review. I have updated the proposal based on each of your comments, as summarized below:

1. Add example status information to examples/README.md

   Addressed. I added an example status matrix to examples/README.md, including the latest validation time and status of each example.

   I also added status_directions.md to the documentation section to explain the meaning of each status.

2. Place Event Trigger and Example Validator under CI/CD instead of Workflow

   Addressed in the updated software architecture diagram.

3. Move resources/tools into Example Validator

   Addressed. The validation-related tooling is now grouped under the Example Validator component instead of being represented as a general-purpose repository-level resources/tools module.

   The proposal now uses the following dedicated location:

   .github/tools/example_validator/
   

   This keeps the CI-specific validation implementation close to the GitHub Actions integration and avoids mixing it with Ianvs runtime resources.

4. Place the other validator functions under validator/tools

   Addressed by reorganizing the proposed validator structure. I placed the supporting modules under example_validator/core/ instead of introducing another tools/ directory, because the parent path already contains a tools directory.

   .github/tools/example_validator/
   ├── main.py
   ├── static_validator.py
   ├── dependency_validator.py
   ├── smoke_test_validator.py
   ├── core/
   │   ├── validation_branch_wrapper.py
   │   ├── inventory.py
   │   ├── regression_detector.py
   │   └── report_generator.py
   └── data/
       └── example_inventory.yaml
   

   This structure makes the ownership and responsibilities of the validation utilities clearer while keeping all related implementation within the Example Validator module.

5. Rename Alpha, Beta, and Gamma

   Addressed. I replaced the placeholder labels with names that more clearly describe the corresponding events:

   Alpha → Previous Tier 3
   Beta  → Tier 2 Merge
   Gamma → Original Scheduled Next Tier 3
   Delta → Rescheduled Next Tier 3
   

In addition, I squashed the previous commits into two focused commits to make the revision history easier to review:

1. Reorganize the proposed file structure.
2. Add and update the proposal content.

The detailed commit history and corresponding changes can be found here:

https://github.com/ken6078/ianvs/commits/temp/week5-changes/

Thank you again for the detailed feedback.