Feature/workflowtest2 - #220
Open
nhanasi wants to merge 26 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a GitHub Actions workflow to compute and publish L1 (unit) and L2 (functional) coverage metrics for PRs, and introduces new markdown summary reports under test/docs/.
Changes:
- Introduces a new PR workflow to run L1 unit coverage (lcov) and L2 functional tests (Docker), extract metrics, and post a PR comment.
- Adds generated-style markdown summaries for L1 and L2 coverage in
test/docs/. - Updates existing coverage analysis docs (
L1_Analysis_Report.md,L2_Analysis_Report.md) via workflow scripting (header refresh + in-block value replacement).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
test/docs/L2_Test_Coverage.md |
Adds an L2 coverage summary report (metrics, gap highlights, notes). |
test/docs/L1_Test_Coverage.md |
Adds an L1 coverage summary report (line/function coverage + notes). |
.github/workflows/L1_L2_CoverageReport.yml |
New workflow to run L1/L2 coverage collection, update docs, and upsert a PR comment. |
|
|
||
| - name: Start mockxconf and native-platform containers | ||
| run: | | ||
| docker run -d --name mockxconf -p 50050:50050 -p 50051:50051 -p 50052:50052 -p 50054:50054 -v ${{ github.workspace }}:/mnt/L2_CONTAINER_SHARED_VOLUME ghcr.io/rdkcentral/docker-device-mgt-service-test/mockxconf:latest |
|
|
||
| - name: Build and run L2 tests | ||
| run: | | ||
| docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib && sh run_l2.sh" |
| (r'(Disabled L2 test functions:\s+)[\d~]+', r'\g<1>' + d['TEST_FUNCTIONS_DISABLED']), | ||
| (r'(Active feature scenarios:\s+)[\d~]+', r'\g<1>' + d['FEATURE_SCENARIOS']), | ||
| (r'(Test files active:\s+)[\d~]+', r'\g<1>' + d['TEST_FILES']), | ||
| (r'(Test files disabled[^:]*:\s+)[\d~]+', r'\g<1>' + d['TEST_FUNCTIONS_DISABLED']), |
Comment on lines
+21
to
+23
| - Coverage is generated by running `sh run_ut.sh --enable-cov`. | ||
| - Source artifact: `rfcMgr/gtest/coverage.info`. | ||
| - This file is updated automatically on pull requests. |
Comment on lines
+52
to
+53
| - L2 results are generated by `run_l2.sh` in the PR workflow. | ||
| - This file is refreshed automatically on pull requests. |
Comment on lines
+7
to
+9
| permissions: | ||
| contents: write | ||
| pull-requests: write |
Code Coverage Summary |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (5)
.github/workflows/L1_L2_CoverageReport.yml:9
- Workflow runs on
pull_requestbut requestscontents: writeglobally. Combined with checking out and executing scripts from the PR (e.g.,run_ut.sh,cov_build.sh,run_l2.sh), this grants an untrusted PR branch the ability to use a write-scopedGITHUB_TOKEN(push commits, modify repo contents), which is a high-risk configuration. Reduce the default token permissions to read-only and grantcontents: writeonly to the minimal step/job that actually needs to push docs (and gate that job to same-repo PRs).
permissions:
contents: write
pull-requests: write
.github/workflows/L1_L2_CoverageReport.yml:121
$LD_LIBRARY_PATHinside the double-quoteddocker exec ... bash -c "..."string is expanded by the runner shell before the command reaches the container. This usually results in the container losing its ownLD_LIBRARY_PATH(often empty on the runner), which can break runtime linking. Escape the$or switch to single quotes so expansion happens inside the container.
- name: Build and run L2 tests
run: |
docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib && sh run_l2.sh"
.github/workflows/L1_L2_CoverageReport.yml:293
- In
LINE_REPLACEMENTS, the "Test files disabled" line is being replaced withTEST_FUNCTIONS_DISABLED, which is a different metric and produces incorrect output intest/docs/L2_Analysis_Report.md(it has a distinct "Test files disabled" line). If you don't have a reliableTEST_FILES_DISABLEDmetric, avoid rewriting this line rather than writing the wrong value.
(r'(Active L2 test functions:\s+)[\d~]+', r'\g<1>' + d['TEST_FUNCTIONS']),
(r'(Disabled L2 test functions:\s+)[\d~]+', r'\g<1>' + d['TEST_FUNCTIONS_DISABLED']),
(r'(Active feature scenarios:\s+)[\d~]+', r'\g<1>' + d['FEATURE_SCENARIOS']),
(r'(Test files active:\s+)[\d~]+', r'\g<1>' + d['TEST_FILES']),
(r'(Test files disabled[^:]*:\s+)[\d~]+', r'\g<1>' + d['TEST_FUNCTIONS_DISABLED']),
(r'(Estimated current L2 functional coverage:\s+)~?[\d.]+%',
test/docs/L1_Test_Coverage.md:23
- This note says the file is refreshed automatically on PRs, but the added workflow updates
test/docs/L1_Analysis_Report.md/test/docs/L2_Analysis_Report.md(and posts a PR comment); it does not regeneratetest/docs/L1_Test_Coverage.md. As written, this will quickly become stale/misleading.
- Coverage is generated by running `sh run_ut.sh --enable-cov`.
- Source artifact: `rfcMgr/gtest/coverage.info`.
- This file is updated automatically on pull requests.
test/docs/L2_Test_Coverage.md:53
- This note says the file is refreshed automatically on PRs, but the added workflow updates
test/docs/L1_Analysis_Report.md/test/docs/L2_Analysis_Report.md(and posts a PR comment); it does not regeneratetest/docs/L2_Test_Coverage.md. As written, this will quickly become stale/misleading.
- L2 results are generated by `run_l2.sh` in the PR workflow.
- This file is refreshed automatically on pull requests.
Code Coverage Summary |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+141
to
+147
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| ref: ${{ github.event.workflow_run.head_sha }} | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Download L1 metrics |
Comment on lines
+147
to
+155
| - name: Download L1 metrics | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: l1-metrics | ||
| run-id: ${{ github.event.workflow_run.id }} | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| path: ci-artifacts/l1 | ||
|
|
||
| - name: Download L2 metrics |
| fi | ||
|
|
||
| git commit -m "docs(coverage): refresh L1/L2 coverage for PR #${{ github.event.workflow_run.pull_requests[0].number }}" | ||
| git push origin HEAD:${{ github.event.workflow_run.head_branch }} |
Comment on lines
+171
to
+174
| python3 test/scripts/update_coverage_docs.py \ | ||
| --reports-dir ci-artifacts/l2-reports \ | ||
| --l1-env ci-artifacts/l1/l1_metrics.env \ | ||
| --l2-env ci-artifacts/l2/l2_metrics.env |
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
|
|
||
| git add test/docs/L1_Analysis_Report.md test/docs/L2_Analysis_Report.md |
Comment on lines
+211
to
+214
| with open("test/docs/L2_Analysis_Report.md", "r") as f: | ||
| content = f.read() | ||
| m = re.search(r"<!-- CI-GENERATED-START -->\n(.*?)\n<!-- CI-GENERATED-END -->", content, re.DOTALL) | ||
| print(m.group(1) if m else "_Component table not available_") |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (3)
.github/workflows/L1_L2_CoverageReport.yml:35
- The runner shell expands
$LD_LIBRARY_PATHbefore invokingdocker exec, so the container sees the runner's value (often empty) rather than the container's currentLD_LIBRARY_PATH. Escape$so the expansion happens inside the container shell.
docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib && sh run_l2.sh"
test/docs/L2_Test_Coverage.md:53
- This doc claims it is refreshed automatically on pull requests, but there is no workflow/script in this PR that updates
test/docs/L2_Test_Coverage.md(the automation updatesL2_Analysis_Report.md). This note will become stale/misleading.
- L2 results are generated by `run_l2.sh` in the PR workflow.
- This file is refreshed automatically on pull requests.
test/docs/L1_Test_Coverage.md:23
- This doc claims it is updated automatically on pull requests, but there is no workflow/script in this PR that updates
test/docs/L1_Test_Coverage.md(the automation updatesL1_Analysis_Report.md). This note will become stale/misleading.
- Coverage is generated by running `sh run_ut.sh --enable-cov`.
- Source artifact: `rfcMgr/gtest/coverage.info`.
- This file is updated automatically on pull requests.
Comment on lines
+253
to
+259
| if START_MARKER not in content or END_MARKER not in content: | ||
| print( | ||
| f"ERROR: CI markers not found in {doc_path}. " | ||
| "Add <!-- CI-GENERATED-START --> and <!-- CI-GENERATED-END --> to the document.", | ||
| file=sys.stderr, | ||
| ) | ||
| return False |
Comment on lines
+13
to
+16
| l2-coverage: | ||
| name: L2 functional coverage metrics | ||
| runs-on: ubuntu-latest | ||
|
|
Comment on lines
+133
to
+138
| update-docs: | ||
| name: Update L1/L2 coverage markdowns | ||
| runs-on: ubuntu-latest | ||
| needs: [l2-coverage] | ||
| # Only update docs when code-coverage.yml succeeded (l1-metrics exists) | ||
| if: ${{ github.event.workflow_run.conclusion == 'success' }} |
Code Coverage Summary |
2 similar comments
Code Coverage Summary |
Code Coverage Summary |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
test/docs/L2_Test_Coverage.md:53
- This document claims it is "refreshed automatically on pull requests", but the newly added workflows/scripts only regenerate and commit
L1_Analysis_Report.md/L2_Analysis_Report.md(notL2_Test_Coverage.md). As written, this file will become stale immediately after merge.
- L2 results are generated by `run_l2.sh` in the PR workflow.
- This file is refreshed automatically on pull requests.
test/docs/L1_Test_Coverage.md:23
- This document claims it is "updated automatically on pull requests", but the new automation only updates
test/docs/L1_Analysis_Report.md/test/docs/L2_Analysis_Report.md. Unless another workflow updates this file, this note is inaccurate and the contents will become stale.
- Coverage is generated by running `sh run_ut.sh --enable-cov`.
- Source artifact: `rfcMgr/gtest/coverage.info`.
- This file is updated automatically on pull requests.
test/scripts/update_coverage_docs.py:257
update_l2_report()hard-fails when CI markers are missing, buttest/docs/L2_Analysis_Report.mdcurrently does not contain these markers. In the current setup this will cause the workflow step that regenerates docs to fail every time. Consider auto-inserting a CI-generated block when markers are absent (or update the doc to include the markers).
if START_MARKER not in content or END_MARKER not in content:
print(
f"ERROR: CI markers not found in {doc_path}. "
"Add <!-- CI-GENERATED-START --> and <!-- CI-GENERATED-END --> to the document.",
file=sys.stderr,
.github/workflows/L1_L2_CoverageReport.yml:36
- The
docker exec ... bash -c "... export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:..."string is double-quoted, so$LD_LIBRARY_PATHis expanded by the runner shell before it reaches the container. That can wipe/override the container’s ownLD_LIBRARY_PATHunexpectedly. Also, onlyrun_l2.shis executed even though the coverage updater expects JSON reports fromrun_l2_reboot_trigger.shas well (e.g.,rfc_unknown_accountid.json,rfc_rfc_webpa.json).
- name: Build and run L2 tests
run: |
docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib && sh run_l2.sh"
Code Coverage Summary |
1 similar comment
Code Coverage Summary |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (4)
.github/workflows/L1_L2_CoverageReport.yml:15
workflow_runexecutes with base-repo permissions; checking out and executing the PR head SHA in this job allows untrusted fork PR code to run with write-scopedGITHUB_TOKEN(and any available secrets). Gate this job to same-repo PRs to avoid privilege escalation.
l2-coverage:
name: L2 functional coverage metrics
runs-on: ubuntu-latest
.github/workflows/L1_L2_CoverageReport.yml:139
- For the same reason as the
l2-coveragejob, this job should not check out and run code from fork PR heads underworkflow_runwith write permissions. Combine the existing success check with a same-repo gate.
needs: [l2-coverage]
# Only update docs when code-coverage.yml succeeded (l1-metrics exists)
if: ${{ github.event.workflow_run.conclusion == 'success' }}
test/docs/L2_Test_Coverage.md:53
- This note says the file is refreshed automatically on pull requests, but no workflow or script in this PR references
test/docs/L2_Test_Coverage.md(automation updatesL2_Analysis_Report.mdinstead). This statement will quickly become misleading.
- L2 results are generated by `run_l2.sh` in the PR workflow.
- This file is refreshed automatically on pull requests.
test/docs/L1_Test_Coverage.md:23
- This note says the file is updated automatically on pull requests, but no workflow or script in this PR references
test/docs/L1_Test_Coverage.md(automation updatesL1_Analysis_Report.mdinstead). This statement will quickly become misleading.
- Coverage is generated by running `sh run_ut.sh --enable-cov`.
- Source artifact: `rfcMgr/gtest/coverage.info`.
- This file is updated automatically on pull requests.
Comment on lines
+8
to
+10
| permissions: | ||
| contents: write | ||
| pull-requests: write |
| test_functions=$(grep -R "^def test_" test/functional-tests/tests/ | wc -l | tr -d ' ') | ||
| test_functions_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*def test_" test/functional-tests/tests/ | wc -l | tr -d ' ' || true) | ||
|
|
||
| src_functions=$(grep -R "^[A-Za-z_][A-Za-z0-9_[:space:]\*]*([^;]*)[[:space:]]*{" rfcMgr/ rfcapi/ tr181api/ utils/ --include='*.cpp' --include='*.c' | wc -l | tr -d ' ') |
Comment on lines
+23
to
+31
| - name: Pull required docker images | ||
| run: | | ||
| docker pull ghcr.io/rdkcentral/docker-device-mgt-service-test/mockxconf:latest | ||
| docker pull ghcr.io/rdkcentral/docker-device-mgt-service-test/native-platform:latest | ||
|
|
||
| - name: Start mockxconf and native-platform containers | ||
| run: | | ||
| docker run -d --name mockxconf -p 50050:50050 -p 50051:50051 -p 50052:50052 -p 50054:50054 -v ${{ github.workspace }}:/mnt/L2_CONTAINER_SHARED_VOLUME ghcr.io/rdkcentral/docker-device-mgt-service-test/mockxconf:latest | ||
| docker run -d --name native-platform --link mockxconf -v ${{ github.workspace }}:/mnt/L2_CONTAINER_SHARED_VOLUME ghcr.io/rdkcentral/docker-device-mgt-service-test/native-platform:latest |
Code Coverage Summary |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.