Create L1_L2_CoverageReport - #217
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions workflow intended to generate L1 (unit) and L2 (functional) coverage metrics on PRs, publish artifacts, regenerate coverage markdown docs, and post/update a PR comment with a summary.
Changes:
- Introduces a multi-job PR workflow to compute and upload L1 and L2 coverage-related artifacts.
- Regenerates L1/L2 coverage markdown files in-repo and (for same-repo PRs) commits them back to the PR branch.
- Upserts a sticky PR comment containing the latest coverage summary.
| - name: Build and run L2 tests | ||
| run: | | ||
| docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME/remote_debugger && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib: && sh run_l2.sh" | ||
|
|
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
| GENERATED_DATE="$(date -u +%Y-%m-%d)" | ||
|
|
||
| cat > test/functional-tests/docs/L1_Test_Coverage.md << EOF | ||
| # Remote Debugger L1 Coverage Report |
| permissions: | ||
| contents: write | ||
| pull-requests: write |
| update-docs: | ||
| name: Update L1/L2 coverage markdowns | ||
| runs-on: ubuntu-latest | ||
| needs: [l1-coverage, l2-coverage] | ||
|
|
| GENERATED_DATE="$(date -u +%Y-%m-%d)" | ||
|
|
||
| cat > test/functional-tests/docs/L1_Test_Coverage.md << EOF |
| test_functions=$(grep -R "^def test_" remote_debugger/test/functional-tests/tests/test_*.py | wc -l | tr -d ' ') | ||
| test_functions_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*def test_" remote_debugger/test/functional-tests/tests/test_*.py | wc -l | tr -d ' ' || true) | ||
|
|
||
| src_functions=$(grep -R "^[A-Za-z_][A-Za-z0-9_[:space:]\*]*([^;]*)[[:space:]]*{" remote_debugger/src/*.c | wc -l | tr -d ' ') |
| c_api_scenarios=$(grep -n "^[[:space:]]*Scenario:" remote_debugger/test/functional-tests/features/rrd_c_api_upload.feature | wc -l | tr -d ' ') | ||
| c_api_tests=$(grep -n "^def test_" remote_debugger/test/functional-tests/tests/test_rrd_c_api_upload.py | wc -l | tr -d ' ') | ||
| c_api_gap=$((c_api_scenarios - c_api_tests)) | ||
| if [ "$c_api_gap" -lt 0 ]; then | ||
| c_api_gap=0 | ||
| fi |
| - name: Upsert PR coverage comment | ||
| uses: actions/github-script@v7 | ||
| env: |
| container: | ||
| image: ghcr.io/rdkcentral/docker-rdk-ci:latest |
| - name: Pull required docker images | ||
| run: | | ||
| docker pull ghcr.io/rdkcentral/docker-device-mgt-service-test/mockxconf:latest | ||
| docker pull ghcr.io/rdkcentral/docker-device-mgt-service-test/native-platform:latest |
Code Coverage Summary |
Code Coverage Summary |
|
|
||
| - name: Build and run L2 tests | ||
| run: | | ||
| docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME/remote_debugger && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib: && sh run_l2.sh" |
| c_api_scenarios=$(grep -n "^[[:space:]]*Scenario:" remote_debugger/test/functional-tests/features/rrd_c_api_upload.feature | wc -l | tr -d ' ') | ||
| c_api_tests=$(grep -n "^def test_" remote_debugger/test/functional-tests/tests/test_rrd_c_api_upload.py | wc -l | tr -d ' ') | ||
| c_api_gap=$((c_api_scenarios - c_api_tests)) | ||
| if [ "$c_api_gap" -lt 0 ]; then | ||
| c_api_gap=0 | ||
| fi |
| GENERATED_DATE="$(date -u +%Y-%m-%d)" | ||
|
|
||
| cat > test/functional-tests/docs/L1_Test_Coverage.md << EOF |
| - name: Upsert PR coverage comment | ||
| uses: actions/github-script@v7 | ||
| env: | ||
| COMMENT_PATH: /tmp/pr_coverage_comment.md | ||
| with: |
| # Remote Debugger L1 Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** \`remotedebugger\` | ||
| **Source:** PR pipeline coverage run |
| # Remote Debugger L2 Test Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** \`remotedebugger\` | ||
| **Test Suite:** \`test/functional-tests\` |
| permissions: | ||
| contents: write | ||
| pull-requests: write |
| container: | ||
| image: ghcr.io/rdkcentral/docker-rdk-ci:latest |
Code Coverage Summary |
| permissions: | ||
| contents: write | ||
| pull-requests: write |
| update-docs: | ||
| name: Update L1/L2 coverage markdowns | ||
| runs-on: ubuntu-latest | ||
| needs: [l1-coverage, l2-coverage] | ||
|
|
| COV_FILE="src/unittest/coverage.info" | ||
| if [ ! -f "$COV_FILE" ]; then | ||
| echo "coverage.info was not generated at $COV_FILE" | ||
| exit 1 | ||
| fi |
| test_functions=$(grep -R "^def test_" remote_debugger/test/functional-tests/tests/test_*.py | wc -l | tr -d ' ') | ||
| test_functions_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*def test_" remote_debugger/test/functional-tests/tests/test_*.py | wc -l | tr -d ' ' || true) | ||
|
|
||
| src_functions=$(grep -R "^[A-Za-z_][A-Za-z0-9_[:space:]\*]*([^;]*)[[:space:]]*{" remote_debugger/src/*.c | wc -l | tr -d ' ') |
| # Remote Debugger L1 Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** `remotedebugger` | ||
| **Source:** PR pipeline coverage run |
| - Coverage is generated by running `sh run_ut.sh --enable-cov`. | ||
| - Source artifact: `src/unittest/coverage.info`. | ||
| - This file is updated automatically on pull requests. |
| # Remote Debugger L2 Test Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** `remotedebugger` | ||
| **Test Suite:** `test/functional-tests` |
| name: l1-coverage-artifacts | ||
| path: | | ||
| src/unittest/coverage.info | ||
| /tmp/coverage_summary.txt |
| 1. `rrd_c_api_upload.feature` scenarios: $C_API_SCENARIOS | ||
| `test_rrd_c_api_upload.py` test functions: $C_API_TESTS | ||
| **Current gap:** $C_API_GAP |
| permissions: | ||
| contents: write | ||
| pull-requests: write |
| docker run -d --name mockxconf -p 50050:50050 -p 50051:50051 -p 50052:50052 -p 50054:50054 -v ${{ github.workspace }}:/mnt/L2_CONTAINER_SHARED_VOLUME ghcr.io/rdkcentral/docker-device-mgt-service-test/mockxconf:latest | ||
| docker run -d --name native-platform --link mockxconf -v ${{ github.workspace }}:/mnt/L2_CONTAINER_SHARED_VOLUME ghcr.io/rdkcentral/docker-device-mgt-service-test/native-platform:latest |
| c_api_scenarios=$(grep -n "^[[:space:]]*Scenario:" remote_debugger/test/functional-tests/features/rrd_c_api_upload.feature | wc -l | tr -d ' ') | ||
| c_api_tests=$(grep -n "^def test_" remote_debugger/test/functional-tests/tests/test_rrd_c_api_upload.py | wc -l | tr -d ' ') | ||
| c_api_gap=$((c_api_scenarios - c_api_tests)) | ||
| if [ "$c_api_gap" -lt 0 ]; then | ||
| c_api_gap=0 | ||
| fi |
| GENERATED_DATE="$(date -u +%Y-%m-%d)" | ||
|
|
||
| cat > test/functional-tests/docs/L1_Test_Coverage.md <<EOF |
| # Remote Debugger L1 Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** `remotedebugger` | ||
| **Source:** PR pipeline coverage run |
| # Remote Debugger L2 Test Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** `remotedebugger` | ||
| **Test Suite:** `test/functional-tests` |
Code Coverage Summary |
2 similar comments
Code Coverage Summary |
Code Coverage Summary |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (4)
.github/workflows/L1_L2_CoverageReport.yml:9
- The workflow upserts PR issue comments via
github.rest.issues.*, which requiresissues: writepermission. With onlypull-requests: write, this step can fail with 403 when listing/creating/updating comments.
permissions:
contents: write
pull-requests: write
.github/workflows/L1_L2_CoverageReport.yml:216
- This job later commits and pushes docs back to the PR branch, but
actions/checkoutonpull_requestdefaults to a detached merge ref (refs/pull/*/merge). Committing on that ref and pushing togithub.head_refcan create an unintended history (or fail). Check out the PR head branch for same-repo PRs, otherwise keep the default ref.
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
.github/workflows/L1_L2_CoverageReport.yml:113
$LD_LIBRARY_PATHis expanded by the runner shell beforedocker execruns, so the container may not inherit its ownLD_LIBRARY_PATHvalue (often resulting in an empty/incorrect path). Quote or escape$so it is evaluated inside the container.
docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib && sh run_l2.sh"
.github/workflows/L1_L2_CoverageReport.yml:130
- With
set -euo pipefail, thisgrep | wc -lpipeline will fail when there are 0 matchingdef test_lines (grep exits 1), even though the desired count is0. Wrapgrepin a group with|| trueso the pipeline succeeds and yields0.
test_functions=$(grep -R "^def test_" test/functional-tests/tests/ | wc -l | tr -d ' ')
| set -euo pipefail | ||
|
|
||
| feature_files=$(find test/functional-tests/features -name '*.feature' | wc -l | tr -d ' ') | ||
| feature_scenarios=$(grep -R "^[[:space:]]*Scenario:" test/functional-tests/features/ | wc -l | tr -d ' ') |
|
|
||
| feature_files=$(find test/functional-tests/features -name '*.feature' | wc -l | tr -d ' ') | ||
| feature_scenarios=$(grep -R "^[[:space:]]*Scenario:" test/functional-tests/features/ | wc -l | tr -d ' ') | ||
| feature_scenarios_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*Scenario:" test/functional-tests/features/ | wc -l | tr -d ' ' || true) |
|
|
||
| test_files=$(find test/functional-tests/tests -maxdepth 1 -name 'test_*.py' | wc -l | tr -d ' ') | ||
| test_functions=$(grep -R "^def test_" test/functional-tests/tests/ | wc -l | tr -d ' ') | ||
| test_functions_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*def test_" test/functional-tests/tests/ | wc -l | tr -d ' ' || true) |
Code Coverage Summary |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
.github/workflows/L1_L2_CoverageReport.yml:9
- Workflow-level permissions grant
contents: write, which exposes a write-capableGITHUB_TOKENto jobs that check out and execute PR code (unit/L2 tests). This is a security risk; keep the default token read-only and grant write scopes only to the job that actually needs to push docs / comment.
permissions:
contents: write
pull-requests: write
.github/workflows/L1_L2_CoverageReport.yml:210
- After scoping the workflow-level token to read-only, the
update-docsjob needs explicit write permissions to push generated markdown and to create/update the PR issue comment via the Issues API.
update-docs:
name: Update L1/L2 coverage markdowns
runs-on: ubuntu-latest
needs: [l1-coverage, l2-coverage]
.github/workflows/L1_L2_CoverageReport.yml:108
mockxconfis mapped to port 50054 here, but the functional tests and test helpers usehttps://mockxconf:50053/...(seetest/functional-tests/tests/rfc_test_helper.py). With the current mapping, L2 tests are likely to fail to reach mockxconf.
docker run -d --name mockxconf -p 50050:50050 -p 50051:50051 -p 50052:50052 -p 50054:50054 -v ${{ github.workspace }}:/mnt/L2_CONTAINER_SHARED_VOLUME ghcr.io/rdkcentral/docker-device-mgt-service-test/mockxconf:latest
.github/workflows/L1_L2_CoverageReport.yml:376
- On fork PRs,
GITHUB_TOKENpermissions are forced to read-only, so this step will 403 when trying to create/update the PR comment. To avoid failing CI for fork PRs, guard the upsert step the same way the push step is guarded (or switch to a saferpull_request_targetdesign).
- name: Upsert PR coverage comment
uses: actions/github-script@v7
Code Coverage Summary |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (5)
.github/workflows/L1_L2_CoverageReport.yml:9
- The workflow upserts a PR comment via
github.rest.issues.*, but the explicitpermissions:block doesn't grantissues: write, which can cause 403s when listing/creating/updating comments.
permissions:
contents: write
pull-requests: write
.github/workflows/L1_L2_CoverageReport.yml:16
- This job runs PR-provided scripts (
run_ut.sh, etc.) but inheritscontents: writefrom the workflow-level token permissions. Restrict this job to read-only permissions to reduce the blast radius of untrusted PR code.
l1-coverage:
name: L1 unit coverage metrics
runs-on: ubuntu-latest
container:
image: ghcr.io/rdkcentral/docker-rdk-ci:latest
.github/workflows/L1_L2_CoverageReport.yml:96
- This job also executes PR-controlled code inside containers and doesn't need write access to the repo. Add job-level read-only token permissions here as well.
l2-coverage:
name: L2 functional coverage metrics
runs-on: ubuntu-latest
.github/workflows/L1_L2_CoverageReport.yml:227
- The L2 job uploads
l2-coverage-artifacts, but this step downloadsl2-metrics. Align the artifact name so the workflow can sourcel2_metrics.env.
- name: Download L2 metrics
uses: actions/download-artifact@v4
with:
name: l2-metrics
path: ci-artifacts/l2
.github/workflows/L1_L2_CoverageReport.yml:336
- On
pull_requestevents,actions/checkouttypically checks out a detached PR merge ref. As written,git push origin HEAD:${{ github.head_ref }}can end up pushing that merge commit onto the PR branch (rewriting it). Switch to the PR head branch before committing/pushing.
git add test/docs/L1_Test_Coverage.md test/docs/L2_Test_Coverage.md
git commit -m "docs(coverage): refresh L1/L2 coverage for PR #${{ github.event.pull_request.number }}"
git push origin HEAD:${{ github.head_ref }}
| functional_coverage_pct=$(awk -v passed="$L2_PASSED" -v total="$feature_scenarios" 'BEGIN { if (total > 0) printf "%.2f", (passed/total)*100; else print "0.00" }') | ||
| gap_to_100_pct=$(awk -v cov="$functional_coverage_pct" 'BEGIN { printf "%.2f", 100-cov }') |
| - name: Download L1 metrics | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: l1-metrics | ||
| path: ci-artifacts/l1 |
Code Coverage Summary |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (5)
.github/workflows/L1_L2_CoverageReport.yml:120
docker exec ... -c "... export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:..."expands$LD_LIBRARY_PATHon the GitHub runner before entering the container, so the container may get an empty/incorrect value (and an empty leading entry can unintentionally add the current directory to the loader search path). Use single quotes (or escape$) so expansion happens inside the container, and build the value without a leading colon.
docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib && sh run_l2.sh"
.github/workflows/L1_L2_CoverageReport.yml:9
- Workflow-level
permissionsgrantscontents: writeandpull-requests: writeto all jobs, including jobs that execute PR-provided code. To reduce blast radius, default the workflow to read-only and grant write permissions only to the job(s) that actually need them (the docs commit + PR comment job).
permissions:
contents: write
pull-requests: write
.github/workflows/L1_L2_CoverageReport.yml:221
- After reducing workflow-level permissions,
update-docsneeds explicit write permissions for pushing docs commits and posting/updating PR comments. Add job-level permissions here to keep other jobs least-privileged.
update-docs:
name: Update L1/L2 coverage markdowns
runs-on: ubuntu-latest
needs: [l1-coverage, l2-coverage]
.github/workflows/L1_L2_CoverageReport.yml:391
- On fork-based PRs,
pull_requestworkflows run with a read-onlyGITHUB_TOKEN, so theactions/github-scriptcomment upsert will fail and can make the whole workflow red even though you already detect fork PRs for doc commits. Gate this step (or make it non-fatal) for fork PRs.
- name: Upsert PR coverage comment
uses: actions/github-script@v7
env:
COMMENT_PATH: /tmp/pr_coverage_comment.md
with:
.github/workflows/L1_L2_CoverageReport.yml:17
- This workflow pulls/runs multiple container images using the moving
:latesttag. For supply-chain safety and reproducibility, pin these images to an immutable version (or ideally a digest, e.g.@sha256:...).
container:
image: ghcr.io/rdkcentral/docker-rdk-ci:latest
| import glob | ||
| import json | ||
|
|
||
| totals = {"collected": 0, "passed": 0, "failed": 0, "skipped": 0} | ||
| for path in glob.glob("/tmp/L2_TEST_RESULTS/*.json"): | ||
| try: | ||
| with open(path, "r", encoding="utf-8") as f: | ||
| data = json.load(f) | ||
| s = data.get("summary", {}) | ||
| totals["collected"] += int(s.get("collected", s.get("total", 0) or 0)) | ||
| totals["passed"] += int(s.get("passed", 0) or 0) | ||
| totals["failed"] += int(s.get("failed", 0) or 0) | ||
| totals["skipped"] += int(s.get("skipped", 0) or 0) | ||
| except Exception: | ||
| continue | ||
|
|
||
| print(f"L2_COLLECTED={totals['collected']}") | ||
| print(f"L2_PASSED={totals['passed']}") | ||
| print(f"L2_FAILED={totals['failed']}") | ||
| print(f"L2_SKIPPED={totals['skipped']}") |
Code Coverage Summary |
PR Coverage Summary
Coverage docs status: updated in PR branch Updated files:
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (5)
.github/workflows/L1_L2_CoverageReport.yml:120
$LD_LIBRARY_PATHis expanded by the runner shell beforedocker execruns, so the container receives the runner's value (often empty) instead of its own environment. Escape the$so expansion happens inside the container.
- name: Build and run L2 tests
run: |
docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib && sh run_l2.sh"
.github/workflows/L1_L2_CoverageReport.yml:134
- With
set -euo pipefail, thegrep | wc -lpipelines will fail when there are 0 matches (grep exits 1), which turns a valid "0 scenarios" case into a hard workflow failure. Wrapgrep/findwith|| truebefore piping intowc.
feature_files=$(find test/functional-tests/features -name '*.feature' | wc -l | tr -d ' ')
feature_scenarios=$(grep -R "^[[:space:]]*Scenario:" test/functional-tests/features/ | wc -l | tr -d ' ')
feature_scenarios_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*Scenario:" test/functional-tests/features/ | wc -l | tr -d ' ' || true)
.github/workflows/L1_L2_CoverageReport.yml:138
- Same
set -euo pipefailissue for the test counts: if there are nodef test_matches,grepexits 1 and the step fails even though the correct count is 0.
test_files=$(find test/functional-tests/tests -maxdepth 1 -name 'test_*.py' | wc -l | tr -d ' ')
test_functions=$(grep -R "^def test_" test/functional-tests/tests/ | wc -l | tr -d ' ')
test_functions_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*def test_" test/functional-tests/tests/ | wc -l | tr -d ' ' || true)
.github/workflows/L1_L2_CoverageReport.yml:140
src_functionsusesgrepwithout-Ebut the pattern relies on ERE grouping ((...)) semantics; in basic regex this changes the meaning and can severely skew the count. Also, this pipeline can fail withpipefailwhen there are 0 matches. Use-E(or adjust the pattern) and guard the pipeline similarly.
src_functions=$(grep -R "^[A-Za-z_][A-Za-z0-9_[:space:]\*]*([^;]*)[[:space:]]*{" rfcMgr/ rfcapi/ tr181api/ utils/ --include='*.cpp' --include='*.c' | wc -l | tr -d ' ')
.github/workflows/L1_L2_CoverageReport.yml:391
- On
pull_requestworkflows, fork PRs do not get a write-capableGITHUB_TOKEN, so this step will fail when trying to create/update a PR comment. Gate the upsert step to same-repo PRs (matching the commit step) or make it non-fatal.
- name: Upsert PR coverage comment
uses: actions/github-script@v7
env:
COMMENT_PATH: /tmp/pr_coverage_comment.md
with:
Code Coverage Summary |
No description provided.