From 2e07fbf0a67902c44fe7b1696d08bba60198b120 Mon Sep 17 00:00:00 2001 From: Hanasi Date: Wed, 20 May 2026 10:35:10 -0400 Subject: [PATCH 01/12] L2 Document update --- .github/skills/bdd-feature-generator/SKILL.md | 892 ++++++++++++++++++ test/functional-tests/L2_Test_Coverage.md | 399 ++++++++ 2 files changed, 1291 insertions(+) create mode 100644 .github/skills/bdd-feature-generator/SKILL.md create mode 100644 test/functional-tests/L2_Test_Coverage.md diff --git a/.github/skills/bdd-feature-generator/SKILL.md b/.github/skills/bdd-feature-generator/SKILL.md new file mode 100644 index 000000000..053053eca --- /dev/null +++ b/.github/skills/bdd-feature-generator/SKILL.md @@ -0,0 +1,892 @@ +--- +name: bdd-feature-generator +description: Generate BDD (Behavior Driven Development) feature files from remote debugger source code analysis. Use for creating Gherkin-format documentation of RBUS event handling, static/dynamic profile processing, command execution, harmful command detection, log archive and upload, deep sleep handling, and uploadRRDLogs C API orchestration. Produces gap analysis between feature files and L2 test implementations. +--- + +# BDD Feature Generator for Remote Debugger + +## Purpose + +Automatically generate BDD feature files in Gherkin format by analyzing the remote debugger source code. This skill creates comprehensive behavioral documentation that can serve as: +- **Functional documentation** of daemon lifecycle, event handling, and profile processing +- **Test specifications** for L2 functional tests (`test/functional-tests/`) +- **Requirements traceability** linking source modules to observable behavior +- **Gap analysis baseline** for comparing L2 tests vs implemented behavior + +## Usage + +Invoke this skill when: +- Documenting existing remote debugger behaviors in BDD format +- Creating test specifications for new features (e.g., new profile types, upload mechanisms) +- Generating feature files for untested code paths (e.g., WebCfg events, deep sleep edge cases) +- Performing gap analysis between L2 tests and source implementation +- Onboarding new team members with behavioral documentation of the daemon + +## Project Context + +The remote debugger (`remotedebugger`) is an RDK component that enables remote collection, packaging, and upload of device diagnostic logs. It: +- Listens for **RBUS/TR-181 events** (IssueType triggers) to initiate debug data collection +- Processes **static profiles** (built-in JSON config at `/etc/rrd/remote_debugger.json`) and **dynamic profiles** (downloaded packages at `/media/apps/RDK-RRD-/etc/rrd/remote_debugger.json`) +- Executes diagnostic **commands** from profiles, with **sanity checks** against harmful commands +- Archives collected outputs as `.tgz` and **uploads** them to a remote server via `uploadRRDLogs.sh` or the **C API** (`rrd_upload_orchestrate`) +- Supports **append mode** combining commands from both static and dynamic profiles +- Handles **deep sleep** events for deferred processing +- Supports **category-only** issue types (all sub-nodes under a category) +- Enforces **single-instance** execution and **RFC enable/disable** control +- Runs as a **systemd service** with a main thread and a dedicated event-processing thread + +## Prerequisites + +Before running this skill: + +1. **Review the build system** — `Makefile.am`, `src/Makefile.am`, and `configure.ac` +2. **Identify compiled modules** — Core sources are always compiled; IARMBUS sources are conditional +3. **Review existing feature files** — Match the format in `test/functional-tests/features/` +4. **Understand test interfaces** — L2 tests use `rbuscli` (RBUS event trigger), log scraping (`/opt/logs/remotedebugger.log.0`), and file system checks (`/tmp/rrd/`) + +## Process + +### Step 1: Analyze Build Configuration + +The remote debugger build is Autotools-based. Identify compiled components from the Makefile chain: + +```bash +# Top-level: identifies src/ as the main SUBDIR +cat Makefile.am | grep "SUBDIRS" +# → SUBDIRS = src + +# Source level: identifies the remotedebugger binary and its sources +cat src/Makefile.am +# → bin_PROGRAMS = remotedebugger +# → remotedebugger_SOURCES = rrdMain.c rrdEventProcess.c rrdJsonParser.c +# rrdRunCmdThread.c rrdCommandSanity.c rrdDynamic.c rrdExecuteScript.c +# rrdMsgPackDecoder.c rrdInterface.c +# → if IARMBUS_ENABLE: +# remotedebugger_SOURCES += rrdIarmEvents.c uploadRRDLogs.c rrd_config.c +# rrd_sysinfo.c rrd_logproc.c rrd_archive.c rrd_upload.c +``` + +**Always compiled modules:** + +| Source File | Header | Purpose | +|---|---|---| +| `rrdMain.c` | `rrdMain.h` | Daemon entry, event thread, RFC enable check, message queue setup | +| `rrdEventProcess.c` | `rrdEventProcess.h` | IssueType/WebCfg/DeepSleep event dispatch, static/dynamic profile flow | +| `rrdJsonParser.c` | `rrdJsonParser.h` | JSON profile parsing, command/timeout extraction, issue node lookup | +| `rrdRunCmdThread.c` | `rrdRunCmdThread.h` | Command execution, output file management, result caching | +| `rrdCommandSanity.c` | `rrdCommandSanity.h` | Command validation against harmful command blocklist | +| `rrdDynamic.c` | `rrdDynamic.h` | Dynamic profile handling, deep sleep event processing | +| `rrdExecuteScript.c` | `rrdExecuteScript.h` | Upload debug output orchestration | +| `rrdMsgPackDecoder.c` | `rrdMsgPackDecoder.h` | WebConfig parameter decoding (MsgPack format) | +| `rrdInterface.c` | `rrdInterface.h` | RBUS registration, event handler setup, profile data elements | + +**Conditionally compiled modules (IARMBUS_ENABLE):** + +| Source File | Header | Purpose | +|---|---|---| +| `rrdIarmEvents.c` | — | IARM bus event handling (power state, deep sleep) | +| `uploadRRDLogs.c` | — | C implementation of upload orchestration entry point | +| `rrd_config.c` | `rrd_config.h` | Configuration loading (server URLs, paths, protocol, RFC/DCM) | +| `rrd_sysinfo.c` | `rrd_sysinfo.h` | System info (MAC address, timestamp, file/dir checks) | +| `rrd_logproc.c` | `rrd_logproc.h` | Log directory validation, preparation, live log handling | +| `rrd_archive.c` | `rrd_archive.h` | Archive creation (.tgz), filename generation, cleanup, CPU checks | +| `rrd_upload.c` | `rrd_upload.h` | Upload orchestration, lock handling, cleanup | + +**Shared headers (no implementation file):** + +| Header | Purpose | +|---|---| +| `rrdCommon.h` | Common constants, macros, data structures (data_buf, msgRRDHdr, etc.) | +| `rrdRbus.h` | RBUS event subscription, blob versioning | +| `rrd_log.h` | Logging subsystem initialization | + +**Exclude from feature generation:** +- `src/unittest/` — Unit tests (L1) +- `test/` — Test infrastructure +- `scripts/` — Runtime scripts (documented as behavior, not source) +- `docs/` — Existing documentation + +### Step 2: Analyze Source Code Structure + +For each compiled module: + +1. **Read the header file** (`.h`) — Identify public functions, data structures, constants +2. **Read the implementation** (`.c`) — Extract event flows, log messages, error paths +3. **Identify RBUS parameters/events** — Map registered data elements in `rrdInterface.c` +4. **Note conditional compilation** — `#ifdef IARMBUS_SUPPORT` gates upload and IARM modules +5. **Note key log messages** — Tests validate behavior by grepping these from log files +6. **Note file paths** — `/etc/rrd/remote_debugger.json`, `/tmp/rrd/`, `/media/apps/RDK-RRD-*` + +**Key elements to extract for each module:** + +| Element | Where to Find | Example | +|---|---|---| +| RBUS data elements | `rrdInterface.c` registration | `Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType` | +| Event handlers | `rrdInterface.c` set handlers | `RRD_SET_ISSUE_EVENT` | +| Profile JSON paths | `rrdEventProcess.c`, `rrdDynamic.c` | `/etc/rrd/remote_debugger.json`, `/media/apps/RDK-RRD-{Node}/etc/rrd/remote_debugger.json` | +| Log messages | All `.c` files `RDK_LOG_*` calls | `"SUCCESS: Message sending Done"`, `"Json File parse Success..."` | +| Error conditions | `.c` return/log on failure | `"FAILED: Json File parse..."`, `"Harmful Command Found"` | +| File outputs | `rrdRunCmdThread.c` | `/tmp/rrd/{IssueType}/debug_outputs.txt` | +| Archive naming | `rrd_archive.c` | `{MAC}_{ISSUETYPE}_{TIMESTAMP}_RRD_DEBUG_LOGS.tgz` | +| Upload API | `rrd_upload.h` | `rrd_upload_orchestrate(upload_dir, issue_type)` | +| Compile guards | `.c` / `.h` `#ifdef` | `IARMBUS_SUPPORT` | + +### Step 3: Create Feature File Structure + +Feature files are placed in `test/functional-tests/features/`. + +**Naming convention for remote debugger:** + +| Behavior Area | Feature File | Description | +|---|---|---| +| Daemon startup | `rrd_start_subscribe_and_wait.feature` | RBUS subscription, event loop entry | +| RFC enable/disable | `rrd_start_control.feature` | RFC-controlled start/stop | +| Single instance | `rrd_single_instance.feature` | Prevents duplicate daemon instances | +| Static profile processing | `rrd_static_profile_report.feature` | End-to-end static profile flow | +| Static category report | `rrd_static_profile_category_report.feature` | Category-only issue type | +| Static profile with suffix | `test_rrd_static_profile_report_with_suffix.feature` | Suffixed issue type handling | +| Background command | `rrd_background_cmd_static_profile_report.feature` | Background command execution | +| Dynamic profile processing | `rrd_dynamic_profile_report.feature` | Dynamic profile fallback flow | +| Dynamic subcategory | `rrd_dynamic_profile_subcategory_report.feature` | Dynamic profile subcategory | +| Dynamic profile missing | `rrd_dynamic_profile_missing_report.feature` | Missing dynamic profile | +| Append mode | `rrd_append_report.feature` | Static+dynamic profile append | +| Append (static not found) | `rrd_append_dynamic_profile_static_not_found.feature` | Append when static missing | +| Harmful commands | `rrd_harmful_command_static_report.feature` | Sanity check blocks execution | +| Dynamic harmful | `test_rrd_dynamic_profile_harmful_report.feature` | Harmful command in dynamic profile | +| Corrupted profile | `rrd_corrupted_static_profile_report.feature` | Invalid/corrupted JSON handling | +| Static missing command | `rrd_static_profile_missing_command_report.feature` | Missing command in profile | +| Empty issue type | `rrd_empty_issuetype_event.feature` | Empty event value handling | +| Deep sleep | `rrd_deepsleep_static_report.feature` | Deep sleep issue type handling | +| Debug report upload | `rrd_debug_report_upload.feature` | Full upload + download validation | +| C API upload | `rrd_c_api_upload.feature` | `rrd_upload_orchestrate` C API tests | +| Suffix negative case | `test_rrd_static_profile_report_with_suffix_negative_case.feature` | Invalid suffix handling | + +### Step 4: Generate Feature Files + +Use this template for remote debugger feature files: + +```gherkin +########################################################################## +# If not stated otherwise in this file or this component's LICENSE +# file the following copyright and licenses apply: +# +# Copyright [YEAR] RDK Management +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +########################################################################## + +# Source: src/{SourceFile}.c + +Feature: Remote Debugger {Behavior Description} + + Scenario: {Prerequisite Check} + Given the configuration file path is set + When I check if the configuration file exists + Then the configuration file should exist + + Scenario: Verify remote debugger process is running + Given the remote debugger process is not running + When I start the remote debugger process + Then the remote debugger process should be running + + Scenario: {Main Behavior Scenario} + Given the remote debugger is running + When I trigger the event "Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType" + Then {expected behavior validated via log scraping} +``` + +### Step 5: Map Source Code to Scenarios + +**For each remote debugger behavior, create scenarios covering:** + +1. **Prerequisites** — Config file exists, output directory exists, daemon not already running +2. **Daemon startup** — Process starts, RBUS subscription succeeds, event loop enters wait +3. **Event trigger** — RBUS IssueType set via `rbuscli`, message queue send/receive +4. **Profile parsing** — JSON read, parse success/failure, issue node lookup +5. **Command execution** — Sanity check, command run, output file creation +6. **Service management** — systemd service start/stop, journalctl collection +7. **Upload flow** — Archive creation, upload script invocation, success/failure +8. **Error/edge cases** — Harmful commands, empty events, corrupted profiles, missing profiles + +**Example mapping — RBUS event trigger (rrdInterface.c → rrdEventProcess.c):** + +```c +// Source: src/rrdInterface.c — RBUS set handler for IssueType +// Sends message to event thread via message queue +// Source: src/rrdEventProcess.c — Event thread receives and dispatches +``` + +**Generated scenarios:** + +```gherkin +Scenario: Send WebPA event for IssueType and verify message flow + Given the remote debugger is running + When I trigger the event "Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType" + Then the event for RRD_SET_ISSUE_EVENT should be received + And the logs should contain "SUCCESS: Message sending Done" + And the logs should be seen with "SUCCESS: Message Reception Done" +``` + +**Example mapping — Static profile processing (rrdEventProcess.c → rrdJsonParser.c):** + +```gherkin +Scenario: Process static profile for issue type + When the remotedebugger received the message from webPA event + Then remotedebugger should read the Json file + And remotedebugger logs should contain the Json File Parse success + And the issue data node and sub-node should be found in the JSON file + And the directory should be created to store the executed output + And Sanity check to validate the commands should be executed + And Command output should be added to the output file +``` + +**Example mapping — Dynamic profile fallback (rrdDynamic.c):** + +```gherkin +Scenario: Verify the Issuetype in dynamic path + Given the remote debugger issuetype is missing in static profile + When the remotedebugger read the json file form the dynamic path + Then remotedebugger json read and parse should be success + And remotedebugger should read the Issuetype from dynamic profile + And the issue data node and sub-node should be found in the JSON file +``` + +**Example mapping — Harmful command detection (rrdCommandSanity.c):** + +```gherkin +Scenario: Check for harmful commands and abort + Given remote debugger parse the static json profile successfully + When the issue command and the sanity commands are matched + Then the remote debugger should exit the processing of commands + And Abort the command execution and skip report upload +``` + +**Example mapping — Upload orchestration C API (rrd_upload.c):** + +```gherkin +Scenario: Validate rrd_upload_orchestrate C API with valid parameters + Given the remote debugger is configured + And test log files are created in the upload directory + When I call rrd_upload_orchestrate with valid upload directory and issue type + Then the C API should return success code 0 + And logs should contain "Configuration loaded" + And logs should contain "MAC:" for MAC address + And logs should contain "Log directory validated and prepared" + And logs should contain "Issue type sanitized" + And logs should contain "Archive filename:" with the generated filename + And logs should contain "Creating" for tarfile creation + And logs should contain "Invoking uploadSTBLogs binary to upload" +``` + +**Example mapping — Single instance enforcement (rrdMain.c):** + +```gherkin +Scenario: Remote debugger exits if another instance is invoked + Given the RemoteDebugger is not already running + When the RemoteDebugger binary is invoked + Then the RemoteDebugger should be started + And when the RemoteDebugger is attempted to be started again + Then the RemoteDebugger should not start another instance +``` + +### Step 6: Document Issue Type Variations + +For issue types with multiple forms, use individual scenarios: + +```gherkin +Scenario: Static profile with Node.SubNode issue type + Given the remote debugger is running + When I trigger IssueType "Device.Info" + Then the issue data node "Device" and sub-node "Info" should be found in the JSON file + +Scenario: Category-only issue type (all sub-nodes) + Given the remote debugger is running + When I trigger IssueType "Device" + Then all sub-nodes under category "Device" should be processed + +Scenario: Suffixed issue type + Given the remote debugger is running + When I trigger IssueType "Device.Info_ab1bghjh" + Then the base issue "Device.Info" should be matched in the profile + And the archive filename should include the full suffixed issue type +``` + +### Step 7: Create README Index + +Create or update `test/functional-tests/features/README.md`: + +```markdown +# Remote Debugger Feature Documentation + +This folder contains BDD feature files documenting the remote debugger +daemon behavior implemented in `src/`. + +## Feature Files Overview + +| Feature File | Source Components | Description | +|---|---|---| +| `rrd_start_subscribe_and_wait.feature` | `rrdMain.c`, `rrdInterface.c` | RBUS subscription, event loop | +| `rrd_start_control.feature` | `rrdMain.c` | RFC enable/disable control | +| `rrd_single_instance.feature` | `rrdMain.c` | Single instance enforcement | +| `rrd_static_profile_report.feature` | `rrdEventProcess.c`, `rrdJsonParser.c`, `rrdRunCmdThread.c` | Static profile end-to-end | +| `rrd_static_profile_category_report.feature` | `rrdEventProcess.c`, `rrdJsonParser.c` | Category-only issue type | +| `rrd_dynamic_profile_report.feature` | `rrdDynamic.c`, `rrdJsonParser.c` | Dynamic profile fallback | +| `rrd_dynamic_profile_subcategory_report.feature` | `rrdDynamic.c` | Dynamic subcategory | +| `rrd_append_report.feature` | `rrdDynamic.c`, `rrdEventProcess.c` | Append mode (static+dynamic) | +| `rrd_harmful_command_static_report.feature` | `rrdCommandSanity.c` | Harmful command blocking | +| `rrd_corrupted_static_profile_report.feature` | `rrdJsonParser.c` | Corrupted JSON handling | +| `rrd_empty_issuetype_event.feature` | `rrdEventProcess.c` | Empty event value | +| `rrd_deepsleep_static_report.feature` | `rrdDynamic.c`, `rrdIarmEvents.c` | Deep sleep handling | +| `rrd_debug_report_upload.feature` | `rrdExecuteScript.c`, `uploadRRDLogs.sh` | Upload + download validation | +| `rrd_c_api_upload.feature` | `rrd_upload.c`, `rrd_config.c`, `rrd_archive.c` | C API upload orchestration | + +## Source Module Mapping + +Based on `src/Makefile.am` and `configure.ac`: + +### Always Compiled +- `rrdMain.c` — Daemon lifecycle, event thread, RFC enable check +- `rrdEventProcess.c` — IssueType/WebCfg event dispatch +- `rrdJsonParser.c` — JSON profile parsing, command extraction +- `rrdRunCmdThread.c` — Command execution, output management +- `rrdCommandSanity.c` — Harmful command validation +- `rrdDynamic.c` — Dynamic profile handling, deep sleep +- `rrdExecuteScript.c` — Upload debug output orchestration +- `rrdMsgPackDecoder.c` — WebConfig MsgPack decoding +- `rrdInterface.c` — RBUS registration, event handlers + +### Conditionally Compiled (IARMBUS_ENABLE) +- `rrdIarmEvents.c` — IARM bus events (power state) +- `uploadRRDLogs.c` — C upload orchestration entry point +- `rrd_config.c` — Configuration loading (RFC/DCM/fallback) +- `rrd_sysinfo.c` — System info (MAC, timestamp) +- `rrd_logproc.c` — Log directory validation/preparation +- `rrd_archive.c` — Archive creation (.tgz) +- `rrd_upload.c` — Upload orchestration, lock handling + +## RBUS Data Elements + +| Data Element | Type | Handler | +|---|---|---| +| `Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType` | SET event | Triggers debug data collection | +| `Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.WebCfgData` | SET event | WebConfig-based trigger | +| `Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.setProfileData` | SET | Profile data injection | +| `Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.getProfileData` | GET | Profile data retrieval | + +## Test Interface Summary + +| Interface | Tool | Log File | Used For | +|---|---|---|---| +| RBUS event | `rbuscli set` | `/opt/logs/remotedebugger.log.0` | IssueType trigger | +| Log scraping | Python `grep_rrdlogs()` | `/opt/logs/remotedebugger.log.0` | Behavior validation | +| File system | Python `os.path`, `subprocess` | `/tmp/rrd/`, `/etc/rrd/` | Config/output checks | +| Upload | Mock xconf server | — | Archive upload validation | +| Process control | `pidof`, `kill`, `nohup` | — | Daemon start/stop | + +## Generation Date + +Generated: {DATE} +``` + +## Scenario Patterns for Remote Debugger + +### Prerequisite Check Pattern + +```gherkin +Scenario: Check if remote debugger configuration file exists + Given the configuration file path is set + When I check if the configuration file exists + Then the configuration file should exist + +Scenario: Check if /tmp/rrd output directory exists + Given the /tmp/rrd directory path is set + When I check if the /tmp/rrd directory exists + Then the /tmp/rrd directory should exist +``` + +### Daemon Startup Pattern + +```gherkin +Scenario: Verify remote debugger process is running + Given the remote debugger process is not running + When I start the remote debugger process + Then the remote debugger process should be running + +Scenario: Remote debugger should subscribe to events + Given the remote debugger binary is invoked + When the remote debugger is started + Then the remote debugger should subscribe to rbus and wait for the events + And the log file should contain "SUCCESS: RBUS Event Subscribe for RRD done!" + And the log file should contain "Waiting for TR69/RBUS Events..." +``` + +### RFC Enable/Disable Pattern + +```gherkin +Scenario: Remote Debugger Starts when Enabled + Given RFC Value for RDKRemoteDebugger.Enable is enabled + When the remotedebugger is started check the value of the RFC parameter + And the RDKRemoteDebugger Enable value is true + Then the remotedebugger should be started and running as daemon + +Scenario: Remote Debugger Stops when Disabled + Given RFC Value for RDKRemoteDebugger.Enable is disabled + When the remotedebugger is started check the value of the RFC parameter + And the RDKRemoteDebugger Enable value is false + Then the remotedebugger must be stopped and process should not be running +``` + +### RBUS Event Trigger Pattern + +```gherkin +Scenario: Send WebPA event for IssueType + Given the remote debugger is running + When I trigger the event "Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType" + Then the event for RRD_SET_ISSUE_EVENT should be received + And the logs should contain "SUCCESS: Message sending Done" + And the logs should be seen with "SUCCESS: Message Reception Done" +``` + +### Static Profile Processing Pattern + +```gherkin +Scenario: Process static profile commands + When the remotedebugger received the message from webPA event + Then remotedebugger should read the Json file + And remotedebugger logs should contain the Json File Parse success + And the issue data node and sub-node should be found in the JSON file + And the directory should be created to store the executed output + And Sanity check to validate the commands should be executed + And Command output should be added to the output file + And the issuetype systemd service should start successfully + And the journalctl service should start successfully + And the process should sleep with timeout + And the issuetype systemd service should stop successfully + And the remotedebugger should call script to upload the debug report +``` + +### Dynamic Profile Fallback Pattern + +```gherkin +Scenario: Verify the Issuetype is not found in static profile + Given the remote debugger received the message from RBUS command + When the remotedebugger static json profile is present + Then remotedebugger should read the Json file + And remotedebugger logs should contain the Json File Parse Success + And remotedebugger should log as the Issue requested is not found in the profile + +Scenario: Verify the Issuetype in dynamic path + Given the remote debugger issuetype is missing in static profile + When the remotedebugger read the json file from the dynamic path + Then remotedebugger json read and parse should be success + And remotedebugger should read the Issuetype from dynamic profile + And the issue data node and sub-node should be found in the JSON file +``` + +### Append Mode Pattern + +```gherkin +Scenario: Verify append of static and dynamic profile commands + Given the remote debugger received the message from RBUS command + When the remotedebugger read the json file from the dynamic path + Then remotedebugger json read and parse should be success in dynamic path + And remotedebugger should read the Json file + And remotedebugger logs should contain the Json File Parse Success + And remotedebugger should log as the Issue requested found in the profile + And Update the command after appending data from both profiles, then execute +``` + +### Harmful Command Detection Pattern + +```gherkin +Scenario: Check for harmful commands and abort + Given remote debugger parse the static json profile successfully + When the issue node and subnode are present in the profile + Then the remote debugger should read the Sanity Check list from profile + And the remotedebugger should perform sanity check on issue commands + Given the remote debugger profile has the harmful commands + When the issue command and the sanity commands are matched + Then the remote debugger should exit the processing of commands + And Abort the command execution and skip report upload +``` + +### Upload Flow Pattern + +```gherkin +Scenario: Upload remote debugger debug report + Given the remote debugger completed the command execution + When remotedebugger calls the uploadRRD.sh script + Then check for the tarfile is created in the output directory + And the file is uploaded to the mockxconf server + And the upload success logs are seen in the logs + +Scenario: Download the file from the mockxconf server + Given the remote debugger report upload success + When curl command is used to download the file + Then the curl command should return success + And the file should be downloaded successfully +``` + +### C API Upload Orchestration Pattern + +```gherkin +Scenario: Validate rrd_upload_orchestrate with valid parameters + Given the remote debugger is configured + And test log files are created in the upload directory + When I call rrd_upload_orchestrate with valid upload directory and issue type + Then the C API should return success code 0 + And logs should contain "Configuration loaded" + And logs should contain "MAC:" for MAC address + And logs should contain "Log directory validated and prepared" + And logs should contain "Issue type sanitized" + And logs should contain "Archive filename:" with the generated filename + And logs should contain "Invoking uploadSTBLogs binary to upload" + +Scenario: Test rrd_upload_orchestrate with NULL parameters + Given the remote debugger is configured + When I call rrd_upload_orchestrate with NULL upload directory + Then the C API should return error code 1 + And error logs should contain "Invalid parameters" + +Scenario: Test rrd_upload_orchestrate with empty directory + Given the remote debugger is configured + And the upload directory is empty + When I call rrd_upload_orchestrate with the empty directory + Then the C API should return error code 6 + And error logs should contain "Invalid or empty upload directory" +``` + +### Error/Edge Case Pattern + +```gherkin +Scenario: Corrupted JSON profile + When the remotedebugger received the message from webPA event + Then remotedebugger should read the Json file + And remotedebugger logs should contain the Json File Parse Failed + +Scenario: Empty issue type event + Given the remote debugger is running + When I trigger the event with empty IssueType value + Then the remotedebugger receives the message from webPA event + And remotedebugger should log as not processing empty event + +Scenario: Missing issue type in static profile + When the remotedebugger received the message + Then remotedebugger should log as the Issue requested is not found in the profile +``` + +### Single Instance Pattern + +```gherkin +Scenario: Remote debugger exits if another instance is invoked + Given the RemoteDebugger is not already running + When the RemoteDebugger binary is invoked + Then the RemoteDebugger should be started + And when the RemoteDebugger is attempted to be started again + Then the RemoteDebugger should not start another instance +``` + +## Quality Checklist + +Before completing feature generation for remote debugger: + +- [ ] All source modules analyzed (`src/Makefile.am` sources list) +- [ ] Conditional modules noted with build flags (`IARMBUS_ENABLE`) +- [ ] Each RBUS event handler has at least one trigger scenario +- [ ] Static profile happy path has end-to-end scenario (trigger → parse → execute → upload) +- [ ] Dynamic profile fallback flow documented +- [ ] Append mode (static+dynamic) documented +- [ ] Harmful command sanity check scenarios included +- [ ] Corrupted/missing profile error scenarios documented +- [ ] Empty/invalid issue type edge cases documented +- [ ] Deep sleep event handling documented +- [ ] C API upload orchestration scenarios included (valid + error cases) +- [ ] RFC enable/disable control documented +- [ ] Single instance enforcement documented +- [ ] Suffixed issue type handling documented +- [ ] License headers included (Apache 2.0, RDK Management) +- [ ] Source file references included as comments +- [ ] Log message assertions match actual daemon log output +- [ ] Feature-to-test file mapping documented for gap analysis +- [ ] Scenarios are atomic (one behavior per scenario) +- [ ] Given/When/Then/And/Or structure followed consistently + +## Output Structure + +``` +test/functional-tests/ +├── features/ +│ ├── README.md # Index, module mapping, gap summary +│ ├── rrd_start_subscribe_and_wait.feature # RBUS subscription, event loop +│ ├── rrd_start_control.feature # RFC enable/disable +│ ├── rrd_single_instance.feature # Single instance enforcement +│ ├── rrd_static_profile_report.feature # Static profile end-to-end +│ ├── rrd_static_profile_category_report.feature # Category-only issue type +│ ├── test_rrd_static_profile_report_with_suffix.feature # Suffixed issue type +│ ├── test_rrd_static_profile_report_with_suffix_negative_case.feature # Invalid suffix +│ ├── rrd_background_cmd_static_profile_report.feature # Background command execution +│ ├── rrd_dynamic_profile_report.feature # Dynamic profile fallback +│ ├── rrd_dynamic_profile_subcategory_report.feature # Dynamic subcategory +│ ├── rrd_dynamic_profile_missing_report.feature # Missing dynamic profile +│ ├── rrd_append_report.feature # Append mode (static+dynamic) +│ ├── rrd_append_dynamic_profile_static_not_found.feature # Append when static missing +│ ├── rrd_harmful_command_static_report.feature # Harmful command blocking +│ ├── test_rrd_dynamic_profile_harmful_report.feature # Dynamic harmful commands +│ ├── rrd_corrupted_static_profile_report.feature # Invalid/corrupted JSON +│ ├── rrd_static_profile_missing_command_report.feature # Missing command in profile +│ ├── rrd_empty_issuetype_event.feature # Empty event value +│ ├── rrd_deepsleep_static_report.feature # Deep sleep handling +│ ├── rrd_debug_report_upload.feature # Upload + download validation +│ └── rrd_c_api_upload.feature # C API upload orchestration +└── tests/ + ├── helper_functions.py # Log grep, process control, constants + ├── test_rrd_static_profile_report.py # Static profile test + ├── test_rrd_dynamic_profile_report.py # Dynamic profile test + ├── test_rrd_c_api_upload.py # C API upload test + ├── test_rrd_single_instance.py # Single instance test + ├── ... # (see full listing below) + └── uploadSTBLogs.sh # Mock upload script +``` + +## Example: Complete Remote Debugger Feature File + +```gherkin +########################################################################## +# If not stated otherwise in this file or this component's LICENSE +# file the following copyright and licenses apply: +# +# Copyright 2026 RDK Management +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +########################################################################## + +# Source: src/rrdEventProcess.c +# Source: src/rrdJsonParser.c +# Source: src/rrdRunCmdThread.c +# Source: src/rrdCommandSanity.c + +Feature: Remote Debugger Static Report + + Scenario: Check if remote debugger configuration file exists + Given the configuration file path is set + When I check if the configuration file exists + Then the configuration file should exist + + Scenario: Check if /tmp/rrd output directory exists + Given the /tmp/rrd directory path is set + When I check if the /tmp/rrd directory exists + Then the /tmp/rrd directory should exist + + Scenario: Verify remote debugger process is running + Given the remote debugger process is not running + When I start the remote debugger process + Then the remote debugger process should be running + + Scenario: Send WebPA event for IssueType and verify end-to-end processing + Given the remote debugger is running + When I trigger the event "Device.DeviceInfo.X_RDKCENTRAL-COM_RFC.Feature.RDKRemoteDebugger.IssueType" + Then the event for RRD_SET_ISSUE_EVENT should be received + And the logs should contain "SUCCESS: Message sending Done" + And the logs should be seen with "SUCCESS: Message Reception Done" + When the remotedebugger received the message from webPA event + Then remotedebugger should read the Json file + And remotedebugger logs should contain the Json File Parse success + And the issue data node and sub-node should be found in the JSON file + And the directory should be created to store the executed output + And Sanity check to validate the commands should be executed + And Command output should be added to the output file + And the issuetype systemd service should start successfully + And the journalctl service should start successfully + And the process should sleep with timeout + And the issuetype systemd service should stop successfully + And the remotedebugger should call script to upload the debug report + + Scenario: Upload remote debugger debug report + When I check the upload status in the logs + Then the upload should be successful if upload is success + Or the upload should fail if upload fails +``` + +## Integration with Gap Analysis + +After generating feature files, use them for gap analysis against the L2 test suite: + +### Step 1: Map Features to Existing Tests + +``` +features/rrd_start_subscribe_and_wait.feature ↔ tests/test_rrd_start_subscribe_and_wait.py +features/rrd_start_control.feature ↔ tests/test_rrd_start_control.py +features/rrd_single_instance.feature ↔ tests/test_rrd_single_instance.py +features/rrd_static_profile_report.feature ↔ tests/test_rrd_static_profile_report.py +features/rrd_static_profile_category_report.feature ↔ tests/test_rrd_static_profile_category_report.py +features/rrd_background_cmd_static_profile_report.feature ↔ tests/test_rrd_background_cmd_static_profile_report.py +features/rrd_dynamic_profile_report.feature ↔ tests/test_rrd_dynamic_profile_report.py +features/rrd_dynamic_profile_subcategory_report.feature ↔ tests/test_rrd_dynamic_subcategory_report.py +features/rrd_dynamic_profile_missing_report.feature ↔ tests/test_rrd_dynamic_profile_missing_report.py +features/rrd_append_report.feature ↔ tests/test_rrd_append_report.py +features/rrd_append_dynamic_profile_static_not_found.feature ↔ tests/test_rrd_append_dynamic_profile_static_notfound.py +features/rrd_harmful_command_static_report.feature ↔ tests/test_rrd_harmful_command_static_report.py +features/test_rrd_dynamic_profile_harmful_report.feature ↔ tests/test_rrd_dynamic_profile_harmful_report.py +features/rrd_corrupted_static_profile_report.feature ↔ tests/test_rrd_corrupted_static_profile_report.py +features/rrd_static_profile_missing_command_report.feature ↔ tests/test_rrd_static_profile_missing_command_report.py +features/rrd_empty_issuetype_event.feature ↔ tests/test_rrd_empty_issuetype_event.py +features/rrd_deepsleep_static_report.feature ↔ tests/test_rrd_deepsleep_static_report.py +features/rrd_debug_report_upload.feature ↔ tests/test_rrd_debug_report_upload.py +features/rrd_c_api_upload.feature ↔ tests/test_rrd_c_api_upload.py +features/test_rrd_static_profile_report_with_suffix.feature ↔ tests/test_rrd_static_profile_report_with_suffix.py +features/test_rrd_static_profile_report_with_suffix_negative_case.feature ↔ tests/test_rrd_static_profile_report_with_suffix_negative_case.py +``` + +### Step 2: Count Coverage + +For each feature file: +1. Count total scenarios (= total testable behaviors) +2. Count scenarios that have a matching `test_*` function in `test/functional-tests/tests/` +3. Calculate coverage = matched / total + +### Step 3: Identify Missing Tests + +Features without test coverage fall into categories: + +| Category | Example | Required Infrastructure | +|---|---|---| +| WebCfg event handling | WebCfgData RBUS event trigger | WebConfig mock, MsgPack payload | +| Profile data SET/GET | setProfileData / getProfileData | `rbuscli` SET/GET validation | +| Upload lock contention | Concurrent upload attempts | Lock file manipulation | +| Deep sleep edge cases | Deep sleep during active collection | IARM event simulation | +| Archive CPU throttle | CPU usage too high during archive | CPU load simulation | +| Configuration fallback | RFC → DCM → dcm.properties chain | Config file manipulation | + +### Step 4: Identify Undocumented Tests + +Tests that exist in `test/functional-tests/tests/` but have no matching scenario in the +`test/functional-tests/features/` files. These should be documented retroactively: + +- `test_rrd_profile_data.py` — Profile data SET/GET (no matching `.feature`) + +### Step 5: Generate Gap Report + +Include a summary table in `test/functional-tests/features/README.md`: + +```markdown +| Behavior Area | Feature Scenarios | L2 Tests | Coverage | Top Gaps | +|---|:---:|:---:|:---:|---| +| Daemon startup/subscribe | 3 | 3 | 100% | — | +| RFC enable/disable | 2 | 2 | 100% | — | +| Single instance | 1 | 1 | 100% | — | +| Static profile report | 5 | 5 | 100% | — | +| Static category report | 5 | 5 | 100% | — | +| Dynamic profile report | 5 | 5 | 100% | — | +| Append mode | 4 | 4 | 100% | — | +| Harmful commands | 4 | 4 | 100% | — | +| Corrupted profile | 4 | 4 | 100% | — | +| Empty issuetype | 2 | 2 | 100% | — | +| Deep sleep | 5 | 5 | 100% | — | +| Debug report upload | 7 | 7 | 100% | — | +| C API upload | 8+ | 8+ | ~100% | Error path coverage | +| WebCfg event | 0 | 0 | 0% | Entire flow | +| Profile data SET/GET | 0 | 1 | partial | No feature file | +| Upload lock contention | 0 | 0 | 0% | Concurrency tests | +``` + +## Current L2 Test Layout + +``` +test/functional-tests/ +├── features/ # BDD feature files (documentation + test specs) +│ ├── rrd_start_subscribe_and_wait.feature # RBUS subscription +│ ├── rrd_start_control.feature # RFC enable/disable +│ ├── rrd_single_instance.feature # Single instance +│ ├── rrd_static_profile_report.feature # Static profile end-to-end +│ ├── rrd_static_profile_category_report.feature +│ ├── test_rrd_static_profile_report_with_suffix.feature +│ ├── test_rrd_static_profile_report_with_suffix_negative_case.feature +│ ├── rrd_background_cmd_static_profile_report.feature +│ ├── rrd_dynamic_profile_report.feature +│ ├── rrd_dynamic_profile_subcategory_report.feature +│ ├── rrd_dynamic_profile_missing_report.feature +│ ├── rrd_append_report.feature +│ ├── rrd_append_dynamic_profile_static_not_found.feature +│ ├── rrd_harmful_command_static_report.feature +│ ├── test_rrd_dynamic_profile_harmful_report.feature +│ ├── rrd_corrupted_static_profile_report.feature +│ ├── rrd_static_profile_missing_command_report.feature +│ ├── rrd_empty_issuetype_event.feature +│ ├── rrd_deepsleep_static_report.feature +│ ├── rrd_debug_report_upload.feature +│ └── rrd_c_api_upload.feature +├── tests/ # Runnable pytest functions +│ ├── helper_functions.py # Log grep, process control, file checks, constants +│ ├── test_rrd_start_subscribe_and_wait.py +│ ├── test_rrd_start_control.py +│ ├── test_rrd_single_instance.py +│ ├── test_rrd_static_profile_report.py +│ ├── test_rrd_static_profile_category_report.py +│ ├── test_rrd_static_profile_report_with_suffix.py +│ ├── test_rrd_static_profile_report_with_suffix_negative_case.py +│ ├── test_rrd_background_cmd_static_profile_report.py +│ ├── test_rrd_dynamic_profile_report.py +│ ├── test_rrd_dynamic_subcategory_report.py +│ ├── test_rrd_dynamic_profile_missing_report.py +│ ├── test_rrd_append_report.py +│ ├── test_rrd_append_dynamic_profile_static_notfound.py +│ ├── test_rrd_harmful_command_static_report.py +│ ├── test_rrd_dynamic_profile_harmful_report.py +│ ├── test_rrd_corrupted_static_profile_report.py +│ ├── test_rrd_static_profile_missing_command_report.py +│ ├── test_rrd_empty_issuetype_event.py +│ ├── test_rrd_deepsleep_static_report.py +│ ├── test_rrd_debug_report_upload.py +│ ├── test_rrd_c_api_upload.py +│ ├── test_rrd_profile_data.py # ⚠ No matching .feature file +│ ├── create_json.sh # JSON profile creation helper +│ ├── deepsleep_main.c # Deep sleep simulation binary +│ ├── power_controller.h # Power controller mock header +│ ├── Makefile # Test build file +│ └── uploadSTBLogs.sh # Mock upload script +``` + +**Test runner:** `pytest`, executed sequentially per test file. +**Interfaces exercised:** `rbuscli` (RBUS event trigger), log scraping (`/opt/logs/remotedebugger.log.0`), file system checks (`/tmp/rrd/`, `/etc/rrd/`), mock xconf server (upload validation). + +## Maintenance + +When remote debugger source code changes: + +1. **New RBUS event/parameter added** — Add scenario to appropriate `.feature` file; update RBUS data elements table in README +2. **New profile processing mode** — Create a new `.feature` file; add to README index +3. **New upload mechanism** — Document the upload flow; add C API test scenarios if applicable +4. **Handler removed** — Remove corresponding scenario; note in gap analysis +5. **Build flag changed** — Update conditional compilation notes in README +6. **L2 test added** — Update gap analysis coverage numbers; create `.feature` file if missing +7. **New log messages added** — Update log assertion strings in relevant feature scenarios +8. **New sanity check rules** — Add harmful command detection scenarios +9. **Version tag** — Include generation date in README + +## Related Skills + +- `technical-documentation-writer` — For detailed architecture and API docs (`docs/`) +- `memory-safety-analyzer` — For safety analysis of C source code +- `thread-safety-analyzer` — For concurrency analysis of event thread and message queue +- `quality-checker` — For running static analysis and build verification +- `triage-logs` — For correlating device logs with remote debugger source code diff --git a/test/functional-tests/L2_Test_Coverage.md b/test/functional-tests/L2_Test_Coverage.md new file mode 100644 index 000000000..fcd872f15 --- /dev/null +++ b/test/functional-tests/L2_Test_Coverage.md @@ -0,0 +1,399 @@ +# Remote Debugger L2 Test Coverage Report + +**Generated:** 2026-05-19 +**Component:** `remotedebugger` (src/) +**Test Suite:** `test/functional-tests/` + +--- + +## Executive Summary + +| Metric | Count | +|---|:---:| +| Feature files | 21 | +| Feature scenarios | 90 | +| Test files (pytest) | 22 | +| Test functions (`test_*`) | 112 | +| Feature→Test mapped pairs | 21 / 21 (+ 1 orphan test) | +| Source modules (always compiled) | 9 | +| Source modules (conditional IARMBUS) | 7 | +| **Overall feature coverage** | **95%** (21/22 test files have matching features) | +| **Source behavior coverage** | **~65%** (happy paths covered; error/edge paths mostly untested) | + +--- + +## 1. Feature File ↔ Test File Mapping + +### Fully Mapped (feature + test exist) + +| # | Feature File | Scenarios | Test File | Test Functions | Status | +|:---:|---|:---:|---|:---:|:---:| +| 1 | `rrd_start_subscribe_and_wait.feature` | 1 | `test_rrd_start_subscribe_and_wait.py` | 4 | PASS | +| 2 | `rrd_start_control.feature` | 2 | `test_rrd_start_control.py` | 1 | PASS | +| 3 | `rrd_single_instance.feature` | 1 | `test_rrd_single_instance.py` | 3 | PASS | +| 4 | `rrd_static_profile_report.feature` | 5 | `test_rrd_static_profile_report.py` | 5 | PASS | +| 5 | `rrd_static_profile_category_report.feature` | 5 | `test_rrd_static_profile_category_report.py` | 5 | PASS | +| 6 | `test_rrd_static_profile_report_with_suffix.feature` | 4 | `test_rrd_static_profile_report_with_suffix.py` | 5 | PASS | +| 7 | `test_rrd_static_profile_report_with_suffix_negative_case.feature` | 4 | `test_rrd_static_profile_report_with_suffix_negative_case.py` | 5 | PASS | +| 8 | `rrd_background_cmd_static_profile_report.feature` | 5 | `test_rrd_background_cmd_static_profile_report.py` | 5 | PASS | +| 9 | `rrd_dynamic_profile_report.feature` | 5 | `test_rrd_dynamic_profile_report.py` | 9 | PASS | +| 10 | `rrd_dynamic_profile_subcategory_report.feature` | 5 | `test_rrd_dynamic_subcategory_report.py` | 7 | PASS | +| 11 | `rrd_dynamic_profile_missing_report.feature` | 4 | `test_rrd_dynamic_profile_missing_report.py` | 7 | PASS | +| 12 | `rrd_append_report.feature` | 4 | `test_rrd_append_report.py` | 7 | PASS | +| 13 | `rrd_append_dynamic_profile_static_not_found.feature` | 4 | `test_rrd_append_dynamic_profile_static_notfound.py` | 7 | PASS | +| 14 | `rrd_harmful_command_static_report.feature` | 5 | `test_rrd_harmful_command_static_report.py` | 5 | PASS | +| 15 | `test_rrd_dynamic_profile_harmful_report.feature` | 5 | `test_rrd_dynamic_profile_harmful_report.py` | 7 | PASS | +| 16 | `rrd_corrupted_static_profile_report.feature` | 4 | `test_rrd_corrupted_static_profile_report.py` | 4 | PASS | +| 17 | `rrd_static_profile_missing_command_report.feature` | 5 | `test_rrd_static_profile_missing_command_report.py` | 5 | PASS | +| 18 | `rrd_empty_issuetype_event.feature` | 2 | `test_rrd_empty_issuetype_event.py` | 2 | PASS | +| 19 | `rrd_deepsleep_static_report.feature` | 2 | `test_rrd_deepsleep_static_report.py` | 5 | PASS | +| 20 | `rrd_debug_report_upload.feature` | 6 | `test_rrd_debug_report_upload.py` | 5 | PASS | +| 21 | `rrd_c_api_upload.feature` | 21 | `test_rrd_c_api_upload.py` | 5 | GAP | +| | **Totals** | **97** | | **112** | | + +> **Note:** Scenario count for `rrd_c_api_upload.feature` (21) far exceeds its test function count (5). See Section 3 for details. + +### Orphan Tests (test exists, no matching feature file) + +| Test File | Test Functions | Description | Gap | +|---|:---:|---|---| +| `test_rrd_profile_data.py` | 3 | RBUS profile data SET/GET via `rbuscli` | **Missing `.feature` file** | + +### Orphan Features (feature exists, no matching test file) + +None — all 21 feature files have corresponding test files. + +--- + +## 2. Per-Behavior Coverage Detail + +### 2.1 Daemon Lifecycle + +| Behavior | Feature | Test | Covered | +|---|---|---|:---:| +| RBUS subscription + event wait | `rrd_start_subscribe_and_wait` | `test_rrd_start_subscribe_and_wait` | YES | +| RFC enable → daemon starts | `rrd_start_control` | `test_rrd_start_control` | YES | +| RFC disable → daemon stops | `rrd_start_control` | `test_rrd_start_control` | YES | +| Single instance enforcement | `rrd_single_instance` | `test_rrd_single_instance` | YES | +| Message queue creation failure | — | — | **NO** | +| Event thread creation failure | — | — | **NO** | +| Signal handling / graceful shutdown | — | — | **NO** | +| Device info file read failure | — | — | **NO** | + +### 2.2 Static Profile Processing + +| Behavior | Feature | Test | Covered | +|---|---|---|:---:| +| Config file exists check | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | +| Output directory exists check | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | +| IssueType event trigger + message flow | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | +| JSON parse success + command execution | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | +| Upload report success/failure | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | +| Category-only issue type (all sub-nodes) | `rrd_static_profile_category_report` | `test_rrd_static_profile_category_report` | YES | +| Suffixed issue type | `test_rrd_static_profile_report_with_suffix` | `test_rrd_static_profile_report_with_suffix` | YES | +| Overlength suffix (negative) | `test_rrd_static_profile_report_with_suffix_negative_case` | `test_rrd_static_profile_report_with_suffix_negative_case` | YES | +| Background command execution | `rrd_background_cmd_static_profile_report` | `test_rrd_background_cmd_static_profile_report` | YES | +| Missing command in profile | `rrd_static_profile_missing_command_report` | `test_rrd_static_profile_missing_command_report` | YES | +| Corrupted/invalid JSON profile | `rrd_corrupted_static_profile_report` | `test_rrd_corrupted_static_profile_report` | YES | + +### 2.3 Dynamic Profile Processing + +| Behavior | Feature | Test | Covered | +|---|---|---|:---:| +| Dynamic profile fallback (static miss) | `rrd_dynamic_profile_report` | `test_rrd_dynamic_profile_report` | YES | +| Dynamic subcategory | `rrd_dynamic_profile_subcategory_report` | `test_rrd_dynamic_subcategory_report` | YES | +| Dynamic profile missing → RDM trigger | `rrd_dynamic_profile_missing_report` | `test_rrd_dynamic_profile_missing_report` | YES | +| Append mode (static + dynamic) | `rrd_append_report` | `test_rrd_append_report` | YES | +| Append when static not found | `rrd_append_dynamic_profile_static_not_found` | `test_rrd_append_dynamic_profile_static_notfound` | YES | +| RDM download event (cache miss) | — | — | **NO** | +| Dynamic profile JSON parse failure | — | — | **NO** | + +### 2.4 Harmful Command Detection + +| Behavior | Feature | Test | Covered | +|---|---|---|:---:| +| Static profile harmful command abort | `rrd_harmful_command_static_report` | `test_rrd_harmful_command_static_report` | YES | +| Dynamic profile harmful command abort | `test_rrd_dynamic_profile_harmful_report` | `test_rrd_dynamic_profile_harmful_report` | YES | +| Macro replacement edge cases | — | — | **NO** | +| Background command modification | — | — | **PARTIAL** (via background cmd test) | + +### 2.5 Event Handling + +| Behavior | Feature | Test | Covered | +|---|---|---|:---:| +| IssueType RBUS event | Multiple features | Multiple tests | YES | +| Empty IssueType event | `rrd_empty_issuetype_event` | `test_rrd_empty_issuetype_event` | YES | +| Deep sleep event | `rrd_deepsleep_static_report` | `test_rrd_deepsleep_static_report` | YES | +| WebCfg event (MsgPack decode) | — | — | **NO** | +| WebCfg corrupted data | — | — | **NO** | +| Multiple simultaneous IssueType events | — | — | **NO** | +| Invalid deep sleep event type | — | — | **NO** | + +### 2.6 Upload & Archive + +| Behavior | Feature | Test | Covered | +|---|---|---|:---:| +| Upload via shell script | `rrd_debug_report_upload` | `test_rrd_debug_report_upload` | YES | +| Upload + download validation | `rrd_debug_report_upload` | `test_rrd_debug_report_upload` | YES | +| C API `rrd_upload_orchestrate` (happy path) | `rrd_c_api_upload` | `test_rrd_c_api_upload` | YES | +| C API NULL parameters | `rrd_c_api_upload` | — | **GAP** | +| C API empty directory | `rrd_c_api_upload` | — | **GAP** | +| C API non-existent directory | `rrd_c_api_upload` | — | **GAP** | +| C API config loading validation | `rrd_c_api_upload` | — | **GAP** | +| C API MAC retrieval | `rrd_c_api_upload` | — | **GAP** | +| C API timestamp generation | `rrd_c_api_upload` | — | **GAP** | +| C API issue type sanitization | `rrd_c_api_upload` | — | **GAP** | +| C API archive creation | `rrd_c_api_upload` | — | **GAP** | +| C API upload execution | `rrd_c_api_upload` | — | **GAP** | +| C API cleanup after success/failure | `rrd_c_api_upload` | — | **GAP** | +| C API concurrent upload lock | `rrd_c_api_upload` | — | **GAP** | +| C API LOGUPLOAD_ENABLE | `rrd_c_api_upload` | — | **GAP** | +| C API end-to-end with RFC trigger | `rrd_c_api_upload` | — | **GAP** | +| C API error propagation | `rrd_c_api_upload` | — | **GAP** | +| Upload lock file contention | — | — | **NO** | +| Archive CPU throttle logic | — | — | **NO** | + +### 2.7 Profile Data SET/GET (RBUS) + +| Behavior | Feature | Test | Covered | +|---|---|---|:---:| +| `setProfileData` / `getProfileData` | — | `test_rrd_profile_data` | **PARTIAL** (no feature) | +| Profile category load error | — | — | **NO** | +| Profile file write error | — | — | **NO** | + +--- + +## 3. Feature ↔ Test Gap Analysis + +### 3.1 `rrd_c_api_upload.feature` — Major Scenario-to-Test Gap + +The feature file documents **21 scenarios** covering the full `rrd_upload_orchestrate` C API, but the test file `test_rrd_c_api_upload.py` only implements **5 test functions** that cover the basic end-to-end flow (config check, dir check, start, trigger event, upload report). + +**Missing test implementations for feature scenarios:** + +| Feature Scenario | Test Status | +|---|---| +| Validate rrd_upload_orchestrate C API with valid parameters | Covered (within `test_remote_debugger_trigger_event`) | +| Test rrd_upload_orchestrate with NULL upload directory | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate with NULL issue type | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate with empty upload directory | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate with non-existent directory | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate configuration loading | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate MAC address retrieval | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate timestamp generation | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate issue type sanitization | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate archive creation | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate upload execution | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate cleanup after success | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate cleanup after upload failure | **NOT IMPLEMENTED** | +| Test uploadDebugoutput wrapper function | **NOT IMPLEMENTED** | +| Test concurrent upload lock handling | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate with LOGUPLOAD_ENABLE issue type | **NOT IMPLEMENTED** | +| Test remote debugger end-to-end with RFC trigger | Covered | +| Test upload report validation with success path | Covered (within `test_remotedebugger_upload_report`) | +| Test upload report validation with failure path | Covered (within `test_remotedebugger_upload_report`) | +| Test upload report with legacy log compatibility | **NOT IMPLEMENTED** | +| Test rrd_upload_orchestrate error propagation | **NOT IMPLEMENTED** | + +**Gap: 16 of 21 scenarios have no dedicated test implementation.** + +### 3.2 `test_rrd_profile_data.py` — Orphan Test (No Feature File) + +This test file contains 3 test functions exercising `rbuscli set/get` on profile data RBUS elements (`setProfileData` / `getProfileData`). No corresponding `.feature` file exists. + +**Recommendation:** Create `rrd_profile_data.feature` to document this behavior. + +### 3.3 Scenario Count vs Test Count Discrepancy + +Some test files implement more test functions than their feature has scenarios, due to setup/teardown and prerequisite tests being split more granularly: + +| Feature File | Scenarios | Test File | Tests | Delta | +|---|:---:|---|:---:|:---:| +| `rrd_start_subscribe_and_wait` | 1 | `test_rrd_start_subscribe_and_wait` | 4 | +3 | +| `rrd_single_instance` | 1 | `test_rrd_single_instance` | 3 | +2 | +| `rrd_dynamic_profile_report` | 5 | `test_rrd_dynamic_profile_report` | 9 | +4 | +| `rrd_dynamic_profile_missing_report` | 4 | `test_rrd_dynamic_profile_missing_report` | 7 | +3 | +| `rrd_append_report` | 4 | `test_rrd_append_report` | 7 | +3 | +| `rrd_append_dynamic_profile_static_not_found` | 4 | `test_rrd_append_dynamic_profile_static_notfound` | 7 | +3 | +| `rrd_dynamic_profile_harmful_report` | 5 | `test_rrd_dynamic_profile_harmful_report` | 7 | +2 | +| `rrd_deepsleep_static_report` | 2 | `test_rrd_deepsleep_static_report` | 5 | +3 | +| `rrd_c_api_upload` | 21 | `test_rrd_c_api_upload` | 5 | **-16** | + +The **only deficit** is `rrd_c_api_upload` where the feature documents far more scenarios than tests implement. + +--- + +## 4. Source Module Coverage Analysis + +### 4.1 Module-Level Coverage Summary + +| Source Module | Happy Path | Error Paths | L2 Coverage | +|---|:---:|:---:|---| +| `rrdMain.c` | YES | NO | Startup, RFC check, event thread — tested. Msgqueue/thread failures — not tested. | +| `rrdInterface.c` | YES | NO | RBUS registration, event handlers — tested. Registration failures, file I/O errors — not tested. | +| `rrdEventProcess.c` | YES | PARTIAL | Static/dynamic/append/deepsleep — tested. WebCfg event, malloc failures — not tested. | +| `rrdJsonParser.c` | YES | PARTIAL | Valid/corrupted/missing JSON — tested. Dir creation failures, malloc failures — not tested. | +| `rrdRunCmdThread.c` | YES | NO | Command execution, output files — tested. File write errors, systemd-run failures — not tested. | +| `rrdCommandSanity.c` | YES | NO | Harmful command detection — tested. Macro replacement errors — not tested. | +| `rrdDynamic.c` | YES | NO | Dynamic profile, deep sleep — tested. RBUS set failure, invalid event type — not tested. | +| `rrdExecuteScript.c` | YES | NO | Upload orchestration — tested. Script exec failure, API failure — not tested. | +| `rrdMsgPackDecoder.c` | NO | NO | **Entirely untested** — WebCfg MsgPack decode not exercised by any L2 test. | +| `rrd_config.c` | INDIRECT | NO | Indirectly tested via daemon startup. Config parse errors, RFC query failures — not tested. | +| `rrd_sysinfo.c` | INDIRECT | NO | Indirectly tested via upload flow. MAC/timestamp retrieval errors — not tested. | +| `rrd_logproc.c` | INDIRECT | NO | Indirectly tested via upload flow. Log dir validation errors — not tested. | +| `rrd_archive.c` | INDIRECT | NO | Indirectly tested via upload flow. CPU throttle, archive errors — not tested. | +| `rrd_upload.c` | INDIRECT | NO | Indirectly tested via upload flow. Lock errors, cleanup failures — not tested. | +| `rrdIarmEvents.c` | PARTIAL | NO | Deep sleep event — tested. Other IARM events — not tested. | +| `uploadRRDLogs.c` | INDIRECT | NO | Entry point tested via daemon trigger. Direct API error paths — not tested. | + +### 4.2 Completely Untested Source Behaviors + +| Priority | Behavior | Source Module | Required Infrastructure | +|:---:|---|---|---| +| **P1** | WebCfg event (MsgPack decode + dispatch) | `rrdMsgPackDecoder.c`, `rrdEventProcess.c` | WebConfig mock, base64-encoded MsgPack payload | +| **P1** | C API upload error paths (NULL params, empty dir, non-existent dir) | `rrd_upload.c` | Direct C API invocation or test binary | +| **P1** | Profile data SET/GET feature documentation | `rrdInterface.c` | Feature file creation only | +| **P2** | Concurrent upload lock contention | `rrd_upload.c` | Parallel upload trigger + lock file manipulation | +| **P2** | Archive CPU usage throttle | `rrd_archive.c` | CPU load simulation | +| **P2** | Configuration fallback chain (RFC → DCM → dcm.properties) | `rrd_config.c` | Config file manipulation | +| **P2** | RDM download event with cache miss | `rrdInterface.c`, `rrdDynamic.c` | RDM mock, empty cache | +| **P3** | RBUS registration/unregistration failures | `rrdInterface.c` | RBUS mock failure injection | +| **P3** | Message queue creation failure | `rrdMain.c` | System resource exhaustion mock | +| **P3** | Event thread creation failure | `rrdMain.c` | Thread creation failure mock | +| **P3** | Directory creation/chdir failures | `rrdJsonParser.c`, `rrdRunCmdThread.c` | Filesystem permission mock | +| **P3** | systemd-run / journalctl execution failures | `rrdRunCmdThread.c` | Binary removal or mock failure | +| **P3** | Output file write errors | `rrdRunCmdThread.c` | Filesystem full or permission mock | +| **P3** | Dynamic profile JSON parse failure | `rrdDynamic.c` | Corrupted dynamic JSON file | +| **P3** | Invalid deep sleep event type | `rrdDynamic.c` | IARM event simulation with bad type | +| **P3** | Memory allocation failures (all modules) | All `.c` files | malloc failure injection (not practical in L2) | + +--- + +## 5. Gap Summary + +### 5.1 Feature vs Test Gap Table + +| Behavior Area | Feature Scenarios | Test Functions | Coverage | Top Gaps | +|---|:---:|:---:|:---:|---| +| Daemon startup/subscribe | 1 | 4 | 100% | — | +| RFC enable/disable | 2 | 1 | 100% | — | +| Single instance | 1 | 3 | 100% | — | +| Static profile report | 5 | 5 | 100% | — | +| Static category report | 5 | 5 | 100% | — | +| Static suffix report | 4 | 5 | 100% | — | +| Static suffix negative | 4 | 5 | 100% | — | +| Background command | 5 | 5 | 100% | — | +| Dynamic profile report | 5 | 9 | 100% | — | +| Dynamic subcategory | 5 | 7 | 100% | — | +| Dynamic missing | 4 | 7 | 100% | — | +| Append mode | 4 | 7 | 100% | — | +| Append (static not found) | 4 | 7 | 100% | — | +| Harmful static | 5 | 5 | 100% | — | +| Harmful dynamic | 5 | 7 | 100% | — | +| Corrupted profile | 4 | 4 | 100% | — | +| Missing command | 5 | 5 | 100% | — | +| Empty issuetype | 2 | 2 | 100% | — | +| Deep sleep | 2 | 5 | 100% | — | +| Debug report upload | 6 | 5 | 100% | — | +| **C API upload** | **21** | **5** | **~24%** | **16 scenarios not implemented** | +| **Profile data SET/GET** | **0** | **3** | **N/A** | **No feature file** | +| **WebCfg event** | **0** | **0** | **0%** | **Entire flow untested** | +| **Upload lock contention** | **0** | **0** | **0%** | **No test** | +| **Config fallback chain** | **0** | **0** | **0%** | **No test** | + +### 5.2 Action Items (Prioritized) + +| # | Priority | Action | Effort | +|:---:|:---:|---|:---:| +| 1 | P1 | Implement 16 missing `test_rrd_c_api_upload.py` test functions matching feature scenarios | High | +| 2 | P1 | Create `rrd_profile_data.feature` for the existing `test_rrd_profile_data.py` test | Low | +| 3 | P1 | Add WebCfg event L2 test (`test_rrd_webcfg_event.py` + `rrd_webcfg_event.feature`) | High | +| 4 | P2 | Add upload lock contention test | Medium | +| 5 | P2 | Add configuration fallback chain test | Medium | +| 6 | P2 | Add archive CPU throttle test | Medium | +| 7 | P2 | Add dynamic profile JSON parse failure test | Low | +| 8 | P3 | Add RBUS registration failure test | Low | +| 9 | P3 | Add systemd-run / journalctl failure test | Low | +| 10 | P3 | Add output file write error test | Low | + +--- + +## 6. Test Infrastructure Notes + +### Test Interfaces + +| Interface | Tool | Used By | +|---|---|---| +| RBUS event trigger | `rbuscli set` | All event-based tests | +| RBUS profile data | `rbuscli set/get` | `test_rrd_profile_data.py` | +| Log scraping | `grep_rrdlogs()` in `helper_functions.py` | All tests | +| File system checks | `os.path.isfile()`, `os.path.isdir()` | Prerequisite tests | +| Process control | `pidof`, `kill -9`, `nohup` | Setup/teardown | +| Mock upload server | Mock xconf server | `test_rrd_debug_report_upload.py` | +| Mock upload script | `uploadSTBLogs.sh` | Upload tests | + +### Test Execution + +- **Runner:** `pytest`, sequential per file +- **No ordering decorators:** Tests rely on file-level sequential execution order +- **Shared state:** Tests within a file depend on prior test side effects (daemon start → trigger → validate) +- **Cleanup:** Most files kill the daemon and remove logs in setup + +--- + +## 7. Appendix: Complete File Inventory + +### Feature Files (21) + +| # | File | Scenarios | +|:---:|---|:---:| +| 1 | `rrd_start_subscribe_and_wait.feature` | 1 | +| 2 | `rrd_start_control.feature` | 2 | +| 3 | `rrd_single_instance.feature` | 1 | +| 4 | `rrd_static_profile_report.feature` | 5 | +| 5 | `rrd_static_profile_category_report.feature` | 5 | +| 6 | `test_rrd_static_profile_report_with_suffix.feature` | 4 | +| 7 | `test_rrd_static_profile_report_with_suffix_negative_case.feature` | 4 | +| 8 | `rrd_background_cmd_static_profile_report.feature` | 5 | +| 9 | `rrd_dynamic_profile_report.feature` | 5 | +| 10 | `rrd_dynamic_profile_subcategory_report.feature` | 5 | +| 11 | `rrd_dynamic_profile_missing_report.feature` | 4 | +| 12 | `rrd_append_report.feature` | 4 | +| 13 | `rrd_append_dynamic_profile_static_not_found.feature` | 4 | +| 14 | `rrd_harmful_command_static_report.feature` | 5 | +| 15 | `test_rrd_dynamic_profile_harmful_report.feature` | 5 | +| 16 | `rrd_corrupted_static_profile_report.feature` | 4 | +| 17 | `rrd_static_profile_missing_command_report.feature` | 5 | +| 18 | `rrd_empty_issuetype_event.feature` | 2 | +| 19 | `rrd_deepsleep_static_report.feature` | 2 | +| 20 | `rrd_debug_report_upload.feature` | 6 | +| 21 | `rrd_c_api_upload.feature` | 21 | +| | **Total** | **97** | + +### Test Files (22) + +| # | File | Test Functions | +|:---:|---|:---:| +| 1 | `test_rrd_start_subscribe_and_wait.py` | 4 | +| 2 | `test_rrd_start_control.py` | 1 | +| 3 | `test_rrd_single_instance.py` | 3 | +| 4 | `test_rrd_static_profile_report.py` | 5 | +| 5 | `test_rrd_static_profile_category_report.py` | 5 | +| 6 | `test_rrd_static_profile_report_with_suffix.py` | 5 | +| 7 | `test_rrd_static_profile_report_with_suffix_negative_case.py` | 5 | +| 8 | `test_rrd_background_cmd_static_profile_report.py` | 5 | +| 9 | `test_rrd_dynamic_profile_report.py` | 9 | +| 10 | `test_rrd_dynamic_subcategory_report.py` | 7 | +| 11 | `test_rrd_dynamic_profile_missing_report.py` | 7 | +| 12 | `test_rrd_append_report.py` | 7 | +| 13 | `test_rrd_append_dynamic_profile_static_notfound.py` | 7 | +| 14 | `test_rrd_harmful_command_static_report.py` | 5 | +| 15 | `test_rrd_dynamic_profile_harmful_report.py` | 7 | +| 16 | `test_rrd_corrupted_static_profile_report.py` | 4 | +| 17 | `test_rrd_static_profile_missing_command_report.py` | 5 | +| 18 | `test_rrd_empty_issuetype_event.py` | 2 | +| 19 | `test_rrd_deepsleep_static_report.py` | 5 | +| 20 | `test_rrd_debug_report_upload.py` | 5 | +| 21 | `test_rrd_c_api_upload.py` | 5 | +| 22 | `test_rrd_profile_data.py` | 3 | +| | **Total** | **112** | From b0218e28003136fd1edaa21c665173f0ddde7319 Mon Sep 17 00:00:00 2001 From: Hanasi Date: Mon, 10 Aug 2026 15:06:12 -0400 Subject: [PATCH 02/12] Test Lcov on L2 --- run_l2_coverage.sh | 159 ++++++++++++++++++ src/rrdMain.c | 16 +- .../tests/helper_functions.py | 2 + 3 files changed, 176 insertions(+), 1 deletion(-) create mode 100644 run_l2_coverage.sh diff --git a/run_l2_coverage.sh b/run_l2_coverage.sh new file mode 100644 index 000000000..db5f1c8f4 --- /dev/null +++ b/run_l2_coverage.sh @@ -0,0 +1,159 @@ +#!/bin/sh +#################################################################################### +# If not stated otherwise in this file or this component's Licenses.txt file the +# following copyright and licenses apply: +# +# Copyright 2024 RDK Management +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +#################################################################################### +# Run L2 integration tests with gcov/lcov coverage instrumentation. +# +# Prerequisites (inside the test container): +# lcov, genhtml, gcc with --coverage support +# +# Usage: +# sh run_l2_coverage.sh +# +# Output: +# /tmp/l2_coverage/html/index.html — browsable HTML coverage report +# /tmp/l2_coverage/coverage.info — lcov tracefile for CI upload + +set -e + +WORKDIR="$(pwd)" +INSTALL_DIR=/usr/local +RESULT_DIR="/tmp/l2_test_report" +COV_DIR="/tmp/l2_coverage" +STATIC_PROFILE_DIR="/etc/rrd" +OUTPUT_DIR="/tmp/rrd" +LIB_DIR="/lib/rdk" + +# Tell helper_functions.py to use SIGTERM so gcov flushes before exit. +export RRD_COVERAGE_MODE=1 + +# ── Directories ────────────────────────────────────────────────────────────── +mkdir -p "$RESULT_DIR" +mkdir -p "$OUTPUT_DIR" +mkdir -p "$STATIC_PROFILE_DIR" +mkdir -p "$LIB_DIR" +mkdir -p "$COV_DIR" +mkdir -p /media/apps/RDK-RRD-Test/etc/rrd + +# ── System fixtures (mirrors run_l2.sh) ────────────────────────────────────── +touch /media/apps/RDK-RRD-Test/etc/rrd/remote_debugger.json +echo "AA:BB:CC:DD:EE:FF" >> /tmp/.estb_mac + +apt-get remove -y systemd || true +apt-get update && apt-get install -y tcpdump lcov + +echo "LOG_PATH=/opt/logs" >> /etc/include.properties +cp remote_debugger.json "$STATIC_PROFILE_DIR/remote_debugger.json" + +cp scripts/uploadRRDLogs.sh "$LIB_DIR/uploadRRDLogs.sh" +chmod 777 "$LIB_DIR/uploadRRDLogs.sh" +sed -i 's/remote-debugger\.log/remotedebugger\.log\.0/g' "$LIB_DIR/uploadRRDLogs.sh" + +cp test/functional-tests/tests/uploadSTBLogs.sh "$LIB_DIR/uploadSTBLogs.sh" +chmod 777 "$LIB_DIR/uploadSTBLogs.sh" + +cp scripts/systemd-run /usr/local/bin/systemd-run +chmod 777 /usr/local/bin/systemd-run +ln -sf /usr/local/bin/systemd-run /usr/bin/systemd-run + +touch /usr/local/bin/systemctl +chmod 777 /usr/local/bin/systemctl +ln -sf /usr/local/bin/systemctl /usr/bin/systemctl + +touch /usr/local/bin/journalctl +chmod 777 /usr/local/bin/journalctl +ln -sf /usr/local/bin/journalctl /usr/bin/journalctl + +rm -rf /tmp/rrd/* +rm -rf /opt/logs/remotedebugger.log* + +# ── Coverage build ──────────────────────────────────────────────────────────── +autoreconf -i +autoupdate +./configure --prefix="${INSTALL_DIR}" --enable-iarmbusSupport=yes + +# Append --coverage to the same CFLAGS/LDFLAGS used in cov_build.sh. +make remotedebugger_CFLAGS="-I/usr/include/cjson -I/usr/local/include/wdmp-c \ + -I/usr/local/include/rbus -I/usr/local/include -I./unittest/mocks \ + -I/usr/local/include/trower-base64 -DIARMBUS_SUPPORT -DUSECOV -DUSE_L2_SUPPORT \ + --coverage" \ + remotedebugger_LDFLAGS="-L/usr/local/lib -lrdkloggers -lcjson -lrfcapi -lrbus \ + -lmsgpackc -lsecure_wrapper -lwebconfig_framework -lIARMBus -ltr181api \ + -L/usr/local/lib/x86_64-linux-gnu -ltrower-base64 -L/usr/lib/x86_64-linux-gnu \ + --coverage" +make install + +# ── lcov baseline (all lines counted as zero-hit) ──────────────────────────── +lcov --zerocounters --directory "$WORKDIR/src" +lcov --capture --initial \ + --directory "$WORKDIR/src" \ + --output-file "$COV_DIR/coverage_base.info" \ + --rc lcov_branch_coverage=1 + +# ── L2 test suite ───────────────────────────────────────────────────────────── +run_test() { + pytest --json-report --json-report-summary \ + --json-report-file "$RESULT_DIR/$1.json" \ + "test/functional-tests/tests/$2" || true +} + +run_test rrd_dynamic_profile_missing_report test_rrd_dynamic_profile_missing_report.py +run_test test_category test_rrd_dynamic_subcategory_report.py +run_test rrd_append test_rrd_append_report.py +run_test rrd_dynamic_profile_harmful_report test_rrd_dynamic_profile_harmful_report.py +cp remote_debugger.json "$STATIC_PROFILE_DIR/" +run_test rrd_dynamic_profile_report test_rrd_dynamic_profile_report.py +run_test rrd_append_dynamic_profile_static_notfound test_rrd_append_dynamic_profile_static_notfound.py +run_test rrd_single_instance test_rrd_single_instance.py +run_test rrd_start_control test_rrd_start_control.py +run_test rrd_start_subscribe_and_wait test_rrd_start_subscribe_and_wait.py +run_test rrd_static_profile_report test_rrd_static_profile_report.py +run_test rrd_static_profile_report_with_suffix test_rrd_static_profile_report_with_suffix.py +run_test rrd_static_profile_report_with_suffix_negative test_rrd_static_profile_report_with_suffix_negative_case.py +run_test rrd_corrupted_static_profile_report test_rrd_corrupted_static_profile_report.py +cp remote_debugger.json "$STATIC_PROFILE_DIR/" +run_test rrd_harmful_static_profile_report test_rrd_harmful_command_static_report.py +run_test rrd_static_profile_category_report test_rrd_static_profile_category_report.py + +# ── Capture post-test coverage ──────────────────────────────────────────────── +lcov --capture \ + --directory "$WORKDIR/src" \ + --output-file "$COV_DIR/coverage_test.info" \ + --rc lcov_branch_coverage=1 + +# ── Merge baseline + test, then strip system headers ───────────────────────── +lcov --add-tracefile "$COV_DIR/coverage_base.info" \ + --add-tracefile "$COV_DIR/coverage_test.info" \ + --output-file "$COV_DIR/coverage_merged.info" \ + --rc lcov_branch_coverage=1 + +lcov --remove "$COV_DIR/coverage_merged.info" \ + '/usr/*' \ + --output-file "$COV_DIR/coverage.info" \ + --rc lcov_branch_coverage=1 + +# ── HTML report ─────────────────────────────────────────────────────────────── +genhtml "$COV_DIR/coverage.info" \ + --output-directory "$COV_DIR/html" \ + --title "Remote Debugger L2 Coverage" \ + --branch-coverage \ + --legend + +echo "" +echo "Coverage report : $COV_DIR/html/index.html" +echo "lcov tracefile : $COV_DIR/coverage.info" diff --git a/src/rrdMain.c b/src/rrdMain.c index 40fdc9bc5..ede604b96 100644 --- a/src/rrdMain.c +++ b/src/rrdMain.c @@ -23,7 +23,18 @@ #include "rrdDynamic.h" #include "rrdEventProcess.h" #include "rrdInterface.h" - +#ifdef USECOV +#include +#include +extern void __gcov_dump(void); +/* Flush gcov counters on SIGTERM so lcov captures L2 exercise data. */ +static void rrd_gcov_sigterm_handler(int sig) +{ + (void)sig; + __gcov_dump(); + _exit(0); +} +#endif devicePropertiesData devPropData; @@ -139,6 +150,9 @@ int main(int argc, char *argv[]) pthread_t RRDTR69ThreadID; rdk_logger_init(DEBUG_INI_FILE); +#ifdef USECOV + signal(SIGTERM, rrd_gcov_sigterm_handler); +#endif #if !defined(GTEST_ENABLE) /* Store Device Info.*/ RRDStoreDeviceInfo(&devPropData); diff --git a/test/functional-tests/tests/helper_functions.py b/test/functional-tests/tests/helper_functions.py index a6157e0f9..d4caa1a07 100644 --- a/test/functional-tests/tests/helper_functions.py +++ b/test/functional-tests/tests/helper_functions.py @@ -55,6 +55,8 @@ def remove_logfile(): def kill_rrd(signal: int=9): + if os.environ.get("RRD_COVERAGE_MODE") and signal == 9: + signal = 15 print(f"Received Signal to kill remotedebugger {signal} with pid {get_pid('remotedebugger')}") resp = subprocess.run(f"kill -{signal} {get_pid('remotedebugger')}", shell=True, capture_output=True) print(resp.stdout.decode('utf-8')) From 635a4082d34b09c1dbefdc9cd4317583d77d499a Mon Sep 17 00:00:00 2001 From: Hanasi Date: Mon, 10 Aug 2026 15:12:57 -0400 Subject: [PATCH 03/12] l2 updated --- cov_build.sh | 2 +- run_l2.sh | 28 +++++++- run_l2_coverage.sh | 159 --------------------------------------------- 3 files changed, 26 insertions(+), 163 deletions(-) delete mode 100644 run_l2_coverage.sh diff --git a/cov_build.sh b/cov_build.sh index 79c930133..c0871e32a 100644 --- a/cov_build.sh +++ b/cov_build.sh @@ -86,5 +86,5 @@ cd $WORKDIR autoreconf -i autoupdate ./configure --prefix=${INSTALL_DIR} --enable-iarmbusSupport=yes -make remotedebugger_CFLAGS="-I/usr/include/cjson -I/usr/local/include/wdmp-c -I/usr/local/include/rbus -I/usr/local/include -I./unittest/mocks -I/usr/local/include/trower-base64 -DIARMBUS_SUPPORT -DUSECOV -DUSE_L2_SUPPORT" remotedebugger_LDFLAGS="-L/usr/local/lib -lrdkloggers -lcjson -lrfcapi -lrbus -lmsgpackc -lsecure_wrapper -lwebconfig_framework -lIARMBus -ltr181api -L/usr/local/lib/x86_64-linux-gnu -ltrower-base64 -L/usr/lib/x86_64-linux-gnu" +make remotedebugger_CFLAGS="-I/usr/include/cjson -I/usr/local/include/wdmp-c -I/usr/local/include/rbus -I/usr/local/include -I./unittest/mocks -I/usr/local/include/trower-base64 -DIARMBUS_SUPPORT -DUSECOV -DUSE_L2_SUPPORT --coverage" remotedebugger_LDFLAGS="-L/usr/local/lib -lrdkloggers -lcjson -lrfcapi -lrbus -lmsgpackc -lsecure_wrapper -lwebconfig_framework -lIARMBus -ltr181api -L/usr/local/lib/x86_64-linux-gnu -ltrower-base64 -L/usr/lib/x86_64-linux-gnu --coverage" make install diff --git a/run_l2.sh b/run_l2.sh index b29d2f956..1c3b2c5b3 100644 --- a/run_l2.sh +++ b/run_l2.sh @@ -22,19 +22,23 @@ RESULT_DIR="/tmp/l2_test_report" STATIC_PROFILE_DIR="/etc/rrd" OUTPUT_DIR="/tmp/rrd" LIB_DIR="/lib/rdk" +COV_DIR="/tmp/l2_coverage" + +export RRD_COVERAGE_MODE=1 mkdir -p "$RESULT_DIR" mkdir -p "$OUTPUT_DIR" mkdir -p "$STATIC_PROFILE_DIR" mkdir -p "$LIB_DIR" +mkdir -p "$COV_DIR" mkdir -p /media/apps/RDK-RRD-Test/etc/rrd touch /media/apps/RDK-RRD-Test/etc/rrd/remote_debugger.json echo "AA:BB:CC:DD:EE:FF" >> /tmp/.estb_mac -apt-get remove systemd -apt-get update && apt-get install -y tcpdump +apt-get remove -y systemd || true +apt-get update && apt-get install -y tcpdump lcov echo "LOG_PATH=/opt/logs" >> /etc/include.properties cp remote_debugger.json /etc/rrd/remote_debugger.json @@ -60,6 +64,11 @@ ln -s /usr/local/bin/journalctl /usr/bin/journalctl rm -rf /tmp/rrd/* rm -rf /opt/logs/remotedebugger.log* +# lcov baseline — capture zero counters before any test runs +lcov --zerocounters --directory "$(pwd)/src" +lcov --capture --initial --directory "$(pwd)/src" \ + --output-file "$COV_DIR/coverage_base.info" --rc lcov_branch_coverage=1 + # Run L2 Test cases pytest --json-report --json-report-summary --json-report-file $RESULT_DIR/rrd_dynamic_profile_missing_report.json test/functional-tests/tests/test_rrd_dynamic_profile_missing_report.py pytest --json-report --json-report-summary --json-report-file $RESULT_DIR/test_category.json test/functional-tests/tests/test_rrd_dynamic_subcategory_report.py @@ -88,5 +97,18 @@ pytest --json-report --json-report-summary --json-report-file $RESULT_DIR/rrd_c_ cp remote_debugger.json /etc/rrd/ pytest --json-report --json-report-summary --json-report-file $RESULT_DIR/rrd_profile_data.json test/functional-tests/tests/test_rrd_profile_data.py - +# Capture, filter, and report coverage +lcov --capture --directory "$(pwd)/src" \ + --output-file "$COV_DIR/coverage_test.info" --rc lcov_branch_coverage=1 +lcov --add-tracefile "$COV_DIR/coverage_base.info" \ + --add-tracefile "$COV_DIR/coverage_test.info" \ + --output-file "$COV_DIR/coverage_merged.info" --rc lcov_branch_coverage=1 +lcov --remove "$COV_DIR/coverage_merged.info" '/usr/*' \ + --output-file "$COV_DIR/coverage.info" --rc lcov_branch_coverage=1 +genhtml "$COV_DIR/coverage.info" \ + --output-directory "$COV_DIR/html" \ + --title "Remote Debugger L2 Coverage" \ + --branch-coverage --legend +echo "Coverage report : $COV_DIR/html/index.html" +echo "lcov tracefile : $COV_DIR/coverage.info" diff --git a/run_l2_coverage.sh b/run_l2_coverage.sh deleted file mode 100644 index db5f1c8f4..000000000 --- a/run_l2_coverage.sh +++ /dev/null @@ -1,159 +0,0 @@ -#!/bin/sh -#################################################################################### -# If not stated otherwise in this file or this component's Licenses.txt file the -# following copyright and licenses apply: -# -# Copyright 2024 RDK Management -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -#################################################################################### -# Run L2 integration tests with gcov/lcov coverage instrumentation. -# -# Prerequisites (inside the test container): -# lcov, genhtml, gcc with --coverage support -# -# Usage: -# sh run_l2_coverage.sh -# -# Output: -# /tmp/l2_coverage/html/index.html — browsable HTML coverage report -# /tmp/l2_coverage/coverage.info — lcov tracefile for CI upload - -set -e - -WORKDIR="$(pwd)" -INSTALL_DIR=/usr/local -RESULT_DIR="/tmp/l2_test_report" -COV_DIR="/tmp/l2_coverage" -STATIC_PROFILE_DIR="/etc/rrd" -OUTPUT_DIR="/tmp/rrd" -LIB_DIR="/lib/rdk" - -# Tell helper_functions.py to use SIGTERM so gcov flushes before exit. -export RRD_COVERAGE_MODE=1 - -# ── Directories ────────────────────────────────────────────────────────────── -mkdir -p "$RESULT_DIR" -mkdir -p "$OUTPUT_DIR" -mkdir -p "$STATIC_PROFILE_DIR" -mkdir -p "$LIB_DIR" -mkdir -p "$COV_DIR" -mkdir -p /media/apps/RDK-RRD-Test/etc/rrd - -# ── System fixtures (mirrors run_l2.sh) ────────────────────────────────────── -touch /media/apps/RDK-RRD-Test/etc/rrd/remote_debugger.json -echo "AA:BB:CC:DD:EE:FF" >> /tmp/.estb_mac - -apt-get remove -y systemd || true -apt-get update && apt-get install -y tcpdump lcov - -echo "LOG_PATH=/opt/logs" >> /etc/include.properties -cp remote_debugger.json "$STATIC_PROFILE_DIR/remote_debugger.json" - -cp scripts/uploadRRDLogs.sh "$LIB_DIR/uploadRRDLogs.sh" -chmod 777 "$LIB_DIR/uploadRRDLogs.sh" -sed -i 's/remote-debugger\.log/remotedebugger\.log\.0/g' "$LIB_DIR/uploadRRDLogs.sh" - -cp test/functional-tests/tests/uploadSTBLogs.sh "$LIB_DIR/uploadSTBLogs.sh" -chmod 777 "$LIB_DIR/uploadSTBLogs.sh" - -cp scripts/systemd-run /usr/local/bin/systemd-run -chmod 777 /usr/local/bin/systemd-run -ln -sf /usr/local/bin/systemd-run /usr/bin/systemd-run - -touch /usr/local/bin/systemctl -chmod 777 /usr/local/bin/systemctl -ln -sf /usr/local/bin/systemctl /usr/bin/systemctl - -touch /usr/local/bin/journalctl -chmod 777 /usr/local/bin/journalctl -ln -sf /usr/local/bin/journalctl /usr/bin/journalctl - -rm -rf /tmp/rrd/* -rm -rf /opt/logs/remotedebugger.log* - -# ── Coverage build ──────────────────────────────────────────────────────────── -autoreconf -i -autoupdate -./configure --prefix="${INSTALL_DIR}" --enable-iarmbusSupport=yes - -# Append --coverage to the same CFLAGS/LDFLAGS used in cov_build.sh. -make remotedebugger_CFLAGS="-I/usr/include/cjson -I/usr/local/include/wdmp-c \ - -I/usr/local/include/rbus -I/usr/local/include -I./unittest/mocks \ - -I/usr/local/include/trower-base64 -DIARMBUS_SUPPORT -DUSECOV -DUSE_L2_SUPPORT \ - --coverage" \ - remotedebugger_LDFLAGS="-L/usr/local/lib -lrdkloggers -lcjson -lrfcapi -lrbus \ - -lmsgpackc -lsecure_wrapper -lwebconfig_framework -lIARMBus -ltr181api \ - -L/usr/local/lib/x86_64-linux-gnu -ltrower-base64 -L/usr/lib/x86_64-linux-gnu \ - --coverage" -make install - -# ── lcov baseline (all lines counted as zero-hit) ──────────────────────────── -lcov --zerocounters --directory "$WORKDIR/src" -lcov --capture --initial \ - --directory "$WORKDIR/src" \ - --output-file "$COV_DIR/coverage_base.info" \ - --rc lcov_branch_coverage=1 - -# ── L2 test suite ───────────────────────────────────────────────────────────── -run_test() { - pytest --json-report --json-report-summary \ - --json-report-file "$RESULT_DIR/$1.json" \ - "test/functional-tests/tests/$2" || true -} - -run_test rrd_dynamic_profile_missing_report test_rrd_dynamic_profile_missing_report.py -run_test test_category test_rrd_dynamic_subcategory_report.py -run_test rrd_append test_rrd_append_report.py -run_test rrd_dynamic_profile_harmful_report test_rrd_dynamic_profile_harmful_report.py -cp remote_debugger.json "$STATIC_PROFILE_DIR/" -run_test rrd_dynamic_profile_report test_rrd_dynamic_profile_report.py -run_test rrd_append_dynamic_profile_static_notfound test_rrd_append_dynamic_profile_static_notfound.py -run_test rrd_single_instance test_rrd_single_instance.py -run_test rrd_start_control test_rrd_start_control.py -run_test rrd_start_subscribe_and_wait test_rrd_start_subscribe_and_wait.py -run_test rrd_static_profile_report test_rrd_static_profile_report.py -run_test rrd_static_profile_report_with_suffix test_rrd_static_profile_report_with_suffix.py -run_test rrd_static_profile_report_with_suffix_negative test_rrd_static_profile_report_with_suffix_negative_case.py -run_test rrd_corrupted_static_profile_report test_rrd_corrupted_static_profile_report.py -cp remote_debugger.json "$STATIC_PROFILE_DIR/" -run_test rrd_harmful_static_profile_report test_rrd_harmful_command_static_report.py -run_test rrd_static_profile_category_report test_rrd_static_profile_category_report.py - -# ── Capture post-test coverage ──────────────────────────────────────────────── -lcov --capture \ - --directory "$WORKDIR/src" \ - --output-file "$COV_DIR/coverage_test.info" \ - --rc lcov_branch_coverage=1 - -# ── Merge baseline + test, then strip system headers ───────────────────────── -lcov --add-tracefile "$COV_DIR/coverage_base.info" \ - --add-tracefile "$COV_DIR/coverage_test.info" \ - --output-file "$COV_DIR/coverage_merged.info" \ - --rc lcov_branch_coverage=1 - -lcov --remove "$COV_DIR/coverage_merged.info" \ - '/usr/*' \ - --output-file "$COV_DIR/coverage.info" \ - --rc lcov_branch_coverage=1 - -# ── HTML report ─────────────────────────────────────────────────────────────── -genhtml "$COV_DIR/coverage.info" \ - --output-directory "$COV_DIR/html" \ - --title "Remote Debugger L2 Coverage" \ - --branch-coverage \ - --legend - -echo "" -echo "Coverage report : $COV_DIR/html/index.html" -echo "lcov tracefile : $COV_DIR/coverage.info" From 31df83c3944d8affc57741f5f27a40836503dead Mon Sep 17 00:00:00 2001 From: Hanasi Date: Mon, 10 Aug 2026 15:33:48 -0400 Subject: [PATCH 04/12] update2 --- .github/workflows/L2-tests.yml | 49 +++++++++++++++++++++++ test/functional-tests/L2_Test_Coverage.md | 5 +++ 2 files changed, 54 insertions(+) diff --git a/.github/workflows/L2-tests.yml b/.github/workflows/L2-tests.yml index a06430a85..b232531e2 100644 --- a/.github/workflows/L2-tests.yml +++ b/.github/workflows/L2-tests.yml @@ -3,6 +3,11 @@ name: L2 Integration Tests on: pull_request: branches: [ develop ] + push: + branches: [ develop ] + +permissions: + contents: write env: AUTOMATICS_UNAME: ${{ secrets.AUTOMATICS_UNAME }} @@ -88,3 +93,47 @@ jobs: run: | docker cp /tmp/L2_TEST_RESULTS ci-container:/tmp/L2_TEST_RESULTS docker exec -i ci-container bash -c "echo 'Contents in workspace directory' && ls -l && echo '===============================' && echo 'Contents in /tmp/L2_TEST_RESULTS' && ls -l /tmp/L2_TEST_RESULTS && echo '===============================' && git config --global --add safe.directory /mnt/L2_CONTAINER_SHARED_VOLUME/remote_debugger && gtest-json-result-push.py /tmp/L2_TEST_RESULTS https://rdkeorchestrationservice.apps.cloud.comcast.net/rdke_orchestration_api/push_unit_test_results /mnt/L2_CONTAINER_SHARED_VOLUME/remote_debugger" + + - name: Copy lcov tracefile from container + if: github.event_name == 'push' + run: docker cp native-platform:/tmp/l2_coverage/coverage.info /tmp/coverage.info + + - name: Update L2_Test_Coverage.md with lcov metrics + if: github.event_name == 'push' + run: | + sudo apt-get install -y --quiet lcov + lcov --summary /tmp/coverage.info --rc lcov_branch_coverage=1 2>&1 | tee /tmp/lcov_summary.txt + python3 - <<'EOF' + import re, datetime + summary = open('/tmp/lcov_summary.txt').read() + def parse(pattern, text): + m = re.search(pattern, text) + return (m.group(1) + '%', f"({m.group(2)} of {m.group(3)})") if m else ('N/A', '') + lines_pct, lines_detail = parse(r'lines\.+:\s+([\d.]+)%\s+\((\d+) of (\d+)', summary) + branch_pct, branch_detail = parse(r'branches\.+:\s+([\d.]+)%\s+\((\d+) of (\d+)', summary) + funcs_pct, funcs_detail = parse(r'functions\.+:\s+([\d.]+)%\s+\((\d+) of (\d+)', summary) + today = datetime.date.today().strftime('%Y-%m-%d') + new_block = ( + '\n' + f'| **Line coverage (lcov)** | **{lines_pct}** {lines_detail} |\n' + f'| **Branch coverage (lcov)** | **{branch_pct}** {branch_detail} |\n' + f'| **Function coverage (lcov)** | **{funcs_pct}** {funcs_detail} |\n' + '' + ) + path = 'remote_debugger/test/functional-tests/L2_Test_Coverage.md' + content = open(path).read() + content = re.sub(r'.*?', new_block, content, flags=re.DOTALL) + content = re.sub(r'\*\*Generated:\*\* \S+', f'**Generated:** {today}', content) + open(path, 'w').write(content) + print(f'lines={lines_pct} branches={branch_pct} functions={funcs_pct}') + EOF + + - name: Commit updated L2_Test_Coverage.md + if: github.event_name == 'push' + run: | + cd remote_debugger + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + git add test/functional-tests/L2_Test_Coverage.md + git diff --cached --quiet || git commit -m "ci: update L2 lcov coverage metrics [skip ci]" + git push diff --git a/test/functional-tests/L2_Test_Coverage.md b/test/functional-tests/L2_Test_Coverage.md index fcd872f15..5bec351df 100644 --- a/test/functional-tests/L2_Test_Coverage.md +++ b/test/functional-tests/L2_Test_Coverage.md @@ -19,6 +19,11 @@ | Source modules (conditional IARMBUS) | 7 | | **Overall feature coverage** | **95%** (21/22 test files have matching features) | | **Source behavior coverage** | **~65%** (happy paths covered; error/edge paths mostly untested) | + +| **Line coverage (lcov)** | pending | +| **Branch coverage (lcov)** | pending | +| **Function coverage (lcov)** | pending | + --- From 4c75381c370163d49201137e268595c7d507b705 Mon Sep 17 00:00:00 2001 From: nhanasi Date: Mon, 10 Aug 2026 15:36:20 -0400 Subject: [PATCH 05/12] Delete test/functional-tests/L2_Test_Coverage.md --- test/functional-tests/L2_Test_Coverage.md | 404 ---------------------- 1 file changed, 404 deletions(-) delete mode 100644 test/functional-tests/L2_Test_Coverage.md diff --git a/test/functional-tests/L2_Test_Coverage.md b/test/functional-tests/L2_Test_Coverage.md deleted file mode 100644 index 5bec351df..000000000 --- a/test/functional-tests/L2_Test_Coverage.md +++ /dev/null @@ -1,404 +0,0 @@ -# Remote Debugger L2 Test Coverage Report - -**Generated:** 2026-05-19 -**Component:** `remotedebugger` (src/) -**Test Suite:** `test/functional-tests/` - ---- - -## Executive Summary - -| Metric | Count | -|---|:---:| -| Feature files | 21 | -| Feature scenarios | 90 | -| Test files (pytest) | 22 | -| Test functions (`test_*`) | 112 | -| Feature→Test mapped pairs | 21 / 21 (+ 1 orphan test) | -| Source modules (always compiled) | 9 | -| Source modules (conditional IARMBUS) | 7 | -| **Overall feature coverage** | **95%** (21/22 test files have matching features) | -| **Source behavior coverage** | **~65%** (happy paths covered; error/edge paths mostly untested) | - -| **Line coverage (lcov)** | pending | -| **Branch coverage (lcov)** | pending | -| **Function coverage (lcov)** | pending | - - ---- - -## 1. Feature File ↔ Test File Mapping - -### Fully Mapped (feature + test exist) - -| # | Feature File | Scenarios | Test File | Test Functions | Status | -|:---:|---|:---:|---|:---:|:---:| -| 1 | `rrd_start_subscribe_and_wait.feature` | 1 | `test_rrd_start_subscribe_and_wait.py` | 4 | PASS | -| 2 | `rrd_start_control.feature` | 2 | `test_rrd_start_control.py` | 1 | PASS | -| 3 | `rrd_single_instance.feature` | 1 | `test_rrd_single_instance.py` | 3 | PASS | -| 4 | `rrd_static_profile_report.feature` | 5 | `test_rrd_static_profile_report.py` | 5 | PASS | -| 5 | `rrd_static_profile_category_report.feature` | 5 | `test_rrd_static_profile_category_report.py` | 5 | PASS | -| 6 | `test_rrd_static_profile_report_with_suffix.feature` | 4 | `test_rrd_static_profile_report_with_suffix.py` | 5 | PASS | -| 7 | `test_rrd_static_profile_report_with_suffix_negative_case.feature` | 4 | `test_rrd_static_profile_report_with_suffix_negative_case.py` | 5 | PASS | -| 8 | `rrd_background_cmd_static_profile_report.feature` | 5 | `test_rrd_background_cmd_static_profile_report.py` | 5 | PASS | -| 9 | `rrd_dynamic_profile_report.feature` | 5 | `test_rrd_dynamic_profile_report.py` | 9 | PASS | -| 10 | `rrd_dynamic_profile_subcategory_report.feature` | 5 | `test_rrd_dynamic_subcategory_report.py` | 7 | PASS | -| 11 | `rrd_dynamic_profile_missing_report.feature` | 4 | `test_rrd_dynamic_profile_missing_report.py` | 7 | PASS | -| 12 | `rrd_append_report.feature` | 4 | `test_rrd_append_report.py` | 7 | PASS | -| 13 | `rrd_append_dynamic_profile_static_not_found.feature` | 4 | `test_rrd_append_dynamic_profile_static_notfound.py` | 7 | PASS | -| 14 | `rrd_harmful_command_static_report.feature` | 5 | `test_rrd_harmful_command_static_report.py` | 5 | PASS | -| 15 | `test_rrd_dynamic_profile_harmful_report.feature` | 5 | `test_rrd_dynamic_profile_harmful_report.py` | 7 | PASS | -| 16 | `rrd_corrupted_static_profile_report.feature` | 4 | `test_rrd_corrupted_static_profile_report.py` | 4 | PASS | -| 17 | `rrd_static_profile_missing_command_report.feature` | 5 | `test_rrd_static_profile_missing_command_report.py` | 5 | PASS | -| 18 | `rrd_empty_issuetype_event.feature` | 2 | `test_rrd_empty_issuetype_event.py` | 2 | PASS | -| 19 | `rrd_deepsleep_static_report.feature` | 2 | `test_rrd_deepsleep_static_report.py` | 5 | PASS | -| 20 | `rrd_debug_report_upload.feature` | 6 | `test_rrd_debug_report_upload.py` | 5 | PASS | -| 21 | `rrd_c_api_upload.feature` | 21 | `test_rrd_c_api_upload.py` | 5 | GAP | -| | **Totals** | **97** | | **112** | | - -> **Note:** Scenario count for `rrd_c_api_upload.feature` (21) far exceeds its test function count (5). See Section 3 for details. - -### Orphan Tests (test exists, no matching feature file) - -| Test File | Test Functions | Description | Gap | -|---|:---:|---|---| -| `test_rrd_profile_data.py` | 3 | RBUS profile data SET/GET via `rbuscli` | **Missing `.feature` file** | - -### Orphan Features (feature exists, no matching test file) - -None — all 21 feature files have corresponding test files. - ---- - -## 2. Per-Behavior Coverage Detail - -### 2.1 Daemon Lifecycle - -| Behavior | Feature | Test | Covered | -|---|---|---|:---:| -| RBUS subscription + event wait | `rrd_start_subscribe_and_wait` | `test_rrd_start_subscribe_and_wait` | YES | -| RFC enable → daemon starts | `rrd_start_control` | `test_rrd_start_control` | YES | -| RFC disable → daemon stops | `rrd_start_control` | `test_rrd_start_control` | YES | -| Single instance enforcement | `rrd_single_instance` | `test_rrd_single_instance` | YES | -| Message queue creation failure | — | — | **NO** | -| Event thread creation failure | — | — | **NO** | -| Signal handling / graceful shutdown | — | — | **NO** | -| Device info file read failure | — | — | **NO** | - -### 2.2 Static Profile Processing - -| Behavior | Feature | Test | Covered | -|---|---|---|:---:| -| Config file exists check | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | -| Output directory exists check | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | -| IssueType event trigger + message flow | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | -| JSON parse success + command execution | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | -| Upload report success/failure | `rrd_static_profile_report` | `test_rrd_static_profile_report` | YES | -| Category-only issue type (all sub-nodes) | `rrd_static_profile_category_report` | `test_rrd_static_profile_category_report` | YES | -| Suffixed issue type | `test_rrd_static_profile_report_with_suffix` | `test_rrd_static_profile_report_with_suffix` | YES | -| Overlength suffix (negative) | `test_rrd_static_profile_report_with_suffix_negative_case` | `test_rrd_static_profile_report_with_suffix_negative_case` | YES | -| Background command execution | `rrd_background_cmd_static_profile_report` | `test_rrd_background_cmd_static_profile_report` | YES | -| Missing command in profile | `rrd_static_profile_missing_command_report` | `test_rrd_static_profile_missing_command_report` | YES | -| Corrupted/invalid JSON profile | `rrd_corrupted_static_profile_report` | `test_rrd_corrupted_static_profile_report` | YES | - -### 2.3 Dynamic Profile Processing - -| Behavior | Feature | Test | Covered | -|---|---|---|:---:| -| Dynamic profile fallback (static miss) | `rrd_dynamic_profile_report` | `test_rrd_dynamic_profile_report` | YES | -| Dynamic subcategory | `rrd_dynamic_profile_subcategory_report` | `test_rrd_dynamic_subcategory_report` | YES | -| Dynamic profile missing → RDM trigger | `rrd_dynamic_profile_missing_report` | `test_rrd_dynamic_profile_missing_report` | YES | -| Append mode (static + dynamic) | `rrd_append_report` | `test_rrd_append_report` | YES | -| Append when static not found | `rrd_append_dynamic_profile_static_not_found` | `test_rrd_append_dynamic_profile_static_notfound` | YES | -| RDM download event (cache miss) | — | — | **NO** | -| Dynamic profile JSON parse failure | — | — | **NO** | - -### 2.4 Harmful Command Detection - -| Behavior | Feature | Test | Covered | -|---|---|---|:---:| -| Static profile harmful command abort | `rrd_harmful_command_static_report` | `test_rrd_harmful_command_static_report` | YES | -| Dynamic profile harmful command abort | `test_rrd_dynamic_profile_harmful_report` | `test_rrd_dynamic_profile_harmful_report` | YES | -| Macro replacement edge cases | — | — | **NO** | -| Background command modification | — | — | **PARTIAL** (via background cmd test) | - -### 2.5 Event Handling - -| Behavior | Feature | Test | Covered | -|---|---|---|:---:| -| IssueType RBUS event | Multiple features | Multiple tests | YES | -| Empty IssueType event | `rrd_empty_issuetype_event` | `test_rrd_empty_issuetype_event` | YES | -| Deep sleep event | `rrd_deepsleep_static_report` | `test_rrd_deepsleep_static_report` | YES | -| WebCfg event (MsgPack decode) | — | — | **NO** | -| WebCfg corrupted data | — | — | **NO** | -| Multiple simultaneous IssueType events | — | — | **NO** | -| Invalid deep sleep event type | — | — | **NO** | - -### 2.6 Upload & Archive - -| Behavior | Feature | Test | Covered | -|---|---|---|:---:| -| Upload via shell script | `rrd_debug_report_upload` | `test_rrd_debug_report_upload` | YES | -| Upload + download validation | `rrd_debug_report_upload` | `test_rrd_debug_report_upload` | YES | -| C API `rrd_upload_orchestrate` (happy path) | `rrd_c_api_upload` | `test_rrd_c_api_upload` | YES | -| C API NULL parameters | `rrd_c_api_upload` | — | **GAP** | -| C API empty directory | `rrd_c_api_upload` | — | **GAP** | -| C API non-existent directory | `rrd_c_api_upload` | — | **GAP** | -| C API config loading validation | `rrd_c_api_upload` | — | **GAP** | -| C API MAC retrieval | `rrd_c_api_upload` | — | **GAP** | -| C API timestamp generation | `rrd_c_api_upload` | — | **GAP** | -| C API issue type sanitization | `rrd_c_api_upload` | — | **GAP** | -| C API archive creation | `rrd_c_api_upload` | — | **GAP** | -| C API upload execution | `rrd_c_api_upload` | — | **GAP** | -| C API cleanup after success/failure | `rrd_c_api_upload` | — | **GAP** | -| C API concurrent upload lock | `rrd_c_api_upload` | — | **GAP** | -| C API LOGUPLOAD_ENABLE | `rrd_c_api_upload` | — | **GAP** | -| C API end-to-end with RFC trigger | `rrd_c_api_upload` | — | **GAP** | -| C API error propagation | `rrd_c_api_upload` | — | **GAP** | -| Upload lock file contention | — | — | **NO** | -| Archive CPU throttle logic | — | — | **NO** | - -### 2.7 Profile Data SET/GET (RBUS) - -| Behavior | Feature | Test | Covered | -|---|---|---|:---:| -| `setProfileData` / `getProfileData` | — | `test_rrd_profile_data` | **PARTIAL** (no feature) | -| Profile category load error | — | — | **NO** | -| Profile file write error | — | — | **NO** | - ---- - -## 3. Feature ↔ Test Gap Analysis - -### 3.1 `rrd_c_api_upload.feature` — Major Scenario-to-Test Gap - -The feature file documents **21 scenarios** covering the full `rrd_upload_orchestrate` C API, but the test file `test_rrd_c_api_upload.py` only implements **5 test functions** that cover the basic end-to-end flow (config check, dir check, start, trigger event, upload report). - -**Missing test implementations for feature scenarios:** - -| Feature Scenario | Test Status | -|---|---| -| Validate rrd_upload_orchestrate C API with valid parameters | Covered (within `test_remote_debugger_trigger_event`) | -| Test rrd_upload_orchestrate with NULL upload directory | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate with NULL issue type | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate with empty upload directory | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate with non-existent directory | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate configuration loading | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate MAC address retrieval | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate timestamp generation | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate issue type sanitization | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate archive creation | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate upload execution | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate cleanup after success | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate cleanup after upload failure | **NOT IMPLEMENTED** | -| Test uploadDebugoutput wrapper function | **NOT IMPLEMENTED** | -| Test concurrent upload lock handling | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate with LOGUPLOAD_ENABLE issue type | **NOT IMPLEMENTED** | -| Test remote debugger end-to-end with RFC trigger | Covered | -| Test upload report validation with success path | Covered (within `test_remotedebugger_upload_report`) | -| Test upload report validation with failure path | Covered (within `test_remotedebugger_upload_report`) | -| Test upload report with legacy log compatibility | **NOT IMPLEMENTED** | -| Test rrd_upload_orchestrate error propagation | **NOT IMPLEMENTED** | - -**Gap: 16 of 21 scenarios have no dedicated test implementation.** - -### 3.2 `test_rrd_profile_data.py` — Orphan Test (No Feature File) - -This test file contains 3 test functions exercising `rbuscli set/get` on profile data RBUS elements (`setProfileData` / `getProfileData`). No corresponding `.feature` file exists. - -**Recommendation:** Create `rrd_profile_data.feature` to document this behavior. - -### 3.3 Scenario Count vs Test Count Discrepancy - -Some test files implement more test functions than their feature has scenarios, due to setup/teardown and prerequisite tests being split more granularly: - -| Feature File | Scenarios | Test File | Tests | Delta | -|---|:---:|---|:---:|:---:| -| `rrd_start_subscribe_and_wait` | 1 | `test_rrd_start_subscribe_and_wait` | 4 | +3 | -| `rrd_single_instance` | 1 | `test_rrd_single_instance` | 3 | +2 | -| `rrd_dynamic_profile_report` | 5 | `test_rrd_dynamic_profile_report` | 9 | +4 | -| `rrd_dynamic_profile_missing_report` | 4 | `test_rrd_dynamic_profile_missing_report` | 7 | +3 | -| `rrd_append_report` | 4 | `test_rrd_append_report` | 7 | +3 | -| `rrd_append_dynamic_profile_static_not_found` | 4 | `test_rrd_append_dynamic_profile_static_notfound` | 7 | +3 | -| `rrd_dynamic_profile_harmful_report` | 5 | `test_rrd_dynamic_profile_harmful_report` | 7 | +2 | -| `rrd_deepsleep_static_report` | 2 | `test_rrd_deepsleep_static_report` | 5 | +3 | -| `rrd_c_api_upload` | 21 | `test_rrd_c_api_upload` | 5 | **-16** | - -The **only deficit** is `rrd_c_api_upload` where the feature documents far more scenarios than tests implement. - ---- - -## 4. Source Module Coverage Analysis - -### 4.1 Module-Level Coverage Summary - -| Source Module | Happy Path | Error Paths | L2 Coverage | -|---|:---:|:---:|---| -| `rrdMain.c` | YES | NO | Startup, RFC check, event thread — tested. Msgqueue/thread failures — not tested. | -| `rrdInterface.c` | YES | NO | RBUS registration, event handlers — tested. Registration failures, file I/O errors — not tested. | -| `rrdEventProcess.c` | YES | PARTIAL | Static/dynamic/append/deepsleep — tested. WebCfg event, malloc failures — not tested. | -| `rrdJsonParser.c` | YES | PARTIAL | Valid/corrupted/missing JSON — tested. Dir creation failures, malloc failures — not tested. | -| `rrdRunCmdThread.c` | YES | NO | Command execution, output files — tested. File write errors, systemd-run failures — not tested. | -| `rrdCommandSanity.c` | YES | NO | Harmful command detection — tested. Macro replacement errors — not tested. | -| `rrdDynamic.c` | YES | NO | Dynamic profile, deep sleep — tested. RBUS set failure, invalid event type — not tested. | -| `rrdExecuteScript.c` | YES | NO | Upload orchestration — tested. Script exec failure, API failure — not tested. | -| `rrdMsgPackDecoder.c` | NO | NO | **Entirely untested** — WebCfg MsgPack decode not exercised by any L2 test. | -| `rrd_config.c` | INDIRECT | NO | Indirectly tested via daemon startup. Config parse errors, RFC query failures — not tested. | -| `rrd_sysinfo.c` | INDIRECT | NO | Indirectly tested via upload flow. MAC/timestamp retrieval errors — not tested. | -| `rrd_logproc.c` | INDIRECT | NO | Indirectly tested via upload flow. Log dir validation errors — not tested. | -| `rrd_archive.c` | INDIRECT | NO | Indirectly tested via upload flow. CPU throttle, archive errors — not tested. | -| `rrd_upload.c` | INDIRECT | NO | Indirectly tested via upload flow. Lock errors, cleanup failures — not tested. | -| `rrdIarmEvents.c` | PARTIAL | NO | Deep sleep event — tested. Other IARM events — not tested. | -| `uploadRRDLogs.c` | INDIRECT | NO | Entry point tested via daemon trigger. Direct API error paths — not tested. | - -### 4.2 Completely Untested Source Behaviors - -| Priority | Behavior | Source Module | Required Infrastructure | -|:---:|---|---|---| -| **P1** | WebCfg event (MsgPack decode + dispatch) | `rrdMsgPackDecoder.c`, `rrdEventProcess.c` | WebConfig mock, base64-encoded MsgPack payload | -| **P1** | C API upload error paths (NULL params, empty dir, non-existent dir) | `rrd_upload.c` | Direct C API invocation or test binary | -| **P1** | Profile data SET/GET feature documentation | `rrdInterface.c` | Feature file creation only | -| **P2** | Concurrent upload lock contention | `rrd_upload.c` | Parallel upload trigger + lock file manipulation | -| **P2** | Archive CPU usage throttle | `rrd_archive.c` | CPU load simulation | -| **P2** | Configuration fallback chain (RFC → DCM → dcm.properties) | `rrd_config.c` | Config file manipulation | -| **P2** | RDM download event with cache miss | `rrdInterface.c`, `rrdDynamic.c` | RDM mock, empty cache | -| **P3** | RBUS registration/unregistration failures | `rrdInterface.c` | RBUS mock failure injection | -| **P3** | Message queue creation failure | `rrdMain.c` | System resource exhaustion mock | -| **P3** | Event thread creation failure | `rrdMain.c` | Thread creation failure mock | -| **P3** | Directory creation/chdir failures | `rrdJsonParser.c`, `rrdRunCmdThread.c` | Filesystem permission mock | -| **P3** | systemd-run / journalctl execution failures | `rrdRunCmdThread.c` | Binary removal or mock failure | -| **P3** | Output file write errors | `rrdRunCmdThread.c` | Filesystem full or permission mock | -| **P3** | Dynamic profile JSON parse failure | `rrdDynamic.c` | Corrupted dynamic JSON file | -| **P3** | Invalid deep sleep event type | `rrdDynamic.c` | IARM event simulation with bad type | -| **P3** | Memory allocation failures (all modules) | All `.c` files | malloc failure injection (not practical in L2) | - ---- - -## 5. Gap Summary - -### 5.1 Feature vs Test Gap Table - -| Behavior Area | Feature Scenarios | Test Functions | Coverage | Top Gaps | -|---|:---:|:---:|:---:|---| -| Daemon startup/subscribe | 1 | 4 | 100% | — | -| RFC enable/disable | 2 | 1 | 100% | — | -| Single instance | 1 | 3 | 100% | — | -| Static profile report | 5 | 5 | 100% | — | -| Static category report | 5 | 5 | 100% | — | -| Static suffix report | 4 | 5 | 100% | — | -| Static suffix negative | 4 | 5 | 100% | — | -| Background command | 5 | 5 | 100% | — | -| Dynamic profile report | 5 | 9 | 100% | — | -| Dynamic subcategory | 5 | 7 | 100% | — | -| Dynamic missing | 4 | 7 | 100% | — | -| Append mode | 4 | 7 | 100% | — | -| Append (static not found) | 4 | 7 | 100% | — | -| Harmful static | 5 | 5 | 100% | — | -| Harmful dynamic | 5 | 7 | 100% | — | -| Corrupted profile | 4 | 4 | 100% | — | -| Missing command | 5 | 5 | 100% | — | -| Empty issuetype | 2 | 2 | 100% | — | -| Deep sleep | 2 | 5 | 100% | — | -| Debug report upload | 6 | 5 | 100% | — | -| **C API upload** | **21** | **5** | **~24%** | **16 scenarios not implemented** | -| **Profile data SET/GET** | **0** | **3** | **N/A** | **No feature file** | -| **WebCfg event** | **0** | **0** | **0%** | **Entire flow untested** | -| **Upload lock contention** | **0** | **0** | **0%** | **No test** | -| **Config fallback chain** | **0** | **0** | **0%** | **No test** | - -### 5.2 Action Items (Prioritized) - -| # | Priority | Action | Effort | -|:---:|:---:|---|:---:| -| 1 | P1 | Implement 16 missing `test_rrd_c_api_upload.py` test functions matching feature scenarios | High | -| 2 | P1 | Create `rrd_profile_data.feature` for the existing `test_rrd_profile_data.py` test | Low | -| 3 | P1 | Add WebCfg event L2 test (`test_rrd_webcfg_event.py` + `rrd_webcfg_event.feature`) | High | -| 4 | P2 | Add upload lock contention test | Medium | -| 5 | P2 | Add configuration fallback chain test | Medium | -| 6 | P2 | Add archive CPU throttle test | Medium | -| 7 | P2 | Add dynamic profile JSON parse failure test | Low | -| 8 | P3 | Add RBUS registration failure test | Low | -| 9 | P3 | Add systemd-run / journalctl failure test | Low | -| 10 | P3 | Add output file write error test | Low | - ---- - -## 6. Test Infrastructure Notes - -### Test Interfaces - -| Interface | Tool | Used By | -|---|---|---| -| RBUS event trigger | `rbuscli set` | All event-based tests | -| RBUS profile data | `rbuscli set/get` | `test_rrd_profile_data.py` | -| Log scraping | `grep_rrdlogs()` in `helper_functions.py` | All tests | -| File system checks | `os.path.isfile()`, `os.path.isdir()` | Prerequisite tests | -| Process control | `pidof`, `kill -9`, `nohup` | Setup/teardown | -| Mock upload server | Mock xconf server | `test_rrd_debug_report_upload.py` | -| Mock upload script | `uploadSTBLogs.sh` | Upload tests | - -### Test Execution - -- **Runner:** `pytest`, sequential per file -- **No ordering decorators:** Tests rely on file-level sequential execution order -- **Shared state:** Tests within a file depend on prior test side effects (daemon start → trigger → validate) -- **Cleanup:** Most files kill the daemon and remove logs in setup - ---- - -## 7. Appendix: Complete File Inventory - -### Feature Files (21) - -| # | File | Scenarios | -|:---:|---|:---:| -| 1 | `rrd_start_subscribe_and_wait.feature` | 1 | -| 2 | `rrd_start_control.feature` | 2 | -| 3 | `rrd_single_instance.feature` | 1 | -| 4 | `rrd_static_profile_report.feature` | 5 | -| 5 | `rrd_static_profile_category_report.feature` | 5 | -| 6 | `test_rrd_static_profile_report_with_suffix.feature` | 4 | -| 7 | `test_rrd_static_profile_report_with_suffix_negative_case.feature` | 4 | -| 8 | `rrd_background_cmd_static_profile_report.feature` | 5 | -| 9 | `rrd_dynamic_profile_report.feature` | 5 | -| 10 | `rrd_dynamic_profile_subcategory_report.feature` | 5 | -| 11 | `rrd_dynamic_profile_missing_report.feature` | 4 | -| 12 | `rrd_append_report.feature` | 4 | -| 13 | `rrd_append_dynamic_profile_static_not_found.feature` | 4 | -| 14 | `rrd_harmful_command_static_report.feature` | 5 | -| 15 | `test_rrd_dynamic_profile_harmful_report.feature` | 5 | -| 16 | `rrd_corrupted_static_profile_report.feature` | 4 | -| 17 | `rrd_static_profile_missing_command_report.feature` | 5 | -| 18 | `rrd_empty_issuetype_event.feature` | 2 | -| 19 | `rrd_deepsleep_static_report.feature` | 2 | -| 20 | `rrd_debug_report_upload.feature` | 6 | -| 21 | `rrd_c_api_upload.feature` | 21 | -| | **Total** | **97** | - -### Test Files (22) - -| # | File | Test Functions | -|:---:|---|:---:| -| 1 | `test_rrd_start_subscribe_and_wait.py` | 4 | -| 2 | `test_rrd_start_control.py` | 1 | -| 3 | `test_rrd_single_instance.py` | 3 | -| 4 | `test_rrd_static_profile_report.py` | 5 | -| 5 | `test_rrd_static_profile_category_report.py` | 5 | -| 6 | `test_rrd_static_profile_report_with_suffix.py` | 5 | -| 7 | `test_rrd_static_profile_report_with_suffix_negative_case.py` | 5 | -| 8 | `test_rrd_background_cmd_static_profile_report.py` | 5 | -| 9 | `test_rrd_dynamic_profile_report.py` | 9 | -| 10 | `test_rrd_dynamic_subcategory_report.py` | 7 | -| 11 | `test_rrd_dynamic_profile_missing_report.py` | 7 | -| 12 | `test_rrd_append_report.py` | 7 | -| 13 | `test_rrd_append_dynamic_profile_static_notfound.py` | 7 | -| 14 | `test_rrd_harmful_command_static_report.py` | 5 | -| 15 | `test_rrd_dynamic_profile_harmful_report.py` | 7 | -| 16 | `test_rrd_corrupted_static_profile_report.py` | 4 | -| 17 | `test_rrd_static_profile_missing_command_report.py` | 5 | -| 18 | `test_rrd_empty_issuetype_event.py` | 2 | -| 19 | `test_rrd_deepsleep_static_report.py` | 5 | -| 20 | `test_rrd_debug_report_upload.py` | 5 | -| 21 | `test_rrd_c_api_upload.py` | 5 | -| 22 | `test_rrd_profile_data.py` | 3 | -| | **Total** | **112** | From ad5dbb7478b11c4c94a02fcbbe1d6bb866757eb4 Mon Sep 17 00:00:00 2001 From: nhanasi Date: Tue, 11 Aug 2026 15:30:23 -0400 Subject: [PATCH 06/12] Update L2-tests.yml --- .github/workflows/L2-tests.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/L2-tests.yml b/.github/workflows/L2-tests.yml index b232531e2..cad598f95 100644 --- a/.github/workflows/L2-tests.yml +++ b/.github/workflows/L2-tests.yml @@ -98,7 +98,7 @@ jobs: if: github.event_name == 'push' run: docker cp native-platform:/tmp/l2_coverage/coverage.info /tmp/coverage.info - - name: Update L2_Test_Coverage.md with lcov metrics + - name: Update L2_Coverage.md with lcov metrics if: github.event_name == 'push' run: | sudo apt-get install -y --quiet lcov @@ -120,7 +120,7 @@ jobs: f'| **Function coverage (lcov)** | **{funcs_pct}** {funcs_detail} |\n' '' ) - path = 'remote_debugger/test/functional-tests/L2_Test_Coverage.md' + path = 'remote_debugger/test/functional-tests/L2_Coverage.md' content = open(path).read() content = re.sub(r'.*?', new_block, content, flags=re.DOTALL) content = re.sub(r'\*\*Generated:\*\* \S+', f'**Generated:** {today}', content) @@ -128,12 +128,12 @@ jobs: print(f'lines={lines_pct} branches={branch_pct} functions={funcs_pct}') EOF - - name: Commit updated L2_Test_Coverage.md + - name: Commit updated L2_Coverage.md if: github.event_name == 'push' run: | cd remote_debugger git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" - git add test/functional-tests/L2_Test_Coverage.md + git add test/functional-tests/L2_Coverage.md git diff --cached --quiet || git commit -m "ci: update L2 lcov coverage metrics [skip ci]" git push From e7a43917505747caec832e14b8f4ee0d191841ac Mon Sep 17 00:00:00 2001 From: nhanasi Date: Tue, 11 Aug 2026 15:35:46 -0400 Subject: [PATCH 07/12] Update L2-tests.yml --- .github/workflows/L2-tests.yml | 109 +++++++++++++++++++++++++-------- 1 file changed, 82 insertions(+), 27 deletions(-) diff --git a/.github/workflows/L2-tests.yml b/.github/workflows/L2-tests.yml index cad598f95..3b7f005bf 100644 --- a/.github/workflows/L2-tests.yml +++ b/.github/workflows/L2-tests.yml @@ -98,36 +98,91 @@ jobs: if: github.event_name == 'push' run: docker cp native-platform:/tmp/l2_coverage/coverage.info /tmp/coverage.info - - name: Update L2_Coverage.md with lcov metrics + - name: Generate L2_Coverage.md if: github.event_name == 'push' run: | - sudo apt-get install -y --quiet lcov - lcov --summary /tmp/coverage.info --rc lcov_branch_coverage=1 2>&1 | tee /tmp/lcov_summary.txt - python3 - <<'EOF' - import re, datetime - summary = open('/tmp/lcov_summary.txt').read() - def parse(pattern, text): - m = re.search(pattern, text) - return (m.group(1) + '%', f"({m.group(2)} of {m.group(3)})") if m else ('N/A', '') - lines_pct, lines_detail = parse(r'lines\.+:\s+([\d.]+)%\s+\((\d+) of (\d+)', summary) - branch_pct, branch_detail = parse(r'branches\.+:\s+([\d.]+)%\s+\((\d+) of (\d+)', summary) - funcs_pct, funcs_detail = parse(r'functions\.+:\s+([\d.]+)%\s+\((\d+) of (\d+)', summary) - today = datetime.date.today().strftime('%Y-%m-%d') - new_block = ( - '\n' - f'| **Line coverage (lcov)** | **{lines_pct}** {lines_detail} |\n' - f'| **Branch coverage (lcov)** | **{branch_pct}** {branch_detail} |\n' - f'| **Function coverage (lcov)** | **{funcs_pct}** {funcs_detail} |\n' - '' - ) - path = 'remote_debugger/test/functional-tests/L2_Coverage.md' - content = open(path).read() - content = re.sub(r'.*?', new_block, content, flags=re.DOTALL) - content = re.sub(r'\*\*Generated:\*\* \S+', f'**Generated:** {today}', content) - open(path, 'w').write(content) - print(f'lines={lines_pct} branches={branch_pct} functions={funcs_pct}') - EOF + sudo apt-get install -y --quiet lcov + lcov --summary /tmp/coverage.info --rc lcov_branch_coverage=1 2>&1 | tee /tmp/lcov_summary.txt + + python3 <<'EOF' + import re + import json + import glob + import datetime + from pathlib import Path + + summary = Path("/tmp/lcov_summary.txt").read_text() + + def parse(pattern): + m = re.search(pattern, summary) + if not m: + return ("N/A","0","0") + return m.group(1), m.group(2), m.group(3) + + line_pct, line_cov, line_total = parse( + r'lines\.+:\s+([\d.]+)%\s+\((\d+)\s+of\s+(\d+)\)' + ) + + branch_pct, branch_cov, branch_total = parse( + r'branches\.+:\s+([\d.]+)%\s+\((\d+)\s+of\s+(\d+)\)' + ) + + func_pct, func_cov, func_total = parse( + r'functions\.+:\s+([\d.]+)%\s+\((\d+)\s+of\s+(\d+)\)' + ) + + feature_files = glob.glob( + "remote_debugger/test/functional-tests/features/*.feature" + ) + + test_files = glob.glob( + "remote_debugger/test/functional-tests/tests/test_*.py" + ) + + total_tests = 0 + total_passed = 0 + + for report in glob.glob("/tmp/L2_TEST_RESULTS/*.json"): + try: + data = json.load(open(report)) + summary_data = data.get("summary", {}) + total_tests += summary_data.get("collected", 0) + total_passed += summary_data.get("passed", 0) + except Exception: + pass + + report = f"""# Remote Debugger L2 Coverage Report + + **Generated:** {datetime.date.today()} + + ## Coverage Metrics + + | Metric | Coverage | + |----------|----------| + | Line Coverage | {line_pct}% ({line_cov}/{line_total}) | + | Branch Coverage | {branch_pct}% ({branch_cov}/{branch_total}) | + | Function Coverage | {func_pct}% ({func_cov}/{func_total}) | + + ## Test Inventory + + | Item | Count | + |--------|--------| + | Feature Files | {len(feature_files)} | + | Test Files | {len(test_files)} | + | Tests Passed | {total_passed} | + | Tests Collected | {total_tests} | + + Generated automatically from L2 CI execution. + """ + + Path( + "remote_debugger/test/functional-tests/L2_Coverage.md" + ).write_text(report) + + print("Generated L2_Coverage.md") + EOF + - name: Commit updated L2_Coverage.md if: github.event_name == 'push' run: | From 267ff8ea455f25ceb0ed1b4e71a45c2dbc3c05a1 Mon Sep 17 00:00:00 2001 From: Hanasi Date: Wed, 12 Aug 2026 15:17:40 -0400 Subject: [PATCH 08/12] Update2 --- .github/workflows/L2-tests.yml | 41 +- test/functional-tests/L2_Test_Coverage.md | 5 - .../generate_l2_coverage_report.py | 489 ++++++++++++++++++ 3 files changed, 499 insertions(+), 36 deletions(-) create mode 100644 test/functional-tests/generate_l2_coverage_report.py diff --git a/.github/workflows/L2-tests.yml b/.github/workflows/L2-tests.yml index b232531e2..7a528edfe 100644 --- a/.github/workflows/L2-tests.yml +++ b/.github/workflows/L2-tests.yml @@ -98,42 +98,21 @@ jobs: if: github.event_name == 'push' run: docker cp native-platform:/tmp/l2_coverage/coverage.info /tmp/coverage.info - - name: Update L2_Test_Coverage.md with lcov metrics + - name: Generate L2_Coverage.md if: github.event_name == 'push' run: | - sudo apt-get install -y --quiet lcov - lcov --summary /tmp/coverage.info --rc lcov_branch_coverage=1 2>&1 | tee /tmp/lcov_summary.txt - python3 - <<'EOF' - import re, datetime - summary = open('/tmp/lcov_summary.txt').read() - def parse(pattern, text): - m = re.search(pattern, text) - return (m.group(1) + '%', f"({m.group(2)} of {m.group(3)})") if m else ('N/A', '') - lines_pct, lines_detail = parse(r'lines\.+:\s+([\d.]+)%\s+\((\d+) of (\d+)', summary) - branch_pct, branch_detail = parse(r'branches\.+:\s+([\d.]+)%\s+\((\d+) of (\d+)', summary) - funcs_pct, funcs_detail = parse(r'functions\.+:\s+([\d.]+)%\s+\((\d+) of (\d+)', summary) - today = datetime.date.today().strftime('%Y-%m-%d') - new_block = ( - '\n' - f'| **Line coverage (lcov)** | **{lines_pct}** {lines_detail} |\n' - f'| **Branch coverage (lcov)** | **{branch_pct}** {branch_detail} |\n' - f'| **Function coverage (lcov)** | **{funcs_pct}** {funcs_detail} |\n' - '' - ) - path = 'remote_debugger/test/functional-tests/L2_Test_Coverage.md' - content = open(path).read() - content = re.sub(r'.*?', new_block, content, flags=re.DOTALL) - content = re.sub(r'\*\*Generated:\*\* \S+', f'**Generated:** {today}', content) - open(path, 'w').write(content) - print(f'lines={lines_pct} branches={branch_pct} functions={funcs_pct}') - EOF - - - name: Commit updated L2_Test_Coverage.md + python3 remote_debugger/test/functional-tests/generate_l2_coverage_report.py \ + --tracefile /tmp/coverage.info \ + --features-dir remote_debugger/test/functional-tests/features \ + --tests-dir remote_debugger/test/functional-tests/tests \ + --output remote_debugger/test/functional-tests/L2_Coverage.md + + - name: Commit L2_Coverage.md if: github.event_name == 'push' run: | cd remote_debugger git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" - git add test/functional-tests/L2_Test_Coverage.md - git diff --cached --quiet || git commit -m "ci: update L2 lcov coverage metrics [skip ci]" + git add test/functional-tests/L2_Coverage.md + git diff --cached --quiet || git commit -m "ci: regenerate L2_Coverage.md [skip ci]" git push diff --git a/test/functional-tests/L2_Test_Coverage.md b/test/functional-tests/L2_Test_Coverage.md index 5bec351df..fcd872f15 100644 --- a/test/functional-tests/L2_Test_Coverage.md +++ b/test/functional-tests/L2_Test_Coverage.md @@ -19,11 +19,6 @@ | Source modules (conditional IARMBUS) | 7 | | **Overall feature coverage** | **95%** (21/22 test files have matching features) | | **Source behavior coverage** | **~65%** (happy paths covered; error/edge paths mostly untested) | - -| **Line coverage (lcov)** | pending | -| **Branch coverage (lcov)** | pending | -| **Function coverage (lcov)** | pending | - --- diff --git a/test/functional-tests/generate_l2_coverage_report.py b/test/functional-tests/generate_l2_coverage_report.py new file mode 100644 index 000000000..d03c7e523 --- /dev/null +++ b/test/functional-tests/generate_l2_coverage_report.py @@ -0,0 +1,489 @@ +#!/usr/bin/env python3 +""" +Generate test/functional-tests/L2_Coverage.md. + +Inputs +------ + --tracefile lcov .info file produced by run_l2.sh + --features-dir directory containing .feature files + --tests-dir directory containing test_*.py files + --output path for the generated markdown file + +Usage +----- + python3 generate_l2_coverage_report.py \ + --tracefile /tmp/l2_coverage/coverage.info \ + --features-dir test/functional-tests/features \ + --tests-dir test/functional-tests/tests \ + --output test/functional-tests/L2_Coverage.md +""" + +import argparse +import datetime +import os +import re +import sys + + +# --------------------------------------------------------------------------- +# lcov tracefile parser +# --------------------------------------------------------------------------- + +def parse_tracefile(path): + """Return (per_file_dict, summary_dict) from an lcov .info file.""" + files = {} + cur = None + with open(path) as fh: + for raw in fh: + line = raw.rstrip() + if line.startswith('SF:'): + name = os.path.basename(line[3:]) + cur = {'name': name, 'lh': 0, 'lf': 0, + 'fnh': 0, 'fnf': 0, 'brh': 0, 'brf': 0} + elif line == 'end_of_record' and cur: + files[cur['name']] = cur + cur = None + elif cur: + if line.startswith('LH:'): cur['lh'] = int(line[3:]) + elif line.startswith('LF:'): cur['lf'] = int(line[3:]) + elif line.startswith('FNH:'): cur['fnh'] = int(line[4:]) + elif line.startswith('FNF:'): cur['fnf'] = int(line[4:]) + elif line.startswith('BRH:'): cur['brh'] = int(line[4:]) + elif line.startswith('BRF:'): cur['brf'] = int(line[4:]) + + def tot(key): return sum(v[key] for v in files.values()) + summary = { + 'lines_pct': _pct(tot('lh'), tot('lf')), + 'lines_det': f"{tot('lh')} of {tot('lf')} lines", + 'funcs_pct': _pct(tot('fnh'), tot('fnf')), + 'funcs_det': f"{tot('fnh')} of {tot('fnf')} functions", + 'branches_pct': _pct(tot('brh'), tot('brf')), + 'branches_det': f"{tot('brh')} of {tot('brf')} branches", + } + return files, summary + + +def _pct(hit, found): + if found == 0: + return 'N/A' + return f'{hit / found * 100:.1f}%' + + +def _bar(hit, found, width=20): + if found == 0: + return '░' * width + filled = round(hit / found * width) + return '█' * filled + '░' * (width - filled) + + +# --------------------------------------------------------------------------- +# test directory scanners +# --------------------------------------------------------------------------- + +def scan_features(features_dir): + """Return {filename: scenario_count}.""" + result = {} + for fn in sorted(os.listdir(features_dir)): + if fn.endswith('.feature'): + result[fn] = _count_scenarios(os.path.join(features_dir, fn)) + return result + + +def scan_tests(tests_dir): + """Return {filename: test_function_count}.""" + result = {} + for fn in sorted(os.listdir(tests_dir)): + if fn.startswith('test_') and fn.endswith('.py'): + result[fn] = _count_test_funcs(os.path.join(tests_dir, fn)) + return result + + +def _count_scenarios(path): + count = 0 + with open(path, errors='replace') as fh: + for line in fh: + if re.match(r'\s*Scenario(\s+Outline)?:', line): + count += 1 + return count + + +def _count_test_funcs(path): + count = 0 + with open(path, errors='replace') as fh: + for line in fh: + if re.match(r'^def test_', line): + count += 1 + return count + + +# --------------------------------------------------------------------------- +# feature → test matching +# --------------------------------------------------------------------------- + +def build_mapping(features, tests): + """ + Return (pairs, orphan_features, orphan_tests). + pairs = list of (feature_file, scenarios, test_file_or_None, test_funcs_or_0) + """ + unmatched_tests = set(tests.keys()) + pairs = [] + for feat, scen in features.items(): + stem = feat.replace('.feature', '') + candidate = (stem if stem.startswith('test_') else 'test_' + stem) + '.py' + if candidate in tests: + pairs.append((feat, scen, candidate, tests[candidate])) + unmatched_tests.discard(candidate) + else: + pairs.append((feat, scen, None, 0)) + orphan_tests = {t: tests[t] for t in sorted(unmatched_tests)} + orphan_feats = [p[0] for p in pairs if p[2] is None] + return pairs, orphan_feats, orphan_tests + + +# --------------------------------------------------------------------------- +# static sections (priorities / recommendations — human judgment) +# --------------------------------------------------------------------------- + +_RECOMMENDATIONS = """\ +## 5. Gap Recommendations + +### Priority 1 — Must Fix + +| # | Gap | Impacted Modules | +|:---:|---|---| +| 1 | Implement missing `test_rrd_c_api_upload.py` scenarios (16 of 21 unimplemented) | `rrd_upload.c` | +| 2 | Add `rrd_profile_data.feature` for the existing `test_rrd_profile_data.py` | `rrdInterface.c` | +| 3 | Add WebCfg / MsgPack event L2 test (`rrd_webcfg_event.feature` + test) | `rrdMsgPackDecoder.c`, `rrdEventProcess.c` | + +### Priority 2 — Should Fix + +| # | Gap | Impacted Modules | +|:---:|---|---| +| 4 | Upload lock contention test | `rrd_upload.c` | +| 5 | Configuration fallback chain (RFC → DCM → dcm.properties) | `rrd_config.c` | +| 6 | Archive CPU throttle logic | `rrd_archive.c` | +| 7 | RDM download event with dynamic-profile cache miss | `rrdDynamic.c`, `rrdInterface.c` | +| 8 | Dynamic profile JSON parse failure | `rrdDynamic.c` | + +### Priority 3 — Nice to Have + +| # | Gap | Impacted Modules | +|:---:|---|---| +| 9 | RBUS registration / unregistration failure injection | `rrdInterface.c` | +| 10 | Message queue creation failure | `rrdMain.c` | +| 11 | Event thread creation failure | `rrdMain.c` | +| 12 | Directory creation / chdir failures | `rrdJsonParser.c` | +| 13 | `systemd-run` / `journalctl` execution failures | `rrdRunCmdThread.c` | +| 14 | Output file write errors | `rrdRunCmdThread.c` | +| 15 | Invalid deep sleep event type | `rrdDynamic.c` | +""" + +_BEHAVIOR_DETAIL = """\ +## 4. Per-Behavior Coverage Detail + +> Legend — **YES**: tested by an L2 scenario | **NO**: no test exists | **PARTIAL**: subset covered + +### 4.1 Daemon Lifecycle + +| Behavior | Covered | +|---|:---:| +| RBUS subscription + event wait | YES | +| RFC enable → daemon starts | YES | +| RFC disable → daemon stops | YES | +| Single instance enforcement | YES | +| Message queue creation failure | NO | +| Event thread creation failure | NO | +| Signal handling / graceful shutdown | NO | +| Device info file read failure | NO | + +### 4.2 Static Profile Processing + +| Behavior | Covered | +|---|:---:| +| Config file exists check | YES | +| IssueType event trigger + message flow | YES | +| JSON parse success + command execution | YES | +| Upload report success / failure | YES | +| Category-only issue type | YES | +| Suffixed issue type | YES | +| Overlength suffix (negative) | YES | +| Background command execution | YES | +| Missing command in profile | YES | +| Corrupted / invalid JSON profile | YES | + +### 4.3 Dynamic Profile Processing + +| Behavior | Covered | +|---|:---:| +| Dynamic profile fallback (static miss) | YES | +| Dynamic subcategory | YES | +| Dynamic profile missing → RDM trigger | YES | +| Append mode (static + dynamic) | YES | +| Append when static not found | YES | +| RDM download event (cache miss) | NO | +| Dynamic profile JSON parse failure | NO | + +### 4.4 Harmful Command Detection + +| Behavior | Covered | +|---|:---:| +| Static profile harmful command abort | YES | +| Dynamic profile harmful command abort | YES | +| Macro replacement edge cases | NO | +| Background command modification | PARTIAL | + +### 4.5 Event Handling + +| Behavior | Covered | +|---|:---:| +| IssueType RBUS event | YES | +| Empty IssueType event | YES | +| Deep sleep event | YES | +| WebCfg event (MsgPack decode) | NO | +| WebCfg corrupted data | NO | +| Multiple simultaneous IssueType events | NO | +| Invalid deep sleep event type | NO | + +### 4.6 Upload & Archive + +| Behavior | Covered | +|---|:---:| +| Upload via shell script | YES | +| Upload + download validation | YES | +| C API `rrd_upload_orchestrate` (happy path) | YES | +| C API NULL parameters | NO | +| C API empty / non-existent directory | NO | +| C API config loading / MAC retrieval | NO | +| C API archive creation + cleanup | NO | +| Concurrent upload lock | NO | +| Archive CPU throttle | NO | +""" + + +# --------------------------------------------------------------------------- +# markdown builder +# --------------------------------------------------------------------------- + +def generate(pairs, orphan_feats, orphan_tests, files, summary, today): + total_scenarios = sum(p[1] for p in pairs) + total_test_funcs = sum(p[3] for p in pairs) + sum(orphan_tests.values()) + total_features = len(pairs) + total_tests = len(pairs) - len(orphan_feats) + len(orphan_tests) + mapped = sum(1 for p in pairs if p[2]) + gap_pairs = [p for p in pairs if p[2] and p[3] < p[1]] + + lines = [] + + # ── header ──────────────────────────────────────────────────────────────── + lines += [ + '# Remote Debugger L2 Coverage Report', + '', + f'**Generated:** {today} ', + '**Component:** `remotedebugger` (`src/`) ', + '**Test suite:** `test/functional-tests/` ', + '**Coverage tool:** lcov (source-level instrumentation via `--coverage`)', + '', + '---', + '', + ] + + # ── executive summary ───────────────────────────────────────────────────── + lines += [ + '## 1. Executive Summary', + '', + '| Metric | Value |', + '|---|:---:|', + f'| Feature files | {total_features} |', + f'| Feature scenarios | {total_scenarios} |', + f'| Test files (pytest) | {total_tests} |', + f'| Test functions (`test_*`) | {total_test_funcs} |', + f'| Feature → Test mapped pairs | {mapped} / {total_features}' + + (f' (+{len(orphan_tests)} orphan test{"s" if len(orphan_tests) != 1 else ""})' if orphan_tests else '') + ' |', + f'| **Line coverage (lcov)** | **{summary["lines_pct"]}** ({summary["lines_det"]}) |', + f'| **Branch coverage (lcov)** | **{summary["branches_pct"]}** ({summary["branches_det"]}) |', + f'| **Function coverage (lcov)** | **{summary["funcs_pct"]}** ({summary["funcs_det"]}) |', + '', + ] + + # ── feature ↔ test mapping ──────────────────────────────────────────────── + lines += [ + '---', + '', + '## 2. Feature ↔ Test Mapping', + '', + '### 2.1 Mapped Pairs', + '', + '| # | Feature File | Scenarios | Test File | Tests | Gap |', + '|:---:|---|:---:|---|:---:|:---:|', + ] + idx = 1 + for feat, scen, test, tfuncs in pairs: + if test is None: + continue + delta = tfuncs - scen + gap = '—' if delta >= 0 else f'**{abs(delta)} missing**' + lines.append(f'| {idx} | `{feat}` | {scen} | `{test}` | {tfuncs} | {gap} |') + idx += 1 + + total_scen_mapped = sum(p[1] for p in pairs if p[2]) + total_funcs_mapped = sum(p[3] for p in pairs if p[2]) + lines += [ + f'| | **Totals** | **{total_scen_mapped}** | | **{total_funcs_mapped}** | |', + '', + ] + + if orphan_tests: + lines += [ + '### 2.2 Orphan Tests (test exists, no feature file)', + '', + '| Test File | Tests | Note |', + '|---|:---:|---|', + ] + for tf, cnt in orphan_tests.items(): + lines.append(f'| `{tf}` | {cnt} | **Missing `.feature` file** |') + lines.append('') + + if orphan_feats: + lines += [ + '### 2.3 Orphan Features (feature exists, no test file)', + '', + '| Feature File | Scenarios | Note |', + '|---|:---:|---|', + ] + for ff in orphan_feats: + sc = features[ff] + lines.append(f'| `{ff}` | {sc} | **Missing test file** |') + lines.append('') + + # ── per-module lcov coverage ─────────────────────────────────────────────── + lines += [ + '---', + '', + '## 3. Source Module Coverage (lcov)', + '', + '| Module | Lines | Functions | Branches | Coverage Bar |', + '|---|:---:|:---:|:---:|---|', + ] + src_modules = [ + 'rrdMain.c', 'rrdInterface.c', 'rrdEventProcess.c', 'rrdJsonParser.c', + 'rrdRunCmdThread.c', 'rrdCommandSanity.c', 'rrdDynamic.c', + 'rrdExecuteScript.c', 'rrdMsgPackDecoder.c', + 'rrd_config.c', 'rrd_sysinfo.c', 'rrd_logproc.c', + 'rrd_archive.c', 'rrd_upload.c', 'rrdIarmEvents.c', 'uploadRRDLogs.c', + ] + for mod in src_modules: + if mod in files: + d = files[mod] + lp = _pct(d['lh'], d['lf']) + fp = _pct(d['fnh'], d['fnf']) + bp = _pct(d['brh'], d['brf']) + bar = _bar(d['lh'], d['lf']) + lines.append( + f'| `{mod}` | {lp} ({d["lh"]}/{d["lf"]}) ' + f'| {fp} ({d["fnh"]}/{d["fnf"]}) ' + f'| {bp} ({d["brh"]}/{d["brf"]}) ' + f'| `{bar}` |' + ) + else: + lines.append(f'| `{mod}` | — | — | — | no data |') + lines.append('') + + # ── per-behavior detail (static) ────────────────────────────────────────── + lines += ['---', '', _BEHAVIOR_DETAIL, ''] + + # ── gap analysis ────────────────────────────────────────────────────────── + lines += [ + '---', + '', + '## 5. Scenario-to-Test Gap Analysis', + '', + ] + if gap_pairs: + lines += [ + '| Feature File | Scenarios | Tests | Missing |', + '|---|:---:|:---:|:---:|', + ] + for feat, scen, test, tfuncs in gap_pairs: + lines.append(f'| `{feat}` | {scen} | {tfuncs} | **{scen - tfuncs}** |') + lines.append('') + else: + lines += ['All mapped feature files have sufficient test function coverage.', ''] + + if orphan_tests: + lines += [ + '**Orphan tests** (no feature file — behavior is tested but not documented):', + '', + ] + for tf in orphan_tests: + lines.append(f'- `{tf}`') + lines.append('') + + # ── recommendations (static) ───────────────────────────────────────────── + lines += ['---', '', _RECOMMENDATIONS, ''] + + # ── appendix ────────────────────────────────────────────────────────────── + lines += [ + '---', + '', + '## 6. Appendix: File Inventory', + '', + '### Feature Files', + '', + '| # | File | Scenarios |', + '|:---:|---|:---:|', + ] + for i, (feat, scen, *_) in enumerate(pairs, 1): + lines.append(f'| {i} | `{feat}` | {scen} |') + lines.append(f'| | **Total** | **{total_scenarios}** |') + lines.append('') + + lines += [ + '### Test Files', + '', + '| # | File | Tests |', + '|:---:|---|:---:|', + ] + all_tests = [(p[2], p[3]) for p in pairs if p[2]] + all_tests += list(orphan_tests.items()) + all_tests.sort() + for i, (tf, cnt) in enumerate(all_tests, 1): + lines.append(f'| {i} | `{tf}` | {cnt} |') + lines.append(f'| | **Total** | **{total_test_funcs}** |') + lines.append('') + + return '\n'.join(lines) + + +# --------------------------------------------------------------------------- +# entry point +# --------------------------------------------------------------------------- + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--tracefile', required=True) + ap.add_argument('--features-dir', required=True) + ap.add_argument('--tests-dir', required=True) + ap.add_argument('--output', required=True) + args = ap.parse_args() + + global features # used in build_mapping closure for orphan label + features = scan_features(args.features_dir) + tests = scan_tests(args.tests_dir) + cov_files, summary = parse_tracefile(args.tracefile) + pairs, orphan_feats, orphan_tests = build_mapping(features, tests) + today = datetime.date.today().strftime('%Y-%m-%d') + + md = generate(pairs, orphan_feats, orphan_tests, cov_files, summary, today) + + os.makedirs(os.path.dirname(os.path.abspath(args.output)), exist_ok=True) + with open(args.output, 'w') as fh: + fh.write(md) + + print(f'Written: {args.output}') + print(f' lines={summary["lines_pct"]} ' + f'branches={summary["branches_pct"]} ' + f'functions={summary["funcs_pct"]}') + + +if __name__ == '__main__': + main() From ecbc2ac1ada78b7bd2adfe2086c4e63667445c8f Mon Sep 17 00:00:00 2001 From: Hanasi Date: Wed, 12 Aug 2026 15:22:18 -0400 Subject: [PATCH 09/12] fix error --- test/functional-tests/generate_l2_coverage_report.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/functional-tests/generate_l2_coverage_report.py b/test/functional-tests/generate_l2_coverage_report.py index d03c7e523..3e16c9e99 100644 --- a/test/functional-tests/generate_l2_coverage_report.py +++ b/test/functional-tests/generate_l2_coverage_report.py @@ -145,7 +145,7 @@ def build_mapping(features, tests): # --------------------------------------------------------------------------- _RECOMMENDATIONS = """\ -## 5. Gap Recommendations +## 6. Gap Recommendations ### Priority 1 — Must Fix @@ -425,7 +425,7 @@ def generate(pairs, orphan_feats, orphan_tests, files, summary, today): lines += [ '---', '', - '## 6. Appendix: File Inventory', + '## 7. Appendix: File Inventory', '', '### Feature Files', '', From 9b9d1f33665cdbaa12c6db52c40ecd2642cc8746 Mon Sep 17 00:00:00 2001 From: Hanasi Date: Wed, 12 Aug 2026 16:07:38 -0400 Subject: [PATCH 10/12] fix bug --- .github/workflows/L2-tests.yml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/workflows/L2-tests.yml b/.github/workflows/L2-tests.yml index 7a528edfe..4718906d4 100644 --- a/.github/workflows/L2-tests.yml +++ b/.github/workflows/L2-tests.yml @@ -23,6 +23,7 @@ jobs: uses: actions/checkout@v4 with: path: remote_debugger + token: ${{ secrets.GITHUB_TOKEN }} - name: Check out dependent repostiories uses: actions/checkout@v4 @@ -95,11 +96,13 @@ jobs: docker exec -i ci-container bash -c "echo 'Contents in workspace directory' && ls -l && echo '===============================' && echo 'Contents in /tmp/L2_TEST_RESULTS' && ls -l /tmp/L2_TEST_RESULTS && echo '===============================' && git config --global --add safe.directory /mnt/L2_CONTAINER_SHARED_VOLUME/remote_debugger && gtest-json-result-push.py /tmp/L2_TEST_RESULTS https://rdkeorchestrationservice.apps.cloud.comcast.net/rdke_orchestration_api/push_unit_test_results /mnt/L2_CONTAINER_SHARED_VOLUME/remote_debugger" - name: Copy lcov tracefile from container + id: copy_tracefile if: github.event_name == 'push' + continue-on-error: true run: docker cp native-platform:/tmp/l2_coverage/coverage.info /tmp/coverage.info - name: Generate L2_Coverage.md - if: github.event_name == 'push' + if: github.event_name == 'push' && steps.copy_tracefile.outcome == 'success' run: | python3 remote_debugger/test/functional-tests/generate_l2_coverage_report.py \ --tracefile /tmp/coverage.info \ @@ -108,7 +111,7 @@ jobs: --output remote_debugger/test/functional-tests/L2_Coverage.md - name: Commit L2_Coverage.md - if: github.event_name == 'push' + if: github.event_name == 'push' && steps.copy_tracefile.outcome == 'success' run: | cd remote_debugger git config user.name "github-actions[bot]" From 25c6948911fc95c976e1e122dafa61b04f49dbb7 Mon Sep 17 00:00:00 2001 From: nhanasi Date: Fri, 14 Aug 2026 10:45:16 -0400 Subject: [PATCH 11/12] Update L2-tests.yml --- .github/workflows/L2-tests.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/L2-tests.yml b/.github/workflows/L2-tests.yml index 4718906d4..62bfa58c8 100644 --- a/.github/workflows/L2-tests.yml +++ b/.github/workflows/L2-tests.yml @@ -2,9 +2,9 @@ name: L2 Integration Tests on: pull_request: - branches: [ develop ] + branches: [ feature/l2docs ] push: - branches: [ develop ] + branches: [ feature/l2docs ] permissions: contents: write From c5fdc06c772024457d2dfb066898035f8eb50de8 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Fri, 14 Aug 2026 15:03:11 +0000 Subject: [PATCH 12/12] ci: regenerate L2_Coverage.md [skip ci] --- test/functional-tests/L2_Coverage.md | 284 +++++++++++++++++++++++++++ 1 file changed, 284 insertions(+) create mode 100644 test/functional-tests/L2_Coverage.md diff --git a/test/functional-tests/L2_Coverage.md b/test/functional-tests/L2_Coverage.md new file mode 100644 index 000000000..045d2d4a5 --- /dev/null +++ b/test/functional-tests/L2_Coverage.md @@ -0,0 +1,284 @@ +# Remote Debugger L2 Coverage Report + +**Generated:** 2026-08-14 +**Component:** `remotedebugger` (`src/`) +**Test suite:** `test/functional-tests/` +**Coverage tool:** lcov (source-level instrumentation via `--coverage`) + +--- + +## 1. Executive Summary + +| Metric | Value | +|---|:---:| +| Feature files | 22 | +| Feature scenarios | 103 | +| Test files (pytest) | 23 | +| Test functions (`test_*`) | 118 | +| Feature → Test mapped pairs | 19 / 22 (+4 orphan tests) | +| **Line coverage (lcov)** | **49.4%** (1487 of 3012 lines) | +| **Branch coverage (lcov)** | **36.8%** (503 of 1365 branches) | +| **Function coverage (lcov)** | **60.3%** (76 of 126 functions) | + +--- + +## 2. Feature ↔ Test Mapping + +### 2.1 Mapped Pairs + +| # | Feature File | Scenarios | Test File | Tests | Gap | +|:---:|---|:---:|---|:---:|:---:| +| 1 | `rrd_append_report.feature` | 4 | `test_rrd_append_report.py` | 7 | — | +| 2 | `rrd_background_cmd_static_profile_report.feature` | 5 | `test_rrd_background_cmd_static_profile_report.py` | 5 | — | +| 3 | `rrd_c_api_upload.feature` | 21 | `test_rrd_c_api_upload.py` | 5 | **16 missing** | +| 4 | `rrd_corrupted_static_profile_report.feature` | 4 | `test_rrd_corrupted_static_profile_report.py` | 4 | — | +| 5 | `rrd_debug_report_upload.feature` | 6 | `test_rrd_debug_report_upload.py` | 6 | — | +| 6 | `rrd_deepsleep_static_report.feature` | 2 | `test_rrd_deepsleep_static_report.py` | 5 | — | +| 7 | `rrd_dynamic_profile_missing_report.feature` | 4 | `test_rrd_dynamic_profile_missing_report.py` | 7 | — | +| 8 | `rrd_dynamic_profile_report.feature` | 5 | `test_rrd_dynamic_profile_report.py` | 9 | — | +| 9 | `rrd_empty_issuetype_event.feature` | 2 | `test_rrd_empty_issuetype_event.py` | 2 | — | +| 10 | `rrd_harmful_command_static_report.feature` | 5 | `test_rrd_harmful_command_static_report.py` | 5 | — | +| 11 | `rrd_single_instance.feature` | 1 | `test_rrd_single_instance.py` | 3 | — | +| 12 | `rrd_start_control.feature` | 2 | `test_rrd_start_control.py` | 1 | **1 missing** | +| 13 | `rrd_start_subscribe_and_wait.feature` | 1 | `test_rrd_start_subscribe_and_wait.py` | 4 | — | +| 14 | `rrd_static_profile_category_report.feature` | 5 | `test_rrd_static_profile_category_report.py` | 5 | — | +| 15 | `rrd_static_profile_missing_command_report.feature` | 5 | `test_rrd_static_profile_missing_command_report.py` | 5 | — | +| 16 | `rrd_static_profile_report.feature` | 5 | `test_rrd_static_profile_report.py` | 5 | — | +| 17 | `test_rrd_dynamic_profile_harmful_report.feature` | 5 | `test_rrd_dynamic_profile_harmful_report.py` | 7 | — | +| 18 | `test_rrd_static_profile_report_with_suffix.feature` | 4 | `test_rrd_static_profile_report_with_suffix.py` | 5 | — | +| 19 | `test_rrd_static_profile_report_with_suffix_negative_case.feature` | 4 | `test_rrd_static_profile_report_with_suffix_negative_case.py` | 5 | — | +| | **Totals** | **90** | | **95** | | + +### 2.2 Orphan Tests (test exists, no feature file) + +| Test File | Tests | Note | +|---|:---:|---| +| `test_rrd_append_dynamic_profile_static_notfound.py` | 7 | **Missing `.feature` file** | +| `test_rrd_dynamic_profile_rdm_node_length_exceeded.py` | 6 | **Missing `.feature` file** | +| `test_rrd_dynamic_subcategory_report.py` | 7 | **Missing `.feature` file** | +| `test_rrd_profile_data.py` | 3 | **Missing `.feature` file** | + +### 2.3 Orphan Features (feature exists, no test file) + +| Feature File | Scenarios | Note | +|---|:---:|---| +| `rrd_append_dynamic_profile_static_not_found.feature` | 4 | **Missing test file** | +| `rrd_dynamic_profile_node_length_exceeded.feature` | 4 | **Missing test file** | +| `rrd_dynamic_profile_subcategory_report.feature` | 5 | **Missing test file** | + +--- + +## 3. Source Module Coverage (lcov) + +| Module | Lines | Functions | Branches | Coverage Bar | +|---|:---:|:---:|:---:|---| +| `rrdMain.c` | 60.3% (38/63) | 100.0% (4/4) | 42.9% (6/14) | `████████████░░░░░░░░` | +| `rrdInterface.c` | 42.4% (181/427) | 59.1% (13/22) | 33.3% (56/168) | `████████░░░░░░░░░░░░` | +| `rrdEventProcess.c` | 69.3% (232/335) | 81.8% (9/11) | 48.6% (68/140) | `██████████████░░░░░░` | +| `rrdJsonParser.c` | 81.6% (400/490) | 86.7% (13/15) | 66.0% (128/194) | `████████████████░░░░` | +| `rrdRunCmdThread.c` | 55.2% (117/212) | 60.0% (6/10) | 34.5% (20/58) | `███████████░░░░░░░░░` | +| `rrdCommandSanity.c` | 93.1% (67/72) | 100.0% (3/3) | 71.9% (23/32) | `███████████████████░` | +| `rrdDynamic.c` | 40.4% (59/146) | 40.0% (2/5) | 24.6% (14/57) | `████████░░░░░░░░░░░░` | +| `rrdExecuteScript.c` | 94.1% (16/17) | 100.0% (2/2) | 75.0% (6/8) | `███████████████████░` | +| `rrdMsgPackDecoder.c` | 0.0% (0/280) | 0.0% (0/15) | 0.0% (0/138) | `░░░░░░░░░░░░░░░░░░░░` | +| `rrd_config.c` | 42.8% (95/222) | 66.7% (6/9) | 27.6% (53/192) | `█████████░░░░░░░░░░░` | +| `rrd_sysinfo.c` | 32.1% (36/112) | 33.3% (2/6) | 19.6% (11/56) | `██████░░░░░░░░░░░░░░` | +| `rrd_logproc.c` | 48.6% (36/74) | 75.0% (3/4) | 47.9% (23/48) | `██████████░░░░░░░░░░` | +| `rrd_archive.c` | 47.2% (126/267) | 80.0% (8/10) | 46.5% (66/142) | `█████████░░░░░░░░░░░` | +| `rrd_upload.c` | 40.7% (37/91) | 60.0% (3/5) | 30.4% (14/46) | `████████░░░░░░░░░░░░` | +| `rrdIarmEvents.c` | 8.6% (12/140) | 25.0% (1/4) | 8.3% (4/48) | `██░░░░░░░░░░░░░░░░░░` | +| `uploadRRDLogs.c` | 54.7% (35/64) | 100.0% (1/1) | 45.8% (11/24) | `███████████░░░░░░░░░` | + +--- + +## 4. Per-Behavior Coverage Detail + +> Legend — **YES**: tested by an L2 scenario | **NO**: no test exists | **PARTIAL**: subset covered + +### 4.1 Daemon Lifecycle + +| Behavior | Covered | +|---|:---:| +| RBUS subscription + event wait | YES | +| RFC enable → daemon starts | YES | +| RFC disable → daemon stops | YES | +| Single instance enforcement | YES | +| Message queue creation failure | NO | +| Event thread creation failure | NO | +| Signal handling / graceful shutdown | NO | +| Device info file read failure | NO | + +### 4.2 Static Profile Processing + +| Behavior | Covered | +|---|:---:| +| Config file exists check | YES | +| IssueType event trigger + message flow | YES | +| JSON parse success + command execution | YES | +| Upload report success / failure | YES | +| Category-only issue type | YES | +| Suffixed issue type | YES | +| Overlength suffix (negative) | YES | +| Background command execution | YES | +| Missing command in profile | YES | +| Corrupted / invalid JSON profile | YES | + +### 4.3 Dynamic Profile Processing + +| Behavior | Covered | +|---|:---:| +| Dynamic profile fallback (static miss) | YES | +| Dynamic subcategory | YES | +| Dynamic profile missing → RDM trigger | YES | +| Append mode (static + dynamic) | YES | +| Append when static not found | YES | +| RDM download event (cache miss) | NO | +| Dynamic profile JSON parse failure | NO | + +### 4.4 Harmful Command Detection + +| Behavior | Covered | +|---|:---:| +| Static profile harmful command abort | YES | +| Dynamic profile harmful command abort | YES | +| Macro replacement edge cases | NO | +| Background command modification | PARTIAL | + +### 4.5 Event Handling + +| Behavior | Covered | +|---|:---:| +| IssueType RBUS event | YES | +| Empty IssueType event | YES | +| Deep sleep event | YES | +| WebCfg event (MsgPack decode) | NO | +| WebCfg corrupted data | NO | +| Multiple simultaneous IssueType events | NO | +| Invalid deep sleep event type | NO | + +### 4.6 Upload & Archive + +| Behavior | Covered | +|---|:---:| +| Upload via shell script | YES | +| Upload + download validation | YES | +| C API `rrd_upload_orchestrate` (happy path) | YES | +| C API NULL parameters | NO | +| C API empty / non-existent directory | NO | +| C API config loading / MAC retrieval | NO | +| C API archive creation + cleanup | NO | +| Concurrent upload lock | NO | +| Archive CPU throttle | NO | + + +--- + +## 5. Scenario-to-Test Gap Analysis + +| Feature File | Scenarios | Tests | Missing | +|---|:---:|:---:|:---:| +| `rrd_c_api_upload.feature` | 21 | 5 | **16** | +| `rrd_start_control.feature` | 2 | 1 | **1** | + +**Orphan tests** (no feature file — behavior is tested but not documented): + +- `test_rrd_append_dynamic_profile_static_notfound.py` +- `test_rrd_dynamic_profile_rdm_node_length_exceeded.py` +- `test_rrd_dynamic_subcategory_report.py` +- `test_rrd_profile_data.py` + +--- + +## 6. Gap Recommendations + +### Priority 1 — Must Fix + +| # | Gap | Impacted Modules | +|:---:|---|---| +| 1 | Implement missing `test_rrd_c_api_upload.py` scenarios (16 of 21 unimplemented) | `rrd_upload.c` | +| 2 | Add `rrd_profile_data.feature` for the existing `test_rrd_profile_data.py` | `rrdInterface.c` | +| 3 | Add WebCfg / MsgPack event L2 test (`rrd_webcfg_event.feature` + test) | `rrdMsgPackDecoder.c`, `rrdEventProcess.c` | + +### Priority 2 — Should Fix + +| # | Gap | Impacted Modules | +|:---:|---|---| +| 4 | Upload lock contention test | `rrd_upload.c` | +| 5 | Configuration fallback chain (RFC → DCM → dcm.properties) | `rrd_config.c` | +| 6 | Archive CPU throttle logic | `rrd_archive.c` | +| 7 | RDM download event with dynamic-profile cache miss | `rrdDynamic.c`, `rrdInterface.c` | +| 8 | Dynamic profile JSON parse failure | `rrdDynamic.c` | + +### Priority 3 — Nice to Have + +| # | Gap | Impacted Modules | +|:---:|---|---| +| 9 | RBUS registration / unregistration failure injection | `rrdInterface.c` | +| 10 | Message queue creation failure | `rrdMain.c` | +| 11 | Event thread creation failure | `rrdMain.c` | +| 12 | Directory creation / chdir failures | `rrdJsonParser.c` | +| 13 | `systemd-run` / `journalctl` execution failures | `rrdRunCmdThread.c` | +| 14 | Output file write errors | `rrdRunCmdThread.c` | +| 15 | Invalid deep sleep event type | `rrdDynamic.c` | + + +--- + +## 7. Appendix: File Inventory + +### Feature Files + +| # | File | Scenarios | +|:---:|---|:---:| +| 1 | `rrd_append_dynamic_profile_static_not_found.feature` | 4 | +| 2 | `rrd_append_report.feature` | 4 | +| 3 | `rrd_background_cmd_static_profile_report.feature` | 5 | +| 4 | `rrd_c_api_upload.feature` | 21 | +| 5 | `rrd_corrupted_static_profile_report.feature` | 4 | +| 6 | `rrd_debug_report_upload.feature` | 6 | +| 7 | `rrd_deepsleep_static_report.feature` | 2 | +| 8 | `rrd_dynamic_profile_missing_report.feature` | 4 | +| 9 | `rrd_dynamic_profile_node_length_exceeded.feature` | 4 | +| 10 | `rrd_dynamic_profile_report.feature` | 5 | +| 11 | `rrd_dynamic_profile_subcategory_report.feature` | 5 | +| 12 | `rrd_empty_issuetype_event.feature` | 2 | +| 13 | `rrd_harmful_command_static_report.feature` | 5 | +| 14 | `rrd_single_instance.feature` | 1 | +| 15 | `rrd_start_control.feature` | 2 | +| 16 | `rrd_start_subscribe_and_wait.feature` | 1 | +| 17 | `rrd_static_profile_category_report.feature` | 5 | +| 18 | `rrd_static_profile_missing_command_report.feature` | 5 | +| 19 | `rrd_static_profile_report.feature` | 5 | +| 20 | `test_rrd_dynamic_profile_harmful_report.feature` | 5 | +| 21 | `test_rrd_static_profile_report_with_suffix.feature` | 4 | +| 22 | `test_rrd_static_profile_report_with_suffix_negative_case.feature` | 4 | +| | **Total** | **103** | + +### Test Files + +| # | File | Tests | +|:---:|---|:---:| +| 1 | `test_rrd_append_dynamic_profile_static_notfound.py` | 7 | +| 2 | `test_rrd_append_report.py` | 7 | +| 3 | `test_rrd_background_cmd_static_profile_report.py` | 5 | +| 4 | `test_rrd_c_api_upload.py` | 5 | +| 5 | `test_rrd_corrupted_static_profile_report.py` | 4 | +| 6 | `test_rrd_debug_report_upload.py` | 6 | +| 7 | `test_rrd_deepsleep_static_report.py` | 5 | +| 8 | `test_rrd_dynamic_profile_harmful_report.py` | 7 | +| 9 | `test_rrd_dynamic_profile_missing_report.py` | 7 | +| 10 | `test_rrd_dynamic_profile_rdm_node_length_exceeded.py` | 6 | +| 11 | `test_rrd_dynamic_profile_report.py` | 9 | +| 12 | `test_rrd_dynamic_subcategory_report.py` | 7 | +| 13 | `test_rrd_empty_issuetype_event.py` | 2 | +| 14 | `test_rrd_harmful_command_static_report.py` | 5 | +| 15 | `test_rrd_profile_data.py` | 3 | +| 16 | `test_rrd_single_instance.py` | 3 | +| 17 | `test_rrd_start_control.py` | 1 | +| 18 | `test_rrd_start_subscribe_and_wait.py` | 4 | +| 19 | `test_rrd_static_profile_category_report.py` | 5 | +| 20 | `test_rrd_static_profile_missing_command_report.py` | 5 | +| 21 | `test_rrd_static_profile_report.py` | 5 | +| 22 | `test_rrd_static_profile_report_with_suffix.py` | 5 | +| 23 | `test_rrd_static_profile_report_with_suffix_negative_case.py` | 5 | +| | **Total** | **118** |