Skip to content

feat(cli): add --tc-params-file option to auto-populate test parameters from mapping file#96

Open
rquidute wants to merge 6 commits into
v2.16-cli-developfrom
feature/903-auto-populate-test-params-from-mapping-file
Open

feat(cli): add --tc-params-file option to auto-populate test parameters from mapping file#96
rquidute wants to merge 6 commits into
v2.16-cli-developfrom
feature/903-auto-populate-test-params-from-mapping-file

Conversation

@rquidute

@rquidute rquidute commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements feature request #903: auto-populate test_parameters from a pre-generated TC parameters mapping file.

Users can now pass a --tc-params-file / -m argument to th-cli run-tests pointing to a JSON file that maps TC IDs to their required run arguments (int-arg, string-arg, timeout, etc.). The CLI automatically looks up each selected TC in the file and merges the matching parameters before starting the test run — eliminating the need to manually copy PIXIT values from the QA Verification Steps sheet for every run.


Related Issue

Closes project-chip/certification-tool#903


Changes

th_cli/utils.pyload_tc_params_mapping()

New function that loads a TC parameters mapping JSON file and returns a merged test_parameters dict for the requested TC IDs:

  • Normalises TC ID separators (-, _, .) and applies case-insensitive matching — consistent with build_test_selection().
  • Merges all matched entries into a single flat dict (alphabetical sort for deterministic key-conflict resolution).
  • Returns (merged_params, missing_ids) so callers can emit graceful fallback warnings without aborting execution.

th_cli/validation.pyvalidate_tc_params_file()

New pre-flight validator called before the test run starts:

  • Verifies the file exists and is readable.
  • Parses the JSON and checks the top-level value is a dict.
  • Ensures every entry value is also a dict (params object), listing all offending TC IDs in the error message if not.

th_cli/commands/run_tests.py--tc-params-file / -m option

  • New click option added after --pics-config-folder.

  • Validated early (before the API client is created) so bad files fail fast.

  • Mapping params are applied at priority level 2 in the merge chain, between the project config and explicit overrides:

    project config  <  --tc-params-file  <  --config  <  -- inline args
    
  • TC IDs not found in the mapping emit a warning and continue normally.

  • Matched params are printed to the console as TC Params from Mapping File (Execution Only).

tests/test_tc_params_mapping.py — 28 unit tests

Covers: exact match, separator/case normalisation, multi-ID merge, key-conflict order, missing IDs, empty mapping, empty test list, space-separated int-arg values, mixed separators, superset mapping files, and all error paths for both new functions.


Motivation

Running large clusters of tests (or onboarding new users) required manually extracting PIXIT values from the QA Verification Steps sheet and pasting them into the TH project config for every test cycle. This was:

  • Repetitive and error-prone (wrong PIXITs cause misattributed failures).
  • Especially burdensome for large test runs with many distinct parameter sets.

The mapping file acts as a single source of truth, maintained by QA and distributed with each TH build. Users only need to override values that are device-specific.


Impact

  • No breaking changes. --tc-params-file is fully optional; omitting it leaves all existing behaviour unchanged.
  • No backend, API, or schema changes — parameters land in test_run_config["test_parameters"] exactly as before.
  • Backward compatible — projects and runs that don't use the flag are unaffected.
  • Override chain is preserved — explicit --config file values and inline -- --int-arg args always take precedence over the mapping file.

Testing

Unit Tests (automated)

28 new tests in tests/test_tc_params_mapping.py covering:

Scenario Tests
Exact TC ID match test_exact_match_returns_params
Separator normalisation (- / _ / .) test_dash_underscore_normalisation, test_dot_normalisation
Case-insensitive matching test_case_insensitive_match
Multi-ID merge (disjoint keys) test_multiple_ids_merged, test_non_conflicting_keys_all_present
Key conflict resolution (alphabetical order) test_key_conflict_last_sorted_id_wins
Missing IDs (graceful fallback) test_unmatched_ids_reported_in_missing, test_all_ids_unmatched, test_empty_mapping_all_missing
Empty inputs test_empty_test_ids_list, test_empty_mapping_is_valid
Real-world formats (space-separated int-arg, mixed separators) test_multiple_space_separated_values_in_int_arg, test_mapping_keys_with_mixed_separators
Superset mapping (only requested IDs merged) test_mapping_superset_only_requested_ids_merged
All error paths (missing file, bad JSON, array root, non-dict values) 6 error-path tests across both classes

Run with:

./scripts/run_pytest.sh tests/test_tc_params_mapping.py -v

Manual Testing

To verify end-to-end behaviour, create a mapping file and pass it to run-tests:

{
  "TC-ACE-1.1": { "int-arg": "PIXIT.ACE.APPENDPOINT:1", "timeout": "300" },
  "TC-MCORE-FS-1.3": { "string-arg": "PIXIT.MCORE.ENDPOINT:3" }
}
# Basic usage
th-cli run-tests -t TC-ACE-1.1 -m tc_params_mapping.json

# Mapping + explicit config override (--config wins over mapping)
th-cli run-tests -t TC-ACE-1.1 -m tc_params_mapping.json -c my_config.json

# Mapping + inline arg override (-- args win over everything)
th-cli run-tests -t TC-ACE-1.1 -m tc_params_mapping.json -- --int-arg PIXIT.ACE.APPENDPOINT:2

# TC not in mapping — warning printed, execution continues normally
th-cli run-tests -t TC-ACE-1.1,TC-CC-1.1 -m tc_params_mapping.json

Expected console output when mapping is applied:

TC Params from Mapping File (Execution Only): {
  "int-arg": "PIXIT.ACE.APPENDPOINT:1",
  "timeout": "300"
}

run-test --help arg

Screenshot 2026-06-12 at 14 01 53

rquidute and others added 2 commits June 12, 2026 12:05
…eters

Implements feature request #903: allow users to provide a pre-generated
TC parameters mapping JSON file that automatically populates
test_parameters for each selected TC ID, eliminating the need to
manually copy PIXIT values from the QA Verification Steps sheet.

Changes:
- utils.py: add load_tc_params_mapping() to load and merge test params
  from a JSON mapping file keyed by TC ID. Normalises separators
  (-/_/.) and performs case-insensitive matching, matching the
  same normalisation used in build_test_selection(). Returns a tuple
  of (merged_params, missing_ids) for graceful fallback reporting.

- validation.py: add validate_tc_params_file() for pre-flight format
  checks (file exists, valid JSON, top-level dict, all values are
  dicts). Surfaces bad TC IDs in the error message.

- commands/run_tests.py: add --tc-params-file / -m option. The mapping
  is applied at priority level 2 in the merge chain:
    project config < mapping file < --config < -- inline args
  Missing TC IDs emit a warning but do not abort execution.

- tests/test_tc_params_mapping.py: 28 unit tests covering happy paths,
  separator/case normalisation, merge-order semantics, missing IDs,
  and all error paths for both new functions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dation.py

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@rquidute rquidute self-assigned this Jun 12, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new feature to auto-populate test parameters from a TC parameters mapping JSON file via the --tc-params-file option in the run_tests command. It includes validation logic, key normalization, parameter merging, and comprehensive unit tests. The review feedback highlights a PEP 8 style violation where a statement is placed on the same line as an if condition, and points out code duplication and redundant JSON parsing between the validation and loading functions, suggesting a shared helper to resolve it.

Comment thread th_cli/commands/run_tests.py
Comment thread th_cli/validation.py Outdated
@rquidute rquidute changed the title feat(cli): add --tc-params-file option for auto-populating test param… feat(cli): add --tc-params-file option to auto-populate test parameters from mapping file Jun 12, 2026
@rquidute rquidute requested a review from oxesoft June 12, 2026 15:19
rquidute and others added 3 commits June 12, 2026 12:20
Address Gemini code review comments on PR #96:

1. Eliminate duplicate file I/O and validation logic between
   validate_tc_params_file (validation.py) and load_tc_params_mapping
   (utils.py). Both functions were independently opening, parsing, and
   structurally validating the same mapping file.

2. Extract parse_and_validate_tc_params_file() into validation.py as the
   single source of truth for mapping-file I/O and structural checks.
   - validate_tc_params_file() now delegates to it and returns the
     resolved Path.
   - load_tc_params_mapping() imports and calls it instead of repeating
     the try/except file-reading block.

3. Add 7 tests for parse_and_validate_tc_params_file in
   test_tc_params_mapping.py, including a cross-function test that proves
   both callers produce identical error messages for the same bad file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_run_tests_logger_configuration was asserting configure_logger_for_run
was called with only title=, but the function now also receives
enable_log_streaming=True (passed when --no-streaming is not set).
- Describe normalisation behaviour (case-insensitive, -/_/. separators)
- Note that unmatched TC IDs are silently skipped
- Add a numbered NOTE block showing the full config precedence chain:
    1. project config  2. --tc-params-file  3. --config  4. -- inline args
Use Click's \b escape to prevent the NOTE block from being re-wrapped
into a single run-on paragraph. Each precedence level now renders on
its own line in the terminal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants