fix: USITC HTS-filtered query payloads by Redliana · Pull Request #8 · Redliana/CMM_Tools

Redliana · 2026-04-17T19:07:24Z

Summary

Port of Redliana/cmm-trade-mcp@b85ebcd into the packages/usitc-mcp/ copy
Resolves the documented step-4 validation failure that made get_trade_data raise USITCAPIError whenever an HTS code or country filter was supplied

Key payload corrections (vs. initial implementation)

commoditySelectType / countriesSelectType = "list" (not "entered")
aggregation labels stay at "Aggregate Commodities" / "Aggregate countries"
commoditiesExpanded / countriesExpanded now carry TextValuePair objects mirroring the code arrays
commoditiesManual now set to the comma-joined code string when any HTS filter is present
groupGranularity pinned to "2"

Verification

ruff format --check packages/ passes
ruff check packages/ passes (All checks passed!)
pytest packages/usitc-mcp/tests — 5 passed (including 2 new envelope-shape assertions)
Live probe with a valid USITC_API_TOKEN:
- Unfiltered 2023 US general imports: $3,076,796,376,327 (unchanged)
- Filtered HTS 2836.91.00 (lithium carbonate) 2023 imports: $293,083,540 — previously failed with step-4

🤖 Generated with Claude Code

Port of Redliana/cmm-trade-mcp@b85ebcd. Fixes the known "step 4" validation failure when commoditySelectType was set to "entered" with non-empty commodity or country filters. DataWeb expects: - commoditySelectType / countriesSelectType = "list" (not "entered") - aggregation labels stay at "Aggregate Commodities" / "Aggregate countries" - commoditiesExpanded / countriesExpanded must carry TextValuePair objects mirroring the commodities/countries arrays - commoditiesManual must be the comma-joined code string when any HTS filter is set - groupGranularity stays at "2" Live probe confirms HTS-filtered imports now return data (HTS 2836.91.00 lithium carbonate 2023 imports = \$293.1M). Adds unit tests asserting the envelope shape so the validated payload doesn't regress silently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Ports a validated USITC DataWeb “SavedQuery” envelope shape into packages/usitc-mcp, fixing HTS/country-filtered get_trade_data requests that previously failed server-side validation.

Changes:

Fixes USITC /report2/runReport payload construction for HTS/country filters (list-mode select types, expanded objects, manual string, pinned group granularity).
Adds USITC unit tests asserting the corrected envelope shape for HTS and country filtering.
Includes several small typing/robustness adjustments across other packages (casts for model_dump()/JSON access, World Bank year parsing, loader signature alignment).

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
packages/usitc-mcp/src/usitc_mcp/client.py	Adjusts SavedQuery payload fields to match validated DataWeb templates for filtered queries.
packages/usitc-mcp/tests/test_client.py	Adds assertions for the corrected filtered-query envelope shape.
packages/worldbank-mcp/src/worldbank_mcp/client.py	Makes year parsing tolerant of non-integer `date` values.
packages/fred-mcp/src/fred_mcp/server.py	Adds a cast around `model_dump()` to satisfy typing expectations of tool return values.
packages/fred-mcp/src/fred_mcp/client.py	Adds casts for response list shapes (`categories`, `seriess`).
packages/cmm-fine-tune/src/cmm_fine_tune/training/config.py	Casts `model_dump()` output to a typed dict for downstream flattening.
packages/cmm-embedding/src/cmm_embedding/training/paired_data_loader.py	Tightens typing for tokenizer outputs and sampler variable.
packages/cmm-embedding/src/cmm_embedding/training/alignment_training.py	Tightens typing for scheduler and encoder outputs / lookup.
packages/cmm-data/src/cmm_data/utils/parsing.py	Switches NaN sentinel creation to `float("nan")`.
packages/cmm-data/src/cmm_data/loaders/osti_docs.py	Aligns loader signature with BaseLoader (`load(**kwargs)`) and hardens full_text extraction typing.
packages/arxiv-mcp/src/arxiv_mcp/server.py	Adds a cast around `response.text` to satisfy typing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-17T19:26:01Z

    if value is None or pd.isna(value):
-        return np.nan
+        return float("nan")



parse_numeric_value is annotated as returning float | None and its docstring says it returns None when parsing fails, but these branches now return float('nan'). This also makes the numpy as np import unused in this module. Consider either returning None on missing/unparseable values, or updating the return annotation/docstring to match the intended NaN-based behavior and remove the unused numpy import.

Copilot · 2026-04-17T19:26:01Z

+        # Commodity and country filtering are validated against the
+        # /savedQuery/getAllSystemSavedQueries payloads. The API expects
+        # list-based selection semantics, not "entered", and the aggregation
+        # labels are case-sensitive.
+        commodities_agg = "Aggregate Commodities"
+        commodities_select = "list" if hts_codes else "all"
+        countries_agg = "Aggregate countries"
+        countries_select = "list" if country_codes else "all"


This change fixes the SavedQuery payload to use list-based selection, but the module docstring and get_trade_data docstring still describe the previous 'step 4' validation failure with commoditySelectType='entered' and advise callers to drop HTS filters. Those docs/TODOs should be updated to reflect the new validated payload behavior so consumers aren't misled about expected failures or workarounds.

Copilot · 2026-04-17T19:26:02Z

+                try:
+                    year = int(raw_date) if raw_date is not None else 0
+                except (TypeError, ValueError):
+                    year = 0


The new year parsing now falls back to 0 when item['date'] is non-numeric. Previously, a non-numeric but truthy date would raise during int(...) and the whole record would be skipped by the outer except. Returning observations with year=0 can leak invalid data downstream; consider continue on invalid dates (or make year optional) to preserve prior semantics.

Suggested change

year = 0

continue

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-17T22:52:55Z

+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          version: "latest"


The workflow bumps actions/checkout and astral-sh/setup-uv to new major tags (v6/v7). Please confirm these tags exist and are intended; otherwise CI will fail at runtime. If the goal is supply-chain hardening, consider pinning actions to a commit SHA (or at least a known-good major version already used in this repo).

Suggested change

- uses: actions/checkout@v6

- name: Install uv

uses: astral-sh/setup-uv@v7

with:

version: "latest"

- uses: actions/checkout@v4

- name: Install uv

uses: astral-sh/setup-uv@v4

with:

version: "0.5.29"

Copilot · 2026-04-17T22:52:55Z

+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          version: "latest"


PR title/description focus on the USITC HTS-filter payload fix, but this PR also changes CI action versions (and multiple other packages). Either update the PR description to cover these extra changes or split the unrelated changes into separate PRs to keep review/rollback risk contained.

nmwashton and others added 2 commits April 17, 2026 14:07

Fix mypy failures across package workspace

744b5f7

Copilot AI review requested due to automatic review settings April 17, 2026 19:22

Copilot started reviewing on behalf of Redliana April 17, 2026 19:22 View session

Redliana self-assigned this Apr 17, 2026

Copilot AI reviewed Apr 17, 2026

View reviewed changes

nmwashton added 2 commits April 17, 2026 14:39

Fix lint regressions from mypy cleanup

9c51719

Update CI actions for Node 24 compatibility

2552347

Copilot AI review requested due to automatic review settings April 17, 2026 22:49

Copilot started reviewing on behalf of Redliana April 17, 2026 22:49 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Redliana merged commit 1ceb7c1 into main Apr 17, 2026
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: USITC HTS-filtered query payloads#8

fix: USITC HTS-filtered query payloads#8
Redliana merged 4 commits into
mainfrom
fix/usitc-hts-filter-payload

Redliana commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Redliana commented Apr 17, 2026

Summary

Key payload corrections (vs. initial implementation)

Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants