feat: add worldbank, fred, usitc, and census-aes MCP packages#6
Conversation
Adds four new MCP server packages extending CMM_Tools with public-sector economic and trade-statistics coverage: - worldbank-mcp: World Bank WDI indicators (no key) - fred-mcp: FRED time series (FRED_API_KEY) - usitc-mcp: USITC DataWeb US trade statistics (USITC_API_TOKEN) - census-aes-mcp: Census International Trade exports (CENSUS_API_KEY) Each package mirrors the uncomtrade-mcp layout (src/<snake>/, py.typed, hatchling build, server/client/models split). Registers the four new modules as known-first-party for Ruff isort. USITC client rewritten against a live bearer token: corrected BASE_URL verified, check_status now probes country/getAllCountries, search_hts now uses commodity/commodityDescriptionLookup, and get_trade_data constructs the real nested SavedQuery envelope for /report2/runReport and flattens the dto.tables[*].rowsNew pivot-table response into TradeRecord rows. Schema discovery notes captured in client.py module docstring; commodity-filter step-4 validation gap tracked as an inline TODO. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-reviewed after bc60a08 (parallelized fan-out queries, removed dead
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
| async def search_indicators(self, query: str, per_page: int = 50) -> list[dict[str, Any]]: | ||
| """Search the WDI indicator catalogue by free-text keyword.""" | ||
| params = {"per_page": min(per_page, 500)} | ||
| payload = await self._request("indicator", params=params) |
There was a problem hiding this comment.
search_indicators fetches at most 500 indicators from /v2/indicator and filters client-side, but WDI has ~1,600+ indicators. Anything outside the first page is unsearchable. Either paginate until all indicators are fetched (respecting the total field in payload[0]), or pass a search query parameter to the WDI API which does support server-side search (e.g., https://api.worldbank.org/v2/indicator?search=mining&format=json).
Fix it with Roo Code or mention @roomote and request a fix.
| _DATA_TO_REPORT = { | ||
| "general_imports": "GEN_CUSTOMS_VALUE", | ||
| "imports_for_consumption": "GEN_CUSTOMS_VALUE", | ||
| "domestic_exports": "FAS_VALUE", | ||
| "total_exports": "FAS_VALUE", | ||
| } |
There was a problem hiding this comment.
Both general_imports and imports_for_consumption map to the same GEN_CUSTOMS_VALUE code. This means compare_import_types in the server will return identical results for both data types, making it functionally useless. imports_for_consumption likely needs a distinct dataToReport value (the module docstring mentions CONS_VAL_YR was observed in the DataWeb UI). If the correct code isn't yet validated, the compare_import_types tool should at minimum warn users that results may be identical until this is resolved.
Fix it with Roo Code or mention @roomote and request a fix.
| TODO: commodity filtering via ``commoditySelectType='entered'`` is not | ||
| yet reliably accepted by the server (it raises a "step 4" validation | ||
| error for reasons not yet pinned down). When that happens this method | ||
| re-submits the query with ``commoditySelectType='all'`` and lets the | ||
| caller filter client-side. | ||
| """ | ||
| trade_type = _TRADE_TYPE.get(flow.lower()) | ||
| if not trade_type: | ||
| raise USITCAPIError(f"Unknown flow '{flow}'. Use 'import' or 'export'.") | ||
| dtr = _DATA_TO_REPORT.get(data_type) | ||
| if not dtr: | ||
| raise USITCAPIError( | ||
| f"Unknown data_type '{data_type}'. Use one of {list(_DATA_TO_REPORT)}." | ||
| ) | ||
|
|
||
| payload = self._build_saved_query( | ||
| trade_type=trade_type, | ||
| data_to_report=dtr, | ||
| years=years, | ||
| hts_codes=hts_codes, | ||
| country_codes=country_codes, | ||
| ) | ||
|
|
||
| try: | ||
| data = await self._post("report2/runReport", payload) | ||
| except httpx.HTTPError as e: | ||
| raise USITCAPIError(f"runReport failed: {e}") from e | ||
|
|
||
| dto = data.get("dto") or {} | ||
| errors = dto.get("errors") or [] | ||
| if errors: | ||
| # Server validation (step 2/step 4/etc). Surface one level up. | ||
| raise USITCAPIError(f"Query rejected: {'; '.join(errors)}") |
There was a problem hiding this comment.
The TODO/docstring here says the method "re-submits the query with commoditySelectType='all' and lets the caller filter client-side" on server rejection, but the actual implementation just raises USITCAPIError on any dto.errors. Either implement the documented fallback or update the docstring to reflect the current behavior (raise on validation errors). As-is, anyone reading the docstring will expect graceful degradation that doesn't exist.
Fix it with Roo Code or mention @roomote and request a fix.
| await asyncio.sleep(0.1) | ||
| records = await client.get_observations( | ||
| series_id=series_id, | ||
| observation_start=observation_start, | ||
| observation_end=observation_end, | ||
| limit=5000, | ||
| ) | ||
| latest = None | ||
| for r in reversed(records): | ||
| if r.value is not None: | ||
| latest = {"date": r.date, "value": r.value} | ||
| break | ||
| metadata = await client.get_series_metadata(series_id) |
There was a problem hiding this comment.
The 0.1s sleep is only at the start of each loop iteration, but each iteration makes two API calls (get_observations + get_series_metadata). With ~15 curated series that's ~30 requests with an effective average gap of ~0.05s, which can exceed the FRED rate limit of 120 requests per 60 seconds if the category filter is removed or the curated set is expanded. Moving the sleep between each API call (or adding a second sleep before get_series_metadata) would halve the request density.
| await asyncio.sleep(0.1) | |
| records = await client.get_observations( | |
| series_id=series_id, | |
| observation_start=observation_start, | |
| observation_end=observation_end, | |
| limit=5000, | |
| ) | |
| latest = None | |
| for r in reversed(records): | |
| if r.value is not None: | |
| latest = {"date": r.date, "value": r.value} | |
| break | |
| metadata = await client.get_series_metadata(series_id) | |
| await asyncio.sleep(0.1) | |
| records = await client.get_observations( | |
| series_id=series_id, | |
| observation_start=observation_start, | |
| observation_end=observation_end, | |
| limit=5000, | |
| ) | |
| latest = None | |
| for r in reversed(records): | |
| if r.value is not None: | |
| latest = {"date": r.date, "value": r.value} | |
| break | |
| await asyncio.sleep(0.1) | |
| metadata = await client.get_series_metadata(series_id) |
Fix it with Roo Code or mention @roomote and request a fix.
Run ruff format on the four new packages to satisfy the CI 'Format check' step (uv run ruff format --check packages/). Also replaces an ambiguous Unicode multiplication sign in a worldbank-mcp docstring to clear RUF002 in the Lint step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review follow-up. Three cleanups: - usitc-mcp: drop `aggregate_by` param from get_trade_data (client and MCP tool). It was advertised as accepting year|month|quarter but DataWeb only runs annual; the parameter was plumbed through without effect and leaked into response echo. Update the module-level TODO to match actual behavior (raises on step-4 instead of silently re-submitting). - usitc-mcp: run compare_import_types' two queries in parallel via asyncio.gather — 2x latency win for a tool that always issues both. - fred-mcp & worldbank-mcp: replace serial CMM-bundle loops (dashboard and country profile) with asyncio.Semaphore(5) + gather. Preserves rate-limit intent from the prior 0.1s pacing while cutting wall time for 10+ series/indicator fan-outs by roughly an order of magnitude. Also removes a decorative ASCII divider comment in usitc-mcp/client.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reviewed at bc60a08. 1 new issue found; 2 previously flagged issues remain unresolved.
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
| DEFAULT_EXPORT_VARS = [ | ||
| "E_COMMODITY", | ||
| "E_COMMODITY_LDESC", | ||
| "ALL_VAL_MO", | ||
| "ALL_VAL_YR", | ||
| "CTY_CODE", | ||
| "CTY_NAME", | ||
| "UNIT_QY1", | ||
| "QTY_1_MO", | ||
| "UNIT_QY2", | ||
| "QTY_2_MO", | ||
| ] |
There was a problem hiding this comment.
DEFAULT_EXPORT_VARS (and DEFAULT_IMPORT_VARS) do not include MONTH. When month is None, MONTH is neither in the get list nor added as a predicate filter, so the Census API won't return a MONTH column in the response. row.get("MONTH") then returns None, and every record gets month=0 via month_fallback. This means full-year queries (no month filter) return data for all 12 months but every record loses its actual month -- they all read month=0. Adding "MONTH" to both DEFAULT_EXPORT_VARS and DEFAULT_IMPORT_VARS (or appending it to the get param explicitly) would fix this.
| DEFAULT_EXPORT_VARS = [ | |
| "E_COMMODITY", | |
| "E_COMMODITY_LDESC", | |
| "ALL_VAL_MO", | |
| "ALL_VAL_YR", | |
| "CTY_CODE", | |
| "CTY_NAME", | |
| "UNIT_QY1", | |
| "QTY_1_MO", | |
| "UNIT_QY2", | |
| "QTY_2_MO", | |
| ] | |
| DEFAULT_EXPORT_VARS = [ | |
| "E_COMMODITY", | |
| "E_COMMODITY_LDESC", | |
| "ALL_VAL_MO", | |
| "ALL_VAL_YR", | |
| "CTY_CODE", | |
| "CTY_NAME", | |
| "UNIT_QY1", | |
| "QTY_1_MO", | |
| "UNIT_QY2", | |
| "QTY_2_MO", | |
| "MONTH", | |
| ] |
Fix it with Roo Code or mention @roomote and request a fix.
2 participants
Adds four new MCP server packages extending CMM_Tools with public-sector economic and trade-statistics coverage:
Each package mirrors the uncomtrade-mcp layout (src//, py.typed, hatchling build, server/client/models split).
Registers the four new modules as known-first-party for Ruff isort.
USITC client rewritten against a live bearer token: corrected BASE_URL verified, check_status now probes country/getAllCountries, search_hts now uses commodity/commodityDescriptionLookup, and get_trade_data constructs the real nested SavedQuery envelope for /report2/runReport and flattens the dto.tables[*].rowsNew pivot-table response into TradeRecord rows. Schema discovery notes captured in client.py module docstring; commodity-filter step-4 validation gap tracked as an inline TODO.