Implement system maintenance metadata refresh console#197
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a safety-first “System Maintenance” subsystem to refresh market metadata (index constituents + market events) with a dry-run/apply workflow, validation, backups, and JSON/Markdown reporting, exposed via a new web console, API endpoints, and CLI entrypoint (first slice of Issue #196).
Changes:
- Introduces
web/maintenance/runner, task/report models, validators, and source adapters for S&P 500 / STI / HSI, plus report persistence. - Adds the
/maintenancedashboard page and/api/maintenance/*endpoints to run tasks and view status/reports. - Routes legacy refresh scripts through the new runner; adds docs and a dedicated test module.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| web/templates/maintenance/index.html | New Maintenance console UI with buttons for dry-run/apply and status/report panels. |
| web/routes/maintenance.py | Registers the server-rendered /maintenance page route. |
| web/routes/api_maintenance.py | Adds maintenance status/run/report API endpoints for the UI/CLI. |
| web/maintenance/init.py | Introduces the maintenance package. |
| web/maintenance/main.py | CLI entrypoint (python -m web.maintenance) for running/validating tasks. |
| web/maintenance/tasks.py | Task/result/report datamodels for serialized output and report generation. |
| web/maintenance/validators.py | Constituent validation rules (required columns, counts, duplicates, symbol formats, change thresholds). |
| web/maintenance/runner.py | Orchestrates tasks, performs dry-run/apply behavior, backups, atomic writes, cache invalidation, and report writing. |
| web/maintenance/reports.py | Writes/reads JSON + Markdown reports; lists recent report summaries. |
| web/maintenance/sources/init.py | Source adapter package marker. |
| web/maintenance/sources/sp500.py | Fetches/normalizes S&P 500 constituents from Wikipedia. |
| web/maintenance/sources/sti.py | Fetches/normalizes STI constituents from Wikipedia with resilient column detection. |
| web/maintenance/sources/hsi.py | Fetches/normalizes HSI constituents from Wikipedia via requests + read_html parsing. |
| web/init.py | Registers maintenance page and API blueprints in the Flask app factory. |
| tests/test_maintenance.py | Adds unit/integration tests for validation and safe dry-run/apply behaviors and basic page/API loading. |
| scripts/refresh_sp500_constituents.py | Replaces legacy script implementation with a runner-based compatibility wrapper. |
| deployment_scripts/refresh_hsi_constituents.py | Replaces legacy script implementation with a runner-based compatibility wrapper. |
| docs/MAINTENANCE.md | Documents safety boundaries, UI/CLI usage, validation rules, reports, and cadence. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+105
to
+110
| def finalize(self, *, started_monotonic: float, now_monotonic: float) -> None: | ||
| self.finished_at = utc_now_iso() | ||
| self.duration_seconds = round(max(0.0, now_monotonic - started_monotonic), 3) | ||
| self.warnings = [w for r in self.results for w in r.warnings] | ||
| self.errors = [e for r in self.results for e in r.errors] | ||
| if any(r.status == STATUS_FAILED for r in self.results): |
Comment on lines
+56
to
+61
| dry_run = bool(payload.get("dry_run", True)) | ||
| days_ahead = _bounded_int(payload.get("days_ahead"), default=28, minimum=1, maximum=90) | ||
| event_symbols = _csv_or_list(payload.get("event_symbols")) | ||
| include_portfolio = bool(payload.get("include_portfolio_symbols", True)) | ||
| if include_portfolio: | ||
| event_symbols.extend(_portfolio_symbols()) |
Comment on lines
+124
to
+131
| path = self.data_dir / str(cfg["filename"]) | ||
| row_count = len(_read_csv_rows(path)) if path.exists() else 0 | ||
| mtime = datetime.fromtimestamp(path.stat().st_mtime, timezone.utc).isoformat() if path.exists() else None | ||
| validation = validate_constituent_rows( | ||
| _read_csv_rows(path), | ||
| market=str(cfg["market"]), | ||
| before_count=row_count, | ||
| ) if path.exists() else ValidationResult(status=STATUS_FAILED, errors=["File not found"]) |
Comment on lines
+110
to
+113
| else: | ||
| result = MaintenanceTaskResult(task=task_name, status=STATUS_FAILED, dry_run=dry_run) | ||
| result.errors.append(f"Unknown maintenance task: {task_name}") | ||
| result.finished_at = _utc_now().isoformat() |
- tasks.py: Include validation.errors in finalize() error aggregation - runner.py: Check status==failed before total_errors in _run_market_events_task - runner.py: Read CSV rows once in get_status() to avoid redundant I/O - runner.py: Use microseconds in backup dir timestamp to avoid collisions - runner.py: Call result.finish() for unknown tasks instead of setting finished_at directly - api_maintenance.py: Only expand portfolio symbols when market_events task is requested - api_maintenance.py: Remove raw exception string from 500 response
…_dir to backup_timestamp_dir
This was referenced Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the first full slice of #196: System Maintenance: Market Metadata Refresh & Data Hygiene.
This PR adds a metadata-only maintenance subsystem for safely refreshing market universe metadata and exposing it through CLI, API, and a dashboard page.
What changed
web/maintenance/package:python -m web.maintenance/maintenanceGET /api/maintenance/statusPOST /api/maintenance/runGET /api/maintenance/reportsGET /api/maintenance/reports/<report_id>scripts/refresh_sp500_constituents.pydeployment_scripts/refresh_hsi_constituents.pydocs/MAINTENANCE.mdSafety / Prime Directive notes
data.market_events.Validation behavior
Constituent refreshes validate before replacement:
Apply mode backs up existing files first and invalidates relevant screener caches after successful replacement.
Testing
I added
tests/test_maintenance.pyfor the new module.I could not run the test suite from this chat environment because the repository is only accessible through the GitHub connector here, not as a local checkout with dependencies installed. Please run:
Notes / follow-up candidates
/maintenance; if desired, a follow-up can add it directly into the navbar under Monitoring.Closes #196