diff --git a/docs/development/current-state-audit.md b/docs/development/current-state-audit.md new file mode 100644 index 0000000..70d9d48 --- /dev/null +++ b/docs/development/current-state-audit.md @@ -0,0 +1,189 @@ +# UrbanPy current-state audit for the 0.3 roadmap + +Audit date: 2026-08-09 + +This is a planning baseline, not a claim that every defect has been reproduced on +every platform. Repository facts were inspected locally at commit `ddd0360`; live +GitHub/PyPI/tool facts should be refreshed when implementation begins. + +## Executive assessment + +UrbanPy has a useful domain scope and recent performance improvements, but the +current default branch is not release-ready. The highest risks are the unprotected +merge path, nondeterministic live-service tests, release credential/workflow +history, unresolved dependency-license status, legacy packaging, and an OSRM path +that can report success without a usable server. + +The 0.3 roadmap is viable if work is sequenced as stabilization and release-chain +hardening first, then packaging/typing/OSRM migration, then public API and release +polish. Treating uv, Trunk, Pydantic, or an AI reviewer as a standalone solution +would leave the main production risks untouched. + +## Repository baseline + +- Default branch: `master` at `ddd0360` (merged PR #59). +- Package code plus tests: approximately 3,425 Python lines in the current tree. +- Packaging: legacy `setup.py`; package version is rewritten during release; + `python_requires` still says Python 3.6 or newer. +- Dependency input: `requirements.txt` pins more than 100 direct and transitive + packages together. `environment.yml` and docs requirements are additional, + inconsistent sources. +- Latest OSMnx metadata requires Python 3.11 or newer, while current CI tests 3.9, + 3.10, and 3.11. The declared and tested support policies do not agree. +- PyPI's latest stable release is 0.2.1 from 2021; development releases appeared + in 2023. Repository tags, PyPI releases, and source versioning need reconciliation. +- `urbanpy.__init__` eagerly imports all feature modules. Optional feature imports + therefore cannot be optional, and HDX configuration is created at import time. +- `urbanpy/routing/osrm_routing.py` duplicates functionality in `routing.py` but is + not part of the documented export path. +- Bash and PowerShell files are tracked but are not included as package data by the + current wheel configuration. +- Pydantic is pinned in the dependency file but not used by UrbanPy; it appears as + dependency-tree baggage rather than an intentional contract. + +## Pull-request and automation baseline + +Current repository workflows visibly provide: + +- test/docs/release/deploy workflow; +- SonarQube scan; +- CodeQL; +- GitGuardian; +- organization-level License Compliance status; +- Copilot pull-request review. + +Findings: + +- The organization ruleset requires a PR and blocks deletion/non-fast-forward + updates, but currently requires zero approvals, no thread resolution, and no + successful status checks. +- The latest inspected PR (#60) is unstable: Python build jobs fail, while docs, + SonarQube, CodeQL, and GitGuardian pass. +- Six geometry tests fail because live Nominatim requests return HTTP 403; the + response body is then parsed as JSON. Other tests call live HDX and OSMnx/Overpass. +- The OSRM test is deselected in CI rather than replaced by unit/contract coverage. +- The same PR received duplicate test/docs runs from overlapping push and PR events. +- The License Compliance status reports 104 issues and has no result URL. The exact + findings require owner/tool investigation; the failure must not be waived blindly. +- GitHub reports 42 open Dependabot vulnerabilities on the default branch (30 high, + 9 moderate, and 3 low at audit time). Most originate from the monolithic pinned + `requirements.txt`; one high-severity alert affects the SonarQube scan Action and + requires upgrading to a fixed release as part of Phase 0. +- SonarQube dependency installation and coverage are marked best-effort, failures + are suppressed, and `sonar.sources=.` makes source/test/notebook scope ambiguous. +- The scan job can pass without proving that tests or coverage succeeded and does + not explicitly wait for the quality gate in repository configuration. +- CI still uses old major Actions and several floating `@master`/`@latest` action + references. Current runner logs warn about forced Node runtime compatibility. +- The combined workflow gives release/deploy logic a very large blast radius and + mutates source files to create a version. + +## Test and API baseline + +- The current suite collects roughly 15 tests. Most geometry tests and the HDX + test depend on changing public data/services. +- Tests include a maintainer's personal email address as Nominatim identification. +- There are placeholder/pass tests for important Google and ORS behavior. +- Assertions sometimes compare `.all()` booleans instead of numeric arrays and + several expected live results are inherently unstable. +- HTTP calls generally omit explicit timeouts and consistent status handling. +- Nominatim currently uses `requests`' default User-Agent, contrary to public + service identification requirements, which explains the observed 403 risk. +- Broad `except Exception`, printing, and mixed sentinels (`None`, `-1`, `NaN`, + raw `Response`) make failure behavior hard to automate or type safely. +- Coordinate order, CRS, and seconds/minutes are not consistently encoded in names + or types. Some docstrings and examples disagree with current functions. +- Public exports are controlled by star imports and module `__all__` lists, with no + compatibility snapshot or explicit stability classification. + +## OSRM baseline + +- The API requires hand-authored `country` and `continent` fragments. +- Geofabrik nested paths and slash-containing canonical IDs are unsupported. +- The official Geofabrik v1 catalog already supplies unique IDs, parents, ISO + aliases, and complete PBF URLs with a versioned stable structure. +- Bash and PowerShell implementations diverge and hard-code platform paths. +- Docker image, port, temporary names, sleeps, cleanup, and error output are not + safe for concurrent or automated use. +- The MLD command sequence uses the PBF filename for partition/customize/routed, + rather than the prepared `.osrm` base shown by OSRM's supported flow. +- No manifest proves which PBF/profile/image created existing artifacts. + +## Security, licensing, and community baseline + +- Recent repository history includes removal of a malicious workflow and an open + PR to replace an exposed PyPI token. Incident closure and credential revocation + must be evidenced, not inferred from file deletion alone. +- Secret scanning and push protection are enabled, which is a useful existing + control to preserve. +- GPL-3 is stated, but package metadata and an EL-BID-approved precise SPDX/license + compatibility policy are not present. +- There is no `SECURITY.md`, CODEOWNERS policy, support policy, or release ownership + document in the inspected tree. +- README and Sphinx docs still recommend Python 3.6-era environments and old + operating systems; README also contains obsolete analytics beacon markup. +- Citation exists as BibTeX text but not `CITATION.cff`. +- Contribution guidance references an internal repository/automatic merge process + that does not match the visible public workflow. + +## Existing issue/PR reconciliation + +| Existing item | 0.3 disposition | +| --- | --- | +| PR #60 trusted publishing | Phase 0; fix CI, verify credential revocation, then merge/supersede | +| PR #58 Sonar config | Triage as likely superseded by merged Sonar files; preserve intent | +| PRs #54–#56 dependency bumps | Re-evaluate after direct-dependency/uv migration; do not merge stale full-lock pins blindly | +| PR #46 satellite embeddings | Separate feature review; not on critical path by default | +| #21 nested Geofabrik regions | Absorb into OSRM catalog resolver | +| #26 Windows OSRM | Absorb into one cross-platform Python manager | +| #29 missing OSRM scripts | Close by eliminating script-owned logic and wheel-test the replacement | +| #38 latest H3 incompatibility | Phase 2 compatibility gate | +| #20 OSMnx nearest-node API | Phase 2 compatibility gate | +| #27 CRS on generated frames | Phase 2 correctness gate | +| #19 and #52 HDX failures | Shared typed HTTP/client boundary plus regression fixtures | +| #14 merge reuse bug | Reproduce and fix in Phase 2 | +| #15 download API compatibility | Address through versioning, release notes, and deprecation policy | +| #16 Conda distribution | Submit to conda-forge after stable PyPI 0.3, not a private release job | +| #28 notebook docs | Convert maintained examples; archive stale outputs | +| #30 masked HDX retrieval | Feasibility decision; defer if upstream format prevents real savings | +| #31 progress feedback | Presentation adapter after operations expose progress events | + +## Plan changes resulting from this review + +The original plans were directionally sound but incomplete. This review adds: + +- a Phase 0 stabilization/security gate before new architecture; +- supply-chain, license, SBOM, credential, and provenance work as a separate + workstream; +- exact branch-rules and human-review requirements; +- a vendor-neutral, least-privilege operating model for remote AI agents; +- a measured CodeRabbit/Copilot review pilot rather than an automatic required bot; +- explicit Trunk adoption criteria and standard-tool fallback; +- external-service compliance and scheduled live contracts; +- direct dependency/optional import architecture and Python 3.11+ recommendation; +- correct OSRM `.osm.pbf` to `.osrm` sequencing, manifests, atomic downloads, + locking, readiness, labeling, and safe cleanup; +- public versus internal Pydantic model taxonomy, schema stability, redaction, + static typing, and migration rules; +- measurable phase gates, issue Definition of Ready, PR Definition of Done, and + post-release support. + +## Primary external references + +- uv project locking/syncing: +- uv package build/publish: +- Geofabrik catalog schema: +- OSRM backend: +- Pydantic documentation: +- SonarQube quality gates: + +- GitHub rulesets: + +- GitHub sub-issues and dependencies: + +- Nominatim usage policy: + +- CodeRabbit plans: +- Greptile pricing: +- conda-forge package contribution: + diff --git a/docs/development/osrm-0.3.md b/docs/development/osrm-0.3.md new file mode 100644 index 0000000..56bd9e9 --- /dev/null +++ b/docs/development/osrm-0.3.md @@ -0,0 +1,334 @@ +# OSRM and Geofabrik plan for UrbanPy 0.3 + +## Outcome + +Provide a cross-platform, observable OSRM lifecycle that downloads the correct +Geofabrik extract, prepares it reproducibly, and starts or stops a local service +without embedded Bash or PowerShell implementations. + +This epic closes or supersedes issues #21, #26, and #29. It also removes the +duplicate lifecycle implementation in `urbanpy/routing/osrm_routing.py`. + +## Current defects to eliminate + +- URLs are assembled from `continent` and `country`, which cannot represent the + Geofabrik hierarchy reliably. +- scripts are not included in installed wheels even though Python calls them; +- Bash and PowerShell implement different paths and behavior; +- the current MLD sequence passes `.osm.pbf` to partition/customize/routed where + OSRM's prepared `.osrm` dataset base is required; +- the image is floating, processing container names are global, and port 5000 is + hard-coded; +- slash-containing region IDs are unsafe as filesystem/container names; +- failures are often printed and then reported as success; +- downloads are not atomic or resumable and concurrent preparation is unsafe; +- there is no disk-space preflight, readiness deadline, structured status, + ownership label, or targeted cleanup policy. + +## Architecture decisions + +- Python owns orchestration. Shell files, if retained temporarily, only invoke the + public Python CLI and contain no lifecycle logic. +- Docker commands are argument arrays executed with `shell=False`. +- Use the official OSRM backend image pinned to an explicit supported version and + record the resolved digest in the preparation manifest. Validate supported CPU + architectures before selecting the tag. +- Use Geofabrik's `index-v1-nogeom.json` as the catalog and `urls.pbf` verbatim. + UrbanPy never constructs a PBF URL from region path fragments. +- The exact Geofabrik `properties.id` is canonical, including `/`. ISO 3166-1 and + ISO 3166-2 codes are lookup aliases. Display names are search text only. +- Silent fuzzy matching is prohibited. Ambiguity raises an error containing + candidate IDs and parents. +- Store catalog cache, downloaded PBFs, prepared datasets, state, and logs in + platform-appropriate locations with explicit user overrides. +- Library methods return typed results and raise typed exceptions. Rendering + progress and messages is a CLI/notebook adapter concern. +- Prepared data is immutable for a specific region, profile, PBF identity, OSRM + image, and algorithm. Reuse requires an exact manifest match. + +## Proposed package layout + +The final layout can vary, but responsibilities must remain separated. + +```text +urbanpy/ + geofabrik/ + catalog.py # fetch/cache/index/resolve + models.py # internal transport plus public region model + routing/ + models.py # OSRM configuration/status/results + osrm.py # high-level manager + osrm_client.py # route/table HTTP client, independent of Docker + _docker.py # injectable command runner + _download.py # streamed atomic downloads + _paths.py # platform paths and safe resource identifiers + cli/ + osrm.py # presentation and exit codes +``` + +No network, Docker, or filesystem work occurs at import time. + +## Public contracts + +### Geofabrik catalog + +`GeofabrikCatalog` provides: + +- `get(region_id)` for an exact canonical ID; +- `resolve(value, parent=None)` with priority exact ID, exact ISO code, then exact + case-folded display name only when unambiguous; +- `search(text, parent=None)` returning candidates without choosing one; +- `list(parent=None, downloadable_only=True)`; +- `refresh(force=False)` and explicit offline behavior. + +`GeofabrikRegion` includes canonical ID, display name, parent ID, ISO aliases, +public PBF URL, and optional ancestry derived from the catalog. It does not expose +the internal-authenticated `pbf-internal` URL. + +Catalog validation checks the v1 FeatureCollection envelope, unique IDs, valid +parent references, expected list-valued ISO fields, and HTTPS public URLs. Unknown +third-party fields are tolerated for forward compatibility. The cache honors ETag +and Last-Modified, has an explicit freshness policy, and can fall back to the last +valid copy with a surfaced stale status. Corrupt or partially written catalogs are +never promoted. + +### OSRM configuration + +`OSRMConfig` contains at least: + +- `region_id`; +- `profile`: car, bicycle, or foot; +- `algorithm`: MLD for 0.3; +- host bind address (loopback by default) and host port; +- data/cache path overrides; +- official image reference; +- download, command, and readiness timeouts; +- refresh/rebuild policy; +- optional resource limits supported consistently by Docker. + +Profile configuration distinguishes the Lua preparation profile from the OSRM API +profile token. The mapping is explicit and tested. + +### OSRM manager + +`OSRMManager` provides: + +- `plan()` — resolve inputs and return paths, sizes where known, commands, and + reuse decisions without mutation; +- `prepare()` — download and build a complete prepared dataset; +- `start()` — start or reuse a service and wait for readiness; +- `status()` — return container, manifest, endpoint, and readiness information; +- `stop()` — stop only the matching UrbanPy-owned service; +- `logs()` — stream or read logs without hiding subprocess failures; +- `clean()` — remove explicitly selected UrbanPy-owned containers/artifacts. + +The manager is also a context manager. Context exit stops a container it started +but does not delete downloaded or prepared data unless explicitly requested. + +### OSRM HTTP client + +`OSRMClient` is independent of Docker and accepts any configured OSRM base URL. It +provides typed route/table/nearest operations, explicit timeouts, status/error +translation, response-size limits, and unit-preserving results. Existing routing +functions delegate to this client, which allows the same UrbanPy API to use a +managed local service or an externally operated OSRM endpoint. + +## Lifecycle + +### 1. Resolve and plan + +- Resolve the exact catalog region and use its `urls.pbf`. +- Derive a Docker/filesystem-safe identifier from a readable slug plus a stable + short hash of the canonical ID. Never use a raw `/` in a resource name. +- Resolve image tag/digest, profile mapping, storage paths, and port. +- Inspect existing manifests and containers through exact labels, not substring + matches in `docker ps` output. + +### 2. Preflight + +- Check Docker CLI presence, daemon access, image architecture, writable storage, + port availability, and a conservative free-space requirement. +- Explain Docker-not-installed, daemon-not-running, permission, port, disk, and + unsupported-platform errors separately. +- Ask the CLI user to confirm unexpectedly large downloads/processing estimates + unless a non-interactive confirmation flag is supplied. + +### 3. Download + +- Stream to a `.part` file with progress, connect/read timeouts, and bounded + retries. +- Use ETag/Last-Modified and `If-Range` for safe resume when supported. Restart a + partial download if the remote identity changed. +- Enforce HTTPS, expected Geofabrik public host policy, response-size limits, and + available disk space. +- Atomically rename only after the response is complete and validation succeeds. +- Record source URL, validators, byte size, retrieval time, and local digest in a + manifest. Do not claim upstream checksum verification unless an authoritative + checksum was actually retrieved and verified. + +### 4. Prepare + +- Acquire a per-dataset inter-process lock. +- Run `osrm-extract` with the selected Lua profile against `.osm.pbf`. +- Run `osrm-partition` and `osrm-customize` against the resulting `.osrm` base. +- Use unique, labeled, automatically removed processing containers so an + interrupted run cannot collide with another region/profile. +- Capture stdout/stderr to structured logs while preserving live progress. +- Build in a staging directory and atomically publish the complete prepared set + plus manifest. A failed run never looks reusable. + +### 5. Start and verify + +- Run a detached, labeled service container with a read-only data mount and the + selected host bind/port. Bind to `127.0.0.1` by default; external exposure is an + explicit opt-in with a security warning. +- Use an exact, collision-safe name incorporating region/profile identity. +- Poll until the container is running and the OSRM HTTP endpoint returns a + well-formed OSRM response, with a deadline and captured diagnostic logs. +- A timeout stops only the container created by that invocation and raises an + actionable readiness error. No unbounded busy loops or fixed success sleeps. + +### 6. Stop and clean + +- Find resources by exact UrbanPy ownership/manifest labels. +- `stop()` is idempotent and never removes data. +- `clean()` requires explicit scopes such as container, prepared profile, PBF, or + catalog cache; dry-run is available and default for broad selections. +- Never remove an unknown container or directory merely because its name contains + `osrm`. + +## Compatibility strategy + +For 0.3, current functions remain available: + +```python +start_osrm_server(country, continent, profile) +stop_osrm_server(country, continent, profile) +``` + +They delegate to `OSRMManager`, resolve `country` as an exact catalog ID/alias, +validate legacy continent/path hints against ancestry where possible, and emit a +visible `FutureWarning`. Legacy underscore path workarounds may be recognized only +in this adapter and are never used to construct URLs. + +The preferred API is region-centric: + +```python +config = OSRMConfig(region_id="us/new-york", profile="foot", port=5001) +with OSRMManager(config) as server: + result = OSRMClient(server.endpoint).route(origin, destination) +``` + +If return values or units of `osrm_route`/`compute_osrm_dist_matrix` change, add a +new typed API first and keep legacy tuple/array behavior through the documented +deprecation window. + +## CLI + +An optional `urbanpy osrm` CLI uses the same manager: + +```text +urbanpy osrm regions search new-york +urbanpy osrm plan --region us/new-york --profile foot +urbanpy osrm prepare --region us/new-york --profile foot +urbanpy osrm start --region us/new-york --profile foot --port 5001 +urbanpy osrm status --region us/new-york --profile foot +urbanpy osrm logs --region us/new-york --profile foot +urbanpy osrm stop --region us/new-york --profile foot +urbanpy osrm clean --region us/new-york --profile foot --prepared --dry-run +``` + +Commands support non-interactive operation, machine-readable JSON output, stable +exit codes, Ctrl-C cancellation, and redaction of sensitive environment values. + +## GitHub sub-issues + +1. Write ADR and public lifecycle/error/compatibility contract. +2. Add Pydantic region/config/status models and safe identifiers. +3. Implement and fixture-test the Geofabrik v1 catalog parser/index. +4. Implement exact ID, ISO, ancestry, search, and ambiguity-aware resolution. +5. Implement conditional catalog caching and explicit offline/stale behavior. +6. Implement streamed atomic/resumable PBF downloads and manifests. +7. Implement platform paths, dataset locks, staging, and atomic publication. +8. Implement injectable Docker discovery, inspection, execution, and error mapping. +9. Implement preflight planning for daemon, architecture, port, storage, and reuse. +10. Implement correct pinned MLD extract/partition/customize preparation. +11. Implement labeled start, loopback binding, readiness, status, logs, and stop. +12. Implement a Docker-independent typed OSRM route/table/nearest HTTP client. +13. Implement dry-run and targeted cleanup safeguards. +14. Add CLI adapters, JSON output, progress, cancellation, and exit codes. +15. Delegate legacy functions and add tested deprecation warnings. +16. Remove duplicate module and orchestration scripts after compatibility lands. +17. Add unit, contract, subprocess, concurrency, and failure-injection tests. +18. Add opt-in Docker integration and cross-platform manual acceptance workflow. +19. Verify installed sdist/wheel behavior and publish operations/troubleshooting docs. + +Each sub-issue must define its exact public surface and test doubles. Do not combine +catalog resolution, Docker orchestration, and legacy removal in one PR. + +## Verification matrix + +### Hermetic PR tests + +- fixture regions: `peru`, nested `sul`, slash-containing `us/new-york`, a region + without ISO, multiple ISO aliases, missing PBF URL, invalid parent, duplicate ID, + malformed catalog, ambiguity, and stale cache; +- exact subprocess argument and label assertions for Linux, macOS, and Windows path + forms; +- `.osm.pbf` versus `.osrm` command sequencing; +- partial download resume/change, atomic failure, insufficient disk, occupied port, + missing Docker, daemon denial, image failure, command failure, readiness timeout, + Ctrl-C, concurrent prepare, and targeted cleanup; +- manifest-match reuse and mismatch-triggered rebuild; +- legacy signature/return/deprecation tests. + +PR tests require neither Docker nor network access. + +### Scheduled/manual integration + +- resolve and download a deliberately small Geofabrik extract; +- prepare it with the pinned image; +- start OSRM, call route/table endpoints, validate units, and stop it; +- rerun to prove idempotent reuse; +- inject a second profile/port to prove isolation; +- clean only resources created by the job. + +Automate Linux Docker integration. Record release-candidate manual evidence from +maintainers using Docker Desktop on macOS (including Apple Silicon where supported) +and Windows. A platform is not claimed as supported without evidence. + +## Acceptance criteria + +- `urls.pbf` from the official catalog is the only download URL source. +- Correct OSRM `.osm.pbf` to `.osrm` preparation is asserted in tests. +- macOS, Linux, and Windows use identical Python lifecycle logic. +- image, dataset, profile, and algorithm identity is reproducible in a manifest. +- concurrent and interrupted runs cannot publish corrupt prepared state. +- start succeeds only after bounded readiness verification. +- local services bind to loopback by default and the HTTP client works with both + managed-local and explicitly configured remote endpoints. +- stop/clean operations can affect only exact UrbanPy-owned resources. +- legacy users receive a working adapter and documented migration path. +- installed wheel behavior, not merely repository execution, passes smoke tests. +- SonarQube, deterministic CI, security, and dependency compliance pass. + +## Out of scope for 0.3 + +- hosting or operating a public OSRM service; +- silently fuzzy-matching region names; +- automatic planet-scale preparation; +- arbitrary custom Lua profiles; +- incremental Geofabrik replication updates; +- non-Docker container engines; +- packaging PBF or prepared graph data inside UrbanPy artifacts. + +## Authoritative references + +- Geofabrik technical catalog: + +- Geofabrik no-geometry catalog: + +- OSRM backend and Docker quick start: + +- OSRM HTTP API: + diff --git a/docs/development/typed-models-0.3.md b/docs/development/typed-models-0.3.md new file mode 100644 index 0000000..8870ac6 --- /dev/null +++ b/docs/development/typed-models-0.3.md @@ -0,0 +1,244 @@ +# Pydantic and safe data-type plan for UrbanPy 0.3 + +## Outcome + +Use Pydantic v2 and static annotations to make configuration, external service +payloads, and intentionally stable public results explicit and safe while +preserving native geospatial objects for computation. + +Pydantic is currently present only as one of many pinned packages in +`requirements.txt`; UrbanPy does not use it. If UrbanPy imports Pydantic directly, +it becomes a declared direct dependency with a tested version range. + +## Boundary-first policy + +Use Pydantic at boundaries where untrusted or serialized data enters or leaves: + +- public configuration and CLI input; +- HTTP request parameters and the external JSON fields UrbanPy consumes; +- durable cache records and OSRM preparation manifests; +- stable public result/configuration objects; +- parsing legacy dictionaries, tuples, and environment values. + +Keep these native in computational paths: + +- `GeoDataFrame` and `GeoSeries`; +- pandas and NumPy arrays/scalars; +- Shapely geometries; +- NetworkX/OSMnx graphs; +- rasters and other large domain arrays. + +Do not create one model per GeoDataFrame row or validate large arrays element by +element by default. Validate the container contract once—geometry type, CRS, +required columns, shape, coordinate conventions—and let the native library own +the bulk data. + +Static typing and runtime validation are complementary. All supported public +Python APIs receive precise annotations; only boundary data receives runtime +validation. + +## Model taxonomy and stability + +### Public domain models + +Place intentionally supported contracts under `urbanpy.models` or documented +feature namespaces. Examples: + +- `Coordinate`: named `longitude` and `latitude`, finite and range-checked; +- `BoundingBox`: `west`, `south`, `east`, `north`, with ordering and antimeridian + behavior explicit; +- `TravelProfile`: constrained values and service-specific mappings; +- `GeofabrikRegion`: canonical region identity, parent, ISO aliases, and public + PBF URL; +- `OSRMConfig`: region, profile, image, port, paths, timeouts, and policy; +- `OSRMStatus`: lifecycle state, endpoint, dataset/image identity, and diagnostics; +- `RouteResult`: explicit `distance_m` and `duration_s`, with optional geometry + only if a stable serialization is specified; +- matrix metadata describing row/column identity and units without wrapping the + underlying NumPy array. + +These models follow the normal compatibility/deprecation policy. Model field names, +defaults, JSON serialization, and generated schema are public API. + +### Internal transport models + +Third-party payload schemas live in internal client modules, for example +`urbanpy._clients.geofabrik`. They model only consumed fields, tolerate documented +additional upstream fields, and translate into UrbanPy domain models. They are not +re-exported and can evolve with upstream APIs without creating public API debt. + +### Lightweight typed structures + +Use `TypedDict`, `Protocol`, `TypeAlias`, `NamedTuple`, or a standard dataclass when +runtime validation/serialization is not needed. `TypeAdapter` can validate a +collection or `TypedDict` without inventing a public `BaseModel`. + +Do not make exceptions into Pydantic models. Define a small, stable UrbanPy +exception hierarchy that wraps transport and validation failures while preserving +safe structured context. + +## Validation conventions + +- Use Pydantic v2 APIs and `ConfigDict`; do not add v1 compatibility code. +- Use strict validation when coercion could conceal coordinate, unit, profile, + identifier, boolean, or port mistakes. Permit intentional convenience coercion + only in a named legacy/input adapter with tests. +- UrbanPy-owned configuration uses `extra="forbid"`. Third-party transport models + normally ignore unknown fields but require all fields UrbanPy actually uses. +- Prefer immutable/frozen value models where mutation would invalidate a manifest + or cache identity. +- Do not enable `arbitrary_types_allowed=True` globally. A native object accepted + without validation should stay outside the model or have a focused adapter. +- Reject non-finite coordinates and numeric values. State whether Z coordinates + are ignored, rejected, or preserved. +- Encode units in field names (`distance_m`, `duration_s`) and docs, not only prose. +- Coordinate tuple adapters must state and test order. Public models use named + fields to avoid `(lat, lon)` versus `(lon, lat)` ambiguity. +- CRS-bearing inputs use shared validation helpers and `pyproj.CRS` normalization; + a missing CRS is never silently assumed for an operation where it affects + correctness. +- Secrets use redacted representations and never appear in validation errors, + model dumps, logs, or `repr`. +- Validation errors report a stable error category, field path, expected form, + and remediation without echoing entire external payloads. +- JSON serialization is deterministic where used for manifests/cache keys. + +## API migration rules + +- Normalize a legacy input once at the boundary; internal code receives the new + model or native normalized type. +- New APIs may accept a model plus documented plain-Python conveniences. +- Avoid decorating every public function with `validate_call`; it changes error + semantics and adds overhead. Use explicit boundary adapters where behavior is + controllable and benchmarked. +- When replacing a tuple/dict result would break unpacking or equality, introduce + a parallel typed API first. Keep the legacy result adapter for the documented + deprecation window. +- Public deprecations use a visible warning category, correct `stacklevel`, docs, + tests, changelog entry, and a scheduled removal version. +- Pydantic `ValidationError` may be preserved as the cause, but the public API + raises a documented UrbanPy boundary/configuration exception where consistency + matters. +- Do not silently change coordinate order, CRS, distance unit, duration unit, or + null/sentinel behavior during model adoption. + +## Static typing plan + +- Select one stable CI type checker in an ADR. Mypy is the initial recommendation + because Pydantic maintains a v2 plugin; verify behavior with current geospatial + stubs before committing. +- Add `py.typed` only when annotations shipped to users are checked and supported. +- Start strictness with new model/client/OSRM modules, then migrate one existing + module at a time. Do not hide the entire current package behind blanket ignores. +- Add and pin needed stub packages in the type-check dependency group, not runtime. +- Use protocols at subprocess/HTTP boundaries to make fakes type-safe and avoid + coupling tests to concrete clients. +- Type-check code examples that claim to be supported. + +## Initial shared model slice + +The Pydantic epic should not block all OSRM work. Its first independently +deliverable slice is: + +1. `Coordinate` and `BoundingBox` plus native geometry/container validators; +2. `TravelProfile` and explicit OSRM API profile mapping; +3. `GeofabrikRegion` plus internal v1 catalog transport models; +4. `OSRMConfig`, manifest identity, and `OSRMStatus`; +5. base exceptions and safe validation-error translation. + +Freeze this slice only after the catalog and Docker manager consumers have been +implemented together in tests. Avoid designing unused generic models. + +## Module migration order + +1. **Geofabrik/OSRM:** best-defined new boundary and immediate production need. +2. **Shared HTTP configuration/errors:** timeouts, endpoints, identification, + retry/caching policy, and safe response metadata. +3. **Routing APIs:** coordinates, profiles, distance/duration results, matrix + dimensions/units, and provider responses. +4. **Download clients:** Nominatim, Overpass, and HDX request/response boundaries. +5. **Geometry/accessibility:** validate CRS, geometry kind, columns, resolution, + and numeric parameters without wrapping whole frames. +6. **Plotting:** validate small option/config inputs only if current errors justify + runtime models. + +Every module migration includes behavior characterization first. Existing defects +are not frozen as compatibility unless explicitly documented. + +## GitHub sub-issues + +1. Inventory public inputs/outputs, current sentinels, coordinate order, CRS, and + units across all exported functions. +2. Write ADR for runtime versus static typing and model stability boundaries. +3. Select Pydantic/Python/type-checker ranges and measure dependency impact. +4. Define exception, units, CRS, naming, serialization, and deprecation conventions. +5. Implement/test coordinate, bounds, profile, and safe URL primitives. +6. Implement internal Geofabrik v1 transport and public region model. +7. Implement OSRM configuration, manifest identity, status, and result models. +8. Add schema snapshots and generated schema documentation for public models. +9. Add public package typing baseline and type-safe HTTP/subprocess protocols. +10. Migrate routing boundaries while preserving legacy tuple/array adapters. +11. Migrate Nominatim/Overpass/HDX boundaries and typed service errors. +12. Add GeoDataFrame/GeoSeries precondition helpers for CRS, columns, and geometry. +13. Migrate geometry/accessibility boundaries module by module. +14. Benchmark validation/import/serialization cost on representative workloads. +15. Publish model/API migration guidance and deprecation schedule. + +Each implementation issue must name a real boundary and consumer. “Add models for +module X” is not sufficiently defined. + +## Verification + +- Unit tests cover valid, invalid, boundary, strict/coercion, redaction, + serialization, and round-trip behavior. +- Property-based tests cover finite coordinate ranges, bounding-box ordering, + region identifiers, ports, timeouts, and matrix dimensions. +- Fixture contract tests cover extra/missing/null/wrong-type third-party fields + without live network calls. +- Schema snapshots detect accidental field/default/serialization changes. +- Compatibility tests cover old signatures, tuple/array ordering, warnings, and + exception translation. +- Static type tests cover production modules and supported examples, including + negative/error examples where the checker supports them. +- Benchmarks compare import time, single-object validation, catalog parsing, and + representative large GeoDataFrame operations. Large frames receive no per-row + Pydantic pass by default. +- Built-wheel tests confirm `py.typed`, public imports, and generated schemas are + actually packaged. +- SonarQube continues to analyze migrated code and remains required. + +## Acceptance criteria + +- Every public model has a documented boundary, owner, consumer, stability level, + example, and schema test. +- Every internal transport model translates into a domain/native type and is not + accidentally re-exported. +- The OSRM/Geofabrik path has no unchecked dictionary traversal. +- Coordinates, CRS, distance units, duration units, and null behavior are explicit. +- Large geospatial objects incur no per-row Pydantic validation by default. +- Secrets and full external payloads cannot leak through validation errors/logs. +- Existing callers have a tested adapter and migration path for changed behavior. +- Static annotations, runtime behavior, documentation, and packaged schemas agree. +- Validation overhead stays within an agreed budget and is negligible relative to + network/geospatial work at the chosen boundaries. + +## Out of scope for 0.3 + +- replacing GeoPandas, pandas, Shapely, NumPy, or NetworkX domain objects; +- validating every dataframe row or array element; +- exposing raw third-party response models as permanent UrbanPy API; +- using Pydantic models merely to increase model count or typing coverage; +- environment-driven global settings without a demonstrated use case; +- changing all existing return values in one breaking migration. + +## Authoritative references + +- Pydantic models and configuration: +- Pydantic strict mode: + +- Pydantic performance guidance: + +- Pydantic mypy plugin: + +- PEP 561 typed packages: + diff --git a/docs/development/urbanpy-0.3-project.md b/docs/development/urbanpy-0.3-project.md new file mode 100644 index 0000000..01704c8 --- /dev/null +++ b/docs/development/urbanpy-0.3-project.md @@ -0,0 +1,437 @@ +# UrbanPy 0.3 production-readiness project + +## Mission + +Ship UrbanPy 0.3 as a reproducible, typed, tested, documented, and maintainable +geospatial library that EL-BID can safely recommend to external users and that +new contributors can improve without relying on maintainer-specific machines. + +This is the parent roadmap for the GitHub Project **UrbanPy 0.3 — Production +Readiness**. Large workstreams are parent issues. Work that can be reviewed and +verified independently is a sub-issue and normally one pull request. + +The evidence behind this roadmap is recorded in +[`current-state-audit.md`](current-state-audit.md). The OSRM and runtime-model +workstreams have dedicated plans: + +- [`osrm-0.3.md`](osrm-0.3.md) +- [`typed-models-0.3.md`](typed-models-0.3.md) + +## Release principles + +- SonarQube remains enabled and required for EL-BID compliance. +- Pull requests with human approval are the only path to the default branch. +- PR checks are deterministic and do not call live public geospatial services. +- The package declares direct dependencies; the development lock records a + reproducible environment. Published wheels do not pin every transitive package. +- Public API compatibility is intentional, documented, and tested. +- Runtime validation is concentrated at untrusted and serialized boundaries. +- Native GeoPandas, pandas, NumPy, Shapely, and NetworkX types remain native in + computational paths. +- AI agents may prepare branches and PRs, but may not approve, merge, change + repository security, or publish releases. +- 0.3 scope favors reliability over adding new data sources or algorithms. + +## Decisions to record before implementation + +Each decision becomes a short architecture decision record (ADR), with owner, +date, alternatives, consequences, and reversal plan. + +1. **Python support.** The recommended baseline is Python 3.11 through 3.14. + Current OSMnx releases require Python 3.11 or newer. Confirm wheel availability + for every direct dependency and operating system before finalizing it. +2. **Build and versioning.** Use `pyproject.toml`, `uv`, a PEP 517 build backend, + and one authoritative version source. Remove tag-time source rewriting. +3. **Dependency boundaries.** Decide which capabilities are base installation + requirements and which become extras, then remove eager imports that defeat + those boundaries. +4. **Runtime typing.** Adopt the boundary-first Pydantic policy in the dedicated + plan and select the static type checker used in CI. +5. **OSRM lifecycle.** Adopt the Python-owned, catalog-backed architecture in the + dedicated OSRM plan. +6. **External services.** Define common timeouts, identification, retries, + caching, errors, and live-contract testing for Nominatim, Overpass, HDX, ORS, + Google Maps, Geofabrik, and OSRM. +7. **License policy.** EL-BID confirms the precise project SPDX expression and + approves a dependency-license allowlist and exception process. Tool output is + evidence, not legal advice. +8. **Default branch.** Treat a possible `master` to `main` rename as a separate, + coordinated migration. It is not required to make 0.3 technically sound. + +## Workstreams and parent issues + +### W0. Project governance and issue triage + +**Outcome:** contributors can see the scope, decision owners, dependencies, and +release gates without reconstructing them from old issues. + +- Create the organization-level GitHub Project and the fields/views below. +- Create parent issues W0 through W10 and attach the dedicated plans. +- Link existing issues and PRs instead of creating duplicates. +- Add issue forms, a pull-request template, `CODEOWNERS`, `SECURITY.md`, + `SUPPORT.md`, and a lightweight governance/maintainer policy. +- Record ADRs for the eight decisions above. +- Define supported-version, deprecation, and vulnerability-response policies. +- Assign at least two humans to release-critical ownership where possible. + +### W1. Packaging, `uv`, and dependency architecture + +**Outcome:** users install a small, correct package; contributors reproduce CI +with documented `uv` commands. + +- Replace `setup.py`, the fully pinned runtime `requirements.txt`, and duplicated + metadata with PEP 621 metadata in `pyproject.toml`. +- Declare only direct runtime dependencies with justified lower/upper bounds. +- Define PEP 735 groups for test, lint/type, docs, security, and release tooling. +- Define extras only after import boundaries make them real, for example + `plotting`, `hdx`, `google`, and `cli`. +- Commit the cross-platform `uv.lock`; CI uses `uv sync --locked` and checks + `uv lock --check`. +- Add a `.python-version` for the contributor default without limiting the CI + compatibility matrix. +- Add `urbanpy.__version__`, complete package metadata/project URLs, classifiers, + and `py.typed` if the public typing contract is accepted. +- Stop importing every optional module and initializing HDX configuration during + `import urbanpy`. +- Build sdist and wheel with `uv build`; run metadata, contents, `pip check`, and + clean-install smoke tests on both artifacts. +- Verify that installed-wheel behavior—not only source-tree behavior—is tested. +- Retire `environment.yml` as a dependency authority. Keep a generated example + only if it adds user value. + +### W2. Deterministic CI, Trunk, and required quality gates + +**Outcome:** one stable set of checks answers whether a PR is mergeable. + +- Split the monolithic workflow into PR CI, SonarQube, documentation, scheduled + contracts, and release workflows with narrow triggers. +- Add concurrency cancellation, job timeouts, dependency caching, minimal token + permissions, and `persist-credentials: false` where checkout credentials are + unnecessary. +- Pin third-party Actions to immutable commit SHAs and automate reviewed updates. +- Prevent duplicate `push` and `pull_request` runs for the same contribution. +- Make one stable aggregate check (`CI / required`) represent the supported + Python matrix so branch rules do not depend on changing matrix job names. +- Run formatting/linting, static typing, hermetic unit tests with coverage, docs + with warnings as errors, and artifact checks. +- Perform a time-boxed Trunk Code Quality spike. If it remains suitable, use + Trunk locally and in CI to orchestrate pinned Ruff, format, Markdown/YAML, + ShellCheck/shfmt, actionlint, and other repository checks. Keep the standard + tool configurations authoritative so contributors and remote agents have a + direct `uv run` fallback. +- Do not enable Trunk Flaky Tests quarantine until tests are hermetic and the + team agrees that quarantine cannot conceal regressions. +- Preserve CodeQL, secret scanning/push protection, and applicable EL-BID checks. + +#### SonarQube hardening + +- Keep the SonarQube scan mandatory. +- Remove all `continue-on-error` and `|| true` behavior from required analysis. +- Produce coverage once in the test workflow and pass the exact artifact to the + scan; a missing or invalid report fails the job. +- Set `sonar.sources=urbanpy`, `sonar.tests=tests`, coverage paths, Python versions, + and focused exclusions instead of analyzing the entire repository as source. +- Enable quality-gate waiting or reliable PR decoration so a failed gate fails + the GitHub check. +- Confirm the EL-BID SonarQube edition/configuration supports the intended PR + analysis and record the quality gate applied to new code. + +#### Default-branch rules + +- Require one human approval, resolution of review threads, and approval after + the latest material push (or dismissal of stale approvals). +- Require `CI / required`, the SonarQube quality gate, security analysis, and the + approved license/vulnerability check from their expected GitHub Apps. +- Block deletion and force pushes and require PRs to be up to date or use a + tested merge queue. +- Keep AI review advisory; it never satisfies the human-approval requirement. + +### W3. Supply-chain security, licensing, and release credentials + +**Outcome:** the dependency and release chain has an auditable disposition. + +- Complete incident closure for the exposed PyPI credential and removed + malicious workflow: revoke/rotate credentials, review audit logs and release + history, document impact, and close the relevant security work. +- Merge or supersede PR #60 only after its workflow is green and trusted + publishing is configured end to end. +- Remove long-lived PyPI tokens; use a protected GitHub environment and OIDC + trusted publishing with `id-token: write` only in the publish job. +- Triage the current “License Compliance” failure instead of suppressing it. +- Resolve the 42 open Dependabot alerts observed at roadmap creation, prioritizing + the high-severity SonarQube scan Action alert and direct/runtime exposure. Do not + close alerts merely by deleting a manifest until replacement resolution is + locked, installed, tested, and rescanned. +- Generate a dependency inventory and CycloneDX SBOM from the `uv` resolution. +- Scan locked dependencies for known vulnerabilities and define severity/SLA and + exception-expiry policy. +- Review direct and transitive dependency licenses against the EL-BID-approved + policy; store machine-readable output and human decisions separately. +- Add OpenSSF-friendly metadata, `SECURITY.md`, dependency update automation, and + release provenance/attestations where supported. +- Remove obsolete analytics beacons and accidental personal identifiers from + tests and documentation. + +### W4. Safe, always-on agentic development + +**Outcome:** remote agents can continuously deliver reviewable work without +receiving release or administrative authority. + +- Add concise repository instructions describing bootstrap commands, architecture, + allowed test doubles, style, verification, and protected areas. +- Create an `agent:ready` issue label. Eligible issues must have bounded scope, + dependencies resolved, acceptance criteria, expected files/interfaces, and + exact verification commands. +- Agents work on short-lived branches and open draft PRs. They may not push to the + default branch, merge, approve, modify rulesets/secrets, or publish packages. +- Use least-privilege short-lived credentials; workflows from forks receive no + secrets and untrusted code never runs with `pull_request_target` privileges. +- Require PR bodies to identify generated changes, tests run, unresolved risks, + API/dependency effects, and the issue closed. +- Require CODEOWNER review for workflows, packaging/release, security, public + models, and compatibility shims. +- Measure agent PR rework, escaped defects, review latency, and abandoned work; + adjust task size and instructions based on evidence. + +#### AI code-review pilot + +- Continue to treat GitHub Copilot review as advisory while measuring its signal. +- Pilot one additional reviewer at a time to avoid duplicate noise. CodeRabbit is + the leading free candidate because public open-source repositories currently + receive its OSS review tier. Greptile's OSS program is limited to qualifying + MIT/Apache projects, so GPL-licensed UrbanPy would rely on its individual + Starter allowance rather than the OSS program. +- Run the pilot for at least 20 representative PRs. Track actionable findings, + false-positive rate, duplicate comments, time-to-review, data access, permissions, + and maintainer effort. +- Do not make an AI reviewer a required check. Human review, tests, SonarQube, + and deterministic static analysis remain authoritative. + +### W5. Test architecture and external-service reliability + +**Outcome:** PR failures indicate code regressions, while scheduled jobs detect +upstream service drift. + +- Divide tests into `unit`, `contract`, `integration`, `docker`, and `live` + markers with explicit ownership and budgets. +- Replace PR-time Nominatim, HDX, OSMnx/Overpass, ORS, Google, Geofabrik, and OSRM + calls with recorded minimal fixtures and HTTP/subprocess fakes. +- Block accidental network access in the unit-test job. +- Add small live contract tests on a schedule and manual dispatch with clear + provider-friendly rates; their failure opens or updates an issue but does not + retroactively make unrelated PRs fail. +- Build a shared HTTP boundary with an identifying User-Agent, explicit connect + and read timeouts, bounded retries with jitter, `raise_for_status`, typed error + translation, optional endpoint override, and response-size limits. +- Respect provider policy, rate limits, caching headers, attribution, and privacy. + Nominatim usage in particular must stay at or below its public-service limit, + identify UrbanPy, and be switchable/configurable. +- Add deterministic fixtures for CRS, coordinate order, geometry validity, units, + empty data, nulls, and disconnected networks. +- Use coverage as a ratchet: no regression in total coverage and at least 80% + coverage on new code, aligned with the SonarQube quality gate. +- Add a small benchmark suite for the performance work merged in PR #59 and fail + only on agreed, repeatable regressions. + +### W6. OSRM and Geofabrik lifecycle + +**Outcome:** local OSRM is reproducible and cross-platform, and extract selection +does not depend on fragile path naming. + +Implement the dedicated [`osrm-0.3.md`](osrm-0.3.md). It absorbs issues #21, +#26, and #29 and retires the duplicate `osrm_routing.py` implementation and the +platform-specific orchestration scripts. + +### W7. Pydantic models, static typing, and API contracts + +**Outcome:** untrusted inputs and stable public results have safe, documented +contracts without slowing geospatial computation. + +Implement [`typed-models-0.3.md`](typed-models-0.3.md). Start with shared OSRM and +Geofabrik boundaries, then migrate external service clients module by module. + +### W8. Geospatial correctness and existing defect backlog + +**Outcome:** supported features behave correctly with the current geospatial +ecosystem and state their CRS/coordinate/unit assumptions. + +- Fix and regression-test current H3 compatibility (#38). +- Fix current OSMnx API and coordinate-order compatibility (#20). +- Guarantee or explicitly require CRS on created and consumed objects (#27). +- Resolve HDX failures (#19 and #52) through the shared service boundary. +- Reproduce and fix `merge_shape_hex` re-use behavior (#14). +- Decide whether masked HDX population retrieval (#30) is feasible without + misleading performance claims; otherwise defer it from 0.3. +- Add meaningful progress reporting for genuinely long operations (#31) without + coupling library functions to one terminal renderer. +- Audit every `__all__` export and classify it as supported, deprecated, internal, + or removed. Add public API and signature snapshots. +- Replace broad exception swallowing, sentinel mixes (`None`, `-1`, `NaN`), and + library-level `print` calls with documented results, warnings, and exceptions. +- Standardize longitude/latitude order, CRS expectations, and distance/duration + units across routing, download, geometry, and accessibility modules. + +New feature PR #46 is evaluated separately. It is not on the 0.3 critical path +unless it satisfies the new data-source, dependency, test, and maintenance rules. + +### W9. Documentation and community readiness + +**Outcome:** installation, common workflows, limits, and contribution paths are +accurate for 0.3. + +- Keep Sphinx for 0.3 unless a short ADR demonstrates that migration pays for + itself; update it and build with warnings as errors. +- Rewrite installation and quickstart material against built artifacts and `uv`. +- Add task-oriented guides for downloads, H3 geometry, accessibility, remote + routing APIs, and local OSRM. +- Add an external-service matrix covering credentials, endpoint configuration, + rate limits, caching, attribution, privacy, failure modes, and live-test policy. +- Publish API reference, model JSON Schemas where useful, and a 0.3 migration guide. +- Turn maintained notebooks into deterministic examples or documentation tests; + archive stale outputs and notebooks that cannot be reproduced (#28). +- Add `CITATION.cff`, accurate author/maintainer information, governance, support, + security reporting, and a maintenance-status statement. +- Document OSM/Geofabrik attribution and data licensing separately from UrbanPy's + software license. +- After a stable PyPI release, pursue a community-maintained conda-forge recipe + rather than a private Anaconda channel (#16). + +### W10. Release engineering and post-release operation + +**Outcome:** exactly the reviewed artifacts are promoted, published, and supported. + +- Use separate build and publish workflows; build once and promote the same + immutable sdist/wheel artifacts. +- Trigger releases from protected, signed `v*` tags or an approved release + workflow; validate tag and package versions without modifying source files. +- Publish alpha, beta, and release-candidate artifacts to TestPyPI and exercise + documented clean-install smoke tests before production. +- Publish PyPI through trusted publishing, create GitHub release notes/changelog, + and deploy documentation from the released version. +- Define rollback/yank criteria, post-release smoke tests, and on-call ownership. +- Monitor installation failures, security reports, upstream service contracts, + and top user issues for at least four weeks after 0.3.0. + +## Delivery phases and gates + +### Phase 0 — Stabilize and secure + +Required before new architecture work is merged: + +- credential/security incident closed; +- PR test workflow is green and hermetic; +- SonarQube remains operational and is no longer best-effort; +- default-branch rules require human review and successful checks; +- current license-compliance failure has an owner and documented disposition; +- no known high-severity dependency/workflow alert lacks a patched resolution or + time-bounded EL-BID-approved exception; +- old issues and PRs are triaged into the Project. + +### Phase 1 — 0.3.0a1 foundation + +- `pyproject.toml`, supported Python range, `uv.lock`, and direct dependencies; +- stable lint/type/test/docs/package commands locally and in CI; +- artifact install tests and trusted-publishing dry run; +- Pydantic primitives and Geofabrik catalog models; +- OSRM catalog resolver and command-runner foundations. + +### Phase 2 — 0.3.0b1 functional migration + +- Python-owned OSRM prepare/start/status/stop lifecycle; +- typed external-service boundaries and consistent error behavior; +- priority compatibility issues (#14, #19, #20, #27, #38, #52) resolved; +- user and contributor documentation updated. + +### Phase 3 — 0.3.0rc1 release hardening + +- full supported Python/OS matrix green; +- manual OSRM acceptance evidence on macOS, Linux, and Windows; +- migration guide and API compatibility report complete; +- SBOM, vulnerability, dependency-license, package-content, and provenance gates + pass; +- no unresolved P0/P1 defects or unowned high risks. + +### Phase 4 — 0.3.0 and follow-through + +- publish reviewed artifacts through trusted publishing; +- deploy versioned documentation and release notes; +- run post-release installation and OSRM smoke tests; +- begin conda-forge submission and four-week support observation. + +## GitHub Project configuration + +Recommended fields: + +- Status: Triage, Backlog, Ready, In progress, In review, Blocked, Done +- Workstream: W0 through W10 +- Priority: P0, P1, P2 +- Work type: Decision, Implementation, Test, Documentation, Security, Release +- Target: Phase 0, 0.3.0a1, 0.3.0b1, 0.3.0rc1, 0.3.0, Post-0.3 +- Risk: Low, Medium, High +- Effort: XS, S, M, L; split anything larger than L +- Agent readiness: Not suitable, Needs decision, Ready, Agent working +- Owner + +Enable GitHub's native **Parent issue**, **Sub-issue progress**, and issue +dependency relationships rather than duplicating dependency state in a text +field. + +Recommended views: + +- Roadmap grouped by target and workstream +- Current iteration board +- Release gates (`priority:P0,P1` or `work-type:Release,Security`) +- Agent-ready queue +- Blocked work with dependency relationships +- Existing bugs and community contributions + +Milestones represent published checkpoints. The Project represents execution. +Labels describe stable properties such as `area:osrm`, `area:typing`, +`area:packaging`, `breaking-change`, `good-first-issue`, `needs-design`, and +`agent:ready`. + +## Issue definition of ready + +Before an issue moves to Ready it has: + +- one testable outcome and an explicit non-goal; +- linked parent, dependencies, and existing issue/PR history; +- public API, dependency, security, data-license, and migration impact identified; +- acceptance criteria and exact local verification commands; +- fixture/test strategy that avoids unapproved live services; +- owner/reviewer and estimated effort no larger than L. + +## Pull-request definition of done + +- linked issue acceptance criteria are satisfied; +- tests cover success, failure, and compatibility behavior; +- `uv lock --check` passes and dependency changes are explained; +- required lint, type, unit, docs, package, security, license, and SonarQube checks + pass; +- public behavior, schemas, changelog, and migration notes are updated; +- generated/AI-assisted changes are disclosed and independently reviewed; +- no unrelated cleanup is bundled into the PR; +- one human reviewer approves after the latest material change. + +## Definition of done for UrbanPy 0.3 + +- A new contributor can clone, bootstrap, check, test, build, and preview docs + using documented `uv` and Trunk/fallback commands. +- PRs cannot merge without human approval, deterministic green CI, SonarQube, + security analysis, and approved dependency compliance. +- Supported Python versions and operating systems pass clean wheel-install tests. +- `import urbanpy` has no network side effects and optional capabilities have + honest dependency boundaries. +- OSRM uses one Python implementation on all platforms, canonical Geofabrik IDs + and URLs, pinned container inputs, safe storage, readiness checks, and targeted + cleanup. +- New boundary models provide actionable errors and stable schemas while large + geospatial objects avoid per-row runtime validation. +- External-service calls identify UrbanPy, set timeouts, respect provider policy, + and have hermetic PR tests plus scheduled live contracts. +- Public APIs state coordinate order, CRS, units, null behavior, and exceptions; + compatibility changes have tests and migration notes. +- Built sdist/wheel contents, metadata, license, SBOM, provenance, and clean + installation are verified before trusted publication. +- Documentation, citation, governance, security, support, and release ownership + are current, and no P0/P1 issue or unowned high risk remains.