Skip to content

feat/migrate windwatts-data to app backend#286

Open
sameersk2k wants to merge 12 commits into
developfrom
feat/migrate-windwatts-data-to-backend
Open

feat/migrate windwatts-data to app backend#286
sameersk2k wants to merge 12 commits into
developfrom
feat/migrate-windwatts-data-to-backend

Conversation

@sameersk2k

@sameersk2k sameersk2k commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Description

Issues: #285 #281

Issue 281 backend is implemented, UI is pending.

Changes

  • MODEL CONFIG: added interpolation flag to easily control whether to support windspeed interpolation at diff heights for a given model.
  • VALIDATION: relaxed strict height check to support interpolation. As long as the given height is between min and max available height it will be passed.
  • SPATIAL MODULE: added spatial module under app. It has lookup files under app/spatial/grid_lookup_files and it has cKDTree lookup algorithm too. In future we will add additional algorithms like sparse lookup for ensemble. The __init__ make this as a package. It created a module level registry _lookups that maps model names with their lookup instances so it a singleton shared across the app and just loaded once. AthenaDataFetcher uses exported CKDTreeLookup class and register func to initialize instances. Irrespective of which algo is used (CDKTree or SparseLookup (future))athena fetcher will call find_nearest or find_n_nearest in spatial init.
  • lookup files: Switched to npz as they are smalled than pickle zip files.
  • Athena Data Fetcher: Now handles core functionality of the package. Athena query logic, caching, etc
  • Wind Data Controller: Imports CKDtree and register from the spatial. Creates lookup instances and registers it will data models.
  • wind_processing.py: It's in utils and provides with helper functions to athena data fetcher above to do interpolation, aggregations, rose sectors, swi smoothing for quantiles etc.
  • Tests - Modified tests to mimic package less approach.
  • Dockerfile - removed windwatts-data package fetching logic
  • Updated requirements.txt as we need geopandas which earlier package handled it.
  • Deleted the legacy routers and controllers from main. Later will delete files too if ok.

Concerns

  • Dataframe caching in AthenaDataFetcher is done by grid index. Grid index is much better when dealing multiple concurrent API calls in future compared to on (lat, long) as grid index (string) is reliable and accurate compared to floats. Instead of 1 df per (lat,long) why cant we cache multiple files per grid index. But how many files should we cache? current is 100 maybe it should be decided on the file size which is 2KB per file for era5-quantiles 2.8 KB for Ensemble Quantiles, 28 KB wtk-timeseries.

Checklist

  • PR targets develop
  • I have run make format and make verify
  • No merge conflicts with develop
  • Branch name is descriptive (feat/..., fix/..., docs/..., chore/...)
  • Commit messages are clear (e.g., feat: add new feature, fix: resolve bug, docs: update documentation)
  • Documentation updated where needed
  • I have self-reviewed this PR

Others: e.g. Screenshots if UI change

@sameersk2k sameersk2k marked this pull request as draft July 7, 2026 21:16
@sameersk2k sameersk2k self-assigned this Jul 7, 2026
@sameersk2k sameersk2k added the backend backend api calls, processing, etc. label Jul 7, 2026
@sameersk2k sameersk2k marked this pull request as ready for review July 8, 2026 20:42
@sameersk2k sameersk2k requested a review from RLiNREL July 8, 2026 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend backend api calls, processing, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant