Historical Observations Data Platform

Code for Eagle Rock Analytics' cloud-based, historical weather observations data platform

The Historical Observations Data Platform is a cloud-based, historical weather observations data platform to enable California's energy sector access to high-quality, open climate and weather data. This work is supported by California Energy Commission grant PIR-19-006. This repository contains the code (via Python scripts and Jupyter Notebooks) associated with the full processing pipeline for data ingestion into the Historical Data Platform.

The Platform responds to community partner needs in understanding weather and climate information including the severity, duration, frequency, and rate of change over time of extreme weather events, as well as supporting projections downscaling efforts. We implement stringent Quality Assurance/Quality Control (QA/QC) procedures in line with international protocols and with customized modifications relevant to energy sector (such as temperature and precipitation extremes, winds, and solar radiation).

📊 About the data

The Platform has sourced station data from from 27 publicly available historical data observation networks across the Western Electricity Coordinating Council (WECC) domain from 1980-2022 (time period varies between networks and stations). 15,064 stations total have completed the full quality control and standardization pipelines and are publically available as cloud-optimized zarrs in Amazon s3 storage.

The following figure shows the locations of all the stations (by network) that have completed our quality control and standardization process:

And here you can see the number of observations throughout the project's time period:

HDP 2.0 private rollout note: The public cadcat link above reflects the legacy public publication surface. Current Phase 2 ASOSAWOS operational runs and validations in this repository use s3://auto-hdp/hdp/ASOSAWOS/ with explicit context-lock preflight checks.

---

🗂 Repository Structure

historical-obs-platform/
├── data/                      # Miscellaneous supporting data
├── data-access/               # Code examples for accessing our data
├── environment/               # Files for building the computational environment, including a README with further instructions
├── figures/                   # Visualizations
├── notebooks/                 # Jupyter notebooks for data visualization and analysis
├── scripts/                   # Data processing code for all steps of the QAQC process
│   ├── 1_pull_data/           # Scripts for retrieving/scrape network station data from their respective sources
│   ├── 2_clean_data/          # Scripts for cleaning individual networks to a consistent standard
│   ├── 3_qaqc_data/           # Scripts to QA/QC stations
│   ├── 4_merge_data/          # Scripts to close out processing, and standardize to hourly timesteps. Data at conclusion have been fully processed.
│   ├── misc/                  # Scripts that don't fit into any other categories
│   ├── pcluster/              # Code and shell scripts for running QAQC and merge scripts in an AWS pcluster environment
│   └── tests/                 # Scripts for testing finalized data products
└──

🖥️ For developers: Running QAQC and MERGE steps

HDP 2.0 private Phase 2 context (critical)

For ASOSAWOS private automation and manual validation runs, keep bucket context aligned:

• HDP_STAGING_BUCKET=hdp-staging-pull
• HDP_SOURCE_BUCKET=auto-hdp
• HDP_PUBLISH_BUCKET=auto-hdp
• HDP_PUBLISH_PREFIX=hdp

If source/publish context is mismatched in append mode, merge can drop historical baseline periods. Always verify final station date ranges with xarray after manual test runs.

1. Follow the instructions in the pcluster README to run QAQC for all networks in an AWS pcluster environment.

2. Run the station list generation script for each network (can be run locally). This uploads a stationlist_{NETWORK}_qaqc.csv file to the QAQC bucket in AWS, indicating when QAQC was run and whether each station passed or failed (Y/N), along with any relevant error messages:

   # Single network
   python scripts/3_qaqc_data/stnlist_update_qaqc.py <NETWORK>
   # All networks
   ./scripts/3_qaqc_data/run_stnlist_update_qaqc.sh

3. Run the concatenation script to identify and merge co-located stations (same lat/lon) in ASOSAWOS and MARITIME into single records. This only needs to be run once — it deletes the original input stations from S3 after concatenation:

   python scripts/3_qaqc_data/qaqc_concatenate_stations.py

Note: This script will fail on re-runs because it deletes the original input stations. You'd need to regenerate the original QAQC'd stations first to run it again.

4. Follow the instructions in the pcluster README to run MERGE for all networks in an AWS pcluster environment.

5. Run the merge station list generation script for each network. In HDP 2.0 private ASOSAWOS flow, this publishes stationlist_{NETWORK}_merge.csv under the publish prefix (s3://auto-hdp/hdp/{NETWORK}/) and mirrors legacy location when configured, indicating when merge/hourly standardization was run and whether each station passed or failed (Y/N), along with any relevant error messages:

   # Single network
   python scripts/4_merge_data/stnlist_update_merge.py <NETWORK>
   # All networks
   ./scripts/4_merge_data/run_stnlist_update_merge.sh

🛠️ Computational Environment

See the environment folder for instructions and files for building the computational environment for this project.

🔏 License

This project is licensed under the GNU GPLv3 - see the LICENSE file for details.

🙋 Support

• 📧 Email: info@eaglerockanalytics.com
• 🐛 Issues: GitHub Issues

🧑‍💻 Contributors