FloodNet Work

Hydroinformatics workflows for NYC FloodNet sensor data, including:

• API extraction and parquet checkpointing
• dataset joining and spatial weather enrichment
• storm delineation and event-level EDA
• rain-influenced gage selection for modeling
• single-gage focused modeling on apparently-darling-gecko
• model training and evaluation

Project Structure

• Finalized_Scripts/: primary notebooks and Python scripts for the core pipeline
• Test_Scripts/: exploratory notebooks and standalone experiments
• Data_Files/: parquet datasets and training databases
• Images_or_plots/: exported figures
• results/: run logs and presentation outputs

1. Prerequisites

• Python 3.10+ (recommended: 3.11)
• pip
• Jupyter Notebook or JupyterLab
• Optional for R scripts: R 4.2+

2. Quickstart (5 Minutes)

git clone https://github.com/mwdunlap2004/floodnet_work.git
cd floodnet_work
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt
jupyter lab

Windows (PowerShell):

git clone https://github.com/mwdunlap2004/floodnet_work.git
cd floodnet_work
py -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -r requirements.txt
jupyter lab

Notes:

• geopandas may require system GIS libraries on some machines.
• torch install can vary by OS/GPU. If needed, use the official selector: https://pytorch.org/get-started/locally/

3. Data Inputs (Expected Files)

Most finalized notebooks/scripts expect files under Data_Files/:

Primary shared data location:

• Google Drive folder: FloodNet project data
• Download/sync these files into local Data_Files/ before running the notebooks.

File	Produced By	Used By
floodnet_parquet_data/*.parquet	Finalized_Scripts/parquet_floodnet_download.py	Finalized_Scripts/joining_parquets.ipynb
nyc_precipitation_master.parquet	Finalized_Scripts/merging_precip.ipynb	Finalized_Scripts/spatial_join.ipynb
floodnet_full_dataset_merged.parquet	Finalized_Scripts/joining_parquets.ipynb	Finalized_Scripts/spatial_join.ipynb
floodnet_floods_only.parquet	Finalized_Scripts/joining_parquets.ipynb	Test_Scripts/flood_duration.ipynb
floodnet_full_dataset_merged_with_weather.parquet	Finalized_Scripts/spatial_join.ipynb	Finalized_Scripts/select_precip_influenced_gages.py
rain_influenced_sites_raw.parquet	Finalized_Scripts/select_precip_influenced_gages.py	Finalized_Scripts/delineate_filtered_storms.py
apparently-darling-gecko.parquet	Finalized_Scripts/apparently_darling_gecko.py	Finalized_Scripts/hpo_search.py, Finalized_Scripts/model_training.py
rain_influenced_gages.parquet	Finalized_Scripts/delineate_filtered_storms.py	Finalized_Scripts/rain_influenced_EDA.ipynb, Finalized_Scripts/hpo_search.py, Finalized_Scripts/model_training.py
delineated_storms.parquet	Test_Scripts/updated_method_storm_seperation.ipynb	Finalized_Scripts/floodnet_eda.ipynb
floodnet_hpo_newfilter.db	Finalized_Scripts/hpo_search.py	Finalized_Scripts/model_training.py

Note:

• Finalized_Scripts/merging_precip.ipynb is currently configured to read source CSVs from data_files/request/ and writes nyc_precipitation_master.parquet; place that parquet under Data_Files/ for spatial_join.ipynb.

4. Pipeline Run Order

Use this order for a clean, reproducible workflow:

1. Finalized_Scripts/parquet_floodnet_download.py
2. Finalized_Scripts/joining_parquets.ipynb
3. Finalized_Scripts/merging_precip.ipynb (builds nyc_precipitation_master.parquet)
4. Finalized_Scripts/spatial_join.ipynb
5. Finalized_Scripts/select_precip_influenced_gages.py (default --top-n 3; creates Data_Files/rain_influenced_sites_raw.parquet)
6. Finalized_Scripts/delineate_filtered_storms.py (creates Data_Files/rain_influenced_gages.parquet)
7. Finalized_Scripts/apparently_darling_gecko.py (isolates one deployment and creates Data_Files/apparently-darling-gecko.parquet)
8. Finalized_Scripts/rain_influenced_EDA.ipynb (rain-coupling diagnostics)
9. Finalized_Scripts/hpo_search.py (default input: apparently-darling-gecko.parquet)
10. Finalized_Scripts/model_training.py (default input: apparently-darling-gecko.parquet)
11. Optional full-network analysis notebooks: Test_Scripts/updated_method_storm_seperation.ipynb, Finalized_Scripts/floodnet_eda.ipynb, Test_Scripts/flood_duration.ipynb, Test_Scripts/finalized_lstm_modeling.ipynb

5. Modeling Scope: Why One Gage

For final model development, the workflow uses one high-reliability FloodNet deployment: apparently-darling-gecko (written as apparently_darling_gecko in text and apparently-darling-gecko in file/script IDs).
Finalized_Scripts/apparently_darling_gecko.py filters storm records to that deployment and writes Data_Files/apparently-darling-gecko.parquet, which is the default training input used by both hpo_search.py and model_training.py.

6. Verified Metrics (Accuracy Check)

The current checked metrics are from results/run_log.json with timestamp 2026-05-04T15:38:45.092605.

Test-set skill scores:

Model	KGE	NSE	RMSE (in)	PBIAS (%)	PeakNSE
Log-Ridge	-0.0246	0.1502	1.3014	55.83	-0.4895
Res-ANN	0.0323	0.1273	1.3188	40.55	-0.4663
Attn-LSTM	0.0699	0.1796	1.2963	32.11	-0.4105

Train-set skill scores:

Model	KGE	NSE	RMSE (in)	PBIAS (%)	PeakNSE
Log-Ridge	0.4419	0.1626	0.4988	3.54	-1.1352
Res-ANN	0.5302	0.1229	0.5105	6.47	-0.5716
Attn-LSTM	0.6296	0.5234	0.3820	1.67	-0.1554

7. SLURM Submission

• Unified job script:
  • Test_Scripts/run_rain_influenced.sbatch
• Examples:

sbatch Test_Scripts/run_rain_influenced.sbatch
sbatch --export=STEP=hpo Test_Scripts/run_rain_influenced.sbatch
sbatch --export=STEP=train Test_Scripts/run_rain_influenced.sbatch
sbatch --export=STEP=all Test_Scripts/run_rain_influenced.sbatch

• Legacy scripts are also aligned to the new input file:
  • Finalized_Scripts/run_hpo.slurm
  • Finalized_Scripts/run_training.sbatch

8. Outputs

• Images_or_plots/: static figures (for example model comparison and duration plots)
• results/: model run artifacts/logs
• results/presentation_figures/: exported presentation figures from floodnet_eda.ipynb
• checkpoints/: trained model checkpoints and scalers from training scripts

9. Optional R Dependencies

If you plan to run .R scripts in Test_Scripts/:

install.packages(c("tidyverse", "jsonlite", "leaflet", "htmltools", "lubridate"))

Troubleshooting

• ModuleNotFoundError: install missing packages into the active .venv.
• Jupyter kernel mismatch: select the .venv kernel in notebook UI.
• GeoPandas install issues: try conda/mamba for geospatial dependencies.
• Slow API pulls: the downloader includes retries and checkpointed parquet outputs.