NCIMM White Box Testing

White-box regression tests, calibration files, and statistical comparison harness for the NCIMM build of EPA SWMM5. This repository extends the NCIMM Regression Test Files with the internal (white-box) diagnostics needed to verify that engine changes do not perturb hydrologic or hydraulic state at the sub-element level between versions (notably v12 and v13).

Where black-box regression compares only the final .rpt/.out, white-box testing here compares time series of internal SWMM5 state variables (node depths, link flows, runoff, groundwater, HGL, etc.) and reduces them to statistical moments and goodness-of-fit metrics.

Contents at a glance

• SWMM5 test input models grouped by feature area (Hydraulics, Hydrology, LID, Orifices, Pumps, Weirs, etc.)
• v12 vs v13 calibration / comparison logs
• Pre-built SWMM5 binaries (SWMM5.EXE, SWMM5.DLL, swmm5.lib) used to produce the reference outputs
• Batch automation (NCIMM_REGRESSION_STATS.BAT, make_dir.bat)
• Aggregated run log (All.log) and reference notes (SWC.MD, Hydrology Reference Manual references)

Repository layout

Test model inputs

Folder	Focus area
Hydraulics	Core hydraulic routing test cases (v12/v13)
Hydrology	Rainfall-runoff and subcatchment tests
LID	Low Impact Development control tests
Orifices	Orifice element tests
Weirs	Weir element tests
Pumps	Pump curve and operation tests
SWMM5_NCIMM	NCIMM-specific test models

OWA / branch test suites

Folder	Focus area
OWA_EXAMPLES	OWA reference example models
OWA_ROUTING	Routing tests
OWA_USER	User-contributed OWA tests
OWA_update_v5111	OWA update branch (v5.1.11) tests
LEW_update_v5113	LEW update branch (v5.1.13) tests

Third-party / vendor models

Folder	Focus area
InfoSWMM Model Folders	InfoSWMM regression models
XPSWMM	XPSWMM-derived test models

Supporting data and logs

Folder	Contents
Calibration_Files	White-box calibration time series for v12/v13 (see types below)
DataFiles	Auxiliary data (rainfall, time series, climate) referenced by .inp files
v12	v12 log / output comparison files
v13	v13 log / output comparison files

Root files

File	Purpose
SWMM5.EXE / SWMM5.DLL / swmm5.lib	NCIMM SWMM5 engine binaries used to generate reference outputs
SWMM5.iobj / SWMM5.ipdb	Incremental build artifacts (MSVC)
NCIMM_REGRESSION_STATS.BAT	Runs the full regression suite and produces calibration / stats logs
make_dir.bat	Creates the working directory structure
All.log	Aggregated log of the most recent regression run
SWC.MD	Supporting notes
Storm Water Management Model Reference Manual Volume I – Hydrology (Revised) References.md	Reference list for the EPA SWMM Hydrology manual

White-box calibration file types

Each test case can emit a family of Fcalibration* files containing per-time-step values of a specific internal state variable, plus generic SWMM5 comparison logs:

File	Variable / Purpose
Flog	Other SWMM5 output file comparison log
FoutCompare	Other SWMM5 output file
Fall_log	Combined log of all comparison files
FcalibrationS	Storage volume in a storage node
FcalibrationR	Runoff
FcalibrationE	Groundwater elevation
FcalibrationG	Groundwater Q
FcalibrationH	Node depth
FcalibrationNH	Node head
FcalibrationNF	Node flooding
FcalibrationL	Node lateral Q
FcalibrationQ	Link Q
FcalibrationV	Link velocity
FcalibrationD	Link depth
FcalibrationHGL	Link hydraulic grade line
FcalibrationNA	Node area
FcalibrationNSA	Node dQ/dH

These time series are the "white-box" view into the engine: they allow detecting regressions that would be invisible in the headline .rpt summary.

Statistical metrics (MAX_STATISTICAL_RESULTS)

For each calibration time series, NCIMM computes simulated-vs-observed moments and goodness-of-fit metrics:

Symbol	Metric
SIM	Simulated mean
OBS	Observed mean
RMSE	Root mean square error
MAE	Mean absolute error
MSLSE	Mean simple least-square error
STDSIM	Simulated standard deviation
STDOBS	Observed standard deviation
SkewnessSim / SkewnessObs	Skewness of simulated / observed series
KurtosisSim / KurtosisObs	Kurtosis of simulated / observed series
LogNASH	Log Nash–Sutcliffe efficiency
IndexD	Willmott's index of agreement
NASH	Nash–Sutcliffe efficiency

These metrics are written into the *_log and Fall_log files for each model run and let you spot small numerical drift between engine versions.

Running the regression suite

On Windows, from the repo root:

rem (Re)create working directories
make_dir.bat

rem Run all regression tests and produce stats / calibration logs
NCIMM_REGRESSION_STATS.BAT

The batch script will iterate over the .inp files in each feature folder, run them with the bundled SWMM5.EXE, emit Fcalibration* files plus Flog / FoutCompare, and aggregate into Fall_log and All.log.

Compare new logs against the reference logs under v12/ and v13/ to confirm no regression.

Typical workflow

1. Build a new candidate SWMM5.EXE / SWMM5.DLL.
2. Drop the new binaries into the repo root (or a working copy).
3. Run NCIMM_REGRESSION_STATS.BAT.
4. Diff the resulting Fcalibration* / Fall_log files against v12/ and v13/ references.
5. Inspect the MAX_STATISTICAL_RESULTS block for any metric that has shifted outside its acceptance tolerance.

Related repositories

• NCIMM-Regression-Test-Files – black-box (.inp + .rpt) regression suite.
• SWMM_MD_Documents – SWMM manuals and references in Markdown.
• 1729-SWMM5-Models – large OWA SWMM5 test model collection.

Contributing

Issues and PRs welcome for:

1. New test models exercising specific SWMM5 features.
2. Improved reference logs (with v12 and v13 outputs).
3. Tolerance / pass-fail thresholds for the statistical metrics.
4. Cross-platform port of NCIMM_REGRESSION_STATS.BAT to bash / PowerShell.

Please keep large binary outputs summarized in v12/ or v13/ rather than committing raw .out files.

License

No explicit license has been declared. Test models contributed from EPA SWMM, OWA, InfoSWMM, and XPSWMM retain their respective upstream licenses. NCIMM binaries are included for reference / reproducibility of the published regression results.

Maintainer

Maintained by @dickinsonre under the SWMMEnablement organization.