NOAA-Integrator acquires and processes AIS data from NOAA Marine Cadastre and loads it into an AISdb-aligned database (SQLite or PostgreSQL/TimescaleDB). It is a uv-managed Python tool with a pure-Python core that consumes the NOAA archive at https://coast.noaa.gov/htdata/CMSP/AISDataHandler/{year}/ and supports every published format, including daily ZIP archives (2015-2024), zone archives (2009-2014), legacy geodatabases (.gdb.zip, 2011-2013), and Zstandard-compressed CSV (.csv.zst, 2025 onward). NOAA-Integrator is developed and maintained by the MAPS Lab at Dalhousie University, continuing work that began under the MERIDIAN initiative.
- Downloads every archive NOAA publishes for a year range, resume-friendly by default
- Organizes archives into
{year}{month}folders across all NOAA naming schemes since 2009 - Extracts
.zip(with integrity check) and.csv.zst, and converts legacy.gdb.zipgeodatabases - Filters records to a WGS84 bounding box, skipping and counting malformed rows
- Deduplicates rows in place with atomic replacement
- Simplifies per-vessel trajectories (Visvalingam-Whyatt, Douglas-Peucker, TD-TR) with optional quality metrics
- Loads month batches into SQLite or PostgreSQL/TimescaleDB through
aisdb.decode_msgs, with retries
The pipeline runs as one console command, noaa-integrator, with a subcommand per stage. Every stage is independent, so you can run only the ones you need. Optional extras add database loading, legacy geodatabase conversion, and trajectory simplification without weighing down the core.
NOAA-Integrator uses uv for environment and dependency management. The core install covers download, organize, extract, filter, and dedup; optional extras add the heavier stages.
uv sync # core pipeline (download, organize, extract, filter, dedup)
uv sync --extra load # + aisdb for database loading
uv sync --extra gdb # + geopandas/fiona for legacy .gdb.zip conversion
uv sync --extra simplify # + fastdtw/similaritymeasures/scipy for trajectory simplification
uv sync --all-extras # everything at onceWithout uv, install from a clone with pip (Python 3.10-3.12).
pip install . # core pipeline
pip install ".[load]" # with a chosen extra (load, gdb, simplify)Download one year, organize it, extract it, and load it into a SQLite database.
uv sync --extra load
uv run noaa-integrator download --start-year 2024 --end-year 2024 --dest data/downloads
uv run noaa-integrator organize --base-dir data/downloads
uv run noaa-integrator extract --base-dir data/downloads --dest data/extracted --start 202401 --end 202412
uv run noaa-integrator load --source-dir data/extracted \
--start-year 2024 --end-year 2024 --sqlite marine_cadastre.dbEach stage is a subcommand of noaa-integrator. Stages are independent; run the ones you need, in order.
uv run noaa-integrator download --start-year 2023 --end-year 2024 --dest data/downloadsScrapes the NOAA index per year and streams every archive. Existing non-empty files are skipped (resume-friendly); force a re-download with --no-skip-existing.
uv run noaa-integrator organize --base-dir data/downloadsGroups archives into {year}{month} folders (for example 202406/), matching every NOAA naming scheme from 2009 to 2025 and beyond.
uv run noaa-integrator extract --base-dir data/downloads --dest data/extracted --start 202301 --end 202312Decompresses .zip (with integrity check) and .csv.zst archives into a mirror {yyyymm} tree. Corrupted archives are reported and skipped, never fatal.
Legacy geodatabase archives (2011-2013) convert separately.
uv run noaa-integrator gdb data/downloads/201303/*.gdb.zip --dest data/extracted/201303uv run noaa-integrator filter \
--base-dir data/extracted --dest data/filtered \
--start-year 2023 --end-year 2023 --start-month 1 --end-month 2 \
--min-lon -77.36 --min-lat 36.02 --max-lon -57.62 --max-lat 48.64Keeps only records inside the WGS84 bounding box. Malformed rows and non-numeric coordinates are skipped and counted; months without matching data produce no output files.
uv run noaa-integrator dedup --directory data/mergedRemoves duplicate rows in place (header preserved, atomic replace).
uv run noaa-integrator simplify data/merged/*.csv --dest data/simplified --algorithm vw --metricsPer-vessel trajectory simplification with Visvalingam-Whyatt (vw), Douglas-Peucker (rdp), or Time-Dependent Trajectory Reduction (tdtr). Adding --metrics also writes SR, LLR, DTW, Frechet, and ASED per track.
# SQLite
uv run noaa-integrator load --source-dir data/filtered \
--start-year 2023 --end-year 2023 --start-month 1 --end-month 2 \
--sqlite marine_cadastre.db
# PostgreSQL / TimescaleDB
export NOAA_PG_DSN='postgresql://user:password@localhost:5432/noaa'
uv run noaa-integrator load --source-dir data/filtered \
--start-year 2023 --end-year 2023 --dsn --timescaledbDecodes month batches through aisdb.decode_msgs. Failed batches are retried (three attempts by default). Credentials come from --dsn or the NOAA_PG_DSN environment variable, never from code.
uv sync --all-extras
uv run pytest # test suite
uv run ruff check . # lint
uv run ruff format . # format
uv run pyrefly check # typecheckContinuous integration runs linting, type checking, and the test suite on Ubuntu and macOS against Python 3.10 and 3.12 for every push to main and every pull request.
Docs · Tutorials · API reference · Website
- AISdb, the core Python/Rust platform for storing, querying, and analyzing AIS data
- AISdb-lite, a lightweight AISdb variant built on PostGIS and TimescaleDB hypertables
- Tutorials, hands-on notebook companions to the GitBook tutorials
If you use NOAA-Integrator in your work, please cite it. Citation metadata lives in CITATION.cff, and the BibTeX entry follows.
@software{NOAAIntegrator2026:GSpadon,
author = {Spadon, Gabriel},
title = {NOAA-Integrator},
year = {2026},
version = {1.0.0},
publisher = {MAPS Lab, Dalhousie University},
url = {https://github.com/MAPS-Lab/NOAA-Integrator},
license = {AGPL-3.0}
}This project is distributed under the terms of the GNU Affero General Public License v3.0 (AGPL-3.0). See LICENSE for details.