Code Modules

Code Modules

The Code/ directory contains processing scripts, QA tools, and one-time utilities. Scripts are organised into prefixed subdirectories.

Directory Structure

Code/
├── DS02_DatasetQA/          # Dataset quality assurance scripts
│   ├── QA00_ELMvaliditation.py
│   └── README.md
├── DS03_PlotExtractionCode/  # Plot-level data extraction
│   └── PE00_LIDAR_extraction.py
├── functions/                # Shared utility functions
│   └── core_functions/
│       ├── parse_APPN_dataset_path.py
│       └── tests/
│           └── test_parse_APPN_dataset_path.py
└── OT00_OneTimeScripts/      # One-time migration/maintenance scripts
    ├── OT00_RenameTiertoT.py
    └── OT01_MoveGrawToRaw.py

Naming Convention

Scripts use a prefix system:

Prefix	Meaning	Example
DS	Dataset processing step	DS02_DatasetQA
QA	Quality assurance	QA00_ELMvaliditation.py
PE	Plot extraction	PE00_LIDAR_extraction.py
OT	One-time script	OT00_RenameTiertoT.py, OT01_MoveGrawToRaw.py

---

QA00 — ELM Validation Panel Extraction

File: Code/DS02_DatasetQA/QA00_ELMvaliditation.py
Full documentation: Code/DS02_DatasetQA/README.md
Author: Arden Burrell
Version: v1.0 (03.03.2026)

Automates the extraction and quality control of spectral data from ELM (Empirical Line Method) validation panels in hyperspectral imaging datasets. The summary below covers the essentials — see the linked README for tested package versions, full folder-structure assumptions, output schemas, troubleshooting, and future enhancements.

What It Does

1. Searches for QC panel shapefiles matching *QC_*_Panel*.shp
2. Locates corresponding VNIR and SWIR orthomosaic rasters
3. Extracts spectral reflectance values from pixels within panel geometries
4. Saves extracted data as CSV or Parquet files
5. Generates visualisation plots showing spectral curves across panels and dates
6. Supports data sharing between nodes via save/load directories

Command-Line Arguments

Argument	Description
--path	Path to search for QA shapefiles (default: git root)
-f, --force	Force overwrite of existing output files
--type	Output format: csv or parquet
-s, --skipplot	Skip plot generation
--skip-processing	Skip raster processing; only load existing outputs
--save-dir	Save copies of extracted spectra for sharing
--load-dir	Load previously extracted spectra from another node
-v, --verbose	Detailed output

Usage

# Standard run
python QA00_ELMvaliditation.py

# Extract and save for sharing with other nodes
python QA00_ELMvaliditation.py --path /path/to/data --save-dir /shared/spectra

# Load external spectra and combine with local data
python QA00_ELMvaliditation.py --load-dir /shared/spectra

Dependencies

numpy, pandas, xarray, rioxarray, geopandas, shapely, matplotlib, seaborn, tqdm, GitPython

See the DS02_DatasetQA README for tested package versions and conda environment setup.

---

PE00 — LIDAR Plot Extraction

File: Code/DS03_PlotExtractionCode/PE00_LIDAR_extraction.py
Full documentation: Code/DS03_PlotExtractionCode/README.md
Authors: Arden Burrell & Richard Harwood
Version: v1.0 (09.03.2026)

Extracts plot-level data from LIDAR point clouds (GOBI and CALVIS sensors) by clipping to plot shapefiles and attaching DSM/DTM elevations. The summary below covers the essentials — see the linked README for full folder-structure assumptions, output schemas, troubleshooting, and future enhancements.

What It Does

1. Discovers FieldLog.csv files (or crawls the hierarchy) to build a table of GOBI/CALVIS acquisitions
2. Loads and validates plot-layout shapefiles under Documentation/Plot_Layout/
3. Locates matching *LiDAR_CombinedPointCloud.las, *LiDAR_DSM_*.tif, *LiDAR_DTM_*.tif files
4. Spatial-joins the point cloud to plot polygons and extracts DSM/DTM values per point
5. Computes canopy height (Delta_z = z - DTM)
6. Writes CSV/Parquet tables plus a YAML metadata sidecar (runtime, system, git state)
7. Optionally saves identifiable copies to a shared --save-dir

Command-Line Arguments

Argument	Description
--path	Root path to search (default: git root)
--path-level	Level that --path refers to: root, node, project, site, sensor, date, run (default: site)
-f, --force	Force overwrite of existing output files
--savetype	Output format: csv or parquet (default: parquet)
--save-dir	Save identifiable copies of extracted data for sharing
--load-dir	Load previously extracted data from another node
-v, --verbose	Detailed output

Usage

# Standard run
python PE00_LIDAR_extraction.py

# Process a single site
python PE00_LIDAR_extraction.py --path /data/node/project/2026SiteA --path-level site

# Save shared copies for other nodes
python PE00_LIDAR_extraction.py --save-dir /shared/lidar

Dependencies

numpy, pandas, xarray, rioxarray, laspy, geopandas, shapely, pyyaml, tqdm, GitPython

---

OT00 — Rename Tier to T

File: Code/OT00_OneTimeScripts/OT00_RenameTiertoT.py

A one-time migration script that renames folders starting with Tier to start with T (e.g. Tier0_raw → T0_raw).

Usage

# Use git root as search path
python OT00_RenameTiertoT.py

# Specify a custom path
python OT00_RenameTiertoT.py --path /path/to/data

The script searches recursively, lists all matching folders, asks for confirmation, then performs the rename.

---

OT01 — Move .graw Folders from T1_proc to T0_raw

File: Code/OT00_OneTimeScripts/OT01_MoveGrawToRaw.py

A one-time migration script that relocates *.graw folders from T1_proc into the adjacent T0_raw folder for GOBI and CALVIS sensors only.

What It Does

1. Walks the search root looking for directories named T1_proc
2. Keeps only those whose ancestry includes a folder whose name contains GOBI or CALVIS (case-insensitive)
3. Collects the direct child folders of T1_proc ending in .graw
4. Lists every proposed move and asks for confirmation
5. Moves each .graw folder into the sibling T0_raw/ (created if missing), refusing to overwrite existing targets

Layout Assumption

.../<sensor>/<date>/run_XX/T1_proc/<name>.graw
        ->
.../<sensor>/<date>/run_XX/T0_raw/<name>.graw

Command-Line Arguments

Argument	Description
--path	Optional root directory to search. When supplied, the git repository check is skipped.
--dry-run	List candidate moves without performing them.

Usage

# Use git root as search path
python OT01_MoveGrawToRaw.py

# Preview moves only
python OT01_MoveGrawToRaw.py --dry-run

# Specify a custom path
python OT01_MoveGrawToRaw.py --path /path/to/data

Dependencies

tqdm (and git on PATH when --path is not provided).