Folder Structure

Folder Structure

The APPN data storage follows a strict hierarchical folder layout. Each level encodes metadata in its name, making it possible to programmatically parse paths and extract project, site, sensor, and temporal information.

Hierarchy

node/
├── {NodeName}_ProjectsSummary.csv
└── YYYY_ProjectDesc{_I|E}{_Researcher}{_org}/
    ├── ProjectSummary.yaml
    ├── FieldLog.csv
    ├── YYYYSiteName{_F|C}/
    │   ├── Documentation/
    │   │   ├── Plot_Layout/         # {YYYYSiteName}_plots.geojson (+ optional variants, sampling, gcp, sidecar JSON, README.md)
    │   │   └── Trial_Info/          # {YYYYSiteName}_trial_info.csv (+ README.md)
    │   ├── Code/
    │   └── SensorPlatform/
    │       └── YYYYMMDD/
    │           ├── FieldNotes.txt
    │           ├── RunOverview.csv
    │           └── run_XX/
    │               ├── T0_raw/         # + Vault/ for GOBI, CALVIS (excluded from cleanup)
    │               ├── T1_proc/        # + QC_data/ for GOBI, CALVIS
    │               └── T2_traits/
    ├── Documentation/
    └── Code/

Naming Conventions

Separator

Use underscores (_) to separate blocks of information within a folder name.

Node

The top-level folder identifying the APPN node. The university abbreviation comes first.

Example	Meaning
USYD_Narrabri	University of Sydney, Narrabri site

Project

Format: YYYY_ProjectDesc{_I\|E}{_Researcher}{_org}

Component	Required	Description	Example
YYYY	Yes	Year the project started	2025
ProjectDesc	Yes	Short description, no spaces	SIFcalibration
I or E	No	Internal or External	E
Researcher	No	Lead researcher last name (Title Case)	Salter
org	No	Organisation code (Alphanumeric string)	ABC123

Optional arguments must be programmatically identifiable when present.

Examples: 2025_Chickpea, 2025_TestProject, 2026_OZGrain

Site

Format: YYYYSiteName{_F|C}

Component	Required	Description	Example
YYYY	Yes	4-digit year	2025
SiteName	Yes	Name or code of the site	IAWatson
_F	No	Field site	2025IAWatson_F
_C	No	Controlled environment	2025IAWatson_C

Sensor Platform

The short name of the sensor platform. For field measurements (e.g. biomass cuts), use FIELDOBS.

Currently defined platforms:

Sensor	Description
GOBI	GOBI hyperspectral
HIRES	High-resolution camera
M3M	M3M sensor
CALVIS	CALVIS hyperspectral
PHENOMATE	PhenoMate platform
MOLE	MOLE sensor
TEMS	TEMS sensor
PTEMS	PTEMS sensor
LITERAL	LITERAL sensor
H30T	H30T sensor
RHIZO	Rhizotron
FIELDOBS	Field observations

Note

FIELDOBS covers any measurement where the field technician is the sensor — i.e. data collected manually rather than via an instrument. Examples include manual emergence counts, biomass cut weights, head counts, and other hand-recorded observations. Field sensors that log their own data (e.g. SVC, ASD) should instead be added as their own sensor platforms. Information about what field data is collected should be recorded in the RunOverview.csv and FieldNotes.txt files.

Date Folder

Format: YYYYMMDD — the date of field work or data collection, with leading zeros.

Note

For continuous sensors (weather stations, soil moisture probes), this can be the first/last day of the month, the date logging started, or the date of data retrieval.

Run Folder

Format: run_XX — zero-indexed with leading zeros (run_00, run_01, etc.).

A "run" is a block of data processed together:

• For drones: an individual flight
• For soil radar: one pass with the tractor
• For other instruments: analogous acquisition blocks

Tier Folders (Data Processing Levels)

Each run contains three tier folders:

Folder	Purpose
T0_raw	Raw, unprocessed data as collected
T1_proc	Processed data (e.g. orthorectified, calibrated)
T2_traits	Derived traits and analysis outputs

Some sensors (e.g. GOBI, CALVIS) also have a T1_proc/QC_data/ subfolder for quality control data, and a T0_raw/Vault/ subfolder. Files placed inside Vault/ are flagged as protected and will be excluded from any future programmatic file cleanup or deletion routines — use it to preserve any raw data that must survive automated pruning.

Optional Folders

At the project and site levels, optional Documentation/ and Code/ folders can store level-specific documents and scripts. The site-level Documentation/ and Code/ folders are created automatically by ProjectBuilder.py whenever a site folder is created.

Site-level Documentation/

At the site level, ProjectBuilder.py auto-creates two protocol-defined subfolders inside Documentation/:

Subfolder	Purpose	Auto-created files
Plot_Layout/	Plot polygon vector files that every sensor's processing pipeline keys off.	README.md (template stub)
Trial_Info/	Trial-information spreadsheet(s) that join to the plot file via plot_id.	README.md (template stub)

The README stubs are written once and never overwritten — operators are expected to fill them in with site-specific context (current main file, variants in use, deprecation log, source spreadsheets, contacts, quirks). The actual plot/trial data files are authored by the user and follow the conventions below.

Plot_Layout/ file naming

{YYYYSiteName} is the site folder name without the trailing _F / _C suffix (e.g. 2025IAWatson).

Artefact	Filename pattern	Notes
Main plot file (mandatory)	{YYYYSiteName}_plots.geojson	The single required vector file; entry point for every downstream pipeline.
Plot variants (optional)	{YYYYSiteName}_plots_{descriptor|sensor}[_v{NN}].geojson	e.g. _unbuffered, _rszone, sensor-specific (_HIRES).
Sampling files (optional)	{YYYYSiteName}_sampling_{measurement}_{YYYYMMDD}[_v{NN}].geojson	One per measurement type (biomass, height, emergence, headcount, …).
Permanent GCPs (optional)	{YYYYSiteName}_gcp[_v{NN}].geojson	Permanent surveyed markers only — temporary per-flight GCPs live with the flight (run_XX/T0_raw/Vault).
Tool config sidecar (optional)	{YYYYSiteName}_plots_v{NN}.json	e.g. FIELDimageR settings, saved alongside the plot file for reproducibility.
Deprecated copies	…_deprecated.geojson	Superseded files retained for provenance (terminal _deprecated tag).

Important

{YYYYSiteName}_plots.geojson is the only mandatory vector file. ProjectBuilder.py does not currently check for or create it — that validation is handled separately.

Trial_Info/ file naming

Artefact	Filename pattern	Notes
Trial info (current)	{YYYYSiteName}_trial_info.csv	CSV preferred; XLSX accepted. Must include a plot_id column matching the plot file.
Deprecated copies	{YYYYSiteName}_trial_info_{YYYYMMDD}_deprecated.csv	Same _deprecated / _v{NN} conventions as plot files.

See the Plot Delineation protocol for the full spec.

Per-Day Files

At the date-folder level, ProjectBuilder.py can create:

• FieldNotes.txt — empty notes file for the day's work (controlled by the MakeNotesFile column in FieldLog.csv)
• RunOverview.csv — table with one row per run for tracking per-run status (controlled by MakeTableFile)

Both default to being created unless explicitly set to False in the field log.

RunOverview.csv default columns

The script seeds the file with a single user-editable column. Existing files are never overwritten, so any extra columns added by the operator are preserved on subsequent runs. Additional columns should be used to add extra information about the runs in tabular format (e.g. APEx experimental information).

Column	Type	Default	Description
Run	str	run_XX	Index column matching the run folder name (zero-padded)
RunFailed	bool	False	Set to True if the run failed and the data should be excluded from downstream processing

Path Parsing

The parse_APPN_dataset_path() function in Code/functions/core_functions/ can automatically extract metadata from any path in this structure. See Core-Functions for details.