Key Files

Reference for every metadata, configuration, and log file involved in the APPN Generic File Storage workflow. This includes the configuration files read by ProjectBuilder and every CSV, YAML, JSON, and TXT file that the script creates or maintains on disk.

File Summary

File	Type	Location	Created By	Purpose
`NodeSummary.yaml`	YAML	Repository root	User (template provided)	Defines nodes and their sensor platforms
`FolderStructureInfo.txt`	TXT	Repository root	Manual reference	Human-readable specification of the folder naming convention
`{NodeName}_ProjectsSummary.csv`	CSV	`./{NodeName}/`	`ProjectBuilder.py`	Boolean matrix of projects vs. sensor platforms
`ProjectSummary.yaml`	YAML	`./{NodeName}/{Project}/`	`ProjectBuilder.py`	Project metadata, researcher, and site definitions
`FieldLog.csv`	CSV	`./{NodeName}/{Project}/`	`ProjectBuilder.py`	Log of every field data collection event for a project
`FieldNotes.txt`	TXT	`.../{Sensor}/{YYYYMMDD}/`	`ProjectBuilder.py`	Free-form field notes for a single collection day
`RunOverview.csv`	CSV	`.../{Sensor}/{YYYYMMDD}/`	`ProjectBuilder.py`	Per-run status table for a single collection day
`Plot_Layout/README.md`	MD	`.../{Site}/Documentation/Plot_Layout/`	`ProjectBuilder.py`	Seeded template; site-specific notes for the plot vector files
`{YYYYSiteName}_plots.geojson`	GeoJSON	`.../{Site}/Documentation/Plot_Layout/`	User	Mandatory main plot polygon file — entry point for every downstream pipeline
`{YYYYSiteName}_plots_*.geojson`	GeoJSON	`.../{Site}/Documentation/Plot_Layout/`	User	Optional plot variants (unbuffered, sensor-specific, etc.)
`{YYYYSiteName}_sampling_*.geojson`	GeoJSON	`.../{Site}/Documentation/Plot_Layout/`	User	Optional sampling-area polygons (one per measurement type)
`{YYYYSiteName}_gcp*.geojson`	GeoJSON	`.../{Site}/Documentation/Plot_Layout/`	User	Optional permanent surveyed GCPs
`{YYYYSiteName}_plots_v{NN}.json`	JSON	`.../{Site}/Documentation/Plot_Layout/`	User	Optional tool-config sidecar (e.g. FIELDimageR settings)
`Trial_Info/README.md`	MD	`.../{Site}/Documentation/Trial_Info/`	`ProjectBuilder.py`	Seeded template; source spreadsheets, contacts, column definitions
`{YYYYSiteName}_trial_info.csv`	CSV	`.../{Site}/Documentation/Trial_Info/`	User	Trial-information spreadsheet; must include a `plot_id` column

Note

The current scripts create only the two README.md stubs inside Documentation/Plot_Layout/ and Documentation/Trial_Info/. The plot polygon, sampling, GCP, and trial-info files themselves are authored by the operator following the protocol conventions.

Important

The following key files are still pending documentation on this page:

QC data files (T1_proc/QC_data/, including QC panel shapefiles *QC_*_Panel*.shp)

`NodeSummary.yaml`

Type: YAML | Location: repository root | Maintained by: user

The primary configuration file. Defines the APPN nodes and the sensor platforms available at each. Read by ProjectBuilder.py at startup. See NodeSummary-Configuration for the full schema.

nodes:
  - name: "USYD_Narrabri"
    university: "University of Sydney"
    location: "Narrabri, NSW, Australia"
    SensorPlatforms:
      - GOBI
      - HIRES
      - M3M

`FolderStructureInfo.txt`

Type: TXT | Location: repository root | Maintained by: user

A plain-text reference document that describes the folder naming convention (node, project, site, sensor, date, run, tier). Not read by any script — kept as on-disk documentation for collaborators. See Folder-Structure for the rendered version.

`{NodeName}_ProjectsSummary.csv`

Type: CSV | Location: ./{NodeName}/ | Created by: NodeChecker() in ProjectBuilder.py

A boolean matrix mapping projects (rows) to sensor platforms (columns). The user fills this in to declare which sensors are valid for which project. New columns are added automatically when sensors are added to NodeSummary.yaml.

Project	GOBI	HIRES	M3M	CALVIS
`2025_Chickpea_E_Salter`	TRUE	FALSE	FALSE	TRUE
`2025_SIFcalibration_I`	FALSE	TRUE	TRUE	FALSE

Note

The first column (index) is the project folder name.
Values must be TRUE or FALSE.
Missing sensor columns are added automatically (filled with False) when NodeSummary.yaml changes.
Project folders that exist on disk but are missing from this CSV trigger an orphan warning (see ProjectBuilder § Orphan Project Detection).
Re-running with --enable-sensors will flip a FALSE to TRUE automatically when a FieldLog.csv row references that sensor.

`ProjectSummary.yaml`

Type: YAML | Location: ./{NodeName}/{Project}/ | Created by: _projYAML() in ProjectBuilder.py

Per-project metadata: researcher, funding, status, and the list of sites belonging to the project. Created from the template in _defaultProjectYAML(). Missing keys are added on subsequent runs without overwriting existing values.

project:
  ShortName: "2025_Chickpea"
  FullName: ""
  description: ""
  start_date: ""
  end_date: ""
  funding_source: ""
  status: ""
  ProjectCode: ""
  Internal: null
  researcher:
    FirstName: ""
    LastName: ""
    Title: ""
    email: ""
    institution: ""
    role: "Principal Investigator"
    orcid: ""
  sites:
    - name: ""
      year: -9999
      season: ""
      SubLocation: ""
      latitude: .nan
      longitude: .nan
      description: ""
      ControlledEnvironment: null
      sensors: []

Key fields

Field	Type	Notes
`ShortName`	str	Mirrors the project folder name
`Internal`	bool / null	`True`/`False` matches the `_I` / `_E` folder suffix
`researcher.*`	str	Lead researcher contact details
`sites[].name`	str	Site identifier — must match `Site` values in `FieldLog.csv`
`sites[].year`	int	Must match `Year` in any `FieldLog.csv` row referencing the site
`sites[].ControlledEnvironment`	bool / null	`True` → site folder gets `_C`, `False` → `_F`, `null` → no suffix
`sites[].sensors`	list	Sensors deployed at the site (e.g. `[GOBI, HIRES]`)

`FieldLog.csv`

Type: CSV | Location: ./{NodeName}/{Project}/ | Created by: projBuilder() in ProjectBuilder.py

One row per field data collection event. Drives folder creation for sensor / date / run / tier sub-trees. Validated by Rowchecker() on every run.

Column	Type	Description
`Year`	int	Year of collection
`Month`	int	Month of collection
`Day`	int	Day of collection
`Sensor`	str	Sensor platform name (must be `TRUE` in `{NodeName}_ProjectsSummary.csv`)
`Technician`	str	Name of the technician who collected the data
`Runs`	int	Number of runs (≥ 1); creates `run_00` … `run_{N-1}` folders
`Site`	str	Site name; must match a site in `ProjectSummary.yaml` with the same year
`MakeNotesFile`	bool	Create `FieldNotes.txt` for this day. Defaults to `True` when blank/`NaN`
`MakeTableFile`	bool	Create `RunOverview.csv` for this day. Defaults to `True` when blank/`NaN`
`CheckSum`	float	Auto-computed integrity hash for the row

Note

The MakeNotesFile and MakeTableFile columns accept false, f, 0, no, n (case-insensitive) to skip creation. Anything else (including blanks) resolves to True. The resolved boolean is written back into the CSV and the checksum is recomputed.
Dates older than 14 days require the --historical flag.
If column schema changes (e.g. new columns added), existing checksums are cleared and recomputed automatically.

`FieldNotes.txt`

Type: TXT | Location: ./{NodeName}/{Project}/{Site}/{Sensor}/{YYYYMMDD}/ | Created by: Sitebuilder() in ProjectBuilder.py

An empty plain-text file created alongside each run_XX/ set, intended for free-form notes recorded during the field day (weather, issues, equipment changes, etc.). Created with pathlib.Path.touch(), so it will not overwrite an existing file. Suppressed when MakeNotesFile is False for the corresponding FieldLog.csv row.

`RunOverview.csv`

Type: CSV | Location: ./{NodeName}/{Project}/{Site}/{Sensor}/{YYYYMMDD}/ | Created by: Sitebuilder() in ProjectBuilder.py

A per-run status table created alongside the run_XX/ folders. One row per run with at minimum a RunFailed boolean column for marking failed acquisitions.

Run	RunFailed
`run_00`	`False`
`run_01`	`False`

Note

The index column (Run) holds the run folder names.
Existing RunOverview.csv files are never overwritten — users may add extra columns and they will be preserved.
Suppressed when MakeTableFile is False for the corresponding FieldLog.csv row.

Site-level `Documentation/Plot_Layout/`

Location: ./{NodeName}/{Project}/{Site}/Documentation/Plot_Layout/

Holds the plot polygon vector files that every sensor's processing pipeline keys off. ProjectBuilder.py auto-creates the folder and seeds a README.md stub; all vector files are authored by the operator. The {YYYYSiteName} token in every filename is the site folder name without the trailing _F / _C suffix (e.g. site 2025IAWatson_F → 2025IAWatson).

Artefact	Filename pattern	Required	Notes
Main plot file	`{YYYYSiteName}_plots.geojson`	Yes	The single mandatory vector file; entry point for downstream pipelines.
Plot variants	`{YYYYSiteName}_plots_{descriptor\|sensor}[_v{NN}].geojson`	No	e.g. `_unbuffered`, `_rszone`, sensor-specific (`_HIRES`).
Sampling files	`{YYYYSiteName}_sampling_{measurement}_{YYYYMMDD}[_v{NN}].geojson`	No	One per measurement type (`biomass`, `height`, `emergence`, `headcount`, …).
Permanent GCPs	`{YYYYSiteName}_gcp[_v{NN}].geojson`	No	Permanent surveyed markers only. Temporary per-flight GCPs live with the flight (`run_XX/T0_raw/Vault`).
Tool config sidecar	`{YYYYSiteName}_plots_v{NN}.json`	No	e.g. FIELDimageR settings, saved alongside the plot file for reproducibility.
Deprecated copies	`…_deprecated.geojson`	No	Superseded files retained for provenance (terminal `_deprecated` tag).
Folder `README.md`	`README.md`	Strongly recommended	Auto-seeded template; site-specific context.

Note

ProjectBuilder.py does not currently validate or create the mandatory {YYYYSiteName}_plots.geojson — that check is handled by a separate (future) tool. The script only ensures the folder and README.md stub exist.

See the Plot Delineation protocol for the full specification.

Site-level `Documentation/Trial_Info/`

Location: ./{NodeName}/{Project}/{Site}/Documentation/Trial_Info/

Holds the trial-information spreadsheet(s) that join to the plot file via plot_id. ProjectBuilder.py auto-creates the folder and seeds a README.md stub.

Artefact	Filename pattern	Required	Notes
Trial info (current)	`{YYYYSiteName}_trial_info.csv`	Strongly recommended	CSV preferred; XLSX accepted. Must include a `plot_id` column matching the plot file. APPN mandates only `plot_id`; every other column is trial-specific.
Deprecated copies	`{YYYYSiteName}_trial_info_{YYYYMMDD}_deprecated.csv`	No	Same `_deprecated` / `_v{NN}` conventions as plot files.
Folder `README.md`	`README.md`	Strongly recommended	Auto-seeded template; source spreadsheets, contacts, column definitions, quirks.

See Plot Delineation protocol — Joining Trial Information for the full spec.

Where Each File Lives

repo_root/
├── NodeSummary.yaml                                      # config (user)
├── FolderStructureInfo.txt                               # reference (user)
└── {NodeName}/
    ├── {NodeName}_ProjectsSummary.csv                    # node-level
    └── {Project}/
        ├── ProjectSummary.yaml                           # project metadata
        ├── FieldLog.csv                                  # project-level log
        └── {Site}/
            ├── Documentation/
            │   ├── Plot_Layout/                          # plot vector files + README.md (auto)
            │   └── Trial_Info/                           # trial spreadsheet + README.md (auto)
            └── {Sensor}/{YYYYMMDD}/
                ├── FieldNotes.txt                        # day-level notes
                ├── RunOverview.csv                       # day-level run table
                └── run_XX/{T0_raw,T1_proc,T2_traits}/    # data tiers

Related Pages

Folder-Structure — full folder hierarchy and naming conventions
NodeSummary-Configuration — schema and examples for NodeSummary.yaml
ProjectBuilder — script that creates and maintains all of the above files
Core-Functions — utilities for parsing the encoded folder paths

Repository · Issues · MIT License · See Contributing-to-the-Wiki to edit these pages.

APPN DataStorage Wiki

Start here

Home

APPN Folder Structure

Guides

Reference

Project

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Key Files

Key Files

File Summary

`NodeSummary.yaml`

`FolderStructureInfo.txt`

`{NodeName}_ProjectsSummary.csv`

`ProjectSummary.yaml`

`FieldLog.csv`

`FieldNotes.txt`

`RunOverview.csv`

Site-level `Documentation/Plot_Layout/`

Site-level `Documentation/Trial_Info/`

Where Each File Lives

Related Pages

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

APPN DataStorage Wiki

Clone this wiki locally

Key Files

Key Files

File Summary

NodeSummary.yaml

FolderStructureInfo.txt

{NodeName}_ProjectsSummary.csv

ProjectSummary.yaml

FieldLog.csv

FieldNotes.txt

RunOverview.csv

Site-level Documentation/Plot_Layout/

Site-level Documentation/Trial_Info/

Where Each File Lives

Related Pages

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

APPN DataStorage Wiki

Clone this wiki locally

`NodeSummary.yaml`

`FolderStructureInfo.txt`

`{NodeName}_ProjectsSummary.csv`

`ProjectSummary.yaml`

`FieldLog.csv`

`FieldNotes.txt`

`RunOverview.csv`

Site-level `Documentation/Plot_Layout/`

Site-level `Documentation/Trial_Info/`