ProjectBuilder

ProjectBuilder

ProjectBuilder.py is the main automation script that generates and manages the APPN folder structure. It reads configuration from YAML and CSV files, creates folders, field logs, and project metadata files, and optionally commits changes to git.

Author: Arden Burrell
Version: v1.0 (22.05.2025)

Usage

python ProjectBuilder.py [options]

Command-Line Arguments

Argument	Type	Default	Description
--no-git	flag	False	Disable all git operations (pull, add, commit, push)
--projectsYAML	str	./NodeSummary.yaml	Path to the node YAML file with sensor definitions
-p, --historical	flag	False	Allow historical dates (older than 14 days) in field logs
--enable-sensors	flag	False	If a sensor referenced in a FieldLog.csv row is currently FALSE in {NodeName}_ProjectsSummary.csv, automatically flip it to TRUE and re-save the CSV instead of raising an error. The sensor must already exist as a column in the CSV (i.e. it must be defined in NodeSummary.yaml).

Examples

# Standard run with git integration
python ProjectBuilder.py

# Run without git (useful for testing or offline work)
python ProjectBuilder.py --no-git

# Allow adding field log entries with dates older than 14 days
python ProjectBuilder.py --historical

# Use a custom node YAML
python ProjectBuilder.py --projectsYAML ./custom_nodes.yaml

# Auto-enable any sensors referenced in FieldLog.csv that are still FALSE
# in <node>_ProjectsSummary.csv (CSV is updated and staged for git).
python ProjectBuilder.py --enable-sensors

How It Works

The script follows this sequence:

1. Git pull — Syncs the local repo with the remote (unless --no-git)
2. Load node configuration — Reads NodeSummary.yaml for node and sensor definitions
3. For each node:
   • Creates the node folder if missing
   • Creates or loads {NodeName}_ProjectsSummary.csv
   • Ensures all sensor columns exist in the CSV
   • Warns if any project folders exist on disk that are missing from the CSV
4. For each project in the CSV:
   • Creates Documentation/ and Code/ subfolders
   • Creates or loads ProjectSummary.yaml with project metadata
   • Validates the YAML structure against the default template and adds any missing keys (preserving existing values)
   • Creates or loads FieldLog.csv for tracking field days
   • Creates site folders (with Documentation/ and Code/ subfolders) based on the project YAML
5. For each field log entry:
   • When --enable-sensors is set, any sensor that is still FALSE in {NodeName}_ProjectsSummary.csv is flipped to TRUE before validation; the updated CSV is written back and staged for git
   • Validates the row (date, sensor, site, run count, checksum)
   • Creates the sensor → date → run → tier folder structure
   • For GOBI and CALVIS sensors, also creates T1_proc/QC_data/ and T0_raw/Vault/ (the Vault/ folder is reserved for files that must be preserved from any future programmatic file cleanup or deletion)
   • Optionally creates FieldNotes.txt and RunOverview.csv
   • Resolves MakeNotesFile / MakeTableFile flags back to explicit booleans
   • Updates checksums in the field log
6. Git commit and push — Commits all new files and pushes (unless --no-git)

Generated Files

{NodeName}_ProjectsSummary.csv

A boolean matrix mapping projects to sensor platforms. Each row is a project, each column is a sensor. Values should be TRUE or FALSE.

Project	GOBI	HIRES	M3M	CALVIS	...
2025_Chickpea	TRUE	FALSE	FALSE	TRUE	...

ProjectSummary.yaml

Created in each project folder with template fields:

project:
  ShortName: "2025_Chickpea"
  FullName: ""
  description: ""
  start_date: ""
  end_date: ""
  funding_source: ""
  status: ""
  ProjectCode: ""
  Internal: null
  researcher:
    FirstName: ""
    LastName: ""
    Title: ""
    email: ""
    institution: ""
    role: "Principal Investigator"
    orcid: ""
  sites:
    - name: ""
      year: -9999
      season: ""
      SubLocation: ""
      latitude: .nan
      longitude: .nan
      description: ""
      ControlledEnvironment: null
      sensors: []

FieldLog.csv

Tracks individual field data collection events:

Column	Type	Description
Year	int	Year of collection
Month	int	Month of collection
Day	int	Day of collection
Sensor	str	Sensor platform name
Technician	str	Name of technician
Runs	int	Number of runs (≥ 1)
Site	str	Site name (must match ProjectSummary.yaml)
MakeNotesFile	bool	Whether to create a FieldNotes.txt (default True when blank/NaN; set False to skip)
MakeTableFile	bool	Whether to create a RunOverview.csv (default True when blank/NaN; set False to skip)
CheckSum	float	Integrity checksum (auto-computed)

The MakeNotesFile and MakeTableFile flags accept either booleans or the strings false, f, 0, no, n (case-insensitive). Any other value — including blanks or NaN — is treated as True. The script writes the resolved boolean back into the CSV and recomputes the row checksum.

RunOverview.csv

Created inside each date folder (alongside the run_XX folders) when MakeTableFile is true. One row per run with a RunFailed boolean column for tracking failed acquisitions:

Run	RunFailed
run_00	False
run_01	False

Existing RunOverview.csv files are not overwritten, so additional columns added by users are preserved.

Validation

The Rowchecker function validates every field log entry:

• Data types: Year, Month, Day, Runs must be int; Sensor, Technician, Site must be str
• Date validity: Must parse to a valid date, cannot be in the future, and must be within 14 days (unless --historical)
• Sensor match: Must be a sensor marked TRUE in the project summary CSV
• Run count: Must be ≥ 1
• Site match: Must exist in the ProjectSummary.yaml sites list with a matching year
• Checksum: Ensures row integrity hasn't been corrupted (rows whose checksum changes due to schema updates are auto-recomputed rather than rejected)

YAML Structure Maintenance

When a ProjectSummary.yaml is loaded, its structure is compared against the default template returned by _defaultProjectYAML():

• Missing keys are added with their default values (existing values are never overwritten)
• Keys are reordered to match the canonical template order
• Unknown keys present in the file are preserved at the end of their parent block
• The updated file is rewritten and staged for git

This means new fields added to _defaultProjectYAML() propagate automatically into existing project YAMLs on the next run.

Orphan Project Detection

NodeChecker scans each node folder for sub-directories whose names start with a 4-digit year and underscore (e.g. 2025_). Any such folder that is not listed in {NodeName}_ProjectsSummary.csv triggers a warning so it can be added manually. Items named Documents, Code, sync.ffs_db, or .DS_Store are ignored.

Key Functions

Function	Purpose
main()	Entry point — orchestrates the full workflow
NodeChecker()	Ensures node folder and project summary CSV exist; warns about orphan project folders
projBuilder()	Loads project info, creates folders and metadata
Sitebuilder()	Creates sensor/date/run/tier folders, FieldNotes.txt, and RunOverview.csv for a field day
Rowchecker()	Validates a field log row
_projYAML()	Creates, loads, validates, and (if needed) upgrades the project YAML
_defaultProjectYAML()	Returns the canonical default project YAML structure
check_yaml_structure()	Compares a loaded YAML against the default and returns missing keys
update_yaml_structure()	Adds missing keys to a YAML while preserving existing values and ordering
_sitenamemaker()	Generates standardised site folder names
_df_col_check()	Ensures DataFrames have required columns; clears stale checksums when columns change
GitPull()	Pulls latest changes from remote
GitChanged()	Checks and stages modified files
is_file_staged()	Checks whether a file is already staged in the git index
fileInRepo()	Checks whether a file is tracked in the git repo
pymkdir()	Creates a directory if it does not already exist

Troubleshooting

• Invalid Sensor Error: Ensure the sensor platform is listed in both NodeSummary.yaml and flagged as TRUE in {NodeName}_ProjectsSummary.csv. As a shortcut, re-run with --enable-sensors to have the script automatically set the sensor to TRUE in the project summary CSV (the column must already exist).
• Date Error: If you observe an error for dates older than 14 days, you must use the --historical flag.
• Git Errors: Use the --no-git flag if you want to test folder creation locally without committing to the upstream repository.