Home

APPN DataStorage Wiki

Welcome to the APPN Generic File Storage wiki. This repository provides a standardised data structure and automation scripts for managing research data storage across APPN (Australian Plant Phenomics Network) nodes.

Overview

The project automates the creation and management of a hierarchical folder structure for phenomics research data. It handles folder generation, metadata files (YAML/JSON), field logs, and git integration for tracking changes.

Folder Hierarchy

The APPN data storage follows a strict hierarchical folder layout. Each level encodes metadata in its name, making it possible to programmatically parse paths and extract project, site, sensor, and temporal information. See Folder-Structure for the full naming conventions.

node/
├── {NodeName}_ProjectsSummary.csv
└── YYYY_ProjectDesc{_I|E}{_Researcher}{_org}/
    ├── ProjectSummary.yaml
    ├── FieldLog.csv
    ├── YYYYSiteName{_F|C}/
    │   ├── Documentation/
    │   ├── Code/
    │   └── SensorPlatform/
    │       └── YYYYMMDD/
    │           ├── FieldNotes.txt
    │           ├── RunOverview.csv
    │           └── run_XX/
    │               ├── T0_raw/         # + Vault/ for GOBI, CALVIS (excluded from cleanup)
    │               ├── T1_proc/        # + QC_data/ for GOBI, CALVIS
    │               └── T2_traits/
    ├── Documentation/
    └── Code/

Quick Start

1. Fork and clone the repository
2. Set up a conda environment:

   conda create -n datastorage python=3.12 numpy pandas pyyaml gitpython git tqdm
   conda activate datastorage

3. Edit NodeSummary.yaml to define your nodes and sensor platforms
4. Run python ProjectBuilder.py
5. Fill in the generated {NodeName}_ProjectsSummary.csv with sensor flags (TRUE/FALSE)
6. Re-run python ProjectBuilder.py to build out project folders

Wiki Pages

Page	Description
Folder-Structure	The hierarchical folder layout and naming conventions
ProjectBuilder	The main automation script for folder and metadata creation
NodeSummary-Configuration	How to configure nodes and sensor platforms
Key-Files	Reference for every CSV, YAML, and TXT file created by the script
Code-Modules	Scripts in the Code/ directory (QA, extraction, utilities)
Core-Functions	Shared utility functions used across the project
Changelog	History of changes to the folder hierarchy

Key Files

File	Purpose
ProjectBuilder.py	Main automation script
NodeSummary.yaml	Node and sensor platform definitions
FolderStructureInfo.txt	Reference for the folder naming convention
Code/	Processing scripts, QA tools, and shared functions

License

This project is released under the MIT License.