Core Functions

Core Functions

Shared utility functions used across the project, located in Code/functions/core_functions/.

parse_APPN_dataset_path

File: Code/functions/core_functions/parse_APPN_dataset_path.py

Parses metadata encoded in an APPN dataset folder path. Given any path within the folder structure, it extracts the node, project, site, sensor, date, run, and tier information.

Function Signature

parse_APPN_dataset_path(
    path: pathlib.Path,
    path_level: str = "auto",
) -> Dict[str, Any]

Parameters

Parameter	Type	Default	Description
path	pathlib.Path	—	Path to parse (file or directory at any level)
path_level	str	"auto"	Level represented by the path. One of: auto, root, node, project, site, sensor, date, run, tier, sub_tier

Return Value

A dictionary with the following keys:

Key	Type	Description
root	str or None	Path to the storage root
node	str or None	Node name (e.g. USYD_Narrabri)
project	str or None	Project folder name
site_folder	str or None	Full site folder name
site	str or None	Site name (without year prefix)
year	int or None	Year extracted from the site folder
sensor	str or None	Sensor platform name
date	pd.Timestamp or None	Date of data collection
run_folder	str or None	Run folder name
run	int or None	Run number
tier	str or None	Tier folder name
sub_tier	str or None	Sub-tier folder name
stem	str or None	Remaining path below sub_tier
valid	bool	Whether the path is valid
errors	list[str]	Validation error messages
path_level	str	Detected or specified level
input_path	str	Original input path as string

Auto-Detection

When path_level="auto" (default), the function:

1. Searches the path ancestry for a date folder matching YYYYMMDD
2. If found, walks upward to populate all parent fields (sensor, site, project, node, root) and downward for run/tier
3. If no date folder found, uses heuristics:
   • Tier folder: matches T\d+_.+ pattern
   • Sensor: matches a known sensor name
   • Site: matches \d{4}.+ pattern
   • Project: matches \d{4}_.+ pattern
   • Node vs root: uses glob patterns to detect date folders at expected depths

Known Sensor Names

The function recognises these sensor platform names:

GOBI, HIRES, M3M, CALVIS, PHENOMATE, MOLE, TEMS, PTEMS,
MPROBES, LITERAL, H30T, RHIZO, MAXAR, JILIN, FIELDOBS, IRT, M3T, SVC

Usage Example

from Code.functions.core_functions import parse_APPN_dataset_path
import pathlib

# Parse a run-level path
result = parse_APPN_dataset_path(
    pathlib.Path("/data/USYD_Narrabri/2025_Chickpea/2025IAWatson/GOBI/20250119/run_00/T0_raw")
)
print(result["node"])     # "USYD_Narrabri"
print(result["project"])  # "2025_Chickpea"
print(result["sensor"])   # "GOBI"
print(result["date"])     # Timestamp('2025-01-19')
print(result["run"])      # 0
print(result["tier"])     # "T0_raw"

# Explicit level specification
result = parse_APPN_dataset_path(
    pathlib.Path("/data/USYD_Narrabri"),
    path_level="node"
)

Tests

Tests are in Code/functions/core_functions/tests/test_parse_APPN_dataset_path.py.

Run with:

conda activate fire
pytest Code/functions/core_functions/tests/test_parse_APPN_dataset_path.py -v

The test suite covers:

• Invalid path_level raises ValueError
• Explicit level parsing for root, node, project, site, sensor, date, run, and tier levels
• Auto-detection at various path depths
• Multiple storage roots