diff --git a/AGENTS.md b/AGENTS.md index 873347d..2f39bde 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -124,17 +124,20 @@ Connecting individual frames back to their originating sample, scan, and beamtim ### Catalog and Cache Storage -#### Default: local user data directory +#### Default: catalog and zarr cache under `~/.config/pyref` (single tree) -By default, `pyref` maintains a single persistent catalog that accumulates every beamtime the user has ever ingested. The catalog and its associated zarr cache live in the platform-appropriate user data directory, resolved at runtime by the Rust IO layer using the `directories` crate: +By default, `pyref` maintains a single persistent catalog that accumulates every beamtime the user has ever ingested. The **catalog** and **local zarr cache** share the same config root (not macOS “Application Support” unless you override with `PYREF_CATALOG_DB` / `PYREF_CACHE_ROOT`): -| Platform | Default catalog path | -|----------|----------------------| -| Linux | `$XDG_DATA_HOME/pyref/catalog.db` (falls back to `~/.local/share/pyref/catalog.db`) | -| macOS | `~/Library/Application Support/pyref/catalog.db` | -| Windows | `%APPDATA%\pyref\catalog.db` | -The zarr archive for each beamtime is stored **on local disk under the same platform data directory** as the catalog, not under the system cache directory: `/pyref/.cache//beamtime.zarr`, where `` is the same root as in the table above (`$XDG_DATA_HOME` or `~/.local/share`, `~/Library/Application Support`, or `%APPDATA%` as appropriate) and `` is a stable SHA-256 digest of the beamtime root path recorded at ingestion time. Example on macOS: `~/Library/Application Support/pyref/.cache//beamtime.zarr`. The zarr tree is local-only; NAS-backed FITS are used for ingestion and re-ingestion, not for routine image reads after ingest. +| Scope | Default path | +| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `catalog.db` | On macOS, always `~/.config/pyref/catalog.db`. On Linux and Windows, `$XDG_CONFIG_HOME/pyref/catalog.db` when `XDG_CONFIG_HOME` is set; otherwise `~/.config/pyref/catalog.db` (on Windows, `~` is the user profile, e.g. `C:\Users\\.config\pyref\catalog.db`). | +| Zarr (`beamtime.zarr`) | `/cache//beamtime.zarr`. Example on macOS: `~/.config/pyref/cache//beamtime.zarr`. `` is a stable SHA-256 digest of the beamtime root path recorded at ingestion time. The zarr tree is local-only; NAS-backed FITS are used for ingestion and re-ingestion, not for routine image reads after ingest. | + + +macOS ignores `XDG_CONFIG_HOME` for this default tree so a common misconfiguration (`XDG_CONFIG_HOME=$HOME/Library/Application Support`) cannot relocate pyref into Application Support. Use `PYREF_HOME` (tests) or `PYREF_CATALOG_DB` / `PYREF_CACHE_ROOT` when you need a non-default location. + +When `PYREF_HOME` is set (common in tests), both tooling expectations may still point at that directory for the catalog file (`/catalog.db`) as implemented in the Rust path resolver; production use relies on the defaults above unless overridden. Optional environment overrides: `PYREF_CATALOG_DB` (absolute path to `catalog.db`) and `PYREF_CACHE_ROOT` (parent of `/beamtime.zarr` directories). Parallel FITS reads during ingest honor `PYREF_INGEST_WORKER_THREADS` or `PYREF_INGEST_RESOURCE_FRACTION` when explicit kwargs or TUI config fields are unset. @@ -180,12 +183,14 @@ A zarr archive on a fast local network share (e.g., 10GbE NFS or SMB) is accepta This table lives in the catalog database and is machine-local in semantics, even when the catalog is on a shared drive. It stores one row per registered NAS label for the current machine. The Rust IO layer reads this table on startup and caches the mappings in memory for the duration of the process. Agents must never read `path_aliases` directly from Python; path resolution is an IO-layer concern exposed through the `pyref.io` interface. -| Column | Type | Description | -|--------|------|-------------| -| `id` | `Integer` | Primary key. | -| `label` | `Text` | Short user-assigned NAS label (e.g., `als-data`). Unique per catalog. | -| `physical_path` | `Text` | Absolute filesystem path to the mount point on this machine. | -| `registered_at` | `Text` | ISO 8601 timestamp of last registration. | + +| Column | Type | Description | +| --------------- | --------- | --------------------------------------------------------------------- | +| `id` | `Integer` | Primary key. | +| `label` | `Text` | Short user-assigned NAS label (e.g., `als-data`). Unique per catalog. | +| `physical_path` | `Text` | Absolute filesystem path to the mount point on this machine. | +| `registered_at` | `Text` | ISO 8601 timestamp of last registration. | + ### Cataloging System @@ -198,45 +203,59 @@ The `profiles` table is the primary user-facing entry point. Users browse profil The 115 FITS primary HDU cards per frame are split into two tiers at ingestion time. Eleven cards that directly drive scan classification, beamspot localization, normalization, and profile identity are promoted to first-class typed columns on the `frames` table: `sample_x`, `sample_y`, `sample_z`, `sample_theta`, `ccd_theta`, `beamline_energy`, `epu_polarization`, `exposure`, `ring_current`, `ai3_izero`, and `beam_current`. All remaining cards are stored in the `frame_header_values` EAV table, keyed through the `header_cards` registry. The `header_cards` table is populated automatically on first ingestion from whatever cards are present in the FITS files; subsequent beamtimes with new or renamed channels append rows to this table without requiring a schema migration. If a card that was previously treated as non-critical needs to be queried as a first-class column, the correct remedy is a Diesel migration that adds the column to `frames` and backfills it from `frame_header_values`, not a workaround join. #### `beamtimes` + Root of the catalog hierarchy. Stores two path columns: `nas_uri`, which is the logical `nas://