Harden FileStore path and YAML loading

[ceej640]

Summary

• validate embryo IDs before using them as filesystem path components
• route dose-import path lookup through FileStore's validated embryo path helper
• remove the yaml.unsafe_load fallback for legacy Python/numpy constructor tags and fail closed with migration guidance
• add regression tests for path traversal rejection and unsafe YAML rejection

Related issues

• Related to the broader data-services/storage hardening work in Storage: retire legacy SQLite stores and add streaming volume access #16. This PR is only a narrow safety slice, not a full DataStore refactor.

Stacking

This branch is stacked on #23 so the full suite can collect. Once #23 merges into 0.22-dev, this PR should reduce to the FileStore safety commit.

Verification

• uv run pytest tests/test_file_store_safety.py tests/test_gently_store.py -q
• uv run pytest -q
  • 560 passed, 4 skipped

[pskeshu]

Here what we need to figure out is if the GentlyStore data store architecture is sound. How do we make it more biologist friendly - to access and browse the raw data - are we storing all the data correctly to the disk - any data in the gently system that we are not storing, or using? Do we need a Gently4 datastore api? currently we are at Gently3 store.

…

On Sat, 30 May 2026 at 20:56, ceej640 ***@***.***> wrote: Summary - validate embryo IDs before using them as filesystem path components - route dose-import path lookup through FileStore's validated embryo path helper - remove the yaml.unsafe_load fallback for legacy Python/numpy constructor tags and fail closed with migration guidance - add regression tests for path traversal rejection and unsafe YAML rejection Stacking This branch is stacked on #23 <https://github.com/pskeshu/gently/pull/23> so the full suite can collect. Once #23 <https://github.com/pskeshu/gently/pull/23> merges into 0.22-dev, this PR should reduce to the FileStore safety commit. Verification - uv run pytest tests/test_file_store_safety.py tests/test_gently_store.py -q - uv run pytest -q - 560 passed, 4 skipped ------------------------------ You can view, comment on, or merge this pull request online at: https://github.com/pskeshu/gently/pull/25 Commit Summary - cb1ac02 <pskeshu@cb1ac02> Fix test suite collection and stale expectations - c10c71b <pskeshu@c10c71b> Harden FileStore path and YAML loading File Changes (7 files <https://github.com/pskeshu/gently/pull/25/files>) - *M* gently/app/agent.py <https://github.com/pskeshu/gently/pull/25/files#diff-53aab192d7d1dfade17986a08c92196f59663e49661f1820d09f27a1ac091aad> (12) - *M* gently/core/file_store.py <https://github.com/pskeshu/gently/pull/25/files#diff-c31fb487fb58c22f9113990b6c85e6d31ba74ea49e0be231fd882aa8098e6a4d> (78) - *M* gently/harness/conversation.py <https://github.com/pskeshu/gently/pull/25/files#diff-632feaa216b8cc24e32645b92dc64a0783b6c9f26baeb7b4d8c7f5643930ee88> (41) - *M* gently/mesh/mesh_service.py <https://github.com/pskeshu/gently/pull/25/files#diff-73595f2a2a804444d1374305a1b1e2a55774e2ccbe20bfa23a5a72a462791d4e> (20) - *M* tests/test_campaign_coordination.py <https://github.com/pskeshu/gently/pull/25/files#diff-a1bfd6d13f647b1bbfb7650d9b2c4a89560b01a96a56b8191de8d5b7831a7429> (8) - *M* tests/test_dispim_device_safety.py <https://github.com/pskeshu/gently/pull/25/files#diff-e3bc17744c2f44923aea78a254b613462a99a1307f2c6a5c04c94efdf45ce66e> (33) - *A* tests/test_file_store_safety.py <https://github.com/pskeshu/gently/pull/25/files#diff-58b535a5181381499367ebc857ed04e220a0eb45d09b1933e3b9ed6a163d4c49> (45) Patch Links: - https://github.com/pskeshu/gently/pull/25.patch - https://github.com/pskeshu/gently/pull/25.diff — Reply to this email directly, view it on GitHub <https://github.com/pskeshu/gently/pull/25?email_source=notifications&email_token=ABVNN4FLKQIMUGXJ3OVDZBL45N7KPA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF4ZTONZVGI4TEMZUG2THEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVRTG633UMVZF6Y3MNFRWW>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVNN4CGOTWDZZVZQUV2DXD45N7KPAVCNFSM6AAAAACZUBKC4WVHI2DSMVQWIX3LMV43ASLTON2WKOZUGU2TMMJXGQYDMMI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Kesavan

[ceej640]

Agreed. This PR is only a FileStore safety patch; it does not answer the larger datastore architecture question.

I would treat a possible Gently4 datastore API as the result of an audit, not the starting assumption. The first pass should probably be:

• inventory every artifact Gently produces or consumes: raw images/volumes, calibration, embryo/sample positions, temperature/setpoints, commands, plans, transcripts, dose/light exposure records, derived meshes/analysis, and operator annotations
• classify each artifact as durable source data, derived/recomputable data, runtime-only state, or UI/cache state
• verify which durable data are currently persisted, schema-versioned, and recoverable from disk
• identify data Gently uses but does not store, or stores but cannot make accessible to a biologist
• define a biologist-facing browser around sessions, samples/embryos, timepoints, modalities, previews, provenance, and export paths
• then decide whether Gently3 can evolve by migration or whether a Gently4 store API is justified

The safety work here still matters as a prerequisite: a data browser/API cannot be trustworthy if IDs can escape the store root or if legacy YAML can load unsafe constructors.

[ceej640]

Follow-up implemented from this thread in commit 9e93225.

I added docs/datastore-audit.md, which turns the GentlyStore/Gently3/Gently4 question into a concrete audit checklist: durable vs derived data, persistence paths, browse/export needs, missing/stored-but-unused data, and criteria for evolving Gently3 versus defining a Gently4 API.

This keeps this PR focused on FileStore safety while documenting the larger datastore decision path.

Verification:

• git diff --check

[ceej640]

Follow-up implemented from the datastore-architecture thread in commit aa042ab.

What changed:

• Added python -m gently.core.datastore_audit ROOT [--json] [--output PATH] to inventory a Gently datastore on disk.
• The audit counts sessions and major artifact classes, including metadata, logs, snapshots, volumes, volume sidecars, samples, projections, perception traces, debug exports, profiler spans, campaign plans, incoming files, and log files.
• It flags practical gaps such as missing session metadata, volume TIFFs without .meta.yaml, snapshot TIFFs without sidecars, sample directories without embryo.yaml, and a missing sessions tree.
• Added docs explaining how to use the command as the first pass before deciding whether Gently3 can evolve or whether a Gently4 API is justified.

Verification:

• non-writing compile check for gently/core/datastore_audit.py
• pytest tests/test_datastore_audit.py -q -p no:cacheprovider
• git diff --check