Document sample and hardware domain model

README.md

@@ -174,10 +174,15 @@ That's the loop: **talk → plan → inspect.** With hardware connected (drop `-
 
 | Guide | Audience | What you'll learn |
 |-------|----------|-------------------|
+| [Documentation Home](docs/index.md) | Everyone | Browse the generated-docs structure for Gently |
+| [Full Stack Microscopy](docs/full-stack-microscopy.md) | Everyone | How intent, samples, hardware, perception, data, and operators fit together |
 | [Try Without Hardware](docs/guides/try-offline.md) | Everyone | Run the agent in 10 minutes — conversation, plan mode, perception |
 | [What Gently Can Do](docs/guides/capabilities.md) | Everyone | Perception, detection, plan mode, memory, mesh, safety |
 | [Build a Plugin](docs/guides/build-a-plugin.md) | Developers | Create organism and hardware plugins for other modalities |
 | [Hardware Setup](docs/guides/hardware-setup.md) | Labs | Connect a diSPIM, start the device layer, first acquisition |
+| [Sample & Hardware Model](docs/architecture/sample-hardware-domains.md) | Developers | Generalize samples, hardware operations, and device-profile boundaries |
+| [Sample Tracking Metrics](docs/architecture/sample-tracking-metrics.md) | Developers | Define reusable sample state, exposure, focus, and perception metrics |
+| [Hardware Profile Template](docs/architecture/hardware-profile-template.md) | Developers | Checklist for adding or documenting a new microscope profile |
 
 ## Architecture
 

docs/architecture/hardware-profile-template.md

@@ -0,0 +1,97 @@
+# Hardware Profile Template
+
+Use this template when adding or documenting a new hardware profile.
+
+## Profile Identity
+
+- Profile name:
+- Primary sample type:
+- Primary acquisition modality:
+- Device-layer entry point:
+- Configuration file(s):
+
+## Standard Operations
+
+| Operation | Supported | Implementation | Notes |
+| --- | --- | --- | --- |
+| `sample_overview` | yes/no |  |  |
+| `detect_samples` | yes/no |  |  |
+| `move_to_sample` | yes/no |  |  |
+| `focus_scan` | yes/no |  |  |
+| `position_calibration` | yes/no |  |  |
+| `acquire_snapshot` | yes/no |  |  |
+| `acquire_volume` | yes/no |  |  |
+| `set_illumination` | yes/no |  |  |
+| `read_device_state` | yes/no |  |  |
+
+## Coordinate Frames
+
+List every coordinate frame exposed to the agent or UI.
+
+| Frame | Units | Origin | Axes | Conversion owner |
+| --- | --- | --- | --- | --- |
+| overview pixels | px |  |  |  |
+| stage |  |  |  |  |
+| acquisition volume | voxels |  |  |  |
+
+Required notes:
+
+- Which frame is stored in `position_coarse`.
+- Which frame is stored in `position_fine`.
+- Whether any axis is inverted relative to the overview image.
+- Which code owns pixel-to-stage transforms.
+
+## Safety Limits
+
+| Device/axis | Lower | Upper | Enforcement layer | How verified |
+| --- | --- | --- | --- | --- |
+|  |  |  | device `set()` |  |
+|  |  |  | firmware |  |
+
+Every motion source should have a lowest-layer safety limit. If joystick,
+manual controls, or vendor UI can bypass Python checks, document the firmware or
+operator procedure that closes that path.
+
+## State Reporting
+
+| State field | Source | Poll/callback | Frequency | Read-only |
+| --- | --- | --- | --- | --- |
+| stage position |  |  |  | yes |
+| focus actuator |  |  |  | yes |
+| illumination state |  |  |  | yes |
+| temperature |  |  |  | yes |
+
+State reporting endpoints must not mutate hardware.
+
+## Data Products
+
+| Product | Format | Storage owner | Metadata required |
+| --- | --- | --- | --- |
+| overview image |  |  | sample id, position, pixel size |
+| snapshot |  |  | sample id, channel, exposure |
+| volume |  |  | sample id, voxel size, channel, exposure |
+
+Large arrays should be written by the device layer or persisted once, then
+referenced by uid/path. Avoid repeated JSON/base64 transfer for production data.
+
+## Sample Metrics Populated
+
+Check the metrics this profile updates:
+
+- [ ] `position_coarse`
+- [ ] `position_fine`
+- [ ] `position_history`
+- [ ] `exposure_count`
+- [ ] `total_exposure_ms`
+- [ ] `focus_history`
+- [ ] `signal_intensity_history`
+- [ ] `perception_runs`
+
+## Test Plan
+
+- [ ] Offline import smoke test
+- [ ] Mock device unit tests
+- [ ] Hardware availability diagnostic
+- [ ] Safety limit rejection test
+- [ ] Acquisition dry-run or simulated run
+- [ ] Live acquisition test with explicit operator opt-in

docs/architecture/sample-hardware-domains.md

@@ -0,0 +1,104 @@
+# Sample and Hardware Domain Model
+
+Gently's current production profile is a diSPIM imaging C. elegans embryos, but
+the control model should be described in terms that also fit other microscopes
+and sample types. This document defines that vocabulary.
+
+## Design Rule
+
+Application code should talk about samples, observations, acquisitions, and
+calibration. Hardware modules translate those concepts to device-specific
+plans, device names, and timing details.
+
+The current `Embryo` state is therefore an implementation of a more general
+sample state. It should remain useful for C. elegans while keeping fields and
+operations that other sample models can reuse.
+
+## Concept Mapping
+
+| Gently concept | Current diSPIM implementation | Other modality examples |
+| --- | --- | --- |
+| Sample | C. elegans embryo | well, organoid, cell colony, tissue region |
+| Sample overview | Bottom-camera widefield image | low-mag confocal tile, plate overview, brightfield montage |
+| Sample detection | Embryo detection/marking | well detection, cell segmentation, ROI picking |
+| Position calibration | Pixel-to-stage and SPIM alignment | tile registration, well-to-stage map, objective-specific alignment |
+| Focus scan | Piezo/galvo focus sweep | Z-stack focus curve, autofocus objective sweep |
+| 3D acquisition | Lightsheet volume | confocal Z-stack, widefield deconvolution stack |
+| Timepoint | One scheduled sample observation | one image, stack, or multimodal acquisition at a sample/time |
+| Perception | VLM/classifier stage reasoning | phenotype call, quality control, event detection |
+
+## Standard Operation Names
+
+Hardware profiles should expose conceptually named operations even when their
+device implementation is modality-specific.
+
+| Operation name | Meaning | diSPIM backing operation |
+| --- | --- | --- |
+| `sample_overview` | Capture a field that can locate samples | bottom camera capture |
+| `detect_samples` | Produce sample candidates/ROIs | embryo detector or manual marking |
+| `move_to_sample` | Move the instrument to a sample's resolved position | XY stage move |
+| `focus_scan` | Search focus around a sample or plane | piezo/galvo focus sweep |
+| `position_calibration` | Refine sample position/calibration state | center/verify and SPIM calibration |
+| `acquire_volume` | Acquire a 3D observation | lightsheet volume scan |
+| `acquire_snapshot` | Acquire a 2D observation | lightsheet snap or overview image |
+| `set_illumination` | Configure light source state/power | laser, LED, room light controls |
+| `read_device_state` | Report physical state without mutation | device-layer status endpoints |
+
+Names in tool descriptions, plan metadata, logs, and docs should prefer these
+general concepts. The hardware package may still contain files named for the
+real devices (`bottom_camera`, `galvo`, `piezo`) because those are implementation
+details inside the diSPIM profile.
+
+## Layering
+
+1. `gently.core` defines storage, events, coordinates, and image utilities.
+2. `gently.harness` defines reusable agent, session, tool, prompt, and planning
+   mechanics.
+3. `gently.organisms` defines biological semantics, such as C. elegans stages.
+4. `gently.hardware` defines device profiles and maps standard operations to
+   concrete devices/plans.
+5. `gently.app` composes one organism and one hardware profile into the
+   microscopy agent.
+
+Domain state should move upward as structured sample records. Raw device
+details should stay in the hardware profile unless the UI is explicitly showing
+hardware diagnostics.
+
+## Plan Naming Convention
+
+Use `*_plan` for Bluesky/device-layer plans and include the conceptual operation
+in the metadata when possible:
+
+```python
+_md = {
+    "plan_name": "acquire_volume",
+    "operation": "acquire_volume",
+    "hardware_profile": "dispim",
+}
+```
+
+Recommended naming:
+
+| Preferred | Avoid for new public API | Reason |
+| --- | --- | --- |
+| `sample_overview_plan` | `bottom_camera_plan` | overview generalizes beyond diSPIM |
+| `focus_scan_plan` | `piezo_sweep_plan` | focus is the user-facing concept |
+| `position_calibration_plan` | `center_embryo_plan` | calibration can apply to many samples |
+| `acquire_volume_plan` | `lightsheet_only_plan` | volume acquisition is modality-neutral |
+
+Existing diSPIM-specific names do not need churn. New public tools, docs, and
+metadata should use the conceptual names and point to the diSPIM implementation.
+
+## Extension Checklist
+
+A new hardware profile should document:
+
+- Which device or plan provides `sample_overview`.
+- How sample coordinates map to stage coordinates.
+- Which acquisition operations are supported: snapshot, volume, timelapse,
+  multichannel, burst.
+- Which state readings are available without moving hardware.
+- Which safety limits are enforced in hardware/device classes.
+- Which sample tracking metrics are populated by the profile.
+
+Use `docs/architecture/hardware-profile-template.md` as the starting point.

docs/architecture/sample-tracking-metrics.md

@@ -0,0 +1,122 @@
+# Sample Tracking Metrics
+
+Sample tracking metrics describe what happened to a sample over time. They are
+kept separate from organism-specific interpretation so they can support embryos,
+wells, cells, organoids, or tissue regions.
+
+## Metric Categories
+
+| Category | Purpose | Examples |
+| --- | --- | --- |
+| Position | Where the sample was observed | coarse XY, fine XY, position history |
+| Exposure | How much light/acquisition burden it received | exposure count, total exposure ms |
+| Focus | Whether optical focus is stable | focus history, drift rate |
+| Signal | Quantitative image measurements | channel intensity, photobleaching curve |
+| Perception | Semantic interpretations | developmental stage, health state, event flags |
+| Provenance | How the record was produced | detector id, model version, confidence |
+
+## Universal Fields
+
+These fields are broadly useful across sample types and should remain stable
+where possible.
+
+| Field | Type | Description |
+| --- | --- | --- |
+| `sample_id` | string | Stable id within a session. Current C. elegans code uses `embryo_id`. |
+| `sample_uid` | string/null | Optional globally meaningful id. Current code uses `uid`/`embryo_uid`. |
+| `role` | string | Experimental role such as test, calibration, control, unassigned. |
+| `position_coarse` | object/null | Overview/manual position, usually XY stage coordinates. |
+| `position_fine` | object/null | Refined acquisition position for the primary objective/modality. |
+| `has_fine_position` | bool | True when fine position should override coarse for acquisition. |
+| `position_history` | list | Optional time series of positions and sources. |
+| `exposure_count` | integer | Number of acquisitions or exposure events. |
+| `total_exposure_ms` | number | Integrated illumination/exposure time. |
+| `last_imaged` | ISO datetime/null | Last successful observation time. |
+| `focus_history` | list | Focus measurements keyed by position, modality, and score. |
+| `signal_intensity_history` | object/list | Per-channel measurements over time. |
+| `perception_runs` | list | Links to semantic classifications and reasoning traces. |
+
+## Domain-Specific Fields
+
+Domain fields are valuable, but should be clearly scoped to an organism/sample
+plugin.
+
+| Field | Current meaning | Scope |
+| --- | --- | --- |
+| `developmental_stage` | C. elegans morphology stage | C. elegans organism profile |
+| `hatching_status` | C. elegans hatch state | C. elegans organism profile |
+| `morphology_history` | Stage/shape observations | organism-specific |
+| `custom_classifications` | User-defined labels | experiment-specific |
+
+## Position Semantics
+
+Gently distinguishes coarse and fine positions:
+
+- `position_coarse` comes from an overview image, manual marking, or low-mag
+  sample map.
+- `position_fine` comes from a refined alignment step for the acquisition
+  modality.
+- `stage_position` is a compatibility/read convenience: fine if present,
+  otherwise coarse.
+
+Persistence and import/export code should preserve both positions. Updating the
+coarse position should clear or invalidate fine position unless the update is
+known not to affect fine alignment.
+
+## Exposure Semantics
+
+Exposure tracking should describe burden on the sample, not simply image count.
+
+Minimum record:
+
+```yaml
+sample_id: embryo_1
+timepoint: 12
+modality: lightsheet_volume
+frames: 50
+exposure_ms: 10.0
+channels:
+  488nm:
+    laser_power_pct: 5.0
+    total_ms: 500.0
+created_at: 2026-05-30T12:00:00
+```
+
+For multimodal acquisitions, store one record per channel/modality or include a
+structured `channels` map. Do not collapse illumination wavelength and camera
+exposure into a single ambiguous number.
+
+## Focus Semantics
+
+Focus measurements should include:
+
+- acquisition modality or objective,
+- stage position/context,
+- focus actuator position,
+- score algorithm,
+- score value,
+- timestamp,
+- source: hardware autofocus, FFT, VLM, manual.
+
+This allows drift estimates such as micrometers/hour without tying the schema to
+one microscope.
+
+## Perception Provenance
+
+Every semantic decision should be traceable:
+
+- model or detector id,
+- input image/volume uid,
+- prompt or detector config hash when applicable,
+- confidence or score,
+- reasoning trace path when available,
+- timestamp and triggering event.
+
+This is the bridge between sample tracking and the evaluation/trajectory
+debugging systems.
+
+## Compatibility Notes
+
+Current `EmbryoState` fields map directly onto this schema. Future sample types
+can either implement their own typed state object or reuse a generic
+`SampleState` shape, provided API responses keep the universal fields above.

docs/full-stack-microscopy.md

@@ -0,0 +1,55 @@
+# Full Stack Microscopy
+
+Gently should be documented as a microscopy system, not only as an agent or a
+device controller. The useful view for a biologist or instrument developer is
+the full path from experimental intent to stored evidence.
+
+## Stack Map
+
+| Stack | What it owns | Gently surface |
+| --- | --- | --- |
+| Experimental intent | Scientific question, hypothesis, controls, success criteria | plan mode, campaigns, plan items |
+| Sample preparation | Organism, strain, treatment, mounting, perturbation | sample records, sample-tracking metrics |
+| Hardware integration | Microscope devices, safety limits, device state, calibration | hardware profiles, device layer, profile templates |
+| Acquisition | Snapshots, volumes, timepoints, illumination, temperature | acquisition tools, Bluesky plans, session metadata |
+| Perception | Detection, classification, quality control, event recognition | perception traces, predictions, reasoning records |
+| Closed-loop decisions | When to continue, stop, adapt, or ask the operator | agent tools, event logs, decision logs |
+| Data and provenance | Raw data, derived data, logs, plans, exports | FileStore/GentlyStore, session directories, debug bundles |
+| Operator experience | Setup, monitoring, intervention, recovery | web UI, chat, settings, docs/tutorials |
+
+## Documentation Shape
+
+Generated docs should include three kinds of material:
+
+- Tutorials: task-focused paths such as "run without hardware", "add a hardware
+  profile", and "start a safe timelapse".
+- Concepts: the full-stack map, sample/hardware domain boundaries, and data
+  provenance expectations.
+- References: API surfaces, command-line flags, storage layouts, hardware
+  profile checklists, and test markers.
+
+## Hardware as One Stack
+
+Hardware is a core stack, but it should not dominate the documentation model.
+The device layer matters because it connects intent to physical state safely:
+limits, calibration, timing, illumination, and temperature all shape what
+scientific claims can be made from the data.
+
+Hardware docs should therefore connect each device profile to:
+
+- the sample state it can observe or change
+- the safety boundaries it enforces
+- the metadata it records
+- the simulator or live-hardware tests that cover it
+- the operator workflow for setup and recovery
+
+## Tutorial Roadmap
+
+Priority tutorials:
+
+- run Gently offline and create a plan
+- connect a local diSPIM device layer
+- add a new hardware profile
+- add a new organism/sample type
+- inspect a stored session and export a debug bundle
+- write a hardware contract test and an opt-in live hardware test