Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
0ba4ada
embryos: split XY position into coarse (bottom-cam) and fine (SPIM) s…
pskeshu May 15, 2026
6f28365
embryos: broadcast EMBRYOS_UPDATE on mutation
pskeshu May 15, 2026
4db06d8
Devices > Map: render embryo waypoints as coarse rings / fine discs
pskeshu May 15, 2026
1ed5fc4
detect_embryos: web Marking canvas replaces napari editor, gate contr…
pskeshu May 15, 2026
5b15797
Devices > Map: pick-up / drop / delete to edit embryos in-place
pskeshu May 15, 2026
7995cbd
device_layer: re-enable XY joystick at boot
pskeshu May 15, 2026
feca96a
Devices > Map: image-anchored crosshair + scroll-to-zoom in camera panel
pskeshu May 15, 2026
294f7dc
eval: event capture / replay / shadow scaffolding for orchestrator A/B
pskeshu May 15, 2026
2f4e1a2
eval: capture production decisions per user turn
pskeshu May 15, 2026
1194a1f
eval: operator-action events + ReactiveCandidate (first real shadow)
pskeshu May 15, 2026
ed75933
docs: closed-loop paradigm write-up
pskeshu May 15, 2026
6dd37be
events: bridge Python logs onto the EventBus / Events page
pskeshu May 16, 2026
7017866
switchbot: BLE room-light accessory device + web test GUI
pskeshu May 28, 2026
d42ac6b
temperature: ACUITYnano thermal controller device (serial/MQTT/mock)
pskeshu May 28, 2026
32a8ad2
device_layer: register switchbot + temperature; expose temperature to…
pskeshu May 28, 2026
3c7bfc7
events: EMBRYO_TERMINATED β€” orchestrator emits, UI tracker handles
pskeshu May 28, 2026
a7250ed
Calibration: SPIM thumb expands into a floating popout for a larger l…
pskeshu May 28, 2026
7f241e1
Embryos > Board: clock / stereo / pace columns replace confidence + rate
pskeshu May 28, 2026
cbf4140
Bump version to 0.21.0.dev0
pskeshu May 28, 2026
63c5be0
switchbot: accept newer 0x05 status-frame response as success
pskeshu May 28, 2026
b6bd49c
config: register room_light SwitchBot for the diSPIM rig
pskeshu May 28, 2026
0158004
switchbot: read_status() β€” battery / firmware via 0x57 0x02 query
pskeshu May 28, 2026
a2a82bc
feat(web): add agent chat window + control lock; retire TUI launch
pskeshu May 29, 2026
9460b78
docs: add biologist-readiness plan
pskeshu May 29, 2026
14a62c3
Merge 0.21-dev into worktree-elegant-purring-russell (integrate web-c…
pskeshu May 29, 2026
99fb073
feat(web): self-managed auth + chat activity indicator + branding
pskeshu May 29, 2026
6917a42
fix(launcher): make Ctrl-C stop the server on Windows
pskeshu May 29, 2026
e075dcd
fix(web): viewing is open; login elevates to control (not a page gate)
pskeshu May 29, 2026
645aace
feat(web): gate person-driven control actions by role
pskeshu May 29, 2026
0fc43c6
feat(web): uniform session transcript across all clients
pskeshu May 29, 2026
f6aad74
fix(store): crash-safe atomic YAML/JSON writes (V3)
pskeshu May 29, 2026
7698dec
fix(tools): get_timelapse_status crashed with KeyError (V1)
pskeshu May 29, 2026
8e3531b
fix(launcher): unify storage env var β€” logs follow data (V4)
pskeshu May 29, 2026
2f4e3e6
fix(web): escape event text to close events-table XSS (V6)
pskeshu May 29, 2026
2b94ffe
docs: update progress β€” auth, gating, transcript, verified bugs done
pskeshu May 29, 2026
2038bf7
feat(web): resume a session from the Sessions tab
pskeshu May 29, 2026
a14372d
fix(config): friendly errors for unknown organism/hardware
pskeshu May 29, 2026
638f827
fix(web): serve session review data from FileStore
pskeshu May 29, 2026
cf2c24e
fix(web): resumed sessions show history + correct session id
pskeshu May 29, 2026
df3db33
feat(web): rehydrate viz imagery from disk on session resume
pskeshu May 29, 2026
c316399
feat(web): rehydrate perception reasoning on resume (populates the vi…
pskeshu May 29, 2026
6d3cc12
feat(web): filmstrip reasoning/detail panel on the side, not the bottom
pskeshu May 29, 2026
8e3d266
feat(web): restore the original filmstrip side-panel CSS (from lost WIP)
pskeshu May 29, 2026
0c69e55
fix(imaging): stored projection is the three-view of View A, not A|B
pskeshu May 29, 2026
e6646fd
feat(scripts): regenerate_projections β€” refresh a session's projectio…
pskeshu May 29, 2026
eead638
fix(scripts): add project root to sys.path in regenerate_projections
pskeshu May 29, 2026
b2bf1f9
fix(web): content-aware cache for FileStore projections
pskeshu May 29, 2026
9240631
fix(imaging): don't split 3D volume by width β€” it halved centered emb…
pskeshu May 29, 2026
95224f7
docs: record dual-view readiness as a design constraint
pskeshu May 29, 2026
a356dfd
feat(launcher): open the web UI in Chrome (GENTLY_BROWSER override, d…
pskeshu May 29, 2026
76505f4
feat(scripts): --all to regenerate projections for every session
pskeshu May 29, 2026
f87b3c9
fix(web): filmstrip thumbnail crops to the XY embryo, not the black c…
pskeshu May 29, 2026
9d0790d
Add room-light toggle and device-layer terminal UI
pskeshu May 29, 2026
cdeabb8
Retire napari from the agent; add web-chat autocomplete; prune dead t…
pskeshu May 29, 2026
c5009ac
Integrate the agent with perception: pull tool, prompt context, event…
pskeshu May 29, 2026
c4bfac2
Live acquisition control, observable + permissioned autonomy, and a r…
pskeshu May 29, 2026
f8f9771
Redesign agent chat as a docked, sliding side panel (overlay + pin-to…
pskeshu May 30, 2026
2a36834
Add a Home landing tab; stop the chat auto-running the startup wizard
pskeshu May 30, 2026
befe98e
Cache YAML parses in FileContextStore β€” fixes slow Plans/campaign loa…
pskeshu May 30, 2026
9369407
Home: recent images aggregated across previous sessions
pskeshu May 30, 2026
7d877ba
Login: non-blocking page with a "Continue in view-only" escape hatch
pskeshu May 30, 2026
596edd3
Home: recent-images was starved by empty trailing sessions
pskeshu May 30, 2026
f64406f
Chat: wider default and a roomier resize ceiling
pskeshu May 30, 2026
cbb8f6a
Open 0.22 dev line: bump version to 0.22.0.dev0
pskeshu May 30, 2026
f33847d
docs(readme): web-first quick start + first-run admin/accounts
pskeshu May 30, 2026
406c87d
docs(readme): dual venv + uv setup instructions
pskeshu May 30, 2026
d6c0082
docs(try-offline): web-first install + dual venv/uv
pskeshu May 30, 2026
bfed3e8
docs(eval): replay-harness design for testing the orchestrator offline
pskeshu May 30, 2026
d50f9d6
docs(readme): "Make your first plan" walkthrough (chat -> /plan -> Pl…
pskeshu May 30, 2026
2e36add
Merge main (fix-name hotfix, PR #34) into 0.22-dev
schneidermc Jun 1, 2026
ce8b647
Relicense and update author list
schneidermc Jun 1, 2026
574bc80
fix(web): use Starlette 1.x TemplateResponse(request, name, ...) sign…
schneidermc Jun 1, 2026
82ed966
Merge pull request #35 from gently-project/relicense
pskeshu Jun 1, 2026
c33d033
Merge pull request #36 from gently-project/fix/web-templateresponse
pskeshu Jun 1, 2026
856df4c
Switch environment setup to uv and add offline/UI-only launch
schneidermc Jun 1, 2026
37ca689
Merge pull request #37 from gently-project/feature/env-setup
pskeshu Jun 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
695 changes: 674 additions & 21 deletions LICENSE

Large diffs are not rendered by default.

159 changes: 135 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Agentic harness for microscopy.

**Status**: v0.11.0 β€” actively developed at Shroff Lab, Janelia.
**Status**: 0.22.0.dev0 β€” actively developed at Shroff Lab, Janelia.

![Safety Architecture](docs/images/safety_architecture.png)

Expand Down Expand Up @@ -67,47 +67,156 @@ Currently, the sample abstraction is the `Embryo` object for *C. elegans* work.

### Prerequisites

- Python 3.11+
- [Node.js](https://nodejs.org/) 18+ (for the Ink TUI)
- An `ANTHROPIC_API_KEY` environment variable
- Python 3.10+
- An `ANTHROPIC_API_KEY` β€” either exported in your shell
(`export ANTHROPIC_API_KEY=your-key`) or placed in a `.env` file in the
project root (`ANTHROPIC_API_KEY=your-key`), which is loaded automatically
on launch and is gitignored. *(Not required if you launch with `--no-api`
to browse the UI only β€” see Launch below.)*
- *(Optional)* `GENTLY_STORAGE_PATH` β€” where sessions and data live (default `D:/Gently3`)

Gently is **web-first**: the agent is driven from an in-page chat in your
browser. There is no TUI to build (Node.js is only needed for the paper
diagrams, not the app).

### Setup

This project uses [uv](https://docs.astral.sh/uv/) for environment and
dependency management. If you don't have it yet, install it following the
[uv installation guide](https://docs.astral.sh/uv/getting-started/installation/)
(e.g. `curl -LsSf https://astral.sh/uv/install.sh | sh` on macOS/Linux).

Gently depends on **`gently-perception`** (the VLM perception harness, repo
`pskeshu/gently-perception`), which is not published to PyPI. Clone it as a
**sibling** of `gently` so the path source in `pyproject.toml` can find it:

```bash
# Clone and install Python dependencies
# Clone both repos side by side
git clone https://github.com/pskeshu/gently.git
git clone https://github.com/pskeshu/gently-perception.git

# Layout:
# <parent>/
# gently/ <- you run commands from here
# gently-perception/ <- resolved via [tool.uv.sources]
```

From the project root run:

```bash
cd gently
pip install -r requirements.txt
uv sync
```

This creates a `.venv` in the project directory and installs the runtime + dev
dependencies pinned in `uv.lock` (see `pyproject.toml` for the spec), including
`gently-perception` as an editable install from the sibling clone. Activate it
with `source .venv/bin/activate`, or just prefix commands with `uv run` (e.g.
`uv run python ...`) to use it without activating.

# Build the TUI (one-time, rebuild after TUI code changes)
cd gently/tui
npm install
npm run build
cd ../..
#### Optional extras

```bash
# Device-layer accessories (microscope computer): BLE/serial/MQTT transports
uv sync --extra device

# GPU acceleration for SAM detection β€” install the CUDA build of PyTorch
# (skip on CPU-only machines; the default torch from uv sync works there)
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
```

#### Running tests

```bash
uv run pytest
```

### Launch

> The commands below use `uv run` so they work without activating the env. If you've activated it first (`source .venv/bin/activate`), the `uv run` prefix isn't necessary.

To verify the install, you can start gently without an API key or hardware. The
web UI boots and is browsable, though the agent itself (chat, perception, plan
mode) stays disabled until you add a key:

```bash
uv run python launch_gently.py --offline --no-api
```

For the full launch:

```bash
# 1. Start the device layer (hardware control + SAM detection)
python start_device_layer.py
# 1. Device layer (hardware control + SAM detection) β€” separate process, own terminal
uv run python start_device_layer.py

# 2. Agent + web UI (starts the in-process server and opens your browser)
uv run python launch_gently.py

# Run without hardware (development / review)
uv run python launch_gently.py --offline

# 2. Launch the agent
python launch_gently.py
# UI-only β€” boot the web UI with no API key (chat/perception disabled)
uv run python launch_gently.py --no-api

# Or launch without hardware (for development / review)
python launch_gently.py --offline
# Don't auto-open a browser β€” open the printed URL yourself
uv run python launch_gently.py --no-browser

# Resume a previous session
python launch_gently.py --resume # interactive picker
python launch_gently.py --resume latest # most recent session
python launch_gently.py --resume <id> # specific session
uv run python launch_gently.py --resume # interactive picker
uv run python launch_gently.py --resume latest # most recent session
uv run python launch_gently.py --resume <id> # specific session

# Verbose / debug logging
python launch_gently.py -v # INFO level
python launch_gently.py --debug # DEBUG level
uv run python launch_gently.py -v # INFO level
uv run python launch_gently.py --debug # DEBUG level
```

The launcher prints a banner with the URL (default `http://localhost:8080`),
device status, storage path, and log location. Open that URL in any browser on
the LAN.

### First sign-in (accounts)

**Viewing is open** β€” the dashboard loads read-only for anyone, no login.
Signing in *elevates* you to control (driving hardware, taking the
single-operator lock); it isn't a gate on the page.

On the **first run**, Gently creates one `admin` account and prints a one-time
random password in the startup banner:

```
First-run admin account created β€” sign in at the URL above:
username: admin
password: <random>
```

- **Save it now** β€” the password is printed to the console once and never
written to the log (only a PBKDF2 hash is stored).
- After signing in, add accounts (roles `viewer` / `operator` / `admin`) via the
admin-only `POST /api/auth/users`.
- **Lost it?** There's no reset command yet β€” delete
`<GENTLY_STORAGE_PATH>/auth/users.yaml` and restart to bootstrap a fresh
`admin` (this clears all accounts).
- **Just trying it locally?** `GENTLY_NO_AUTH=1` disables accounts entirely
(legacy mode: localhost gets control, remote callers need `X-Gently-Token`).

Accounts live under `<GENTLY_STORAGE_PATH>/auth/` (`users.yaml` + `secret.key`),
outside the repo.

## Make your first plan

You don't need a microscope to try the core loop β€” **plan mode is pure agent reasoning and works under `--offline`**. The path from launch to an inspectable plan:

1. **Open the agent chat.** Click **Agent** in the header (or press `Ctrl`/`Cmd`+`J`). New here? The **Home** tab's *Start an experiment* button runs a short setup wizard (also available anytime via `/wizard` β€” it sets the organism, the campaign, and what you're trying to learn).
2. **Enter plan mode** β€” type `/plan` in the chat. The agent switches from *operator* to *scientific collaborator*: it won't touch hardware, it helps you design an experiment.
3. **Describe what you want, in plain language.** For example:
> *"Follow GFP-tagged embryos from bean stage through elongation, imaging every 10 minutes, with a no-laser control β€” three embryos per condition."*

The agent drafts a **campaign**: a sequence of typed **plan items** β€” imaging πŸ“·, bench πŸ§ͺ, genetics 🧬, analysis πŸ“Š, decision points 🚦 β€” each with concrete specs (strain, interval, laser power, Z-slices, target window, success criteria). Keep replying to refine it; `/plan status` shows progress and `/plan exit` returns to run mode.
4. **Inspect it in the plan viewer.** Open the **Plans** tab. Your campaign appears as a card β€” click it to open the **plan document**. Each item shows its status (β—‹ planned Β· β—‘ in progress Β· ● done) and specs; click one to see full details in the inspector. Switch layouts (document / board / graph / timeline) from the view controls, and browse plan **versions** as it evolves. (Typing `/campaign` in chat lists campaigns too.)

That's the loop: **talk β†’ plan β†’ inspect.** With hardware connected (drop `--offline` and start the device layer), the same campaign drives acquisition β€” and perception events can wake the agent to adjust it as the embryos develop.

## Guides

| Guide | Audience | What you'll learn |
Expand All @@ -125,7 +234,7 @@ Four layers with strict downward-only dependencies. The **harness** (reusable ag
gently/
β”œβ”€β”€ core/ # Layer 1: Foundation β€” zero domain knowledge
β”‚ β”œβ”€β”€ event_bus.py # Async pub/sub messaging
β”‚ β”œβ”€β”€ store.py # GentlyStore (SQLite + files)
β”‚ β”œβ”€β”€ file_store.py # FileStore (file-based: YAML / JSONL / TIF)
β”‚ β”œβ”€β”€ imaging.py # Projection, normalization, encoding
β”‚ └── coordinates.py # Pixel/stage transforms
β”‚
Expand Down Expand Up @@ -220,4 +329,6 @@ These papers provide theoretical background for gently's approach:

## License

See [LICENSE](LICENSE) file.
Copyright Β© 2026 Howard Hughes Medical Institute.

Gently is licensed under the GNU General Public License v3.0 or later (GPL-3.0-or-later) β€” see the [LICENSE](LICENSE) file.
11 changes: 10 additions & 1 deletion config/config.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,13 @@
organism: "celegans"
hardware: "dispim"
mmconfig: "MMConfig_tracking_screening.cfg"
mmdirectory: "C:/Program Files/Micro-Manager-1.4"
mmdirectory: "C:/Program Files/Micro-Manager-1.4"

# SwitchBot Bot β€” physical button-pusher mounted on the diSPIM room light
# switch. Talks BLE direct (no SwitchBot Hub / cloud). Plans address it by
# name, e.g. `bps.mv(room_light, 'on')`. Remove this block to skip
# registration; the device layer is tolerant of either state.
switchbot:
name: room_light
address: "EC:6F:04:06:5B:23"
timeout: 20.0
Loading