Gently 0.22: web-first operator surface, self-managed auth, and observable permissioned autonomy

[pskeshu]

🌱 This is our first PR opening Gently to outside review — and the first time we're asking the wider community for help. We'd genuinely love yours.

We're getting Gently production-ready for our first user, while making the harness more autonomous and closed-loop between perception and the orchestrator. If you do only one thing: install it, run it offline, and tell us how it feels (see What we'd love your feedback on below). Rough edges are expected — pointing at them is exactly the help we need.

This epoch makes the browser Gently's single control surface and turns the agent into a first-class, observable, permissioned operator: you drive the microscope from an in-page chat, watch from any LAN browser, and see exactly what the agent does on its own and why. It retires the Node/Ink TUI and the desktop napari/Qt windows that were freezing the asyncio loop on the shared instrument PC, adds self-managed accounts with a single-driver control lock, and closes the perception→agent→acquisition loop so the "adaptive" microscope can actually adapt. This opens the 0.22 line (version bumped to 0.22.0.dev0); the prior epoch is tagged v0.21.0.

What we'd love your feedback on

Primary ask: please actually install Gently and run it offline, then drive the web UI. The
hands-on interface feedback is the main thing we want — everything below is ordered by how much it
helps us.

1. Installation & first run (please do this for real). Set up a fresh environment (venv or uv),
   install Gently, and launch it offline (python launch_gently.py --offline). Tell us exactly where
   the README / offline-guide steps snag — a missing dependency, an unclear step, an env var
   (GENTLY_STORAGE_PATH) that isn't explained, anything that made you guess. Offline mode runs the full
   conversational agent + web UI with no microscope attached, so you don't need hardware.

2. The interface (the part we care about most). Once it's up, drive the web UI: open the home page,
   chat with the agent, and click through the tabs. Tell us what's confusing, what's missing, and
   what felt good. Concrete reactions ("I didn't know what this tab was for", "I expected chat to do X")
   are gold.

3. What we built technically (one paragraph). Gently now does agentic perception: the agent reads
   the live Perceiver (developmental-stage calls per embryo per timepoint) and a new wake-router
   (gently/app/wake_router.py) lets it act on developmental events — when perception detects a stage
   transition, arrest, or hatching, the router coalesces/throttles those events and wakes the agent for
   an autonomous reasoning turn between user messages, closing the perceive->decide->acquire loop.

4. Known limitation / open question (we'd value your take). We can't yet easily test the
   orchestrator's perception-driven reasoning offline without a fresh live run — even though we have
   rich recorded sessions on disk (volumes, perception traces, captured event streams). We're designing
   an offline replay harness that republishes recorded perception/lifecycle events onto the agent's
   EventBus on a controllable clock (built on the existing gently/eval/ EventReplay/EventCapture
   substrate) to exercise the real wake-router + agent without hardware. How do you think about testing
   agentic patterns like this — what would make you trust that an offline replay faithfully represents a
   live run?

Where we'd especially value help

Beyond the line-by-line review checklist, three areas where outside perspective matters most:

• The timelapse orchestrator (delicate — actively being streamlined). gently/app/orchestration/timelapse.py drives per-embryo acquisition + perception off wall-clock scheduling (datetime.now() / asyncio.sleep) and has accreted complexity around burst acquisition, per-frame races, cadence, and photodose. It's the most timing-sensitive, safety-critical part of the loop — and the thing we most want to simplify. Ideas for a cleaner scheduling model (and an injectable clock) are very welcome.
• Closing the perception → decide → acquire loop (autonomy). The wake-router + permissioned autonomy (gently/app/wake_router.py) is new: perception events wake the agent to reason and adjust acquisition between user messages. We want this loop genuinely closed and safe — feedback on the wake triggers, coalescing/throttling, the AUTO-mode hard backstop, and the photodose envelope (see What needs review) is high-value.
• Testing agentic patterns offline. We can't yet easily exercise the orchestrator's perception-driven reasoning without a live run, despite rich recorded sessions. We've written up an offline replay harness — docs/EVAL.md, built on the existing gently/eval/ capture/replay substrate. How do you test and trust realtime agent reasoning under simulated conditions? That question is wide open.

Where this is headed

The orchestrator today reasons largely turn-by-turn. We're building toward richer cognitive capabilities — memory, expectations, hypothesis tracking, longer-horizon planning — surfaced and coordinated through a meta-orchestrator: a higher layer that supervises the per-instrument orchestrator(s), holds campaign-level intent, and is where the closed perceive → decide → acquire loop becomes a deliberate, inspectable reasoning process rather than reflexive wakes. If this is your area (cognitive architectures for autonomous microscopy, multi-orchestrator coordination), we'd love to think it through with you.

What we wanted

Gently had two competing, fragile operator surfaces. napari's Qt event loop could synchronously freeze the agent/web asyncio loop mid-tool-call, and the Ink TUI required a Node build step while giving only one person a view. We wanted the browser to be the only console: drive the agent from an in-page chat, view volumes/images in the existing WebGL projection viewer instead of popping desktop windows, let multiple people watch, and give the headless device layer a readable terminal UI of its own.

Once viewing became open to anyone on the LAN, we needed an access model — but one with no external SSO or database dependency, working on plain HTTP. The principle was "viewing is open; login is an elevation to control, not a gate on the page," with default-deny on hardware-moving routes. That also forced us to close a stored-XSS hole in the events table now that arbitrary perception/agent text is viewable by anonymous watchers.

Perception had been a fire-and-forget loop the conversational agent could not see (it subscribed to a STAGE_DETECTED event the perception path never emits), and there was no working tool to change a running timelapse's cadence. We wanted to close that loop: let the agent read the live Perceiver's per-embryo stage/stability/arrest state, give it real live knobs (interval, per-embryo cadence, photodose budget), and let it act between user messages on decision-moment events. Because every action spends light on a living sample, autonomy is deliberately opt-in and gated (OFF/ASK/AUTO), with a human approval round-trip in ASK and a hard registry backstop on irreversible tools.

Underneath, the file-based Gently3 stores needed hardening: the very files /resume depends on were written with an unlink-then-rename pattern that left a crash window; the new Home tab made campaign loading O(N²); and resuming a session restored embryo state but left the UI looking empty.

Finally, the web UI itself needed to feel like a real instrument console: a calm landing page, a dockable resizable chat panel, an open view-only path, and — most consequentially — imagery that faithfully shows the whole embryo in the three-orthogonal-view layout the perceiver sees.

What got made

Web-first migration & napari retirement

launch_gently.py no longer spawns Node/Ink at all — it starts the in-process uvicorn viz server, prints a terminal banner (URL / device status / storage / logs), auto-opens the browser, and blocks on asyncio.Event().wait() until Ctrl-C. The agent is driven from a floating in-page chat over /ws/agent; volumes and images render in the existing WebGL viewer instead of desktop windows.

• --no-browser flag added; the Node/dist requirement and interactive picker removed (run_ink_picker left as dead-for-reference).
• view_volume rewritten to call viz_server.open_volume_in_browser → broadcasts an open_volume WS message → ProjectionViewer.open() (WebGL raymarcher), with file_path mapped back to embryo+timepoint; batch_lightsheet now pushes each image via agent.push_viz.
• Deleted gently/app/tools/data_tools.py (4 Databroker tools) and examples/example_napari_visualization.py (522 lines), with no dangling references.
• @-tool / /-command autocomplete added to the web composer, fed by a new AgentBridge.get_tools_json() sent in the connect frame; tool rows show args + a one-line result_summary with an error heuristic.
• Room-light SwitchBot toggle wired device-layer → client → /api/devices/room_light → a hidden-until-available header button.
• New gently/hardware/console_ui.py: a TTY/Unicode-safe terminal UI (Windows VT, NO_COLOR, cp1252 ASCII fallback) for the device layer's step progress and a plain-language startup-failure panel.

Auth, roles & control gating

A dependency-free AccountStore (accounts.py) stores users in <storage>/auth/users.yaml with PBKDF2-HMAC-SHA256 hashes (200k iterations) and signs stateless session cookies. auth.py gained an account-mode branch to resolve_role: when users exist, identity comes from the signed cookie (operator/admin → control, everyone else → view); otherwise it falls back to legacy loopback / X-Gently-Token mode.

• Stateless cookie: HMAC-SHA256 over username|expiry, urlsafe-b64, 1-week TTL, HttpOnly + SameSite=Lax, Secure only over HTTPS (so it survives plain-HTTP LAN); secret.key created on first run.
• auth_routes.py: /login page plus /api/auth/login|logout|me and an admin-only POST /api/auth/users; launch_gently.py bootstraps a random-password admin on first run and prints it once.
• REST hardware/mutation routes gated by Depends(require_control) (the perception chat POST newly gated); /ws/agent and /ws enforce role from the cookie — viewers can watch but cannot take the single-driver lock or send marking messages.
• events.js XSS fix: escapeHtml applied before injecting <mark> highlight tags at every innerHTML site.

Agent↔perception integration & permissioned autonomy

A new WakeRouter (gently/app/wake_router.py) subscribes to critical events plus DETECTOR_EVALUATED, filters for real developmental transitions/arrest (dropping role=test pseudo-stages, recheck-skips, and the no_object sentinel), coalesces bursts (20s) and throttles non-critical wakes (120s, critical bypasses), then fires agent.run_wake_turn.

• An asyncio.Lock turn-lock serializes user and wake turns over the shared conversation; bridge.stream_response now aclose()s the generator in a finally so the lock always releases on cancel/error.
• A hard backstop in registry.py blocks {set_laser_power, remove_embryo, stop_timelapse} whenever _autonomous_active is set, regardless of mode; wiring verified end-to-end.
• New tools: set_autonomy (off/ask/auto), modify_timelapse_interval, set_embryo_cadence, set_photodose_budget/get_photodose_status, read-only get_recent_perceptions; get_stage_history/predict_hatching now prefer the live Perceiver.
• ASK mode round-trips an Approve/Modify/Skip picker (300s timeout → skip); a deterministic ## Perception (live) snapshot is injected into the system prompt bypassing the cache.
• Autonomous turns are observable: bracketed with autonomous_start/stream_end, broadcast to all web clients, and persisted distinctly as "Gently · autonomous" in chat_display.json.

Storage, session resume & performance

FileStore._write_yaml and save_conversation now fsync the temp file and use os.replace() (atomic, overwrites on Windows), closing the crash window on the files /resume depends on. FileContextStore._read_yaml gained a parse cache keyed by (mtime, size) handing out deepcopies, collapsing campaign-tree builds from O(N²) to O(N).

• New browser resume flow: GET /api/sessions + GET /api/sessions/{id} read the live FileStore; control-gated POST /api/sessions/{id}/resume calls agent.resume_session and broadcasts session_changed.
• rehydrate_session repopulates the in-memory image store from on-disk projections and rebuilds the tracker's detection_reasoning/projection_uids/per-embryo stage from predictions.jsonl; /ws connect overrides the stale tracker session id with the live agent session.
• Resumed sessions derive a best-effort chat transcript from conversation_history when chat_display.json is missing.
• Projection responses switched to a content-aware mtime+size ETag (was immutable + uid) so regenerated JPEGs refresh; new cheap helpers (recent_session_ids, list_embryo_ids, list_projection_timepoints) power the Home aggregator with a component-wise path-traversal guard.

UI surfaces: Home, chat panel, login, imaging

A new HomeApp (home.js) plus a Home tab is the default landing page with three read-only cards (recent sessions with Resume, recent plans with progress chips, a recent-images strip). The chat was rewritten from a floating FAB popup into a docked <aside> side panel inside a new .app-shell flex row.

• Chat panel: overlay-by-default slide-over (no reflow) with opt-in pin-to-dock (a real pushing column), left-edge pointer-capture drag-resize (clamped, persisted to localStorage, double-click reset), header "Agent" toggle with connection dot + unseen-activity badge, Ctrl/Cmd+J, a sticky ASK-approval slot, and pin-to-bottom autoscroll with a "↓ N new" pill.
• imaging.py: 3D volumes now render the full three-orthogonal-view layout via projection_three_view (XY|YZ over XZ) instead of the old A|B side-by-side flat XY max — the wrong image for biologists.
• Two new routes in sessions.py: GET /api/home/recent-images (walks the store on disk, no YAML parse / no pixel decode, scan budget default 200 / clamp 500) and GET /api/sessions/{id}/projection (component-wise ancestor check defeating sibling-prefix dirs).
• Wizard no longer auto-runs on WS connect (gated behind server.wizard_autorun, default off; launched on demand from Home); login.html gained a "Continue without signing in →" view-only escape hatch.
• ProjectionViewer gained a ResizeObserver + gently:layout-changed listener (the 3D WebGL canvas previously never resized); filmstrip got a side reasoning panel and an object-position:left-center thumbnail crop.

What needs review

High risk

• Autonomy backstop completeness — the hard backstop blocks only {set_laser_power, remove_embryo, stop_timelapse}. During an AUTO turn the agent can still call modify_parameters (per-embryo 488 power within the 2–6% clamp, slices, exposure), modify_timelapse_interval/set_embryo_cadence (interval as low as 1s = high dose), and queue_burst. set_photodose_budget is the only dose ceiling and is DISABLED by default. Confirm an unsupervised AUTO agent's photodose envelope matches intent.
• Resume state consistency: per-session subsystems not re-pointed — interaction_logger, event_capture, decision_log, the timeline manager, and AgentMemory are bound to the original session_id at agent construction and are NOT re-initialized on resume. After a browser resume, subsequent activity can still log into the OLD session folder, silently splitting records across two directories. Most material correctness gap in the storage theme.
• Control lock does not gate take_control — the observer gate covers only chat/command/cancel; any observer can seize the wheel with no auth/confirmation, and wizard choice responses are ungated. Two clients can race during the startup wizard on the single shared bridge; disconnect cleanup only clears _choice_futures when the last client leaves, so a mid-flight choice future from a departed client can linger.

Medium risk

• Default-deny gated on has_users() — resolve_role enters account mode only when store.has_users() is true. If users.yaml fails to load (_load_users returns {} and logs), the server silently reverts to legacy mode (localhost gets full control) with no hard failure.
• WebSocket auth lacks Origin/CSRF check — /ws/agent and /ws derive control from the cookie; SameSite=Lax helps but there's no explicit Origin allow-list, and on the intended plain-HTTP LAN the session token travels in cleartext (Secure only set under HTTPS).
• No login rate limiting / lockout — POST /api/auth/login has no throttle/backoff/lockout; the 200k-iteration PBKDF2 cost is the only brute-force mitigation, and the bootstrapped admin username is the fixed string admin.
• Stateless cookie has no revocation — tokens stay valid until 1-week expiry; logout only deletes the client cookie. A demoted operator's role updates, but an exfiltrated token can only be killed by deleting secret.key (invalidates everyone). No per-session jti or rotation-on-privilege-change.
• require_control coverage is per-route, not global — campaigns.py uses a separate parallel auth dependency and volumes.py POST /api/volumes3d (CV subagent push) has no require_control. Any new write route ships unprotected by default — audit that every state-mutating/hardware route is covered or deliberately exempt.
• Unauthenticated cross-session read exposure — GET /api/home/recent-images and GET /api/sessions/{id}/projection are NOT behind require_control; any LAN client can enumerate session/embryo ids and fetch projection JPEGs from ANY session on disk. The broad anonymous GET/SSE/WS surface (sessions, embryos, traces, chat transcripts, campaigns) is open by design — confirm none is sensitive.
• Resume while a timelapse is live — POST .../resume mutates the live agent in place (resets ImageStore, tracker reasoning/uids) with no busy/active-timelapse guard; restarting a running timelapse is explicitly unwired. A resume mid-acquisition could reset state under the running loop.
• ASK picker resolution authority vs control arbitration — register_display_broadcaster is router-scoped with "last registration wins," so the wake choice factory points at the most-recently-connected client, while choice_response is gated to _control['holder'] — these need not be the same client. The picker can render on clients that cannot answer; 300s timeout→skip is the only net.
• Wake-router latency under an in-flight turn — a critical event arriving while the turn-lock is held is deferred and re-armed on the 20s coalesce window, not preempted; "critical bypasses the throttle" does not mean it preempts an active turn. A fast transient (hatching) could be delayed tens of seconds. Intentional correctness/latency tradeoff — confirm acceptable.
• Lock-release / cancellation on run_wake_turn — relies on aclose() in finally; if _resolve_wake_choice is awaiting the picker future when the outer turn cancels, the future is discarded by request_id but not explicitly cancelled, so a stray choice_response could set it later. The CancelledError branch + discard mitigate; trace future ownership across aclose/timeout/disconnect.
• imaging.py projection correctness — generate_jpeg_projection now sends the full 3D volume to projection_three_view and uses View A only for explicit 4D. This was flip-flopped (0c69e55c then 92406310). Correctness hinges on storage convention: if some 3D volumes really are dual-view A|B concatenated along X, they'll now project as a side-by-side A|B three-view. Verified only against a centered embryo in a 2:1 field; whether both conventions coexist isn't established.
• recent-images scan budget vs cost — scan default jumped 6 → 200 (clamp 500); each session pays an iterdir and a per-embryo glob. Worst case (many embryos, few projections) does real disk work on an unauthenticated GET with no server-side cache (the 15s TTL lives only in the browser).
• Web-only launch removes the only blocking foreground process — main() blocks on asyncio.Event().wait() forever; if the viz server fails to start, the banner says so but the process still hangs with no UI and no health re-check / exit-on-failure path.

Low risk

• Username enumeration via timing — verify_password returns immediately for unknown users and runs PBKDF2 only when the user exists; error text is uniform but the exists/not-exists timing channel remains.
• path-traversal guard ordering — get_session_projection calls get_projection_path(...).exists() BEFORE the resolved in sd.parents check, so a crafted embryo triggers a stat() outside the session dir before rejection (not served, but a pre-validation probe). Confirm _session_dir can't be coaxed with a traversal session_id.
• WS vs REST enforcement drift — _ws_can_control (legacy "anyone connected can drive") and require_control (loopback/token) are independent mechanisms with overlapping but not identical logic; confirm they can't disagree such that a remote viewer gains drive via one channel.
• Parse cache cross-process invalidation — the (mtime,size) cache won't invalidate an out-of-process edit preserving byte size within the mtime resolution window (low-probability on NTFS, higher on FAT/network volumes); st_mtime float compared with ==.
• Atomic-write temp files — neither _write_yaml path fsyncs the parent dir after rename; FileContextStore._write_yaml uses a single fixed .tmp name (collision hazard under true concurrency), and orphan .tmp files are only cleaned in the except path.
• Client error heuristic / summary truncation — looksLikeError() regexes the first line and can mislabel benign-but-failed or alarming-but-fine results; result_summary is the first non-empty line truncated to 140 chars, so multi-line errors lose detail.
• open_volume / room_light delivery & status semantics — view_volume reports success on len(active_connections) even if the browser is unattended; room_light/status returns available:false/unknown on ANY exception, hiding the toggle until the next 15s poll, and reads cached bot.read() state with no BLE round-trip.
• Dead code retained — run_ink_picker/pick_session (now just resumes latest), sam_detection.show_in_napari, and multi_embryo.py napari branch are intentional deprecated shims; confirm truly unreachable (e.g. multi_embryo's interactive path raises without a viz_server).
• Rehydrated transcript is lossy — the chat fallback flattens Claude blocks to text-only (drops tool_use/tool_result) and pulls stage/reasoning from predictions.jsonl with detector_name hardcoded to perception. Fine for display — confirm no downstream consumer treats it as authoritative.
• Minor UI: Home /wizard 250ms setTimeout races WS connect + control-status; _resize3D hardcodes h=400; chat width not re-clamped on viewport shrink until next drag; projection-viewer listener could double-bind on re-init without dispose.

Try it

# Collaborators
git fetch origin
git checkout 0.22-dev

Then run the two processes:

# Device layer (separate process, gets its own terminal UI)
python start_device_layer.py

# Agent + web (starts the in-process viz server and opens the browser)
python launch_gently.py

Auth note: on first run the server bootstraps a random-password admin and prints the one-time password once in the startup banner — capture it then. Set GENTLY_NO_AUTH=1 to disable accounts entirely (legacy mode: loopback gets control, remote callers need X-Gently-Token). Use --no-browser to skip auto-opening the browser.

Notes

• napari is fully retired from the agent runtime. It remains only in diagnostics/examples and as deprecated guarded shims (sam_detection.show_in_napari, the multi_embryo plan); no live agent path opens a Qt window.
• No SQLite. All data stays in file-based stores under D:\Gently3\ (human-browsable YAML/JSONL/TIF). This epoch hardened those writes (atomic os.replace + fsync) and added a parse cache.

[pskeshu]

First-run setup & admin password — reviewed, docs added (f33847d)

Reviewed the first-timer instructions against the actual code. The README Quick Start was stale for this web-first line — it told newcomers to install Node and npm run build the Ink TUI (retired in this PR), used pip install -r requirements.txt, and said nothing about the web UI or the admin bootstrap. Fixed in f33847dd (new "First sign-in (accounts)" section, web-first launch steps, file-based-store note).

How the admin password works on first run

launch_gently.py → AccountStore.bootstrap_admin_if_empty():

• Unless GENTLY_NO_AUTH=1, on startup, if users.yaml has no users, Gently creates a single admin with a random secrets.token_urlsafe(12) password and prints it once in the console banner:

  First-run admin account created — sign in at the URL above:
      username: admin
      password: <random>
  

• The password is console-only — it is never written to the log file (only a PBKDF2-HMAC-SHA256 hash, 200k iterations, lives in <GENTLY_STORAGE_PATH>/auth/users.yaml). The cookie-signing key is auth/secret.key, created on first run.
• Add more accounts (roles viewer / operator / admin) via the admin-only POST /api/auth/users.

How to run (first-timer)

pip install -e .
python start_device_layer.py      # device layer (separate process)
python launch_gently.py           # agent + web UI -> http://localhost:8080

--offline (no hardware), --no-browser, and GENTLY_NO_AUTH=1 (disable accounts) as needed.

Gap worth a follow-up

There's no built-in password reset. If the one-time password is missed, the only recovery is deleting auth/users.yaml and restarting (re-bootstraps a fresh admin, clearing all accounts). A --reset-admin / --set-password launcher flag would be a friendlier path — and ties into the auth review items already flagged in the description (fixed admin username, no login rate-limiting, no token revocation). Happy to add that in a follow-up commit if we want it in 0.22.

[pskeshu]

Dual venv + uv setup instructions (406c87d, d6c0082)

Follow-up to the first-run docs: support both environment managers so newcomers can use whichever they have.

• 406c87d — README Setup now offers both paths: python -m venv .venv + pip install -e ., or uv venv + uv pip install -e .. Added a Launch note so uv users know to either activate .venv or prefix with uv run.
• d6c0082 — the 10-minute offline guide (docs/guides/try-offline.md) carried the same stale steps as the README (pip install -r requirements.txt + npm run build for the retired TUI); brought it to web-first and mirrored the dual venv/uv paths.

Why lockfile-free uv: there's no uv.lock / [tool.uv] in the repo and it's a standard PEP 621 pyproject.toml, so uv venv + uv pip install -e . works as a drop-in faster pip with nothing extra to maintain. If we later want reproducible, pinned environments, a follow-up could add uv lock + uv sync (which commits a uv.lock) — glad to do that if the team wants it in 0.22.

[subindevs]

Bug: _autonomous_active set before lock acquired — blocks human hardware commands

gently/app/agent.py:1014

run_wake_turn sets self._autonomous_active = True before the generator is iterated, but _turn_lock is not acquired until inside handle_message_stream (line 950). If a user turn is already holding the lock at that moment, the registry backstop at registry.py:435 sees _autonomous_active=True while the user turn is still executing — silently refusing stop_timelapse, remove_embryo, or set_laser_power on a human-directed command.

Failure scenario: operator sends a message that calls stop_timelapse; WakeRouter fires concurrently, sets the flag, then blocks waiting for the lock. The user turn's stop_timelapse hits the backstop and is refused with "irreversible action cannot run autonomously" — no error is shown to the operator.

Fix: move the flag into handle_message_stream behind an autonomous: bool = False parameter — set it after lock.acquire(), clear it in the same finally before lock.release(). We've opened PR #33 with the patch.