diff --git a/.env.example b/.env.example index 1dc480c..462a595 100644 --- a/.env.example +++ b/.env.example @@ -3,6 +3,9 @@ # OPENCLAW_BASE_URL=http://127.0.0.1:18789 # OPENCLAW_API_KEY= +# OpenRouter (voice messages, Ask mode, smart titles) is configured in the app +# under Settings → "Voice & Ask", not here — the key is stored in the local DB. + PORT=3000 # Default (when unset): ~/.iclaw/data/iclaw.db # DB_PATH=./data/iclaw.db diff --git a/AGENTS.md b/AGENTS.md index 8e1128d..283e6b1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,7 +19,7 @@ These are intentional non-goals. Don't add them. If a request lands here that ne - **Not a generic OpenAI-compat client.** We migrated off `/v1/chat/completions` in commit `a92c10e`. Do not re-introduce an HTTP chat completion path or SSE-based streaming — everything chat-related is native WS. - **Not a multi-provider chat tool.** No fallback to Ollama, vLLM, Claude API, OpenAI direct, etc. OpenClaw is the only backend; switch model providers in `openclaw.json`. - **Not a remote/hosted product.** Loopback-only. No multi-user, no auth layer, no team features. Threat model is "this user on this machine". -- **Not a re-implementation of OpenClaw's agent runtime.** We don't compact transcripts, run tools, manage `MEMORY.md`, or wrap the LLM. We display what the gateway produces and (separately) maintain a project-scoped fact layer that's *additional* to OpenClaw memory, not a replacement. +- **For the default Full Power chat, not a re-implementation of OpenClaw's agent runtime.** On that path we don't compact transcripts, run tools, or wrap the LLM — we display what the gateway produces and (separately) maintain a project-scoped fact layer *additional* to OpenClaw memory. (The Work / Safe work / Incognito modes are the deliberate exception — a small, self-contained agent loop in `packages/iclaw-runtime`; see below. Don't grow it into a general OpenClaw replacement.) - **Not heavyweight on the frontend.** No build step beyond `tsc`. Plain CSS, vanilla JS on the client, EJS for views. If you want to add React/Vue/Svelte, open an issue first — the answer is probably no for the current scope. ## Architecture you need to know @@ -62,6 +62,26 @@ These are intentional non-goals. Don't add them. If a request lands here that ne Browser ↔ server is **one persistent WebSocket** at `/ws`. Chat traffic (send, abort, exec approval, streaming turn events, cross-tab sync) flows through it. HTTP routes exist for page rendering (EJS), form actions, the `/media/*` proxy, and JSON endpoints under `/api/gateway/*`. Every HTTP mutation that touches a chat also emits the matching `chat-updated` / `chat-deleted` over WS so other tabs catch up instantly. +### Runtime modes (Work / Safe work / Incognito) + +The default **Full Power** mode goes to the gateway (above). The other three modes +are served by the bundled **iClaw runtime** (`packages/iclaw-runtime`), a small +agent runtime kept deliberately minimal: + +- The host calls it over HTTP on `127.0.0.1:7430` (`src/services/workRuntime.ts` → + runtime `index.ts`). The model loop runs **on the host** via OpenRouter. +- Tool/shell execution is isolated in a **Docker sandbox**, one model per file: + `secure-runner.ts` (Safe work / Incognito — everything runs in a per-turn + container) and `work-container.ts` (Work — agent loop + file tools on the host, + only `run_command` containerized with per-folder `:ro`/`:rw` mounts). Both share + ONE image, `container/secure-sandbox.Dockerfile` (tag `iclaw-secure:latest`, + built via `npm run build:secure-image`); `node:22` is only an emergency fallback. +- The live runtime is just 9 files (`index.ts`, `sessions.ts`, `secure-runner.ts`, + `work-container.ts`, `agent/{loop,tools,security,prompt-dump}.ts`, `log.ts`). + Path/secret enforcement lives in `agent/security.ts`. Needs Docker + an + OpenRouter key; without either, these modes are unavailable and fall back to + Full Power. This is NOT the OpenClaw path — keep the two cleanly separate. + ### The canonical client paths | What you want | Use | diff --git a/README.md b/README.md index 5a87464..ec7069d 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,30 @@ Hit **Share** in any chat to get an encrypted link. The chat is encrypted in you Powered by [iClaw-cloud](https://github.com/iClawApp/iClaw-cloud) — defaults to `https://app.iclaw.digital`. +## Chat modes + +The composer has a small mode selector. The selected mode is stored per message +(`messages.mode`) and rides the `frontend → WS → chatRunner` path. Missing/unknown +modes fall back to `execute`, so old chats and older clients keep working. Modes +are config-driven in [`src/services/chatModes.ts`](src/services/chatModes.ts) (the +catalog also lists disabled placeholders so new modes can be added without +touching call sites or the DB — the column is plain `TEXT`). + +| Mode | What it does | Backend | +| --- | --- | --- | +| **Full Power** (default) | The full OpenClaw agent — files, tools, shell, browser. | OpenClaw Gateway | +| **Work** | AI edits files in folders you pick; you approve every change. | iClaw runtime | +| **Safe work & Internet research** | Locked Docker sandbox — run untrusted code and research the web, isolated from your system. | iClaw runtime | +| **Incognito** | Private, read-only research — reads files & the web, never writes, nothing saved. | iClaw runtime | + +**Full Power** routes to the gateway exactly as before. The other three run on +the bundled **iClaw runtime** (`packages/iclaw-runtime`): the agent loop runs on +the host and reaches [OpenRouter](https://openrouter.ai), while tool/shell +execution is isolated in a per-turn Docker sandbox. They therefore need **Docker** +running and an **OpenRouter key** (Settings); without either they're unavailable +and the composer falls back to Full Power. See [AGENTS.md](AGENTS.md) for the +runtime architecture. + ## Remote Access (alpha) Open the iClaw UI from another device through an **iclaw-relay** tunnel (Settings → Remote Access). diff --git a/docs/project-skills-spec.md b/docs/project-skills-spec.md new file mode 100644 index 0000000..2559025 --- /dev/null +++ b/docs/project-skills-spec.md @@ -0,0 +1,563 @@ +# Per-Project Skills (Inbox-Gated Self-Learning) — Implementation Spec + +> Audience: an engineer/agent implementing this feature in the iClaw monorepo. +> Read this top-to-bottom before writing code. It tells you exactly which +> existing code to mirror, what to build, and in what order. + +## 0. One-paragraph summary + +Give iClaw a **closed learning loop scoped to projects**: after a chat turn (or +session), a **background reviewer** distills reusable *procedural skills* from +what just happened. Distilled skills do **not** activate automatically — they +land in a **per-project inbox** as *suggestions*. The user accepts/edits/rejects +them (exactly like the existing **project fact suggestions**). Accepted skills +become **active** project skills, stored as **`SKILL.md` markdown** (agentskills.io +format), and their one-line `description` is injected as a **summary index** into +the Work/Secure/Execute system prompt; the full body is loaded on demand. The +user sees a **Skills panel per project** showing each skill's name + description +(the "summary per skill" the product owner asked for). This mirrors Hermes's +self-improvement loop but adds a human-acceptance gate, which neutralizes the two +big risks (skill sprawl and prompt-injection-planted skills). + +This is the **procedural** half of memory. iClaw already has the **declarative** +half (project *facts*). We are duplicating the facts pattern for skills, then +layering Hermes's distillation + the SKILL.md format on top. + +--- + +## 1. Existing patterns to MIRROR (read these first) + +The whole feature is "facts, but richer." Study these before designing: + +| Concern | Facts implementation (mirror it) | +|---|---| +| DB tables | `src/db/database.ts` → `project_facts`, `project_fact_suggestions` (lines ~52–77) | +| Data layer | `src/services/store.ts` → `projectFacts` (~368) and `projectFactSuggestions` (~452) objects | +| Types | `src/types/index.ts` → `ProjectFact`, `ProjectFactSuggestion` | +| Distillation pipeline | `src/services/projectMemory.ts` → `scheduleProjectFactExtraction`, `runProjectFactExtraction`, `runThrowawayTurn`, `isDuplicateFact`, `parseExtractedFactLines` | +| Pipeline trigger | `src/services/chatRunner.ts:676` calls `scheduleProjectFactExtraction({...})` after a successful assistant reply | +| Prompt injection (Work/Secure) | `src/services/chatRunner.ts` `buildWorkSystemPrompt` (~849) prepends facts; the runtime composes the final system prompt in `packages/iclaw-runtime/src/agent/loop.ts` `buildSystemPrompt` | +| Prompt injection (Execute/gateway) | `src/services/projectMemory.ts` `buildGatewayUserMessage` prepends facts to the gateway turn | +| REST: inbox accept/reject | `src/routes/chats.ts` → `GET /:id/fact-suggestions`, `POST /:id/fact-suggestions/:sid/accept`, `.../reject` (~98–158) | +| REST: active-item CRUD | `src/routes/projects.ts` → `PATCH /:id/facts/:factId`, `POST /:id/facts/:factId/delete` (~210–237) | +| WS events | `src/types/protocol.ts` → `project-fact-suggestions`, `project-fact-suggestion-removed`, `project-fact-added/updated/deleted`, `project-facts-synced` (~159–179) | +| UI: inbox cards | `public/js/iclaw.js` → `fact-suggestions-card`, `fact-suggestion-row[data-suggestion-id]`, `removeFactSuggestionRow`, `existingFactSuggestionIds`, `cancelFactSuggestionRowExpiry` | +| UI: active-item panel | the project facts list (search `project-fact` in `public/js/iclaw.js` and the projects/settings view) | + +**Rule of thumb:** for every `fact` symbol there should be a parallel `skill` +symbol. Keep names consistent (`projectSkills`, `projectSkillSuggestions`, +`scheduleProjectSkillReview`, `project-skill-suggestions`, etc.). + +--- + +## 2. What to take from Hermes (the "best of") + +Local clone: `~/.hermes/hermes-agent`. Key references: + +1. **Background-review fork with a restricted toolset.** + `agent/background_review.py` → `spawn_background_review` forks a daemon that + replays the conversation snapshot in a copy of the agent whose tools are + **whitelisted to memory + skill writes only** (everything else denied at + runtime). Main conversation + prompt cache are untouched. + - **Take:** the *pattern* of a separate, least-privilege reviewer that only + emits memory/skill writes. In iClaw this maps to `runThrowawayTurn` + (`projectMemory.ts`) — a throwaway gateway session — except the reviewer + output is parsed into **inbox suggestions**, never written live. + +2. **Be ACTIVE but CLASS-LEVEL (anti-sprawl).** + `_SKILL_REVIEW_PROMPT` in `background_review.py`: "most sessions produce at + least one skill update… Target shape: CLASS-LEVEL skills, each with a rich + `SKILL.md` and a `references/` directory… Not a long flat list of narrow + one-session-one-skill entries. Patch the existing skill if it covers the + learning." + - **Take:** the reviewer must prefer **patching an existing skill** over + creating a new one, and skills should be broad/reusable, not one-offs. + This is what keeps the library from rotting. + +3. **`SKILL.md` format (agentskills.io standard).** + Example: `~/.hermes/hermes-agent/skills/yuanbao/SKILL.md`. Frontmatter: + ```yaml + --- + name: + description: "" + version: 1.0.0 + metadata: + iclaw: + tags: [..] + related_skills: [..] + --- + # + ## <procedure sections...> + ``` + - **Take:** store skills in exactly this format. `description` is the summary + shown in the UI and injected into the prompt index. Body is the procedure. + +4. **Nudge intervals (optional, phase 2).** + `agent/agent_init.py` → `_skill_nudge_interval = 10`. A lighter in-band + reminder every N turns, complementary to background review. + - **Take later:** start with background review only; add nudges if needed. + +5. **Skill manager tool.** `tools/skill_manager_tool.py` → `_create_skill(name, + content, category)`. We don't need the agent to write skills live (the + reviewer does), but we DO need a **read** tool (`get_skill`) so the acting + agent can load a full skill body on demand. See §7. + +What we deliberately **diverge** from Hermes on: +- Hermes skills are **global + category-foldered**; ours are **per-project** + (keyed by `project_id`), matching iClaw facts. (Add an optional `global` scope + at acceptance time — see §5.) +- Hermes writes skills **live**; we route through an **inbox** (human gate). + +--- + +## 3. Core design decisions (locked) + +1. **Inbox-gated.** Distilled skills are `project_skill_suggestions` rows, never + active until accepted. This is the safety spine — do not skip it. +2. **Per-project scope**, with an optional `global` scope chosen at acceptance. +3. **`SKILL.md` markdown** is the storage format for both active skills and + suggestions (the suggestion already holds the full proposed body). +4. **Two suggestion kinds:** `new` (a new skill) and `patch` (an update to an + existing skill `target_skill_id`, carrying the full proposed new body; show a + diff in the UI). +5. **Provenance + trust.** Every suggestion records `source_chat_id` and an + `untrusted` flag (true when the source turn ingested untrusted content — for + now set heuristically: secure-mode turns, or turns that fetched web/email/ + Telegram content). Untrusted suggestions are **always** inbox-gated and + visually flagged. (Auto-accept of trusted suggestions is a future toggle — + see §14; default = everything is gated.) +6. **Summaries-only in the prompt.** Inject `name: description` lines for active + skills (token-bounded, like facts). Full body via `get_skill` tool on demand. +7. **Cost-aware trigger.** Do NOT run the reviewer every turn. Trigger on: + session end / chat idle, OR every N (config, default 8) *substantive* turns + (turns that used tools), whichever comes first. (Facts extraction runs per + turn because it's cheap and append-only; skill review is heavier, so throttle.) + +--- + +## 4. Data model + +Add to `src/db/database.ts` (mirror the facts tables; the schema is applied with +`CREATE TABLE IF NOT EXISTS`, so just append — no migration framework needed, +same as facts): + +```sql +-- Active, accepted project skills (procedural memory). Stored as SKILL.md. +CREATE TABLE IF NOT EXISTS project_skills ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + project_id INTEGER REFERENCES projects(id) ON DELETE CASCADE, -- NULL = global skill + name TEXT NOT NULL, -- kebab-case, unique within scope + description TEXT NOT NULL, -- one-line summary (shown to user + prompt index) + body TEXT NOT NULL, -- full SKILL.md (frontmatter + procedure) + tags TEXT, -- JSON array of strings (optional) + source_chat_id INTEGER REFERENCES chats(id) ON DELETE SET NULL, + usage_count INTEGER NOT NULL DEFAULT 0, + version INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) +); +CREATE INDEX IF NOT EXISTS idx_skills_project ON project_skills(project_id, id); +-- Uniqueness within a scope (project_id may be NULL for global). SQLite treats +-- NULLs as distinct, which is fine — enforce global uniqueness in the store layer. +CREATE UNIQUE INDEX IF NOT EXISTS idx_skills_scope_name + ON project_skills(project_id, name); + +-- Inbox: proposed skills awaiting user acceptance. +CREATE TABLE IF NOT EXISTS project_skill_suggestions ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + project_id INTEGER NOT NULL REFERENCES projects(id) ON DELETE CASCADE, + chat_id INTEGER NOT NULL REFERENCES chats(id) ON DELETE CASCADE, + kind TEXT NOT NULL DEFAULT 'new', -- 'new' | 'patch' + target_skill_id INTEGER REFERENCES project_skills(id) ON DELETE CASCADE, -- for 'patch' + name TEXT NOT NULL, + description TEXT NOT NULL, + body TEXT NOT NULL, -- full proposed SKILL.md + tags TEXT, -- JSON array (optional) + untrusted INTEGER NOT NULL DEFAULT 0, -- 1 if source turn ingested untrusted content + assistant_message_id INTEGER REFERENCES messages(id) ON DELETE SET NULL, + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); +CREATE INDEX IF NOT EXISTS idx_skill_suggestions_chat ON project_skill_suggestions(chat_id, id); +CREATE INDEX IF NOT EXISTS idx_skill_suggestions_project ON project_skill_suggestions(project_id, id); +``` + +--- + +## 5. Store layer (`src/services/store.ts`) + +Add two objects next to `projectFacts` / `projectFactSuggestions`. Mirror their +method shapes exactly. + +```ts +export const projectSkills = { + listByProject(projectId: number): ProjectSkill[] // active skills for a project (+ global merged in by caller if desired) + listGlobal(): ProjectSkill[] + get(id: number): ProjectSkill | undefined + getByName(projectId: number | null, name: string): ProjectSkill | undefined + // index = name + description only (for prompt injection / panel list) + listIndex(projectId: number): { id: number; name: string; description: string }[] + create(opts: { projectId: number | null; name; description; body; tags?; sourceChatId?: number|null }): ProjectSkill + update(id: number, patch: { description?; body?; tags?; name? }): void // bumps version + updated_at + incrementUsage(id: number): void + remove(id: number): void +}; + +export const projectSkillSuggestions = { + listByChat(chatId: number): ProjectSkillSuggestion[] + listByProject(projectId: number): ProjectSkillSuggestion[] + get(id: number): ProjectSkillSuggestion | undefined + insert(opts: { projectId; chatId; kind:'new'|'patch'; targetSkillId?: number|null; name; description; body; tags?; untrusted?: boolean; assistantMessageId: number|null }): ProjectSkillSuggestion + remove(id: number): void +}; +``` + +Notes: +- `update` increments `version` and sets `updated_at`; on accepting a `patch`, + call `projectSkills.update(target, {...})`. +- Enforce scope+name uniqueness in `create` (look up `getByName` first; if it + exists, treat as an update rather than insert — the route layer decides). +- `tags` round-trips as `JSON.stringify` / `JSON.parse`. + +--- + +## 6. Types (`src/types/index.ts`) + +```ts +export interface ProjectSkill { + id: number; + project_id: number | null; // null = global + name: string; + description: string; + body: string; + tags: string | null; // JSON + source_chat_id: number | null; + usage_count: number; + version: number; + created_at: string; + updated_at: string; +} + +export interface ProjectSkillSuggestion { + id: number; + project_id: number; + chat_id: number; + kind: 'new' | 'patch'; + target_skill_id: number | null; + name: string; + description: string; + body: string; + tags: string | null; + untrusted: number; // 0 | 1 + assistant_message_id: number | null; + created_at: string; +} +``` + +--- + +## 7. Skill exposure to the agent (read path) + +Active skills must reach the agent in two layers, matching how facts do it: + +### 7a. Summary index in the system prompt +- **Work/Secure (iclaw-runtime):** the host builds the project block in + `chatRunner.ts buildWorkSystemPrompt`. Add a **"Skills available"** section: + one `- <name>: <description>` line per active skill (project + global), + token-bounded the same way facts are (see `buildGatewayUserMessage`'s budget + logic). Tell the model: *"To use a skill, call `get_skill` with its name to + load the full procedure."* + - The runtime's `packages/iclaw-runtime/src/agent/loop.ts buildSystemPrompt` + already appends the host-supplied `systemPrompt`. No runtime change needed + for the index itself — it rides in via `buildWorkSystemPrompt`. +- **Execute (gateway):** add the same index block in `buildGatewayUserMessage` + (`projectMemory.ts`) alongside the facts block. + +### 7b. `get_skill` tool (load full body on demand) +- **Work/Secure runtime:** add a `get_skill` tool to + `packages/iclaw-runtime/src/agent/tools.ts` (`TOOL_DEFINITIONS` + a handler in + `executeTool`). Problem: the runtime is a separate process and does not have + the iClaw DB. Two options — pick (A) for the MVP: + - **(A) Inject bodies at session create.** When the host creates the work + session (`createWorkSession` in `workRuntime.ts` / `runWorkModeTurn` in + `chatRunner.ts`), pass the active skills (name + body) for the chat's + project as a new `skills` field; the runtime stores them on the session and + `get_skill(name)` reads from that in-memory map. Re-inject when skills + change (the session is already recreated on folder/mode changes — see + `workSessions` signature logic; add skills to that signature so accepting a + skill recreates the session with the new set). Simple, no cross-process DB. + - (B) Add an HTTP endpoint on the host that the runtime calls to fetch a skill + body by `(projectId, name)`. More moving parts; defer. +- **Execute (gateway):** the gateway agent already has tools; expose `get_skill` + via the same mechanism facts use, or simply inline the full bodies of the + top-N most relevant skills if token budget allows. MVP: index-only + instruct + the agent that bodies can be requested; wire the actual fetch in phase 2 if + the gateway can't call back. (Coordinate with how Execute mode tools are + registered — see `openclawWs`.) + +> MVP shortcut acceptable: if `get_skill` plumbing is heavy, start by injecting +> the **full bodies** of up to ~3 small active skills (most-recently-used) and +> just the index for the rest. Make the body-vs-index cutoff a constant. + +On a skill being used (agent calls `get_skill`), bump `usage_count` (host side). + +--- + +## 8. The distillation pipeline (`src/services/projectSkills.ts` — NEW) + +Model it on `projectMemory.ts`. New module, parallel functions. + +### 8a. Trigger +- Add `scheduleProjectSkillReview(opts)` and call it from the **same place** + facts are scheduled (`chatRunner.ts:676` area) **but throttled**: + - Maintain a per-chat counter of substantive (tool-using) turns. + - Fire the reviewer when counter ≥ `SKILL_REVIEW_TURN_INTERVAL` (default 8) OR + on session end (hook where a Work/Secure/Execute session is torn down). + - Also expose a manual "review now" affordance later (phase 2). +- Guard like facts: only when `chat.shares_to_project`, project exists, etc. + +### 8b. Reviewer call (least-privilege, throwaway) +- Reuse `runThrowawayTurn` (gateway 'main' agent) OR a direct OpenRouter call + (see `loadOpenRouterConfig`) — pick whichever the surrounding mode already + uses; gateway throwaway is the established pattern. +- Feed it: the recent turn(s) transcript (user + assistant + which tools ran), + PLUS the **existing skill index** (names + descriptions) for the project so it + can decide patch-vs-new and avoid duplicates. +- The reviewer must output **structured** results so we can build suggestions. + Use a strict, parseable format (JSON preferred; fall back to a line protocol + if the model is unreliable). Target schema: + ```json + { + "skills": [ + { + "action": "new" | "patch", + "target": "<existing-skill-name, only for patch>", + "name": "<kebab-case>", + "description": "<one line>", + "tags": ["..."], + "body": "<full SKILL.md markdown>" + } + ] + } + ``` + If nothing qualifies, the model returns `{"skills": []}` (or literal `NONE`). + +### 8c. Reviewer prompt (adapt Hermes `_SKILL_REVIEW_PROMPT`) +Encode these rules (paraphrase from `~/.hermes/hermes-agent/agent/background_review.py`): +- Prefer **patching** an existing skill over creating a near-duplicate. +- Skills should be **class-level / reusable**, not one-shot task logs. +- A skill captures a *procedure/convention/workflow* discovered this session + that will help next time (commands that worked, gotchas, project conventions, + tool quirks) — NOT transient task state, NOT secrets/paths/credentials (same + exclusions as the facts prompt; reuse that exclusion list). +- Output strictly the JSON above. Same language as the technical content. + +### 8d. Build suggestions +- Parse the JSON. For each entry: + - Dedup against existing **active skills** and **pending suggestions** for the + project (reuse `isDuplicateFact`-style normalization on `name`+`description`). + - For `patch`: resolve `target` name → `target_skill_id`; if not found, + downgrade to `new`. + - Determine `untrusted` (see §10). + - Insert into `project_skill_suggestions`. +- Broadcast a WS event (`project-skill-suggestions`) like + `project-fact-suggestions`, so the chat UI shows inbox cards immediately. + +### 8e. Compaction (phase 2) +Facts compact when the table grows (`compactProjectFacts`). Skills mostly +self-limit via class-level discipline + user pruning, so defer compaction. + +--- + +## 9. REST routes + +### Inbox (per chat) — mirror `src/routes/chats.ts` fact-suggestion routes +- `GET /chats/:id/skill-suggestions` → list for chat. +- `POST /chats/:id/skill-suggestions/:sid/accept` + - body: `{ scope?: 'project' | 'global', body?, description?, name?, tags? }` + (the optional fields let the UI submit user **edits** before accepting). + - if `kind === 'patch'`: `projectSkills.update(target_skill_id, {...})`. + - if `kind === 'new'`: `projectSkills.create({...scope...})`. + - remove the suggestion; broadcast `project-skill-suggestion-removed` + + `project-skill-added`/`project-skill-updated`. + - If accepting changes the active skill set for any open Work/Secure chat in + that project, invalidate those sessions so the new skill is injected (tie + into the `workSessions` signature — see §7b option A). +- `POST /chats/:id/skill-suggestions/:sid/reject` → remove + broadcast removed. + +### Active skills (per project) — mirror `src/routes/projects.ts` facts CRUD +- `GET /projects/:id/skills` → list active (project + global) with index info. +- `GET /projects/:id/skills/:skillId` → full body (for the view/edit modal). +- `PATCH /projects/:id/skills/:skillId` → edit description/body/tags/name. +- `POST /projects/:id/skills/:skillId/delete`. + +--- + +## 10. Security (do not cut corners here) + +This feature is a **prompt-injection amplifier** if done naively: untrusted +content (a web page, an email, a Telegram message) could contain text that the +reviewer distills into a *standing instruction*. The inbox is the primary +defense; reinforce it: + +1. **Everything is inbox-gated by default.** No code path writes an active skill + without an explicit `accept` call from the user. +2. **`untrusted` provenance flag.** Set `untrusted = 1` when the source turn + ingested external content. MVP heuristic: `true` if the turn ran in **Secure + mode** with network enabled, OR used a fetch/`run_command curl`/email/Telegram + tool. Surface this on the inbox card ("learned from untrusted content — + review carefully"). +3. **The reviewer is least-privilege by construction.** It only emits suggestion + text; it has no tools and cannot act. (This is the iClaw analogue of Hermes's + whitelisted reviewer.) +4. **Never distill secrets.** Reuse the facts exclusion list (no tokens, keys, + `.env`, paths, hostnames). Additionally, strip/refuse any skill body that + contains secret-looking patterns (reuse `findBlockedPattern` from + `packages/iclaw-runtime/src/agent/security.ts` conceptually, or a host-side + equivalent). +5. **No auto-accept in MVP.** The trust-level auto-accept (phase 2, §14) must + only ever apply to **trusted** (non-ingested) sessions, never `untrusted=1`. + +Cross-reference: this connects to the broader threat-model discussion (lethal +trifecta, host-mediated creds, egress allowlist). A planted skill that says +"always send X to Y" is only dangerous if the acting agent also has a send +capability — keep capability scoping per task regardless of this feature. + +--- + +## 11. WS protocol (`src/types/protocol.ts`) + +Add events mirroring the fact events: +```ts +| { type: 'project-skill-suggestions'; chatId: number; projectId: number; + projectName: string; + suggestions: { id: number; kind: 'new'|'patch'; name: string; + description: string; untrusted: boolean; targetSkillId: number|null }[] } +| { type: 'project-skill-suggestion-removed'; chatId: number; suggestionId: number } +| { type: 'project-skill-added'; projectId: number; skill: ProjectSkill } +| { type: 'project-skill-updated'; projectId: number; skill: ProjectSkill } +| { type: 'project-skill-deleted'; projectId: number; skillId: number } +``` +(Card list carries only summary fields; the full body is fetched via REST when +the user expands/edits — keeps WS frames small.) + +--- + +## 12. UI (`public/js/iclaw.js` + views) + +### 12a. Inbox cards (in the chat) — extend the fact-suggestion card +Mirror `fact-suggestions-card` / `fact-suggestion-row`. Differences vs facts: +- Each row shows **name + description** (the summary) and a **kind badge** + (`New skill` / `Updates "<target>"`). +- An **untrusted** badge when `untrusted` (amber, "from untrusted content"). +- A **Preview/Edit** affordance: clicking fetches the full body + (`GET /chats/:id/skill-suggestions` returns bodies, or fetch per-suggestion) + and shows it in an expandable area / modal with an editable textarea. +- Accept button → `POST .../accept` with any edits + a scope toggle + (Project / Global). Reject button → `.../reject`. +- Reuse `existingSuggestionIds` / expiry helpers (`cancelFactSuggestionRowExpiry` + analogue) so the inbox self-cleans. + +### 12b. Project Skills panel — mirror the project facts list +- A "Skills" section on the project view (next to Facts): list each active skill + as **name — description**, with view (open body modal), edit, delete. +- This list IS the "short summary per skill" deliverable. +- Live-update via the new `project-skill-*` WS events. + +### 12c. Composer indicator (optional, phase 2) +A small "N skills" chip in the composer for Work/Execute, like the work-folders +count, so the user knows skills are active for this project. + +--- + +## 13. Wiring checklist (file-by-file) + +1. `src/db/database.ts` — add the two tables + indexes (§4). +2. `src/types/index.ts` — add `ProjectSkill`, `ProjectSkillSuggestion` (§6). +3. `src/services/store.ts` — add `projectSkills`, `projectSkillSuggestions` (§5). +4. `src/services/projectSkills.ts` (NEW) — reviewer pipeline (§8). +5. `src/services/chatRunner.ts` — call `scheduleProjectSkillReview` (throttled), + add the skills index to `buildWorkSystemPrompt`, and (option A §7b) pass + active skills into `createWorkSession` + include them in the `workSessions` + recreation signature. +6. `src/services/workRuntime.ts` — add `skills` to `CreateSessionOptions` and + send it. +7. `packages/iclaw-runtime/src/index.ts` + `sessions.ts` — accept `skills` on + session create, store on the session. +8. `packages/iclaw-runtime/src/agent/tools.ts` + `loop.ts` — add `get_skill` + tool reading the session's skill map; bump usage via an event/log the host + can observe (or skip usage tracking in MVP). +9. `src/services/projectMemory.ts` — add skills index to `buildGatewayUserMessage` + (Execute path). +10. `src/routes/chats.ts` — skill-suggestion inbox routes (§9). +11. `src/routes/projects.ts` — active-skill CRUD routes (§9). +12. `src/types/protocol.ts` — WS events (§11). +13. `public/js/iclaw.js` (+ relevant `views/*.ejs`, `public/css/style.css`) — + inbox cards + project Skills panel (§12). + +--- + +## 14. Phasing + +**MVP (ship first):** +- Tables, store, types. +- Reviewer pipeline (throttled, JSON output, dedup, patch-vs-new). +- Inbox suggestions + WS + chat cards (accept/edit/reject, scope, untrusted badge). +- Project Skills panel (list/view/edit/delete) — the summary view. +- Prompt **index** injection (Work/Secure via `buildWorkSystemPrompt`, Execute + via `buildGatewayUserMessage`). For bodies, use the "inline top-N small skills" + shortcut (§7b) — defer real `get_skill`. + +**Phase 2:** +- `get_skill` tool (runtime in-memory map, option A) + usage tracking. +- Trust-level auto-accept of **trusted** suggestions (config per project; + untrusted always gated). +- Nudge intervals (Hermes-style in-band reminder). +- Skill compaction / merge when a project accumulates many. +- Composer "N skills" chip. + +--- + +## 15. Acceptance criteria (definition of done for MVP) + +- [ ] After ~8 tool-using turns in a project chat (or on session end), a skill + suggestion can appear in the chat as an inbox card with name + description. +- [ ] No skill ever becomes active without the user clicking Accept. +- [ ] Accepting a `new` suggestion creates a `project_skills` row (SKILL.md + body); accepting a `patch` updates the target and bumps `version`. +- [ ] The user can edit the body before accepting. +- [ ] Suggestions from untrusted sessions show the untrusted badge. +- [ ] The project Skills panel lists each active skill as name — description, + with view/edit/delete, live-updating over WS. +- [ ] Active skills' `name: description` index appears in the Work/Secure and + Execute system prompts, token-bounded. +- [ ] Rejecting / deleting works and broadcasts the removal. +- [ ] Type-checks clean: `npm run typecheck` (host) and runtime `tsc --noEmit` + (ignore the pre-existing `agent-to-agent` / `db-v2.test` errors). + +--- + +## 16. Open decisions (resolve with product owner if blocked) + +1. **Reviewer model:** gateway 'main' agent (consistent with facts) vs a cheap + OpenRouter model (`SUMMARY_MODEL` style). Default: reuse the facts pattern + (gateway throwaway) for MVP. +2. **Trigger cadence:** every-N-turns vs session-end vs both. Default: both, + N=8. +3. **Global scope at acceptance:** include the Project/Global toggle in MVP, or + project-only first? Default: include the toggle (cheap, and useful). +4. **`get_skill` vs inline bodies:** MVP uses inline-top-N; confirm that's + acceptable or prioritize the tool. + +--- + +## 17. Don't break + +- Facts must keep working untouched — skills are additive and parallel. +- The Work/Secure session recreation logic (`workSessions` signature in + `chatRunner.ts`) already recreates on folder/mode change; if you add skills to + the injected session state, **add skills to that signature** so accepting a + skill takes effect without a manual restart (this is the same class of bug we + already fixed for folder-access and mode changes — see git history + `fix(work): recreate session on Work<->Secure mode switch`). +- Keep the reviewer **off the hot path** (fire-and-forget, like + `scheduleProjectFactExtraction`); never block the user's turn on it. diff --git a/package-lock.json b/package-lock.json index 976c7b2..3ba966b 100644 --- a/package-lock.json +++ b/package-lock.json @@ -8,6 +8,9 @@ "name": "@iclawapp/iclaw", "version": "0.2.2", "license": "MIT", + "workspaces": [ + "packages/*" + ], "dependencies": { "@noble/hashes": "^2.2.0", "@serenity-kit/opaque": "^1.1.0", @@ -573,6 +576,10 @@ "node": ">=18" } }, + "node_modules/@iclaw/runtime": { + "resolved": "packages/iclaw-runtime", + "link": true + }, "node_modules/@jridgewell/resolve-uri": { "version": "3.1.2", "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", @@ -1299,6 +1306,182 @@ "url": "https://opencollective.com/vitest" } }, + "node_modules/@vscode/ripgrep": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep/-/ripgrep-1.18.0.tgz", + "integrity": "sha512-ns5lWe44tSfbTMbVUsyB+I1819PVSw4AdpgK0RNkzfWfwy6+3IUNSxwSrfTno1/oWaS/hERNz+XLWVyga2aJBQ==", + "license": "MIT", + "optionalDependencies": { + "@vscode/ripgrep-darwin-arm64": "1.18.0", + "@vscode/ripgrep-darwin-x64": "1.18.0", + "@vscode/ripgrep-linux-arm": "1.18.0", + "@vscode/ripgrep-linux-arm64": "1.18.0", + "@vscode/ripgrep-linux-ia32": "1.18.0", + "@vscode/ripgrep-linux-ppc64": "1.18.0", + "@vscode/ripgrep-linux-riscv64": "1.18.0", + "@vscode/ripgrep-linux-s390x": "1.18.0", + "@vscode/ripgrep-linux-x64": "1.18.0", + "@vscode/ripgrep-win32-arm64": "1.18.0", + "@vscode/ripgrep-win32-ia32": "1.18.0", + "@vscode/ripgrep-win32-x64": "1.18.0" + } + }, + "node_modules/@vscode/ripgrep-darwin-arm64": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-darwin-arm64/-/ripgrep-darwin-arm64-1.18.0.tgz", + "integrity": "sha512-r3ktHSvbFycQNF6sl7sNDPocpsI7J+mEzh1IaZFkY0spm3k2Z9t8hPAeOK7+p0l6p6/swkQC14XWX01low+94Q==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@vscode/ripgrep-darwin-x64": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-darwin-x64/-/ripgrep-darwin-x64-1.18.0.tgz", + "integrity": "sha512-25b4gWbL138dGuQU244ebCKKc0q05ULBMoFSz9oAEUHNeqK/lOJViDS7DRvbDazzAzSEdan391Znks/R5mkaTQ==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@vscode/ripgrep-linux-arm": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-linux-arm/-/ripgrep-linux-arm-1.18.0.tgz", + "integrity": "sha512-GDAvufNDHu8zqLEmXstalQF0Wh6wQvdsBi/Vg3Yi3CK4a8XoFXqqXVEHEZ9xQz3t0NfoSEc9JbvK9DDS6FxyxQ==", + "cpu": [ + "arm" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@vscode/ripgrep-linux-arm64": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-linux-arm64/-/ripgrep-linux-arm64-1.18.0.tgz", + "integrity": "sha512-lQ/5zTG++U0E3IhVgS4EPTTn/U4okncaRMM5GOFfOYZywS4nuD31GhkHbNYlDk5CuDC68+hYJ0/eQeyCKJDA+g==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@vscode/ripgrep-linux-ia32": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-linux-ia32/-/ripgrep-linux-ia32-1.18.0.tgz", + "integrity": "sha512-YWLkSUtFd4Jh5EepIhA9RJSfv3uMAVMo+2rBIGHPBnvgLrZciIs2cDKei1/p6Wc/aCzUoHyMAg2R6tw4ZCBKGg==", + "cpu": [ + "ia32" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@vscode/ripgrep-linux-ppc64": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-linux-ppc64/-/ripgrep-linux-ppc64-1.18.0.tgz", + "integrity": "sha512-quXVY8fwQ8O/lvU1yrSqSl3jlUzysRSb+AfUfCL/tRtphxsKlFvPAejryZ6vg4Bgvn8XL74xb4qMCDmWgYrT5w==", + "cpu": [ + "ppc64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@vscode/ripgrep-linux-riscv64": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-linux-riscv64/-/ripgrep-linux-riscv64-1.18.0.tgz", + "integrity": "sha512-f5kBQBrWfQt8Q7OhSORuNDei5dkYagBj3y4jImSUXGMy8B/Ke7SltSRcUtjPv166FAFfHCAmWuZp3+cWnX2/Vw==", + "cpu": [ + "riscv64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@vscode/ripgrep-linux-s390x": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-linux-s390x/-/ripgrep-linux-s390x-1.18.0.tgz", + "integrity": "sha512-rTOcJFGGcl2c07RUOWUo4U1ndnemKhY6A9hnMB18uk7jSgJc0d/QLBGWMWpumdtoJtpizn/wIv5mXIisJukusQ==", + "cpu": [ + "s390x" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@vscode/ripgrep-linux-x64": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-linux-x64/-/ripgrep-linux-x64-1.18.0.tgz", + "integrity": "sha512-mQ3bVrUpnD2vs7QT0vX90Lt0cnUq467uFtEktIdsJJmW296RoSULRGqWgzG1AKxyBpNDD6l4ZO4qKf6SgyC23Q==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@vscode/ripgrep-win32-arm64": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-win32-arm64/-/ripgrep-win32-arm64-1.18.0.tgz", + "integrity": "sha512-vfTIjq1OHnzUjxZcHVQAMbnggp8dpGf+0QKFOZHwWPqFwXxQC8eCWM+5NUdoJ6yrElCeMzoUTXoK/LdZaniB+Q==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@vscode/ripgrep-win32-ia32": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-win32-ia32/-/ripgrep-win32-ia32-1.18.0.tgz", + "integrity": "sha512-//rfAE+BOw5AC2EMmepmiE36jUuevtQYNQqqlw1s3m9FlRxjxEut97RkRPHAu9BG4mSojatZx+kXZXNdyI9caQ==", + "cpu": [ + "ia32" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@vscode/ripgrep-win32-x64": { + "version": "1.18.0", + "resolved": "https://registry.npmjs.org/@vscode/ripgrep-win32-x64/-/ripgrep-win32-x64-1.18.0.tgz", + "integrity": "sha512-KNPvtElldqILHdnAetujPaowkNbpqJy3ssIGGN6F6Kve9Qi+nNLI2DN01O83JjCEVQbCzl8Ov3QZ9Eov3BR8Dg==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, "node_modules/accepts": { "version": "1.3.8", "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz", @@ -2847,6 +3030,24 @@ "wrappy": "1" } }, + "node_modules/openai": { + "version": "6.41.0", + "resolved": "https://registry.npmjs.org/openai/-/openai-6.41.0.tgz", + "integrity": "sha512-IGWPopZq6Rjoynjfb3NSLf/z2MTw7UiOsm9TAjPGAjUESH7Uq41Trg4QWehBEn58p74i+m7uoRPV2vXcpPXhyA==", + "license": "Apache-2.0", + "peerDependencies": { + "ws": "^8.18.0", + "zod": "^3.25 || ^4.0" + }, + "peerDependenciesMeta": { + "ws": { + "optional": true + }, + "zod": { + "optional": true + } + } + }, "node_modules/parseurl": { "version": "1.3.3", "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", @@ -3828,6 +4029,23 @@ "optional": true } } + }, + "packages/iclaw-runtime": { + "name": "@iclaw/runtime", + "version": "0.1.0", + "license": "MIT", + "dependencies": { + "@vscode/ripgrep": "^1.18.0", + "openai": "^6.41.0" + }, + "devDependencies": { + "@types/node": "^22.10.0", + "tsx": "^4.19.0", + "typescript": "^5.7.0" + }, + "engines": { + "node": ">=20" + } } } } diff --git a/package.json b/package.json index d3e325f..e3f9989 100644 --- a/package.json +++ b/package.json @@ -14,6 +14,9 @@ "engines": { "node": ">=20" }, + "workspaces": [ + "packages/*" + ], "bin": { "iclaw": "bin/iclaw.js" }, @@ -29,6 +32,7 @@ "prebuild": "node scripts/copy-opaque-vendor.mjs", "build": "tsc", "start": "node dist/index.js", + "build:secure-image": "bash packages/iclaw-runtime/container/build-secure.sh", "typecheck": "tsc --noEmit", "pretest": "node scripts/copy-opaque-vendor.mjs", "test": "vitest run", diff --git a/packages/iclaw-runtime/container/.dockerignore b/packages/iclaw-runtime/container/.dockerignore new file mode 100644 index 0000000..598ce19 --- /dev/null +++ b/packages/iclaw-runtime/container/.dockerignore @@ -0,0 +1,2 @@ +agent-runner/node_modules +agent-runner/dist diff --git a/packages/iclaw-runtime/container/build-secure.sh b/packages/iclaw-runtime/container/build-secure.sh new file mode 100755 index 0000000..c6bb66a --- /dev/null +++ b/packages/iclaw-runtime/container/build-secure.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# Build the iClaw Secure-Mode sandbox image (Chromium + agent-browser). +# Usage: ./build-secure.sh [--cjk] +set -euo pipefail + +cd "$(dirname "$0")" + +IMAGE_TAG="${ICLAW_SECURE_IMAGE:-iclaw-secure:latest}" +CJK="false" +[[ "${1:-}" == "--cjk" ]] && CJK="true" + +echo "Building ${IMAGE_TAG} (CJK fonts: ${CJK})..." +DOCKER_BUILDKIT=1 docker build \ + -f secure-sandbox.Dockerfile \ + --build-arg INSTALL_CJK_FONTS="${CJK}" \ + -t "${IMAGE_TAG}" \ + . + +echo "Done. Secure Mode will use ${IMAGE_TAG}." diff --git a/packages/iclaw-runtime/container/secure-sandbox.Dockerfile b/packages/iclaw-runtime/container/secure-sandbox.Dockerfile new file mode 100644 index 0000000..6337c57 --- /dev/null +++ b/packages/iclaw-runtime/container/secure-sandbox.Dockerfile @@ -0,0 +1,63 @@ +# syntax=docker/dockerfile:1.7 +# iClaw sandbox image — the SINGLE image shared by Safe work and Work mode. +# +# A long-lived `sleep` container the host docker-execs into, one turn at a time +# (Safe work: src/secure-runner.ts) or bind-mounts the user's folders into for +# `run_command` (Work: src/work-container.ts → resolveWorkImage prefers this tag). +# +# Toolset is curated 80/20: the ~20% of CLIs that ~80% of agent turns reach for, +# and nothing heavy. Deliberately EXCLUDED: a browser (Chromium + agent-browser +# ~870MB — web access is via curl/wget, and our search runs a different path), +# language toolchains/compilers, and other large runtimes. The agent self-installs +# the long tail at runtime WITHOUT root into /workspace/.tools (on PATH), which +# lives in the bind-mounted, TTL-reaped workspace — installs persist across +# container restarts within a session and auto-delete when the workspace expires. +# For a heavier base (e.g. a Go/Rust toolchain), point ICLAW_WORK_IMAGE at a +# custom image instead of bloating this shared one. +# +# Built via container/build-secure.sh → tag `iclaw-secure:latest`. +# Runs as the non-root `node` user. + +FROM node:22-slim + +# Curated 80/20 CLIs. curl/wget + ca-certificates power web access and non-root +# self-installs (downloading static binaries). git, python3 and an editor cover +# the bulk of real agent work. All baked in (not apt-installed at runtime) +# because the container runs as non-root `node` and is recreated after idle reap +# — a runtime `apt install` can't work and wouldn't persist anyway. +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates \ + curl \ + wget \ + git \ + ripgrep \ + python3 \ + nano \ + unzip \ + zip \ + xz-utils \ + bzip2 \ + jq \ + less \ + tree \ + && rm -rf /var/lib/apt/lists/* + +# ---- Self-installed tools (no root) ---------------------------------------- +# Drop static binaries into /workspace/.tools/bin, or `npm i -g <pkg>` (prefix +# below). Inside the bind-mounted, TTL-reaped workspace, so they persist across +# container restarts within a session and are auto-deleted when the workspace +# expires (~7 days) — see createSecureWorkspace / destroySecureWorkspace. +ENV NPM_CONFIG_PREFIX=/workspace/.tools/npm +ENV PATH="/workspace/.tools/bin:/workspace/.tools/npm/bin:$PATH" + +# ---- Workspace + non-root user --------------------------------------------- +# /workspace is the host bind-mount target. Home must be writable for tool caches. +RUN mkdir -p /workspace && chown -R node:node /workspace && chmod 777 /home/node + +USER node +WORKDIR /workspace + +# Default command; the host overrides argv with `sleep 3600` at run time. +CMD ["sleep", "3600"] diff --git a/packages/iclaw-runtime/package.json b/packages/iclaw-runtime/package.json new file mode 100644 index 0000000..ba36468 --- /dev/null +++ b/packages/iclaw-runtime/package.json @@ -0,0 +1,26 @@ +{ + "name": "@iclaw/runtime", + "version": "0.1.0", + "description": "iClaw agent runtime — agent loop on the host, Docker sandbox per turn (Work / Safe work / Incognito), OpenRouter-only.", + "license": "MIT", + "type": "module", + "main": "dist/index.js", + "scripts": { + "build": "tsc", + "dev": "tsx src/index.ts", + "start": "node dist/index.js", + "typecheck": "tsc --noEmit" + }, + "dependencies": { + "@vscode/ripgrep": "^1.18.0", + "openai": "^6.41.0" + }, + "devDependencies": { + "@types/node": "^22.10.0", + "tsx": "^4.19.0", + "typescript": "^5.7.0" + }, + "engines": { + "node": ">=20" + } +} diff --git a/packages/iclaw-runtime/src/agent/loop.ts b/packages/iclaw-runtime/src/agent/loop.ts new file mode 100644 index 0000000..7d0a009 --- /dev/null +++ b/packages/iclaw-runtime/src/agent/loop.ts @@ -0,0 +1,436 @@ +/** + * Model-agnostic agent loop. + * + * Calls any OpenRouter model that supports tool calling. + * Streams text deltas and tool events back to the caller via async generator. + */ +import OpenAI from 'openai'; + +import { TOOL_DEFINITIONS, WEB_FETCH_TOOL, WEB_SEARCH_TOOL, READ_SUMMARY_TOOL, ANALYZE_LINK_TOOL, executeTool, type ToolContext, type ToolName, type SavingsNote } from './tools.js'; +import { dumpPrompt, newTurnId } from './prompt-dump.js'; + +export interface AgentOptions { + apiKey: string; + model: string; + allowedFolders: string[]; + /** Per-folder access levels. When omitted, all allowed folders are writable. */ + folderAccess?: { path: string; readonly: boolean }[]; + /** Shell backend for run_command (Docker sandbox). Omit to disable commands. */ + runShell?: (command: string, cwd: string) => Promise<string>; + /** + * Sandbox backend for analyze_link (yt-dlp). Runs in the session's container + * so yt-dlp never parses untrusted data on the host. Omit to drop the tool + * (no Docker → analyze_link not offered, falls back to web_fetch/web_search). + */ + linkSandbox?: (command: string) => Promise<string>; + /** + * Incognito (read-only, ephemeral): file reads are unrestricted (read + * anywhere; secrets still refused), write_file is disabled, run_command is + * sandboxed read-only, and the `web_fetch` research tool is exposed. + */ + incognito?: boolean; + systemPrompt?: string; + /** + * Image data URLs (`data:<mime>;base64,…`) for THIS turn's user message — + * files the user dropped into the chat. Sent once as vision blocks so the + * model literally sees them; NOT stored in history (one-shot, expensive). + */ + images?: string[]; + onWriteApproval?: (filePath: string, content: string) => Promise<boolean>; + /** Abort the in-flight turn (user pressed Stop). Stops the model stream and + * ends the loop cleanly between rounds. */ + signal?: AbortSignal; +} + +export type AgentEvent = + | { type: 'text'; content: string } + | { type: 'tool_start'; name: string; input: unknown } + | { type: 'tool_result'; name: string; result: string } + | { type: 'note'; note: SavingsNote } + | { type: 'approval_request'; changeId: string; path: string; content: string } + | { type: 'done'; tokens?: number; cached?: number } + | { type: 'error'; message: string }; + +export type Message = OpenAI.Chat.ChatCompletionMessageParam; + +/** Max tool-call rounds per turn before we stop (env-tunable for long tasks). */ +const MAX_ROUNDS = Math.max(1, Number(process.env.ICLAW_MAX_ROUNDS) || 40); + +/** Turn a provider/SDK error into a concise, user-facing message. */ +function describeApiError(err: unknown): string { + const e = err as { + status?: number; + code?: number | string; + error?: { code?: number; metadata?: { error_type?: string } }; + message?: string; + }; + const code = e?.status ?? e?.code ?? e?.error?.code; + const isRateLimit = code === 429 || e?.error?.metadata?.error_type === 'rate_limit_exceeded'; + if (isRateLimit) { + return 'Rate limit reached on the model provider (429). Wait a few seconds and try again, ' + + 'or switch to a model with higher limits (ICLAW_MODEL).'; + } + return e?.message || String(err); +} + +// Mid-turn compaction budget: once the in-flight message array passes this many +// chars, stub out all but a few tool outputs (they're already acted on; the model +// can re-run a tool if it truly needs the data again). Lowered to 16k — recent +// work turns plateau around 15–20k, so a 32k gate almost never fired and the +// whole history was resent every round (O(n²) tokens). Like Hermes' compressor we +// keep the FIRST few tool outputs (early task context) as well as the last few. +const INTURN_COMPACT_CHARS = Number(process.env.ICLAW_INTURN_COMPACT_CHARS) || 16_000; +const INTURN_KEEP_TOOL_MSGS = Number(process.env.ICLAW_INTURN_KEEP_TOOL_MSGS) || 6; +const INTURN_KEEP_FIRST_TOOL_MSGS = Number(process.env.ICLAW_INTURN_KEEP_FIRST_TOOL_MSGS) || 2; + +/** + * Shrink old tool-result messages in-place when the turn's context grows too + * large. Keeps message structure (assistant↔tool pairing + tool_call_ids) intact + * for API validity — only replaces stale tool *content* with a short stub. No + * extra model call. Shared by the Work/Incognito loop and the Sandbox loop. + * + * Protects the first `INTURN_KEEP_FIRST_TOOL_MSGS` and last `INTURN_KEEP_TOOL_MSGS` + * tool outputs; only the middle gets stubbed (mirrors Hermes' protect_first_n / + * protect_last_n). + */ +export function shrinkOldToolOutputs(messages: Message[]): void { + let total = 0; + for (const m of messages) total += typeof m.content === 'string' ? m.content.length : 0; + if (total <= INTURN_COMPACT_CHARS) return; + + const toolIdx: number[] = []; + for (let i = 0; i < messages.length; i++) if (messages[i].role === 'tool') toolIdx.push(i); + const keepFirst = INTURN_KEEP_FIRST_TOOL_MSGS; + const keepLast = INTURN_KEEP_TOOL_MSGS; + for (let k = keepFirst; k < toolIdx.length - keepLast; k++) { + const i = toolIdx[k]; + const content = (messages[i] as { content?: unknown }).content; + if (typeof content === 'string' && content.length > 160) { + messages[i] = { + ...messages[i], + content: `[earlier tool output omitted to save context — was ${content.length.toLocaleString()} chars; re-run the tool if you need it again]`, + } as Message; + } + } +} + +/** + * Prompt caching (#2): mark the static prefix (system prompt + tool schemas) with + * an OpenRouter/Anthropic `cache_control: ephemeral` breakpoint so it isn't re- + * billed at full price on every round. The system block is the one guaranteed- + * stable prefix every round (tools ride with it), so one breakpoint there caches + * ~1.2k tokens of overhead per round. OpenRouter strips the hint for providers + * that don't support it (e.g. DeepSeek already caches automatically), so this is + * safe across models. Off only when ICLAW_PROMPT_CACHE=off. + * + * Returns a NEW array (doesn't mutate `messages`) with the system content + * converted to the parts form Anthropic needs for a cache breakpoint. + */ +export function withPromptCaching(messages: Message[]): Message[] { + if (process.env.ICLAW_PROMPT_CACHE === 'off') return messages; + const i = messages.findIndex((m) => m.role === 'system'); + if (i === -1) return messages; + const sys = messages[i]; + if (typeof sys.content !== 'string' || !sys.content) return messages; + const out = messages.slice(); + out[i] = { + role: 'system', + content: [ + { type: 'text', text: sys.content, cache_control: { type: 'ephemeral' } }, + ], + } as unknown as Message; + return out; +} + +/** + * Tool-loop guardrail (#5): a turn can waste rounds (and tokens) re-issuing the + * SAME tool call with identical args — a stuck model retrying a failed read, or + * looping. Track call signatures; once one repeats past the limit, short-circuit + * with a nudge instead of executing it again (mirrors Hermes' tool_loop_guardrails). + */ +const TOOL_REPEAT_LIMIT = Math.max(1, Number(process.env.ICLAW_TOOL_REPEAT_LIMIT) || 3); + +export function makeToolGuard(): { check(name: string, rawArgs: string): string | null } { + const counts = new Map<string, number>(); + return { + check(name: string, rawArgs: string): string | null { + const sig = `${name}:${rawArgs}`; + const n = (counts.get(sig) ?? 0) + 1; + counts.set(sig, n); + if (n > TOOL_REPEAT_LIMIT) { + return `Guardrail: you've already called ${name} with these exact arguments ${n - 1} times this turn ` + + `and got the same result. Not running it again. Change the arguments, try a different tool, ` + + `or tell the user what's blocking you.`; + } + return null; + }, + }; +} + +const DEFAULT_SYSTEM = `Work Mode: read/search/edit/create files in the selected folders, run shell commands, and research the web (web_search then web_fetch). +Prefer edit_file over rewriting whole files. For a large file you only need the gist of, use read_summary (cheap) instead of read_file. The user approves every write in the UI — don't paste file contents or ask "is this correct?"; just act, then briefly say what you did. +Be efficient: chain shell steps with && in one call, don't repeat commands. Never go outside the allowed folders. +Keep replies short — don't echo back long file listings or file contents; summarize in a line or two.`; + +/** + * System-install policy (Section 3): the agent must not change the user's + * computer. Shared by Work and Safe prompts. run_command runs in an isolated + * sandbox, so installs there are harmless AND don't reach the host — which is + * exactly why we say it in the prompt rather than guarding run_command (a guard + * would wrongly block legitimate in-sandbox installs). + */ +export const HOST_INSTALL_POLICY = `System changes (important): your run_command runs in an isolated sandbox, so installing tools there is fine for the task but does NOT install anything on the user's actual computer. Never attempt or claim to make host system changes — installing Python, Node, Git, Docker, Homebrew or system packages on their machine, or editing their PATH. If a task genuinely needs something installed on their computer, tell them it changes their system, give the official install command or link, and ask them to run it themselves.`; + +const INCOGNITO_SYSTEM = `Incognito: private, READ-ONLY research. You can read files anywhere, search, run a read-only shell in the selected folders, and use web_search/web_fetch. +You CANNOT write — never claim you saved or changed anything. This chat is ephemeral: nothing is stored. Be concise; put findings in your reply.`; + +/** + * Build the per-turn system message. The base rules and the folder-access + * summary are ALWAYS included so the model knows which folders are read-only + * (and won't waste calls writing to them); any host-supplied prompt (project + * context) is appended rather than replacing the base. + */ +/** One-line identity so the model doesn't hallucinate being Claude/GPT. */ +function identityLine(model: string): string { + return `You are iClaw, a private AI assistant${model ? `, powered by ${model}` : ''}. If asked what you are, say exactly that — never claim to be ChatGPT, Claude, Gemini or any other product.`; +} + +function buildSystemPrompt(opts: AgentOptions): string { + if (opts.incognito) { + const parts = [identityLine(opts.model), INCOGNITO_SYSTEM]; + if (opts.allowedFolders.length) { + parts.push( + `\nShell folders (read-only): ${opts.allowedFolders.join(', ')}. File reads aren't limited to these; secrets are always refused.`, + ); + } else { + parts.push('\nNo shell folders selected — run_command is off; use read_file / search_files / web_fetch.'); + } + if (opts.systemPrompt?.trim()) parts.push(`\n${opts.systemPrompt.trim()}`); + return parts.join('\n'); + } + + const parts = [identityLine(opts.model), DEFAULT_SYSTEM, HOST_INSTALL_POLICY]; + + const folders = opts.folderAccess?.length + ? opts.folderAccess + : opts.allowedFolders.map((path) => ({ path, readonly: false })); + if (folders.length) { + const lines = folders.map((f) => `- ${f.path} (${f.readonly ? 'read-only' : 'rw'})`); + parts.push( + `\nFolders (use these exact paths):\n${lines.join('\n')}\n` + + `Never write or run commands in a read-only folder — say it's read-only and offer a rw folder instead.`, + ); + } + + if (opts.systemPrompt?.trim()) parts.push(`\n${opts.systemPrompt.trim()}`); + return parts.join('\n'); +} + +/** + * Run one user turn. Yields events as the agent works. + * Handles multi-step tool loops automatically. + */ +export async function* runAgentTurn( + history: Message[], + userMessage: string, + opts: AgentOptions, +): AsyncGenerator<AgentEvent> { + const client = new OpenAI({ + baseURL: 'https://openrouter.ai/api/v1', + apiKey: opts.apiKey, + }); + + // A tool may emit a savings note mid-call (analyze_link). Collect it here and + // flush as a `note` event right after the tool_result it belongs to. + const pendingNotes: SavingsNote[] = []; + const toolCtx: ToolContext = { + allowedFolders: opts.allowedFolders, + folderAccess: opts.folderAccess, + runShell: opts.runShell, + linkSandbox: opts.linkSandbox, + readOnly: opts.incognito, + readAnywhere: opts.incognito, + requestWriteApproval: opts.onWriteApproval ?? (async () => true), + onNote: (note) => pendingNotes.push(note), + }; + + // Per-mode tool set. Incognito is read-only, so don't ship write_file/edit_file + // schemas it can't use (saves prompt tokens). Web research tools run host-side + // and are never exposed to Secure Mode (it has its own loop + container network + // gate that a host-side fetch would bypass). + const fileTools = opts.incognito + ? TOOL_DEFINITIONS.filter((t) => t.function.name !== 'write_file' && t.function.name !== 'edit_file') + : TOOL_DEFINITIONS; + // analyze_link (yt-dlp in the session container) is offered only when a + // sandbox backend is wired (Docker up). Without it, the model uses web_fetch. + const tools = [ + ...fileTools, READ_SUMMARY_TOOL, WEB_FETCH_TOOL, WEB_SEARCH_TOOL, + ...(opts.linkSandbox ? [ANALYZE_LINK_TOOL] : []), + ]; + + // When the user dropped image(s), send the turn's user message as a + // multimodal content array (text + image blocks) so a vision model sees them. + // History keeps only the text form (the caller stores `userMessage`), so the + // images aren't re-sent every round. + const userMsg: OpenAI.Chat.ChatCompletionUserMessageParam = opts.images?.length + ? { + role: 'user', + content: [ + { type: 'text', text: userMessage }, + ...opts.images.map((url) => ({ type: 'image_url' as const, image_url: { url } })), + ], + } + : { role: 'user', content: userMessage }; + + const messages: Message[] = [ + { role: 'system', content: buildSystemPrompt(opts) }, + ...history, + userMsg, + ]; + + // Token usage across all rounds (dev-mode display). `turnCached` = how many + // prompt tokens were served from the provider's prefix cache. + let turnTokens = 0; + let turnCached = 0; + const dumpTurnId = newTurnId(); + const dumpMode = opts.incognito ? 'incognito' : 'work'; + const guard = makeToolGuard(); + + // Max tool-call rounds to prevent infinite loops (env-tunable: ICLAW_MAX_ROUNDS). + for (let round = 0; round < MAX_ROUNDS; round++) { + // User pressed Stop between rounds → end cleanly (partial text already sent). + if (opts.signal?.aborted) { + yield { type: 'done', tokens: turnTokens || undefined, cached: turnCached || undefined }; + return; + } + let textBuffer = ''; + const toolCallBuffers: Record<string, { name: string; arguments: string }> = {}; + + // Dev mode: persist exactly what we're about to send (incl. tool schemas). + dumpPrompt({ turnId: dumpTurnId, mode: dumpMode, model: opts.model, round, messages, tools }); + + let stream: AsyncIterable<OpenAI.Chat.ChatCompletionChunk>; + try { + stream = await client.chat.completions.create( + { + model: opts.model, + messages: withPromptCaching(messages), + tools: tools as unknown as OpenAI.Chat.ChatCompletionTool[], + tool_choice: 'auto', + stream: true, + stream_options: { include_usage: true }, // final chunk carries token usage + }, + { signal: opts.signal }, + ); + } catch (err) { + // Aborted by the user → not an error; end the turn cleanly. + if (opts.signal?.aborted) { + yield { type: 'done', tokens: turnTokens || undefined, cached: turnCached || undefined }; + return; + } + yield { type: 'error', message: describeApiError(err) }; + return; + } + + let finishReason: string | null = null; + + // The provider can inject an error MID-STREAM (e.g. a 429 surfaced as a JSON + // error in the SSE body). Catch it here so the turn ends with a clean error + // event instead of throwing out of the generator and leaving a truncated reply. + try { + for await (const chunk of stream) { + // Usage rides in a final chunk that has no choices — capture it first. + if (chunk.usage?.total_tokens) turnTokens += chunk.usage.total_tokens; + const cached = (chunk.usage as { prompt_tokens_details?: { cached_tokens?: number } } | undefined) + ?.prompt_tokens_details?.cached_tokens; + if (cached) turnCached += cached; + const choice = chunk.choices[0]; + if (!choice) continue; + + finishReason = choice.finish_reason ?? finishReason; + const delta = choice.delta; + + // Text content + if (delta.content) { + textBuffer += delta.content; + yield { type: 'text', content: delta.content }; + } + + // Tool call accumulation + if (delta.tool_calls) { + for (const tc of delta.tool_calls) { + const idx = String(tc.index ?? 0); + if (!toolCallBuffers[idx]) { + toolCallBuffers[idx] = { name: '', arguments: '' }; + } + if (tc.function?.name) toolCallBuffers[idx].name += tc.function.name; + if (tc.function?.arguments) toolCallBuffers[idx].arguments += tc.function.arguments; + } + } + } + } catch (err) { + // Aborted by the user → not an error; end the turn cleanly. + if (opts.signal?.aborted) { + yield { type: 'done', tokens: turnTokens || undefined, cached: turnCached || undefined }; + return; + } + yield { type: 'error', message: describeApiError(err) }; + return; + } + + const toolCalls = Object.values(toolCallBuffers); + + // No tool calls → model is done + if (toolCalls.length === 0 || finishReason === 'stop') { + if (textBuffer) { + messages.push({ role: 'assistant', content: textBuffer }); + } + yield { type: 'done', tokens: turnTokens || undefined, cached: turnCached || undefined }; + return; + } + + // Build assistant message with tool calls + const assistantMsg: OpenAI.Chat.ChatCompletionAssistantMessageParam = { + role: 'assistant', + content: textBuffer || null, + tool_calls: toolCalls.map((tc, i) => ({ + id: `call_${i}`, + type: 'function' as const, + function: { name: tc.name, arguments: tc.arguments }, + })), + }; + messages.push(assistantMsg); + + // Execute each tool call + for (let i = 0; i < toolCalls.length; i++) { + const tc = toolCalls[i]; + let parsedArgs: Record<string, unknown> = {}; + try { + parsedArgs = JSON.parse(tc.arguments); + } catch { + parsedArgs = {}; + } + + yield { type: 'tool_start', name: tc.name, input: parsedArgs }; + + // Guardrail: refuse to re-run an identical call that's already looping. + const blocked = guard.check(tc.name, tc.arguments); + const result = blocked ?? (await executeTool(tc.name as ToolName, parsedArgs, toolCtx)); + + yield { type: 'tool_result', name: tc.name, result }; + while (pendingNotes.length) yield { type: 'note', note: pendingNotes.shift()! }; + + messages.push({ + role: 'tool', + tool_call_id: `call_${i}`, + content: result, + }); + } + // Mid-turn compaction: on a long multi-round task the accumulated tool + // outputs would be resent every round (O(n²) tokens). Stub out old ones. + shrinkOldToolOutputs(messages); + // Continue loop — model will see tool results and respond + } + + yield { type: 'error', message: `Reached the step limit (${MAX_ROUNDS} tool rounds). Send "continue" to keep going, or raise ICLAW_MAX_ROUNDS.` }; +} diff --git a/packages/iclaw-runtime/src/agent/prompt-dump.ts b/packages/iclaw-runtime/src/agent/prompt-dump.ts new file mode 100644 index 0000000..f4d5c21 --- /dev/null +++ b/packages/iclaw-runtime/src/agent/prompt-dump.ts @@ -0,0 +1,121 @@ +/** + * Dev-mode prompt dumper. + * + * When ICLAW_DEV_MODE=true, every request we send to the model (the full + * messages array + tool schemas, i.e. exactly what burns prompt tokens) is + * appended to a per-turn JSONL file, with a per-section size breakdown so you + * can see WHAT is heavy (system prompt vs tool schemas vs history). + * + * Self-limiting: files older than 24h are deleted, and the directory is trimmed + * (oldest-first) to stay under 50 MB. Best-effort and fully wrapped in try/catch + * — dumping must never break or slow down a real turn. + * + * Location: $ICLAW_DEV_PROMPT_DIR or ~/.iclaw/dev-prompts/<turnId>.jsonl + */ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +const TTL_MS = 24 * 60 * 60 * 1000; +const MAX_BYTES = 50 * 1024 * 1024; +const SWEEP_INTERVAL_MS = 30_000; + +let lastSweep = 0; + +function enabled(): boolean { + return process.env.ICLAW_DEV_MODE === 'true'; +} + +function dir(): string { + return process.env.ICLAW_DEV_PROMPT_DIR || path.join(os.homedir(), '.iclaw', 'dev-prompts'); +} + +export function newTurnId(): string { + return `${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`; +} + +interface DumpRecord { + turnId: string; + mode: string; + model: string; + round: number; + messages: readonly unknown[]; + tools?: readonly unknown[]; +} + +function summarize(messages: readonly unknown[], tools?: readonly unknown[]) { + const charsByRole: Record<string, number> = {}; + let messageChars = 0; + for (const raw of messages) { + const m = raw as { role?: string; content?: unknown }; + const role = m.role || 'unknown'; + const c = typeof m.content === 'string' + ? m.content.length + : JSON.stringify(m.content ?? '').length; + charsByRole[role] = (charsByRole[role] || 0) + c; + messageChars += c; + } + const toolChars = tools ? JSON.stringify(tools).length : 0; + return { + chars_by_role: charsByRole, + chars_messages: messageChars, + chars_tools: toolChars, + // ~4 chars/token — a rough guide, not the billed count. + approx_tokens: Math.ceil((messageChars + toolChars) / 4), + }; +} + +export function dumpPrompt(rec: DumpRecord): void { + if (!enabled()) return; + try { + const d = dir(); + fs.mkdirSync(d, { recursive: true }); + const line = JSON.stringify({ + ts: new Date().toISOString(), + mode: rec.mode, + model: rec.model, + round: rec.round, + sizes: summarize(rec.messages, rec.tools), + messages: rec.messages, + tools: rec.tools, + }) + '\n'; + fs.appendFileSync(path.join(d, `${rec.turnId}.jsonl`), line); + sweep(d); + } catch { + // dev-only; never surface + } +} + +function sweep(d: string): void { + const now = Date.now(); + if (now - lastSweep < SWEEP_INTERVAL_MS) return; + lastSweep = now; + try { + const files = fs.readdirSync(d) + .filter((f) => f.endsWith('.jsonl')) + .map((f) => { + const p = path.join(d, f); + try { const st = fs.statSync(p); return { p, mtime: st.mtimeMs, size: st.size }; } + catch { return null; } + }) + .filter((x): x is { p: string; mtime: number; size: number } => x != null); + + // 1) TTL: drop anything older than 24h. + const live: { p: string; mtime: number; size: number }[] = []; + for (const f of files) { + if (now - f.mtime > TTL_MS) { try { fs.rmSync(f.p, { force: true }); } catch { /* */ } } + else live.push(f); + } + // 2) Size cap: trim oldest until under 50 MB. + let total = live.reduce((a, f) => a + f.size, 0); + if (total > MAX_BYTES) { + live.sort((a, b) => a.mtime - b.mtime); + for (const f of live) { + if (total <= MAX_BYTES) break; + try { fs.rmSync(f.p, { force: true }); total -= f.size; } catch { /* */ } + } + } + } catch { + // ignore + } +} diff --git a/packages/iclaw-runtime/src/agent/security.ts b/packages/iclaw-runtime/src/agent/security.ts new file mode 100644 index 0000000..a0c7c54 --- /dev/null +++ b/packages/iclaw-runtime/src/agent/security.ts @@ -0,0 +1,137 @@ +/** + * Path security for Work Mode. + * All file operations must pass validatePath() before execution. + */ +import fs from 'node:fs'; +import path from 'node:path'; + +const BLOCKED_PATTERNS = [ + '.env', '.ssh', '.gnupg', '.gpg', '.aws', '.azure', '.gcloud', '.kube', '.docker', + 'id_rsa', 'id_ed25519', 'id_dsa', 'private_key', + '.netrc', '.npmrc', '.pypirc', '.secret', +]; + +const BLOCKED_EXTENSIONS = ['.pem', '.key', '.p12', '.pfx', '.cert', '.crt']; + +const BLOCKED_NAMES = [ + 'credentials', 'credentials.json', 'credentials.yaml', 'credentials.yml', + 'secrets', 'secrets.json', 'secrets.yaml', 'secrets.yml', + 'token', 'tokens', +]; + +export class SecurityError extends Error { + constructor(message: string) { + super(message); + this.name = 'SecurityError'; + } +} + +/** Expand a leading ~ and resolve to an absolute path (no symlink resolution). */ +function expandPath(inputPath: string): string { + const expanded = inputPath.startsWith('~/') || inputPath === '~' + ? path.join(process.env.HOME ?? '', inputPath.slice(1)) + : inputPath; + return path.resolve(expanded); +} + +/** + * Return the secret-bearing pattern a real path matches (in any component, by + * extension, or by basename), or null if clean. Shared by validatePath (host + * file tools) and validateMountRoot (container mounts) so both apply the SAME + * deny-list — a path the file tools refuse to touch is also never mounted. + */ +export function findBlockedPattern(realPath: string): string | null { + const parts = realPath.split(path.sep); + for (const part of parts) { + for (const blocked of BLOCKED_PATTERNS) { + if (part === blocked || part.startsWith(blocked + '.')) return blocked; + } + } + const ext = path.extname(realPath).toLowerCase(); + if (BLOCKED_EXTENSIONS.includes(ext)) return ext; + const basename = path.basename(realPath).toLowerCase(); + for (const blocked of BLOCKED_NAMES) { + if (basename === blocked || basename.startsWith(blocked + '.')) return blocked; + } + return null; +} + +/** Resolve and validate a path against allowed folders. Throws SecurityError if blocked. */ +export function validatePath(inputPath: string, allowedFolders: string[]): string { + if (!inputPath) throw new SecurityError('Empty path'); + + const resolved = expandPath(inputPath); + + // Symlink resolution — prevent escape via symlinks + let realPath: string; + try { + realPath = fs.realpathSync(resolved); + } catch { + // File doesn't exist yet (write case) — use resolved path + realPath = resolved; + } + + const blocked = findBlockedPattern(realPath); + if (blocked) { + throw new SecurityError(`Path matches blocked pattern "${blocked}": ${realPath}`); + } + + // Must be under at least one allowed folder + if (allowedFolders.length > 0) { + const allowed = allowedFolders.some((folder) => { + try { + const realFolder = fs.realpathSync(path.resolve(folder)); + return realPath === realFolder || realPath.startsWith(realFolder + path.sep); + } catch { + return false; + } + }); + + if (!allowed) { + throw new SecurityError( + `Path "${realPath}" is outside allowed folders: ${allowedFolders.join(', ')}`, + ); + } + } + + return realPath; +} + +/** + * Validate a folder that's about to be bind-mounted into the command sandbox. + * Resolves ~ and symlinks, then rejects any secret-bearing root (e.g. ~/.ssh, + * a folder named `credentials`) via the SAME deny-list the host file tools use. + * Returns the real (symlink-resolved) path to mount. Throws SecurityError. + * + * Note: this guards the mount ROOT. It deliberately does not try to hide secret + * files nested inside an otherwise-legitimate folder — granting a shell into a + * folder inherently exposes its contents. Callers therefore only mount folders + * the user explicitly selected (never a broad fallback like all of $HOME). + */ +export function validateMountRoot(folderPath: string): string { + if (!folderPath) throw new SecurityError('Empty mount path'); + const realPath = fs.realpathSync(expandPath(folderPath)); + const blocked = findBlockedPattern(realPath); + if (blocked) { + throw new SecurityError(`Folder matches blocked pattern "${blocked}": ${realPath}`); + } + return realPath; +} + +/** Check if a path is read-write allowed (vs read-only). */ +export function isWriteAllowed( + realPath: string, + allowedFolders: { path: string; readonly?: boolean }[], +): boolean { + for (const folder of allowedFolders) { + try { + const realFolder = fs.realpathSync(path.resolve(folder.path)); + if (realPath === realFolder || realPath.startsWith(realFolder + path.sep)) { + return !folder.readonly; + } + } catch { + continue; + } + } + return false; +} diff --git a/packages/iclaw-runtime/src/agent/tools.ts b/packages/iclaw-runtime/src/agent/tools.ts new file mode 100644 index 0000000..5711362 --- /dev/null +++ b/packages/iclaw-runtime/src/agent/tools.ts @@ -0,0 +1,1094 @@ +/** + * Tool definitions (JSON schema for the model) + implementations. + * All file operations are validated against allowedFolders. + */ +import fs from 'node:fs'; +import path from 'node:path'; +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; +import { createRequire } from 'node:module'; + +import { validatePath, isWriteAllowed, SecurityError } from './security.js'; + +// Bundled ripgrep binary — shipped as a dependency so it's ALWAYS present (no +// per-host "is rg installed?" branching, no grep fallback). We drive it with +// our own ignore list (below), so it works the same with or without git. +// @vscode/ripgrep is CommonJS with no type declarations, so pull rgPath through +// createRequire rather than an ESM import (avoids the export-shape mismatch). +const { rgPath } = createRequire(import.meta.url)('@vscode/ripgrep') as { rgPath: string }; + +const execFileAsync = promisify(execFile); + +// Refuse to slurp a huge file into memory at all (use search_files instead). +const MAX_FILE_BYTES = Number(process.env.ICLAW_MAX_FILE_BYTES) || 5_000_000; +const COMMAND_TIMEOUT = 30_000; +const WEB_FETCH_TIMEOUT = 20_000; +const WEB_FETCH_MAX_CHARS = 20_000; + +// ── Token-saving output caps ────────────────────────────────────────────────── +// Tool outputs are the biggest token sink in multi-turn chats: they land in the +// history and get resent every round. Cap what we hand back to the model. +const MAX_FILE_READ_CHARS = Number(process.env.ICLAW_MAX_FILE_READ) || 16_000; +const MAX_CMD_OUTPUT_CHARS = Number(process.env.ICLAW_MAX_CMD_OUTPUT) || 8_000; +const MAX_LIST_ENTRIES = Number(process.env.ICLAW_MAX_LIST_ENTRIES) || 200; +// search_files: cap matching lines returned per file so a high-frequency term +// can't dump a whole file into history. +const MAX_MATCH_LINES_PER_FILE = Number(process.env.ICLAW_MAX_MATCH_LINES) || 30; + +// Directories to never descend into when searching. Different stacks bury huge +// generated trees in different places (JS node_modules, Rust target, Python +// .venv/__pycache__, Java .gradle, …); scanning them is what made unscoped +// searches time out. This is OUR list, fed to ripgrep as ignore globs, so it +// applies whether or not the user has git. ripgrep additionally skips binary +// files (videos/photos/archives/office docs) and follows no symlinks — exactly +// the behaviour a non-technical user's media-heavy folders need. +// Override the whole set with ICLAW_SEARCH_EXCLUDE_DIRS (comma-separated). +const SEARCH_EXCLUDE_DIRS = (process.env.ICLAW_SEARCH_EXCLUDE_DIRS + ? process.env.ICLAW_SEARCH_EXCLUDE_DIRS.split(',').map((s) => s.trim()).filter(Boolean) + : ['node_modules', '.git', '.svn', '.hg', '.cache', '.next', '.nuxt', 'dist', 'build', + 'out', 'target', 'vendor', '.venv', 'venv', '.tox', '__pycache__', '.mypy_cache', + '.pytest_cache', '.gradle', '.idea', 'Pods', '.terraform', '.tools']); + +// Cheap-model summarizer (read_summary, web_fetch summarize). Moves the cost of +// reading big content onto a cheap model; the expensive model + history only +// carry the short summary. +const SUMMARY_MODEL = process.env.ICLAW_SUMMARY_MODEL || 'google/gemini-2.5-flash-lite'; +const SUMMARY_MAX_INPUT_CHARS = Number(process.env.ICLAW_SUMMARY_MAX_INPUT) || 60_000; + +export const TOOL_OUTPUT_MAX_CHARS = MAX_CMD_OUTPUT_CHARS; + +/** + * Compress raw command output before it lands in history (rtk-inspired, but + * generic + lossless-for-signal — no per-command filters that could hide a + * failing test or an error the agent needs). Runs BEFORE clampMiddle so the cap + * keeps real content instead of progress-bar/ANSI noise. Four safe passes: + * 1. strip ANSI escape codes (colours, cursor moves) — pure noise to a model, + * and confirmed present in real run_command output (e.g. "\x1b[95m…\x1b[0m"); + * 2. collapse carriage-return progress lines to their final state (npm/pip/ + * docker/git "Downloading 12%…100%" → just the last frame); + * 3. dedupe runs of ≥3 identical consecutive lines into one + "(×N)"; + * 4. squeeze 3+ blank lines down to one. + */ +// eslint-disable-next-line no-control-regex +const ANSI_RE = /\x1b\[[0-9;?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|\x1b[@-Z\\-_]/g; + +export function compressCommandOutput(s: string): string { + if (!s) return s; + let text = s.replace(ANSI_RE, ''); + // Carriage-return progress: keep only what survives the last \r on each line. + text = text + .split('\n') + .map((line) => (line.includes('\r') ? line.slice(line.lastIndexOf('\r') + 1) : line)) + .join('\n'); + // Dedupe runs of ≥3 identical, non-blank consecutive lines into one + "(×N)". + const lines = text.split('\n'); + const out: string[] = []; + for (let i = 0; i < lines.length; ) { + let j = i + 1; + while (j < lines.length && lines[j] === lines[i]) j++; + const run = j - i; + if (run >= 3 && lines[i].trim()) out.push(`${lines[i]} …(×${run})`); + else for (let k = i; k < j; k++) out.push(lines[k]); + i = j; + } + return out.join('\n').replace(/\n{3,}/g, '\n\n'); +} + +/** Keep the head and tail of long output (errors are usually at the end). */ +export function clampMiddle(s: string, max: number): string { + if (s.length <= max) return s; + const head = Math.ceil(max * 0.6); + const tail = max - head; + return s.slice(0, head) + + `\n\n…[truncated ${(s.length - max).toLocaleString()} of ${s.length.toLocaleString()} chars — refine the command or use search_files/read_file]…\n\n` + + s.slice(s.length - tail); +} + +// ── Tool JSON schemas (sent to the model) ──────────────────────────────────── + +export const TOOL_DEFINITIONS = [ + { + type: 'function' as const, + function: { + name: 'list_files', + description: 'List a directory. Dirs show as "[dir] name/", files as "[file] name".', + parameters: { + type: 'object', + properties: { path: { type: 'string', description: 'Directory path' } }, + required: ['path'], + }, + }, + }, + { + type: 'function' as const, + function: { + name: 'read_file', + description: 'Read a file.', + parameters: { + type: 'object', + properties: { path: { type: 'string', description: 'File path' } }, + required: ['path'], + }, + }, + }, + { + type: 'function' as const, + function: { + name: 'search_files', + description: 'Find files. To LOCATE a file by its name (e.g. "where is config.json?"), pass "name" — it returns just the matching paths, cheaply. To search file CONTENTS, pass "query". Omit "path" to cover ALL folders you have access to in one call — do that instead of searching them one by one.', + parameters: { + type: 'object', + properties: { + path: { type: 'string', description: 'Directory to search. Omit to search every accessible folder in one call.' }, + name: { type: 'string', description: 'Locate files by NAME or fragment (case-insensitive), returning paths only. Use this to find where a file lives — not "query".' }, + query: { type: 'string', description: 'Search file CONTENTS for this string. Use "name" instead when you just want to locate a file.' }, + filePattern: { type: 'string', description: 'Optional glob to limit which files "query" searches, e.g. "*.ts"' }, + }, + required: [], + }, + }, + }, + { + type: 'function' as const, + function: { + name: 'write_file', + description: 'Create or overwrite a whole file (needs approval). Use edit_file to change existing files.', + parameters: { + type: 'object', + properties: { + path: { type: 'string', description: 'File path' }, + content: { type: 'string', description: 'Full file content' }, + }, + required: ['path', 'content'], + }, + }, + }, + { + type: 'function' as const, + function: { + name: 'edit_file', + description: 'Replace an exact, UNIQUE snippet in a file (old_string→new_string). Preferred for edits; needs approval.', + parameters: { + type: 'object', + properties: { + path: { type: 'string', description: 'File path' }, + old_string: { type: 'string', description: 'Exact unique text to replace' }, + new_string: { type: 'string', description: 'Replacement' }, + }, + required: ['path', 'old_string', 'new_string'], + }, + }, + }, + { + type: 'function' as const, + function: { + name: 'run_command', + description: 'Run a shell command in an allowed folder. Chain steps with && to save calls.', + parameters: { + type: 'object', + properties: { + command: { type: 'string', description: 'Command' }, + cwd: { type: 'string', description: 'Working dir (allowed folder)' }, + }, + required: ['command', 'cwd'], + }, + }, + }, +] as const; + +/** + * Web research tool. Kept OUT of TOOL_DEFINITIONS and appended by the agent loop + * only when enabled (Incognito), so it never reaches Secure Mode — there, all + * network must go through the container's `--network` flag, and a host-side + * fetch would bypass that boundary. + */ +export const WEB_FETCH_TOOL = { + type: 'function' as const, + function: { + name: 'web_fetch', + description: 'Fetch an http(s) URL and return its text (HTML stripped). Read-only. Set summarize:true for a short gist (cheaper) instead of the full page.', + parameters: { + type: 'object', + properties: { + url: { type: 'string', description: 'Absolute http(s) URL' }, + summarize: { type: 'boolean', description: 'Return a short summary instead of the full text' }, + focus: { type: 'string', description: 'What the summary should focus on (optional)' }, + }, + required: ['url'], + }, + }, +} as const; + +/** + * read_summary — read a file and return a SHORT summary via a cheap model, + * instead of dumping the whole file into the expensive model's context (and + * history). Host-loop only (Work / Incognito); kept out of Secure, since the + * summary call is a host-side network request that would bypass the sandbox's + * container network gate. + */ +export const READ_SUMMARY_TOOL = { + type: 'function' as const, + function: { + name: 'read_summary', + description: 'Read a file and return a SHORT summary (cheap model) — for when you just need the gist of a large file. Use read_file when you need exact content to edit.', + parameters: { + type: 'object', + properties: { + path: { type: 'string', description: 'File path' }, + focus: { type: 'string', description: 'What to focus the summary on (optional)' }, + }, + required: ['path'], + }, + }, +} as const; + +/** + * Web search — kept OUT of TOOL_DEFINITIONS (like web_fetch) and appended by the + * agent loop only for the host-loop modes (Work / Incognito), never Secure + * (host-side network would bypass the sandbox's container network gate). + */ +export const WEB_SEARCH_TOOL = { + type: 'function' as const, + function: { + name: 'web_search', + description: 'Search the web; returns top results (title, url, snippet). Then web_fetch a URL for details.', + parameters: { + type: 'object', + properties: { + query: { type: 'string', description: 'Search query' }, + count: { type: 'number', description: 'Max results (default 6)' }, + }, + required: ['query'], + }, + }, +} as const; + +/** + * analyze_link — extract the *content* behind a URL (not the raw HTML). + * + * Kept OUT of TOOL_DEFINITIONS and appended by the loop, like the web tools. + * Unlike web_fetch/web_search (host-side), this runs ENTIRELY inside the + * sandbox: the loop injects a `runInSandbox` callback wired to the container, + * so the network egress respects the same container network gate. First handler + * is YouTube (subtitles via yt-dlp); the registry is built to grow to more + * sites without touching the schema. + * + * `mode` is the token lever: "summary" (default) hands the extracted text to a + * cheap model focused on `purpose` and returns only the gist; "full" returns the + * cleaned text (clamped). Always prefer "summary" unless you need exact wording. + */ +export const ANALYZE_LINK_TOOL = { + type: 'function' as const, + function: { + name: 'analyze_link', + description: + 'Extract and read the content behind a URL (currently: YouTube video subtitles/transcript). ' + + 'Runs in the sandbox. Use mode:"summary" (default, cheap) for the gist; mode:"full" only when ' + + 'you need exact wording. Always pass a short "purpose" so the summary keeps what you actually need. ' + + 'For plain web articles use web_fetch instead.', + parameters: { + type: 'object', + properties: { + url: { type: 'string', description: 'Absolute http(s) URL (e.g. a YouTube video link)' }, + mode: { + type: 'string', + enum: ['summary', 'full'], + description: 'summary = cheap, purpose-focused gist (default); full = cleaned transcript (clamped)', + }, + purpose: { + type: 'string', + description: 'One short sentence on what you need from the link — drives the summary extraction', + }, + }, + required: ['url'], + }, + }, +} as const; + +export type ToolName = + | 'list_files' | 'read_file' | 'read_summary' | 'search_files' | 'write_file' | 'edit_file' + | 'run_command' | 'web_fetch' | 'web_search' | 'analyze_link'; + +/** + * A token/cost saving worth surfacing in the chat. Emitted when a tool fed the + * main model a cheap SUMMARY instead of the full extracted content (e.g. + * analyze_link summary mode reads a long video transcript but hands the model + * only the gist). The loop forwards it as a `note` event → a chat system note. + */ +export interface SavingsNote { + /** Which tool produced the saving — drives the chat-note wording. */ + kind: 'analyze_link' | 'read_file' | 'read_summary' | 'run_command' | 'search'; + /** Short human label for the source, e.g. "YouTube transcript". */ + source: string; + /** + * Whole-percent of content NOT sent to the main model (1 - delivered/full). + * Present only when we actually hold BOTH numbers as a byproduct of normal + * work (read_file/read_summary/run_command/analyze_link). Omitted for `search`, + * where ripgrep trims long lines but never reports how many bytes it dropped — + * so we surface a quantity-free "trimmed oversized output" note instead of a + * fabricated percentage. + */ + savedPct?: number; + /** Chars the model would have ingested otherwise (full). */ + fullChars?: number; + /** Chars actually delivered to the model. */ + deliveredChars?: number; +} + +// Only post a "saved you X%" note when the win is real and worth mentioning — +// a tiny truncation isn't worth a chat row. Both gates must pass. +const SAVINGS_MIN_FULL_CHARS = 4_000; +const SAVINGS_MIN_PCT = 30; + +/** + * Emit a "saved you X%" note IFF we already hold both the full and delivered + * sizes (zero extra work — just a subtraction) and the saving clears the gates. + * Used by read_file / read_summary / run_command truncation. Never fabricates a + * baseline; if the numbers aren't both in hand, the caller simply doesn't call. + */ +function emitTruncationSaving( + ctx: ToolContext, + kind: SavingsNote['kind'], + source: string, + fullChars: number, + deliveredChars: number, +): void { + if (!ctx.onNote || fullChars < SAVINGS_MIN_FULL_CHARS || deliveredChars >= fullChars) return; + const savedPct = Math.round((1 - deliveredChars / fullChars) * 100); + if (savedPct < SAVINGS_MIN_PCT) return; + ctx.onNote({ kind, source, savedPct, fullChars, deliveredChars }); +} + +// ── Tool context (injected per-session) ────────────────────────────────────── + +export interface ToolContext { + allowedFolders: string[]; + /** + * Per-folder access levels (path + readonly flag). When provided, write_file + * is denied for paths under a read-only folder. When omitted (e.g. restored + * sessions) all allowed folders are treated as writable. + */ + folderAccess?: { path: string; readonly: boolean }[]; + /** + * Runs a shell command for run_command. Injected so the host never executes + * bash directly: Work Mode wires this to a Docker container with per-folder + * :ro/:rw mounts (the kernel enforces read-only). When omitted (no Docker), + * run_command is disabled and returns a guidance message — the strict + * fallback that keeps read-only an honest guarantee. + */ + runShell?: (command: string, cwd: string) => Promise<string>; + /** + * Runs an analyze_link helper command inside the session's sandbox container + * (warm-reused; yt-dlp self-installs once). Injected so yt-dlp — which parses + * untrusted YouTube data — never runs on the host. Omitted when no Docker → + * analyze_link returns a guidance message. + */ + linkSandbox?: (command: string) => Promise<string>; + /** + * Incognito (read-only): write_file is denied outright (nothing ever hits + * disk), and run_command is only reachable via a read-only sandbox. + */ + readOnly?: boolean; + /** + * Incognito: file reads (read/list/search) are NOT restricted to + * allowedFolders — the agent may read anywhere on the host. The secret + * deny-list (BLOCKED_PATTERNS in security.ts) still applies, so .ssh/.env/ + * credentials etc. are refused regardless. + */ + readAnywhere?: boolean; + /** Called when agent wants to write — returns true if approved, false if rejected. */ + requestWriteApproval: (filePath: string, content: string) => Promise<boolean>; + /** + * Optional sink for a user-visible "saved N% cost" note (analyze_link summary + * mode). The agent loop wires this to forward a `note` event to the chat. + */ + onNote?: (note: SavingsNote) => void; +} + +/** Folders to validate reads against — empty (anywhere) for Incognito. */ +function readFolders(ctx: ToolContext): string[] { + return ctx.readAnywhere ? [] : ctx.allowedFolders; +} + +// ── Tool implementations ────────────────────────────────────────────────────── + +export async function executeTool( + name: ToolName, + args: Record<string, unknown>, + ctx: ToolContext, +): Promise<string> { + try { + switch (name) { + case 'list_files': return await listFiles(args, ctx); + case 'read_file': return await readFile(args, ctx); + case 'read_summary': return await readSummary(args, ctx); + case 'search_files': return await searchFiles(args, ctx); + case 'write_file': return await writeFile(args, ctx); + case 'edit_file': return await editFile(args, ctx); + case 'run_command': return await runCommand(args, ctx); + case 'web_fetch': return await webFetch(args); + case 'web_search': return await webSearch(args); + case 'analyze_link': + if (!ctx.linkSandbox) { + return 'analyze_link needs a sandbox container (Docker), which is unavailable here. Use web_fetch/web_search instead.'; + } + return await analyzeLink(args, { runInSandbox: ctx.linkSandbox, networkEnabled: true, onNote: ctx.onNote }); + default: return `Unknown tool: ${name}`; + } + } catch (err) { + if (err instanceof SecurityError) return `Security error: ${err.message}`; + return `Error: ${err instanceof Error ? err.message : String(err)}`; + } +} + +async function listFiles(args: Record<string, unknown>, ctx: ToolContext): Promise<string> { + const dir = validatePath(args.path as string, readFolders(ctx)); + const entries = fs.readdirSync(dir, { withFileTypes: true }); + // Explicit [dir]/[file] labels (+ trailing slash on dirs) so the model can + // reliably tell directories from files and recurse without guessing. + const lines = entries + .sort((a, b) => Number(b.isDirectory()) - Number(a.isDirectory()) || a.name.localeCompare(b.name)) + .map((e) => (e.isDirectory() ? `[dir] ${e.name}/` : `[file] ${e.name}`)); + if (lines.length === 0) return '(empty directory)'; + // Cap big listings — the full list otherwise lands in history and is resent + // every round. + if (lines.length > MAX_LIST_ENTRIES) { + const shown = lines.slice(0, MAX_LIST_ENTRIES); + return `${shown.join('\n')}\n…[+${lines.length - MAX_LIST_ENTRIES} more entries — narrow with search_files]`; + } + return lines.join('\n'); +} + +async function readFile(args: Record<string, unknown>, ctx: ToolContext): Promise<string> { + const filePath = validatePath(args.path as string, readFolders(ctx)); + const stat = fs.statSync(filePath); + if (stat.size > MAX_FILE_BYTES) { + return `File too large to read whole (${stat.size.toLocaleString()} bytes). Use search_files to find the part you need.`; + } + const content = fs.readFileSync(filePath, 'utf-8'); + if (content.length > MAX_FILE_READ_CHARS) { + // We already read the whole file; handing the model only the head is a real, + // free-to-measure saving (full = content.length, delivered = the cap). + emitTruncationSaving(ctx, 'read_file', path.basename(filePath), content.length, MAX_FILE_READ_CHARS); + return content.slice(0, MAX_FILE_READ_CHARS) + + `\n\n…[truncated: showing first ${MAX_FILE_READ_CHARS.toLocaleString()} of ${content.length.toLocaleString()} chars. Use search_files for specific content, or read_summary for the gist.]`; + } + return content; +} + +async function readSummary(args: Record<string, unknown>, ctx: ToolContext): Promise<string> { + const filePath = validatePath(args.path as string, readFolders(ctx)); + const stat = fs.statSync(filePath); + if (stat.size > MAX_FILE_BYTES) { + return `File too large (${stat.size.toLocaleString()} bytes). Use search_files for specific content.`; + } + const content = fs.readFileSync(filePath, 'utf-8'); + if (!content.trim()) return '(empty file)'; + const summary = await summarizeText(content, args.focus ? String(args.focus) : undefined); + // Model ingests the short summary instead of the full file — both sizes in hand. + emitTruncationSaving(ctx, 'read_summary', path.basename(filePath), content.length, summary.length); + return `Summary of ${path.basename(filePath)} (${content.length.toLocaleString()} chars). ` + + `Call read_file for the exact content if you need to edit it.\n\n${summary}`; +} + +async function searchFiles(args: Record<string, unknown>, ctx: ToolContext): Promise<string> { + const query = (args.query as string | undefined) ?? ''; + const name = (args.name as string | undefined)?.trim(); + const pattern = (args.filePattern as string | undefined) ?? ''; + const rawPath = (args.path as string | undefined)?.trim(); + + if (!name && !query) { + return 'Provide "name" to locate a file by its name, or "query" to search file contents.'; + } + + // Resolve the search roots. With an explicit path, validate it lies inside the + // allowed folders (unchanged). WITHOUT one, sweep EVERY allowed folder in a + // single ripgrep call — a Work chat can grant many folders, and making the + // model search them one-by-one burns rounds and bloats context. ripgrep takes + // multiple path args, so all folders are covered in one process. + // (Incognito reads anywhere and has no fixed root set, so it still needs an + // explicit path — there's no sane "search the whole disk" default.) + let roots: string[]; + if (rawPath) { + try { + roots = [validatePath(rawPath, readFolders(ctx))]; + } catch (e) { + // Don't just bounce the model with a bare error — point it at the cheap + // fix (omit path → all allowed folders) so it doesn't waste a round + // guessing another path. + if (e instanceof SecurityError) { + const allowed = readFolders(ctx); + return `Security error: ${e.message}. Tip: omit "path" to search every allowed folder at once` + + `${allowed.length ? ` (${allowed.join(', ')})` : ''}.`; + } + throw e; + } + } else { + const allowed = readFolders(ctx); + if (allowed.length === 0) { + return 'Provide a "path" to search — no folder is granted to search across.'; + } + roots = allowed; + } + + const excludeGlobs = SEARCH_EXCLUDE_DIRS.flatMap((d) => ['-g', `!**/${d}/**`]); + + // NAME mode: locate files by filename across all roots and return paths only — + // no content, no per-line context. This is the right primitive for "where is + // file X" (a content grep for a filename matches every log/trace that mentions + // it, dumping huge irrelevant lines). `rg --files` lists files honouring our + // excludes; --iglob filters by name (case-insensitive, friendlier for users). + if (name) { + const glob = /[*?]/.test(name) ? name : `*${name}*`; + const fileArgs = ['--files', '--hidden', '--no-messages', ...excludeGlobs, '--iglob', glob, '--', ...roots]; + let out = ''; + try { + ({ stdout: out } = await execFileAsync(rgPath, fileArgs, { timeout: COMMAND_TIMEOUT, maxBuffer: 8 * 1024 * 1024 })); + } catch (err) { + const e = err as { killed?: boolean; signal?: string; stdout?: string }; + if (e.killed || e.signal === 'SIGTERM' || e.signal === 'SIGKILL') { + return 'Search did not finish in time (the tree is large). This is NOT a confirmation that the file is missing — search a more specific subfolder.'; + } + out = e.stdout ?? ''; + } + const found = out.trim().split('\n').filter(Boolean).slice(0, 40); + if (found.length === 0) return 'No files found with that name.'; + const more = out.trim().split('\n').filter(Boolean).length > 40 ? '\n…[more — narrow the name]' : ''; + return found.join('\n') + more; + } + + // CONTENT mode: find the files that contain `query`. ripgrep skips binary + // files (videos / photos / archives / office docs) and never follows symlinks + // for free; we pass our own exclude list as ignore globs so generated/ + // dependency trees are pruned with or without git. This keeps the search fast + // and — paired with the honest timeout below — truthful (an unscoped grep over + // a 40GB+ tree used to time out, misreported as "No matches found", i.e. a + // false "file doesn't exist"). + // --hidden searches dotfiles too (rg still always skips .git); -F = literal + // (plain substring); --max-filesize guards against slurping a giant file. + const rgArgs = ['--hidden', '--no-messages', '-l', '-F', '-m', '1', + '--max-filesize', `${MAX_FILE_BYTES}`, + ...(pattern ? ['-g', pattern] : []), + ...excludeGlobs, + '--', query, ...roots]; + + let stdout = ''; + try { + ({ stdout } = await execFileAsync(rgPath, rgArgs, { timeout: COMMAND_TIMEOUT, maxBuffer: 8 * 1024 * 1024 })); + } catch (err) { + const e = err as { killed?: boolean; signal?: string; code?: number; stdout?: string }; + // CRITICAL: a timeout/kill is NOT "no matches". Say so explicitly so the + // model narrows the search instead of telling the user the file is absent. + if (e.killed || e.signal === 'SIGTERM' || e.signal === 'SIGKILL') { + return ( + `Search did not finish in time (the tree is large` + + `${pattern ? '' : ' and no filePattern was given, so every file was scanned'}). ` + + `This is NOT a confirmation that the file is missing. Narrow it: search a more ` + + `specific subfolder, or pass filePattern (e.g. "*.sh").` + ); + } + // ripgrep exits 1 on no matches, 2 on partial read errors (matches on + // readable files may still be in stdout). Keep whatever it printed. + stdout = e.stdout ?? ''; + } + + const files = stdout.trim().split('\n').filter(Boolean).slice(0, 20); + if (files.length === 0) return 'No matches found.'; + + // Show matching lines for the first 5 files. Cap matches PER FILE (-m) so a + // file with thousands of hits can't dump itself into history (resent every + // round), and clamp the combined output as a backstop. + const results: string[] = []; + let trimmedLines = 0; // long match lines ripgrep clamped (markers it printed) + for (const file of files.slice(0, 5)) { + // --max-columns caps each match LINE's length: minified JS / JSON / log + // (.jsonl trace) files can have single lines tens of thousands of chars + // long, and dumping even a few of them blew the token budget. -preview keeps + // a short head of the line instead of omitting it outright. + const { stdout: ctx2 } = await execFileAsync( + rgPath, ['-n', '--no-messages', '-F', '-m', String(MAX_MATCH_LINES_PER_FILE), + '--max-columns', '200', '--max-columns-preview', '--', query, file], { timeout: 5000 }, + ).catch(() => ({ stdout: '' })); + trimmedLines += (ctx2.match(/\[\.\.\. omitted end of long line\]/g) || []).length; + const trimmed = ctx2.trim(); + const lineCount = trimmed ? trimmed.split('\n').length : 0; + const more = lineCount >= MAX_MATCH_LINES_PER_FILE ? `\n…[more matches — refine the query]` : ''; + results.push(`${file}:\n${trimmed}${more}`); + } + if (files.length > 5) results.push(`...and ${files.length - 5} more files`); + // Qualitative saving: ripgrep never tells us HOW MANY bytes it dropped from a + // long line, only that it did. So when several oversized lines were trimmed we + // post a quantity-free note (no fabricated %), gated so it's not noise. + if (ctx.onNote && trimmedLines >= 2) { + ctx.onNote({ kind: 'search', source: 'your files' }); + } + return clampMiddle(results.join('\n\n'), MAX_CMD_OUTPUT_CHARS); +} + +async function writeFile(args: Record<string, unknown>, ctx: ToolContext): Promise<string> { + if (ctx.readOnly) { + return 'write_file is disabled in Incognito mode. Incognito is read-only and never writes to disk — ' + + 'summarize or return the content in your reply instead.'; + } + const filePath = validatePath(args.path as string, ctx.allowedFolders); + const content = args.content as string; + + if (ctx.folderAccess && !isWriteAllowed(filePath, ctx.folderAccess)) { + return `Write denied: "${filePath}" is in a read-only folder. Ask the user to grant read & write access to this folder.`; + } + + const approved = await ctx.requestWriteApproval(filePath, content); + if (!approved) return `Write rejected by user: ${filePath}`; + + fs.mkdirSync(path.dirname(filePath), { recursive: true }); + fs.writeFileSync(filePath, content, 'utf-8'); + return `Written: ${filePath}`; +} + +async function editFile(args: Record<string, unknown>, ctx: ToolContext): Promise<string> { + if (ctx.readOnly) { + return 'edit_file is disabled in Incognito mode (read-only). Return the change in your reply instead.'; + } + const filePath = validatePath(args.path as string, ctx.allowedFolders); + const oldStr = String(args.old_string ?? ''); + const newStr = String(args.new_string ?? ''); + if (!oldStr) return 'edit_file requires old_string — the exact text to replace.'; + + if (ctx.folderAccess && !isWriteAllowed(filePath, ctx.folderAccess)) { + return `Edit denied: "${filePath}" is in a read-only folder. Ask the user to grant read & write access.`; + } + + let content: string; + try { + content = fs.readFileSync(filePath, 'utf-8'); + } catch { + return `File not found (read it / create with write_file first): ${filePath}`; + } + + const first = content.indexOf(oldStr); + if (first === -1) { + return 'old_string not found. Read the file and copy the exact text to replace (including whitespace/indentation).'; + } + if (content.indexOf(oldStr, first + oldStr.length) !== -1) { + return 'old_string is not unique — it appears more than once. Include more surrounding context so it matches exactly one place.'; + } + + const next = content.slice(0, first) + newStr + content.slice(first + oldStr.length); + + // Reuse the write-approval flow; show the resulting full content so the UI + // diff/preview reflects what will land on disk. + const approved = await ctx.requestWriteApproval(filePath, next); + if (!approved) return `Edit rejected by user: ${filePath}`; + + fs.writeFileSync(filePath, next, 'utf-8'); + return `Edited: ${filePath}`; +} + +async function runCommand(args: Record<string, unknown>, ctx: ToolContext): Promise<string> { + // Validate cwd is inside an allowed folder up front (clear error before we + // hand off to the sandbox, and the container only mounts allowed folders). + const cwd = validatePath(args.cwd as string, ctx.allowedFolders); + const command = args.command as string; + + // No sandbox available → run_command is disabled. We never fall back to host + // bash, because that can't enforce per-folder read-only. File tools still + // work (write_file is path-checked on the host). + if (!ctx.runShell) { + return 'run_command is unavailable. Shell commands run in a Docker sandbox, which needs both ' + + '(1) Docker installed and running, and (2) at least one folder explicitly selected for this chat. ' + + 'Ask the user to start Docker and/or add a folder. Meanwhile read_file / search_files / write_file still work.'; + } + + // The sandbox mounts read-only folders as :ro, so the kernel — not us — + // rejects any write outside the read & write folders. Commands may freely + // read from read-only folders. + const out = await ctx.runShell(command, cwd); + // Compress noise (ANSI, progress bars, repeated lines) THEN cap verbose output + // (test runs, build logs) so the cap keeps real signal, not flooding history. + const delivered = clampMiddle(compressCommandOutput(out), MAX_CMD_OUTPUT_CHARS); + // The command's full output is already in hand; if we capped it, that's a real + // saving (the model would otherwise ingest all of it, every round). + emitTruncationSaving(ctx, 'run_command', 'command output', out.length, delivered.length); + return delivered; +} + +// ── web_fetch (read-only research) ──────────────────────────────────────────── + +/** Crude HTML → readable text. Good enough for research summaries, not parsing. */ +function htmlToText(html: string): string { + return html + .replace(/<script[\s\S]*?<\/script>/gi, ' ') + .replace(/<style[\s\S]*?<\/style>/gi, ' ') + .replace(/<!--[\s\S]*?-->/g, ' ') + .replace(/<\/(p|div|li|h[1-6]|tr|section|article|header|footer)>/gi, '\n') + .replace(/<br\s*\/?>/gi, '\n') + .replace(/<[^>]+>/g, ' ') + .replace(/ /g, ' ') + .replace(/&/g, '&') + .replace(/</g, '<') + .replace(/>/g, '>') + .replace(/"/g, '"') + .replace(/'/g, "'") + .replace(/[ \t]+/g, ' ') + .replace(/\n{3,}/g, '\n\n') + .trim(); +} + +async function webFetch(args: Record<string, unknown>): Promise<string> { + const url = String(args.url ?? '').trim(); + if (!/^https?:\/\/\S+$/i.test(url)) { + return 'Only absolute http(s) URLs are allowed.'; + } + const ctrl = new AbortController(); + const timer = setTimeout(() => ctrl.abort(), WEB_FETCH_TIMEOUT); + try { + const res = await fetch(url, { + signal: ctrl.signal, + redirect: 'follow', + headers: { 'User-Agent': 'iClaw-Incognito/1.0', Accept: 'text/html,application/json;q=0.9,*/*;q=0.8' }, + }); + const ct = res.headers.get('content-type') || ''; + const raw = await res.text(); + let text = /html/i.test(ct) ? htmlToText(raw) : raw.trim(); + // Optional: hand the page to the cheap model and return just the gist. + if (args.summarize === true && text) { + const summary = await summarizeText(text, args.focus ? String(args.focus) : undefined); + return `Summary of ${url} (HTTP ${res.status}):\n\n${summary}`; + } + if (text.length > WEB_FETCH_MAX_CHARS) { + text = text.slice(0, WEB_FETCH_MAX_CHARS) + `\n\n[truncated at ${WEB_FETCH_MAX_CHARS} chars]`; + } + return `HTTP ${res.status} — ${url}\n\n${text || '(empty response body)'}`; + } catch (err) { + const msg = err instanceof Error && err.name === 'AbortError' + ? `timed out after ${WEB_FETCH_TIMEOUT / 1000}s` + : err instanceof Error ? err.message : String(err); + return `Fetch failed (${url}): ${msg}`; + } finally { + clearTimeout(timer); + } +} + +// ── cheap-model summarizer (read_summary, web_fetch summarize) ──────────────── + +/** + * Summarize text with a cheap model via OpenRouter. Faithful + dense; preserves + * exact names/numbers. Degrades gracefully to a truncation if there's no key or + * the call fails, so callers always get usable output. + */ +async function summarizeText(text: string, focus?: string): Promise<string> { + const key = process.env.ICLAW_OPENROUTER_API_KEY || ''; + if (!key || !text.trim()) return clampMiddle(text, MAX_CMD_OUTPUT_CHARS); + const base = (process.env.OPENROUTER_BASE_URL || 'https://openrouter.ai/api/v1').replace(/\/+$/, ''); + const input = text.length > SUMMARY_MAX_INPUT_CHARS ? text.slice(0, SUMMARY_MAX_INPUT_CHARS) : text; + const ctrl = new AbortController(); + const timer = setTimeout(() => ctrl.abort(), WEB_FETCH_TIMEOUT); + try { + const res = await fetch(`${base}/chat/completions`, { + method: 'POST', + signal: ctrl.signal, + headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${key}` }, + body: JSON.stringify({ + model: SUMMARY_MODEL, + temperature: 0.2, + messages: [ + { + role: 'system', + content: + 'You compress a document for another AI agent. Produce a dense, faithful summary: ' + + 'what it is, its structure, and the key facts. Preserve exact names, numbers, paths and ' + + 'identifiers. No preamble, no fluff. If the input was truncated, say so at the end.', + }, + { role: 'user', content: (focus ? `Focus on: ${focus}\n\n---\n` : '') + input }, + ], + }), + }); + if (!res.ok) throw new Error(`summary HTTP ${res.status}`); + const data = await res.json() as { choices?: { message?: { content?: string } }[] }; + const out = data.choices?.[0]?.message?.content; + return (typeof out === 'string' && out.trim()) + ? out.trim() + (text.length > SUMMARY_MAX_INPUT_CHARS ? '\n\n[note: input was truncated before summarizing]' : '') + : clampMiddle(text, MAX_CMD_OUTPUT_CHARS); + } catch { + return clampMiddle(text, MAX_CMD_OUTPUT_CHARS); + } finally { + clearTimeout(timer); + } +} + +// ── web_search (OpenRouter by default, DuckDuckGo as keyless fallback) ──────── + +interface SearchHit { title: string; url: string; snippet: string } + +function formatHits(query: string, hits: SearchHit[], provider: string): string { + if (hits.length === 0) return `No results for "${query}".`; + const lines = hits.map((h, i) => `${i + 1}. ${h.title}\n ${h.url}${h.snippet ? `\n ${h.snippet}` : ''}`); + return `Web search (${provider}) — "${query}":\n\n${lines.join('\n\n')}`; +} + +/** + * Zero-config default: use OpenRouter's built-in web search via the SAME key the + * runtime already uses for chat — no separate search account/key to set up. The + * `web` plugin runs a search and the response carries `url_citation` annotations + * (title/url/snippet). Costs a small per-result fee on the user's existing + * OpenRouter credits. + */ +async function openRouterSearch(query: string, count: number, signal: AbortSignal): Promise<SearchHit[]> { + const key = process.env.ICLAW_OPENROUTER_API_KEY || ''; + if (!key) throw new Error('no OpenRouter key'); + const base = (process.env.OPENROUTER_BASE_URL || 'https://openrouter.ai/api/v1').replace(/\/+$/, ''); + const model = process.env.ICLAW_SEARCH_MODEL || process.env.ICLAW_MODEL || 'google/gemini-2.5-flash'; + const res = await fetch(`${base}/chat/completions`, { + method: 'POST', + signal, + headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${key}` }, + body: JSON.stringify({ + model, + plugins: [{ id: 'web', max_results: count }], + messages: [{ role: 'user', content: `Find the most relevant, recent web results for: ${query}` }], + }), + }); + if (!res.ok) throw new Error(`OpenRouter HTTP ${res.status}`); + const data = await res.json() as { + choices?: { message?: { annotations?: { type?: string; url_citation?: { url?: string; title?: string; content?: string } }[] } }[]; + }; + const anns = data.choices?.[0]?.message?.annotations ?? []; + return anns + .filter((a) => a.type === 'url_citation' && a.url_citation?.url) + .map((a) => ({ + title: a.url_citation!.title || a.url_citation!.url!, + url: a.url_citation!.url!, + snippet: (a.url_citation!.content ?? '').replace(/\s+/g, ' ').trim().slice(0, 300), + })) + .slice(0, count); +} + +async function duckDuckGoSearch(query: string, count: number, signal: AbortSignal): Promise<SearchHit[]> { + const u = `https://html.duckduckgo.com/html/?q=${encodeURIComponent(query)}`; + const res = await fetch(u, { signal, headers: { 'User-Agent': 'Mozilla/5.0 iClaw-Incognito/1.0' } }); + const html = await res.text(); + const hits: SearchHit[] = []; + const re = /<a[^>]*class="result__a"[^>]*href="([^"]+)"[^>]*>([\s\S]*?)<\/a>/g; + let m: RegExpExecArray | null; + while ((m = re.exec(html)) && hits.length < count) { + let url = m[1]; + const uddg = /[?&]uddg=([^&]+)/.exec(url); // DDG wraps links in a redirect + if (uddg) url = decodeURIComponent(uddg[1]); + const title = m[2].replace(/<[^>]+>/g, '').replace(/&/g, '&').trim(); + if (url.startsWith('http')) hits.push({ title, url, snippet: '' }); + } + return hits; +} + +async function webSearch(args: Record<string, unknown>): Promise<string> { + const query = String(args.query ?? '').trim(); + if (!query) return 'web_search requires a query.'; + const count = Math.min(10, Math.max(1, Number(args.count) || 6)); + const ctrl = new AbortController(); + const timer = setTimeout(() => ctrl.abort(), WEB_FETCH_TIMEOUT); + try { + // 1) OpenRouter web search — the zero-config default (reuses the chat key). + if (process.env.ICLAW_OPENROUTER_API_KEY) { + try { + const hits = await openRouterSearch(query, count, ctrl.signal); + if (hits.length) return formatHits(query, hits, 'OpenRouter'); + } catch { /* fall through */ } + } + // 2) Keyless last resort. + return formatHits(query, await duckDuckGoSearch(query, count, ctrl.signal), 'DuckDuckGo'); + } catch (err) { + const msg = err instanceof Error && err.name === 'AbortError' + ? `timed out after ${WEB_FETCH_TIMEOUT / 1000}s` + : err instanceof Error ? err.message : String(err); + return `Search failed for "${query}": ${msg}.`; + } finally { + clearTimeout(timer); + } +} + +// ── analyze_link (sandboxed link → content extraction) ─────────────────────── +// +// Engine: yt-dlp (1000+ site extractors → easy to grow past YouTube). Shipped as +// a self-contained binary self-installed into /workspace/.tools/bin on first use +// — deliberately NOT baked into the image (keeps it lean; the binary persists in +// the workspace across container restarts, like uv). All network egress happens +// inside the sandbox via the injected runInSandbox callback, so it honours the +// same container network gate as run_command. + +const ANALYZE_LINK_FULL_MAX_CHARS = Number(process.env.ICLAW_ANALYZE_LINK_MAX) || 16_000; +// Subtitle language priority (yt-dlp glob syntax). Tunable; first available wins. +const ANALYZE_LINK_SUB_LANGS = process.env.ICLAW_ANALYZE_LINK_LANGS || 'en.*,uk.*'; +// Only surface a "saved N%" note when the win is real: the transcript is at +// least this long AND we trimmed at least this fraction. Avoids noise on clips. +const ANALYZE_LINK_SAVINGS_MIN_CHARS = 2_000; +const ANALYZE_LINK_SAVINGS_MIN_PCT = 25; + +/** Single-quote a string for safe embedding in a bash command. */ +function shQuote(s: string): string { + return `'${s.replace(/'/g, `'\\''`)}'`; +} + +/** Pull the 11-ish-char video id out of any YouTube URL shape, else null. */ +function extractYouTubeId(url: string): string | null { + try { + const u = new URL(url); + const host = u.hostname.replace(/^www\./, ''); + if (host === 'youtu.be') return u.pathname.slice(1).split('/')[0] || null; + if (host.endsWith('youtube.com') || host.endsWith('youtube-nocookie.com')) { + if (u.pathname === '/watch') return u.searchParams.get('v'); + const m = u.pathname.match(/^\/(?:shorts|embed|live|v)\/([^/?#]+)/); + if (m) return m[1]; + } + return null; + } catch { + return null; + } +} + +/** + * WebVTT → dense plain text. Drops the header, cue timestamps/ids and inline + * styling tags, and de-duplicates consecutive identical lines (YouTube + * auto-captions emit each segment twice — once styled, once plain). + */ +function cleanVtt(vtt: string): string { + const out: string[] = []; + let last = ''; + for (const raw of vtt.split(/\r?\n/)) { + if (!raw.trim()) continue; + if (/^WEBVTT/.test(raw)) continue; + if (/^(Kind|Language):/i.test(raw)) continue; + if (raw.includes('-->')) continue; // timestamp cue line + if (/^\d+$/.test(raw.trim())) continue; // numeric cue id + const line = raw + .replace(/<[^>]+>/g, '') // inline <00:00.000> / <c> tags + .replace(/ /g, ' ') + .replace(/&/g, '&') + .replace(/</g, '<') + .replace(/>/g, '>') + .replace(/'/g, "'") + .trim(); + if (!line || line === last) continue; + out.push(line); + last = line; + } + return out.join(' ').replace(/\s{2,}/g, ' ').trim(); +} + +/** + * Bash run inside the sandbox: ensure yt-dlp, fetch subtitles (+ info json) for + * the video without downloading it, then print a marker-delimited payload the + * host parses. `set -e`-safe: every fallible step is guarded so we always exit 0 + * with a parseable result (success, no-subs, or install error). + */ +function buildYouTubeSubsCommand(videoId: string, url: string): string { + const out = `/workspace/.cache/links/${videoId}`; // videoId is [\w-]-validated + return [ + 'set -e', + 'BIN=/workspace/.tools/bin', + 'mkdir -p "$BIN" /workspace/.cache/links', + 'export PATH="$BIN:$PATH"', + // Install yt-dlp if missing OR if a cached binary can\'t actually run (e.g. a + // wrong-arch download from a previous version) — self-heals the stale binary. + 'if ! (command -v yt-dlp >/dev/null 2>&1 && yt-dlp --version >/dev/null 2>&1); then', + // Pick the binary matching the container arch. yt-dlp_linux is x86_64-only; + // arm64 containers (e.g. Docker on Apple Silicon) need the aarch64 build. + ' case "$(uname -m)" in aarch64|arm64) YDL=yt-dlp_linux_aarch64 ;; armv7l|armhf) YDL=yt-dlp_linux_armv7l ;; *) YDL=yt-dlp_linux ;; esac', + ' curl -fsSL "https://github.com/yt-dlp/yt-dlp/releases/latest/download/$YDL" -o "$BIN/yt-dlp" 2>/dev/null && chmod +x "$BIN/yt-dlp" || { echo "__ERR__ could not download yt-dlp (network blocked?)"; exit 0; }', + ' hash -r 2>/dev/null || true', + ' yt-dlp --version >/dev/null 2>&1 || { echo "__ERR__ yt-dlp not runnable after install (arch=$(uname -m))"; exit 0; }', + 'fi', + `OUT=${shQuote(out)}`, + 'rm -f "$OUT"*.vtt "$OUT".info.json 2>/dev/null || true', + `ERR=$(yt-dlp --quiet --no-warnings --skip-download --write-subs --write-auto-subs --write-info-json --sub-langs ${shQuote(ANALYZE_LINK_SUB_LANGS)} --sub-format vtt -o "$OUT.%(ext)s" ${shQuote(url)} 2>&1) || true`, + 'f=$(ls "$OUT"*.vtt 2>/dev/null | head -1)', + 'if [ -n "$f" ]; then', + ' echo "__META__"', + ' jq -r \'[.title, .uploader, .duration_string] | map(select(. != null)) | join(" | ")\' "$OUT.info.json" 2>/dev/null || true', + ' echo "__VTT__"', + ' cat "$f"', + 'else', + ' echo "__NOSUBS__"', + ' echo "$ERR" | tail -n 5', + 'fi', + ].join('\n'); +} + +/** + * analyze_link implementation. `runInSandbox` runs a bash command inside the + * session's container; `networkEnabled` mirrors the chat's network gate. + */ +export async function analyzeLink( + args: Record<string, unknown>, + deps: { + runInSandbox: (command: string) => Promise<string>; + networkEnabled: boolean; + onNote?: (note: SavingsNote) => void; + }, +): Promise<string> { + const url = String(args.url ?? '').trim(); + if (!/^https?:\/\/\S+$/i.test(url)) return 'analyze_link needs an absolute http(s) URL.'; + const mode = args.mode === 'full' ? 'full' : 'summary'; + const purpose = args.purpose ? String(args.purpose) : undefined; + + if (!deps.networkEnabled) { + return 'analyze_link needs network, which is currently OFF for this chat. Ask the user to enable network, then retry.'; + } + + const videoId = extractYouTubeId(url); + if (!videoId || !/^[\w-]{6,}$/.test(videoId)) { + return 'analyze_link currently supports only YouTube video links. For web pages or articles, use web_fetch instead.'; + } + + let raw: string; + try { + raw = await deps.runInSandbox(buildYouTubeSubsCommand(videoId, url)); + } catch (err) { + return `analyze_link failed to run in the sandbox: ${err instanceof Error ? err.message : String(err)}`; + } + + if (raw.includes('__ERR__')) { + return `analyze_link: ${raw.split('__ERR__')[1].trim().slice(0, 300) || 'sandbox error'}`; + } + if (raw.includes('__NOSUBS__') || (!raw.includes('__VTT__'))) { + const detail = (raw.split('__NOSUBS__')[1] ?? raw).trim().slice(0, 250); + const blocked = /sign in|confirm.*bot|429|HTTP Error 403|blocked|not a bot/i.test(detail); + return blocked + ? `analyze_link: YouTube blocked the request from the sandbox IP, or the video is restricted. ${detail}` + : `analyze_link: no subtitles available in ${ANALYZE_LINK_SUB_LANGS} (video may have none, or only other languages). ${detail}`; + } + + const meta = raw.includes('__META__') ? raw.split('__META__')[1].split('__VTT__')[0].trim() : ''; + const transcript = cleanVtt(raw.split('__VTT__')[1] ?? ''); + if (!transcript) return 'analyze_link: subtitles were fetched but empty after cleanup.'; + const header = meta ? `Video: ${meta}\n\n` : ''; + + if (mode === 'full') { + const body = + transcript.length > ANALYZE_LINK_FULL_MAX_CHARS + ? transcript.slice(0, ANALYZE_LINK_FULL_MAX_CHARS) + + `\n\n…[truncated ${(transcript.length - ANALYZE_LINK_FULL_MAX_CHARS).toLocaleString()} chars — re-call with mode:"summary" and a purpose]` + : transcript; + return `${header}Transcript (${transcript.length.toLocaleString()} chars):\n\n${body}`; + } + + const summary = await summarizeText(transcript, purpose); + + // Cost win: the main model ingests this short summary instead of the whole + // transcript (which it would also re-read every subsequent round). Surface the + // saving as a chat note. Char-count is a transparent token proxy; gate on a + // real win so we don't post noise for tiny clips. + if (deps.onNote && transcript.length >= ANALYZE_LINK_SAVINGS_MIN_CHARS && summary.length < transcript.length) { + const savedPct = Math.round((1 - summary.length / transcript.length) * 100); + if (savedPct >= ANALYZE_LINK_SAVINGS_MIN_PCT) { + deps.onNote({ + kind: 'analyze_link', + source: 'video transcript', + savedPct, + fullChars: transcript.length, + deliveredChars: summary.length, + }); + } + } + + return `${header}Transcript summary${purpose ? ` (focus: ${purpose})` : ''}:\n\n${summary}`; +} diff --git a/packages/iclaw-runtime/src/docker-lifecycle.ts b/packages/iclaw-runtime/src/docker-lifecycle.ts new file mode 100644 index 0000000..369b7e3 --- /dev/null +++ b/packages/iclaw-runtime/src/docker-lifecycle.ts @@ -0,0 +1,233 @@ +/** + * On-demand Docker lifecycle. + * + * Docker is only needed when a task actually runs a sandboxed command (Work + * run_command / analyze_link, or any Safe-Mode turn). So instead of nagging the + * user up front, we start Docker ourselves the first moment a task needs it, and + * — only if WE started it — stop it again once it's been idle and there are no + * containers left running (so we never kill the user's own Docker work). + * + * ensureDockerForTask() → true if Docker is usable now (started it if needed) + * startDockerIdleReaper() → background loop that stops a self-started, idle, + * container-free daemon after ICLAW_DOCKER_IDLE_STOP_MS + * + * Linux is start-only-by-the-user (the daemon needs sudo), so there we never + * auto-start or auto-stop. + */ +import { execFile } from 'node:child_process'; +import { existsSync, mkdirSync, rmSync, statSync, utimesSync, writeFileSync } from 'node:fs'; +import { homedir, platform as osPlatform } from 'node:os'; +import { join } from 'node:path'; +import { promisify } from 'node:util'; + +import { log } from './log.js'; + +const execFileAsync = promisify(execFile); + +const PROBE_TIMEOUT = 8_000; +const START_POLL_ATTEMPTS = 30; +const START_POLL_INTERVAL = 2_000; +/** Idle window before stopping a daemon we started (default 15 min). */ +const IDLE_STOP_MS = Number(process.env.ICLAW_DOCKER_IDLE_STOP_MS) || 15 * 60_000; +const REAPER_INTERVAL = 60_000; + +/** + * Durable "iClaw started Docker" marker. Its EXISTENCE means some iClaw instance + * started the daemon (so we may stop it); its MTIME is the last time a task used + * Docker (the idle-stop countdown). On disk — NOT in memory — so the ownership + * survives a runtime restart/crash: a fresh process inherits it and its reaper + * still stops the daemon a previous process started. Cleared when we stop Docker + * (or could be removed by the user). Single file, shared across installs; the + * "no running containers" guard is what actually keeps it safe. + */ +const OWN_MARKER = process.env.ICLAW_DOCKER_OWN_MARKER || join(homedir(), '.iclaw', '.docker-owned'); + +let ensureInFlight: Promise<boolean> | null = null; + +function sleep(ms: number): Promise<void> { + return new Promise((r) => setTimeout(r, ms)); +} + +/** Claim Docker ownership durably (we started it). */ +function markOwned(): void { + try { + mkdirSync(join(OWN_MARKER, '..'), { recursive: true }); + writeFileSync(OWN_MARKER, ''); + } catch { + /* best-effort — without the marker we just won't auto-stop */ + } +} + +/** Refresh last-use (marker mtime) — only if we own Docker. */ +function touchOwned(): void { + if (!existsSync(OWN_MARKER)) return; + try { + const now = new Date(); + utimesSync(OWN_MARKER, now, now); + } catch { + /* ignore */ + } +} + +/** Last-use time (ms) if iClaw owns the daemon, else null. */ +function ownedSince(): number | null { + try { + return statSync(OWN_MARKER).mtimeMs; + } catch { + return null; + } +} + +/** Release ownership (after we stop Docker). */ +function clearOwned(): void { + try { + rmSync(OWN_MARKER, { force: true }); + } catch { + /* ignore */ + } +} + +async function reachable(): Promise<boolean> { + try { + await execFileAsync('docker', ['info'], { timeout: PROBE_TIMEOUT }); + return true; + } catch { + return false; + } +} + +async function installed(): Promise<boolean> { + try { + await execFileAsync('docker', ['--version'], { timeout: PROBE_TIMEOUT }); + return true; + } catch { + return false; + } +} + +async function launchDaemon(): Promise<void> { + const plat = osPlatform(); + if (plat === 'darwin') { + await execFileAsync('open', ['-a', 'Docker'], { timeout: PROBE_TIMEOUT }); + } else if (plat === 'win32') { + await execFileAsync('cmd', ['/c', 'start', '', 'Docker Desktop'], { timeout: PROBE_TIMEOUT }); + } + // Linux: starting dockerd needs sudo — leave it to the user. +} + +/** + * Ensure Docker is usable for a task RIGHT NOW. If it's down but installed, start + * it ourselves and wait (up to ~60s). Returns false when Docker is missing or + * couldn't be started (caller then disables run_command / fails the Safe turn). + * Single-flight so concurrent tools don't each try to launch the daemon. + */ +export function ensureDockerForTask(): Promise<boolean> { + if (!ensureInFlight) { + ensureInFlight = (async () => { + if (await reachable()) { + // Already up. Refresh the idle countdown only if WE own it (a prior + // iClaw run started it) — never claim a daemon the user started. + touchOwned(); + return true; + } + if (!(await installed()) || osPlatform() === 'linux') return false; + log.info('Docker is down but a task needs it — starting it'); + try { + await launchDaemon(); + } catch { + return false; // Docker.app not found / launch failed + } + for (let i = 0; i < START_POLL_ATTEMPTS; i++) { + await sleep(START_POLL_INTERVAL); + if (await reachable()) { + markOwned(); // durable: survives a runtime restart + log.info('Docker started by iClaw (will auto-stop when idle + empty)'); + return true; + } + } + return false; + })().finally(() => { + ensureInFlight = null; + }); + } + return ensureInFlight; +} + +/** Reset the idle-stop countdown — call whenever a container is used. */ +export function markDockerUse(): void { + touchOwned(); +} + +/** Number of running containers, or -1 if the probe failed. */ +async function runningContainerCount(): Promise<number> { + try { + const { stdout } = await execFileAsync('docker', ['ps', '-q'], { timeout: PROBE_TIMEOUT }); + return stdout.split('\n').filter((s) => s.trim()).length; + } catch { + return -1; + } +} + +async function quitDocker(): Promise<void> { + const plat = osPlatform(); + try { + if (plat === 'darwin') { + await execFileAsync('osascript', ['-e', 'quit app "Docker"'], { timeout: 15_000 }); + } else if (plat === 'win32') { + await execFileAsync('taskkill', ['/IM', 'Docker Desktop.exe', '/F'], { timeout: 15_000 }); + } + } catch (err) { + log.warn('Stopping Docker failed', { error: err instanceof Error ? err.message : String(err) }); + } +} + +/** + * Background loop: stop Docker once it's been idle past IDLE_STOP_MS AND has no + * running containers AND iClaw owns it (the durable marker — set when WE started + * it, surviving restarts). The container check is the key safety net: if the + * user (or one of our warm sandboxes) still has a container up, we leave the + * daemon alone. + */ +export function startDockerIdleReaper(): void { + setInterval(async () => { + const since = ownedSince(); + if (since == null) return; // not started by iClaw → never touch + if (Date.now() - since < IDLE_STOP_MS) return; + const n = await runningContainerCount(); + if (n !== 0) return; // containers still running (or probe failed) → don't touch + log.info('Docker idle + no containers — stopping the daemon iClaw started'); + await quitDocker(); + clearOwned(); + }, REAPER_INTERVAL).unref(); +} + +let shutdownStopDone = false; + +/** + * On graceful shutdown, stop a Docker daemon WE started if it's container-free — + * so quitting iClaw doesn't leave our daemon running. Skipped (fast exit, no + * churn) when we don't own it, or when any container is up (a warm sandbox from + * a task that just ran, or the user's own work): in that case the durable marker + * is left behind so the next runtime can stop it later. Hard-capped so a hung + * Docker can't block the exit. + */ +export async function stopDockerOnShutdown(maxMs = 8_000): Promise<void> { + if (shutdownStopDone) return; + shutdownStopDone = true; + if (ownedSince() == null) return; // not ours → leave it + await Promise.race([ + (async () => { + const n = await runningContainerCount(); + if (n !== 0) return; // containers up → don't kill work; marker persists + log.info('Shutting down — stopping the idle Docker iClaw started'); + await quitDocker(); + clearOwned(); + })(), + sleep(maxMs), + ]); +} + +/** Test/inspection helper. */ +export function _state(): { owned: boolean; ownedSince: number | null } { + return { owned: ownedSince() != null, ownedSince: ownedSince() }; +} diff --git a/packages/iclaw-runtime/src/index.ts b/packages/iclaw-runtime/src/index.ts new file mode 100644 index 0000000..f20721f --- /dev/null +++ b/packages/iclaw-runtime/src/index.ts @@ -0,0 +1,230 @@ +/** + * iClaw Runtime — HTTP server for the Work / Safe work / Incognito modes. + * + * Model-agnostic agent loop runs HERE on the host (via OpenRouter); tool/shell + * execution is isolated in a per-turn Docker sandbox (secure-runner.ts / + * work-container.ts). No SQLite — sessions live in memory + the secure-mode + * workspace dir under ~/.iclaw/secure. + * + * API: + * POST /sessions → { sessionId } + * POST /sessions/:id/messages → 202 + * POST /sessions/:id/abort → 200 (stop in-flight turn, keep session) + * GET /sessions/:id/events → SSE stream + * DELETE /sessions/:id → 200 + * GET /health → 200 + */ +import http from 'node:http'; + +import { createSession, getSession, deleteSession, abortSession, attachSseClient, detachSseClient, sendMessage, getSessionInfo, exportSessionWorkspace, applySessionChanges, sweepExpiredSessions, startContainerReaper, loadPersistedSessions, type RuntimeAttachment } from './sessions.js'; +import { killOrphanContainers } from './secure-runner.js'; +import { startDockerIdleReaper, stopDockerOnShutdown } from './docker-lifecycle.js'; +import { killOrphanWorkContainers } from './work-container.js'; + +const PORT = parseInt(process.env.ICLAW_RUNTIME_PORT || '7430', 10); +const SECRET = process.env.ICLAW_RUNTIME_SECRET || ''; +const API_KEY = process.env.ICLAW_OPENROUTER_API_KEY || ''; +// Default agent model. Override per-install via ICLAW_MODEL in .env. +const DEFAULT_MODEL = process.env.ICLAW_MODEL || 'deepseek/deepseek-v4-flash'; + +function authOk(req: http.IncomingMessage): boolean { + if (!SECRET) { + const addr = req.socket.remoteAddress ?? ''; + return addr === '127.0.0.1' || addr === '::1' || addr === '::ffff:127.0.0.1'; + } + return req.headers['x-iclaw-token'] === SECRET; +} + +function json(res: http.ServerResponse, status: number, body: unknown): void { + const payload = JSON.stringify(body); + res.writeHead(status, { 'Content-Type': 'application/json' }); + res.end(payload); +} + +function readBody(req: http.IncomingMessage): Promise<unknown> { + return new Promise((resolve, reject) => { + let raw = ''; + req.on('data', (c) => { raw += c; }); + req.on('end', () => { + try { resolve(JSON.parse(raw || '{}')); } + catch { resolve({}); } + }); + req.on('error', reject); + }); +} + +const server = http.createServer(async (req, res) => { + if (!authOk(req)) return json(res, 401, { error: 'unauthorized' }); + + const url = new URL(req.url ?? '/', `http://localhost`); + const parts = url.pathname.split('/').filter(Boolean); + + // GET /health + if (req.method === 'GET' && parts[0] === 'health') { + return json(res, 200, { ok: true }); + } + + // GET /sessions/:id/info + if (req.method === 'GET' && parts[0] === 'sessions' && parts[2] === 'info') { + const info = getSessionInfo(parts[1]); + if (!info) return json(res, 404, { error: 'session not found' }); + return json(res, 200, info); + } + + // POST /sessions + if (req.method === 'POST' && parts[0] === 'sessions' && parts.length === 1) { + const body = await readBody(req) as { allowedFolders?: string[]; folderAccess?: { path: string; readonly: boolean }[]; copyFolders?: string[]; model?: string; secure?: boolean; incognito?: boolean; systemPrompt?: string; key?: string; history?: { role: string; content: string }[] }; + // folderAccess (when present) is the source of truth for per-folder read/ + // write; derive allowedFolders paths from it so the two never drift. + const folderAccess = Array.isArray(body.folderAccess) + ? body.folderAccess + .filter((f) => f && typeof f.path === 'string' && f.path) + .map((f) => ({ path: f.path, readonly: f.readonly !== false })) + : undefined; + const allowedFolders = folderAccess + ? folderAccess.map((f) => f.path) + : (body.allowedFolders ?? []); + const copyFolders = Array.isArray(body.copyFolders) + ? body.copyFolders.filter((p) => typeof p === 'string' && p) + : undefined; + const sessionId = createSession({ + allowedFolders, + folderAccess, + copyFolders, + model: body.model ?? DEFAULT_MODEL, + apiKey: API_KEY, + secure: body.secure ?? false, + incognito: body.incognito ?? false, + systemPrompt: body.systemPrompt, + key: body.key, + history: body.history, + }); + return json(res, 201, { sessionId }); + } + + // POST /sessions/:id/messages + if (req.method === 'POST' && parts[0] === 'sessions' && parts[2] === 'messages') { + const sessionId = parts[1]; + if (!getSession(sessionId)) return json(res, 404, { error: 'session not found' }); + const body = await readBody(req) as { content?: string; networkEnabled?: boolean; ttlDays?: number; attachments?: RuntimeAttachment[]; copyFolders?: string[] }; + if (!body.content?.trim()) return json(res, 400, { error: 'content required' }); + const attachments = Array.isArray(body.attachments) + ? body.attachments.filter( + (a): a is RuntimeAttachment => + !!a && typeof a.path === 'string' && !!a.path && + typeof a.mimeType === 'string' && typeof a.fileName === 'string', + ) + : undefined; + const copyFolders = Array.isArray(body.copyFolders) + ? body.copyFolders.filter((p) => typeof p === 'string' && p) + : undefined; + sendMessage(sessionId, body.content, body.networkEnabled, body.ttlDays, attachments, copyFolders).catch(console.error); + return json(res, 202, { queued: true }); + } + + // POST /sessions/:id/abort — stop the in-flight turn (keeps the session). + if (req.method === 'POST' && parts[0] === 'sessions' && parts[2] === 'abort') { + const sessionId = parts[1]; + if (!getSession(sessionId)) return json(res, 404, { error: 'session not found' }); + return json(res, 200, { aborted: abortSession(sessionId) }); + } + + // GET /sessions/:id/events (SSE) + if (req.method === 'GET' && parts[0] === 'sessions' && parts[2] === 'events') { + const sessionId = parts[1]; + if (!getSession(sessionId)) return json(res, 404, { error: 'session not found' }); + res.writeHead(200, { + 'Content-Type': 'text/event-stream', + 'Cache-Control': 'no-cache', + 'Connection': 'keep-alive', + 'X-Accel-Buffering': 'no', + }); + res.write(': connected\n\n'); + attachSseClient(sessionId, res); + req.on('close', () => detachSseClient(sessionId)); + return; + } + + // POST /sessions/:id/export — copy the Safe sandbox to a host folder. + if (req.method === 'POST' && parts[0] === 'sessions' && parts[2] === 'export') { + if (!getSession(parts[1])) return json(res, 404, { error: 'session not found' }); + const body = await readBody(req) as { destDir?: string }; + const result = exportSessionWorkspace(parts[1], typeof body.destDir === 'string' ? body.destDir : undefined); + if (!result) return json(res, 400, { error: 'not a Safe session' }); + return json(res, 200, result); + } + + // POST /sessions/:id/apply — copy the sandbox's changes back to the originals. + if (req.method === 'POST' && parts[0] === 'sessions' && parts[2] === 'apply') { + if (!getSession(parts[1])) return json(res, 404, { error: 'session not found' }); + const results = applySessionChanges(parts[1]); + if (!results) return json(res, 400, { error: 'not a Safe session' }); + return json(res, 200, { results }); + } + + // DELETE /sessions/:id + if (req.method === 'DELETE' && parts[0] === 'sessions' && parts.length === 2) { + deleteSession(parts[1]); + return json(res, 200, { stopped: true }); + } + + json(res, 404, { error: 'not found' }); +}); + +server.on('error', (err: NodeJS.ErrnoException) => { + if (err.code === 'EADDRINUSE') { + console.error(`[iclaw-runtime] port ${PORT} already in use — waiting 3s and retrying`); + setTimeout(() => server.listen(PORT, '127.0.0.1'), 3000); + } else { + console.error('[iclaw-runtime] server error', err); + process.exit(1); + } +}); + +server.listen(PORT, '127.0.0.1', () => { + console.error(`[iclaw-runtime] listening on port ${PORT}, model=${DEFAULT_MODEL}`); +}); + +// Periodic cleanup of expired sessions (every hour) +setInterval(() => { + const removed = sweepExpiredSessions(); + if (removed > 0) console.error(`[iclaw-runtime] swept ${removed} expired session(s)`); +}, 3600_000).unref(); + +// Reap idle Secure-Mode sandbox containers (keeps RAM in check across chats). +startContainerReaper(30_000); + +// Stop a Docker daemon WE started once it's idle and container-free (never +// touches a daemon the user started, nor one with running containers). +startDockerIdleReaper(); + +// Restore persisted Secure sessions so workspaces + TTL survive restarts +// (expired ones are deleted), then kill stray containers from the old process. +const { restored, expired } = loadPersistedSessions(); +if (restored > 0 || expired > 0) { + console.error(`[iclaw-runtime] restored ${restored} secure session(s), deleted ${expired} expired`); +} +killOrphanContainers() + .then((n) => { if (n > 0) console.error(`[iclaw-runtime] killed ${n} orphan container(s)`); }) + .catch((err) => console.error('[iclaw-runtime] container cleanup failed', err)); +killOrphanWorkContainers() + .then((n) => { if (n > 0) console.error(`[iclaw-runtime] killed ${n} orphan work container(s)`); }) + .catch((err) => console.error('[iclaw-runtime] work container cleanup failed', err)); + +let shuttingDown = false; +async function shutdown(): Promise<void> { + if (shuttingDown) return; + shuttingDown = true; + // Stop a Docker daemon WE started if it's idle/container-free (capped so a + // hung Docker can't block the exit). No-op when we don't own it or a + // container is still up — the durable marker is left for the next runtime. + try { + await stopDockerOnShutdown(); + } catch { + /* best-effort */ + } + server.close(); + process.exit(0); +} +process.on('SIGTERM', shutdown); +process.on('SIGINT', shutdown); diff --git a/packages/iclaw-runtime/src/install-id.ts b/packages/iclaw-runtime/src/install-id.ts new file mode 100644 index 0000000..adffc2a --- /dev/null +++ b/packages/iclaw-runtime/src/install-id.ts @@ -0,0 +1,26 @@ +/** + * Per-install identity for container labelling. + * + * Every sandbox container is stamped with `--label iclaw-install=<slug>` so the + * orphan reapers only ever touch THIS install's containers — two iClaw installs + * (different checkouts) on one machine can't kill each other's sandboxes. + * + * The slug is sha1 of this install's absolute path (like NanoClaw's + * install-slug): deterministic across restarts, unique across checkouts. + * Overridable via ICLAW_INSTALL_ID for unusual setups. + */ +import { createHash } from 'node:crypto'; +import { dirname, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +function computeSlug(): string { + const override = process.env.ICLAW_INSTALL_ID?.trim(); + if (override) return override.replace(/[^A-Za-z0-9_.-]/g, '').slice(0, 32) || 'iclaw'; + // This file lives at <install>/packages/iclaw-runtime/(src|dist)/install-id.* + const here = dirname(fileURLToPath(import.meta.url)); + const installRoot = resolve(here, '..'); + return createHash('sha1').update(installRoot).digest('hex').slice(0, 8); +} + +/** `iclaw-install=<slug>` — the value passed to `docker run --label` / `ps --filter`. */ +export const INSTALL_LABEL = `iclaw-install=${computeSlug()}`; diff --git a/packages/iclaw-runtime/src/log.ts b/packages/iclaw-runtime/src/log.ts new file mode 100644 index 0000000..d1e820c --- /dev/null +++ b/packages/iclaw-runtime/src/log.ts @@ -0,0 +1,64 @@ +const LEVELS = { debug: 20, info: 30, warn: 40, error: 50, fatal: 60 } as const; +type Level = keyof typeof LEVELS; + +const COLORS: Record<Level, string> = { + debug: '\x1b[34m', + info: '\x1b[32m', + warn: '\x1b[33m', + error: '\x1b[31m', + fatal: '\x1b[41m\x1b[37m', +}; +const KEY_COLOR = '\x1b[35m'; +const MSG_COLOR = '\x1b[36m'; +const RESET = '\x1b[39m'; +const FULL_RESET = '\x1b[0m'; + +const threshold = LEVELS[(process.env.LOG_LEVEL as Level) || 'info'] ?? LEVELS.info; + +function formatErr(err: unknown): string { + if (err instanceof Error) { + return `{ type: "${err.constructor.name}", message: "${err.message}", stack: ${err.stack} }`; + } + return JSON.stringify(err); +} + +function formatData(data: Record<string, unknown>): string { + const parts: string[] = []; + for (const [k, v] of Object.entries(data)) { + parts.push(`${KEY_COLOR}${k}${RESET}=${k === 'err' ? formatErr(v) : JSON.stringify(v)}`); + } + return parts.length ? ' ' + parts.join(' ') : ''; +} + +function ts(): string { + const d = new Date(); + const hh = String(d.getHours()).padStart(2, '0'); + const mm = String(d.getMinutes()).padStart(2, '0'); + const ss = String(d.getSeconds()).padStart(2, '0'); + const ms = String(d.getMilliseconds()).padStart(3, '0'); + return `${hh}:${mm}:${ss}.${ms}`; +} + +function emit(level: Level, msg: string, data?: Record<string, unknown>): void { + if (LEVELS[level] < threshold) return; + const tag = `${COLORS[level]}${level.toUpperCase()}${level === 'fatal' ? FULL_RESET : RESET}`; + const stream = LEVELS[level] >= LEVELS.warn ? process.stderr : process.stdout; + stream.write(`[${ts()}] ${tag} ${MSG_COLOR}${msg}${RESET}${data ? formatData(data) : ''}\n`); +} + +export const log = { + debug: (msg: string, data?: Record<string, unknown>) => emit('debug', msg, data), + info: (msg: string, data?: Record<string, unknown>) => emit('info', msg, data), + warn: (msg: string, data?: Record<string, unknown>) => emit('warn', msg, data), + error: (msg: string, data?: Record<string, unknown>) => emit('error', msg, data), + fatal: (msg: string, data?: Record<string, unknown>) => emit('fatal', msg, data), +}; + +process.on('uncaughtException', (err) => { + log.fatal('Uncaught exception', { err }); + process.exit(1); +}); + +process.on('unhandledRejection', (reason) => { + log.error('Unhandled rejection', { err: reason }); +}); diff --git a/packages/iclaw-runtime/src/secure-export.ts b/packages/iclaw-runtime/src/secure-export.ts new file mode 100644 index 0000000..5bb9773 --- /dev/null +++ b/packages/iclaw-runtime/src/secure-export.ts @@ -0,0 +1,198 @@ +/** + * Safe Mode export / apply. + * + * After working in the sandbox the user may want the results OUT of it: + * export → copy the whole sandbox workspace to a host folder (a fresh place; + * nothing of theirs is overwritten). Good for "give me the output". + * apply → for each folder that was copied IN (see secure-ingest's + * .iclaw-ingest.json), copy the new/changed files back to the + * ORIGINAL folder. Additive + overwrite only — never deletes the + * user's files — and refuses secret roots. This is the explicit + * "apply the sandbox's changes to my real project" action. + * + * Both are user-initiated (button + confirm); apply is the only path that ever + * writes to the user's real files, and only to folders they themselves chose to + * ingest. + */ +import { + cpSync, mkdirSync, readFileSync, writeFileSync, existsSync, statSync, readdirSync, +} from 'node:fs'; +import { homedir } from 'node:os'; +import { join, relative, sep, dirname } from 'node:path'; + +import { validateMountRoot } from './agent/security.js'; +import { log } from './log.js'; + +/** Workspace internals that must never be exported/applied. */ +const INTERNAL = new Set(['.tools', '.iclaw-ingest.json']); +const SKIP_DIRS = new Set(['node_modules', '.git']); + +export interface ExportResult { + ok: boolean; + /** Host path the workspace was copied to (on success). */ + path?: string; + files?: number; + error?: string; +} + +export interface ApplyFileChange { + /** Original-folder-relative path that was written. */ + path: string; + kind: 'created' | 'modified'; +} + +export interface ApplyResult { + ok: boolean; + /** Original host folder the changes were applied to. */ + source?: string; + applied?: ApplyFileChange[]; + error?: string; +} + +interface IngestLogEntry { + kind: string; + source: string; + ok: boolean; + target?: string; +} + +/** Bounded recursive file count. */ +function countFiles(dir: string, cap = 50_000): number { + let total = 0; + const stack = [dir]; + while (stack.length && total < cap) { + const cur = stack.pop()!; + let entries; + try { entries = readdirSync(cur, { withFileTypes: true }); } + catch { continue; } + for (const e of entries) { + if (e.isDirectory()) stack.push(join(cur, e.name)); + else total++; + } + } + return total; +} + +/** Copy the whole sandbox (minus workspace internals) to a fresh host folder. */ +export function exportWorkspace(workspaceDir: string, destDir?: string): ExportResult { + try { + const base = destDir && destDir.trim() ? destDir : join(homedir(), 'Downloads'); + mkdirSync(base, { recursive: true }); + const target = join(base, `iclaw-sandbox-${Date.now()}`); + cpSync(workspaceDir, target, { + recursive: true, + filter: (src) => { + const rel = relative(workspaceDir, src); + if (!rel) return true; // the root itself + const top = rel.split(sep)[0]; + return !INTERNAL.has(top); + }, + }); + return { ok: true, path: target, files: countFiles(target) }; + } catch (err) { + return { ok: false, error: err instanceof Error ? err.message : String(err) }; + } +} + +function readIngestLog(workspaceDir: string): IngestLogEntry[] { + try { + const raw = JSON.parse(readFileSync(join(workspaceDir, '.iclaw-ingest.json'), 'utf-8')); + return Array.isArray(raw?.results) ? (raw.results as IngestLogEntry[]) : []; + } catch { + return []; + } +} + +/** True when two files differ by size or content (cheap size check first). */ +function differs(a: string, b: string): boolean { + try { + const sa = statSync(a); + const sb = statSync(b); + if (sa.size !== sb.size) return true; + return !readFileSync(a).equals(readFileSync(b)); + } catch { + return true; // missing target / unreadable → treat as changed + } +} + +/** + * Copy new/changed files from `sandboxRoot` back into `originalRoot`. Additive + + * overwrite only (never deletes); skips node_modules/.git. Returns what changed. + */ +function syncBack(sandboxRoot: string, originalRoot: string): ApplyFileChange[] { + const applied: ApplyFileChange[] = []; + const stack = ['']; + while (stack.length) { + const rel = stack.pop()!; + const cur = rel ? join(sandboxRoot, rel) : sandboxRoot; + let entries; + try { entries = readdirSync(cur, { withFileTypes: true }); } + catch { continue; } + for (const e of entries) { + const childRel = rel ? join(rel, e.name) : e.name; + if (e.isDirectory()) { + if (SKIP_DIRS.has(e.name)) continue; + stack.push(childRel); + continue; + } + const from = join(sandboxRoot, childRel); + const to = join(originalRoot, childRel); + const existed = existsSync(to); + if (existed && !differs(from, to)) continue; // unchanged + mkdirSync(dirname(to), { recursive: true }); + cpSync(from, to); + applied.push({ path: childRel, kind: existed ? 'modified' : 'created' }); + } + } + return applied; +} + +/** + * Apply the sandbox's changes back to the original folders that were ingested. + * Only `folder` sources have an original to write to (repo/zip/url don't). Each + * original is re-validated (refuses secret roots) before any write. + */ +export function applyChanges(workspaceDir: string): ApplyResult[] { + const entries = readIngestLog(workspaceDir).filter( + (e) => e.ok && e.kind === 'folder' && e.target, + ); + const out: ApplyResult[] = []; + for (const e of entries) { + try { + // Re-validate the destination root (refuses secret-bearing roots, resolves + // symlinks) — the same gate ingest used, now guarding the write-back. + const originalRoot = validateMountRoot(e.source); + const sandboxRoot = join(workspaceDir, e.target!); + if (!existsSync(sandboxRoot)) { + out.push({ ok: false, source: e.source, error: 'sandbox copy missing' }); + continue; + } + const applied = syncBack(sandboxRoot, originalRoot); + out.push({ ok: true, source: originalRoot, applied }); + } catch (err) { + const error = err instanceof Error ? err.message : String(err); + log.warn('Safe-mode apply failed', { source: e.source, error }); + out.push({ ok: false, source: e.source, error }); + } + } + return out; +} + +/** One-line summaries for the UI. */ +export function describeExport(r: ExportResult): string { + return r.ok ? `Exported ${r.files ?? '?'} files to ${r.path}` : `Export failed: ${r.error}`; +} + +export function describeApply(results: ApplyResult[]): string { + if (results.length === 0) return 'Nothing to apply — no folders were copied into this sandbox.'; + const lines: string[] = []; + for (const r of results) { + if (!r.ok) { + lines.push(`✗ ${r.source}: ${r.error}`); + continue; + } + const n = r.applied?.length ?? 0; + lines.push(n === 0 ? `• ${r.source}: no changes` : `• ${r.source}: applied ${n} file(s)`); + } + return lines.join('\n'); +} diff --git a/packages/iclaw-runtime/src/secure-ingest.ts b/packages/iclaw-runtime/src/secure-ingest.ts new file mode 100644 index 0000000..ec14ddf --- /dev/null +++ b/packages/iclaw-runtime/src/secure-ingest.ts @@ -0,0 +1,202 @@ +/** + * Safe Mode source ingest. + * + * Safe Mode = "I don't trust this thing. Work on a COPY in a sandbox." So before + * the agent touches anything, the untrusted source is brought INTO the isolated + * sandbox workspace (~/.iclaw/secure/<id>/), and the container only ever mounts + * that workspace. The user's original files/repos are never modified — we copy, + * clone, extract, or download into the sandbox; the originals stay put. + * + * Supported sources: + * folder → recursive copy of a local directory (secret roots refused) + * repo → shallow `git clone` of a remote http(s)/git URL + * zip → extract a local .zip into the workspace + * url → download a single artifact into the workspace + * + * Each ingest lands in its own subdirectory so multiple sources don't collide, + * and a `.iclaw-ingest.json` log is written for the report / audit trail. + */ +import { execFile } from 'node:child_process'; +import { + cpSync, mkdirSync, statSync, writeFileSync, readdirSync, existsSync, realpathSync, +} from 'node:fs'; +import { basename, join } from 'node:path'; +import { promisify } from 'node:util'; + +import { validateMountRoot } from './agent/security.js'; +import { log } from './log.js'; + +const execFileAsync = promisify(execFile); + +const CLONE_TIMEOUT = Number(process.env.ICLAW_INGEST_CLONE_TIMEOUT) || 180_000; +const DOWNLOAD_TIMEOUT = Number(process.env.ICLAW_INGEST_DOWNLOAD_TIMEOUT) || 120_000; +/** Cap the file count we report (a cheap, bounded walk — not a full inventory). */ +const COUNT_CAP = 50_000; + +export type IngestSource = + | { kind: 'folder'; path: string } + | { kind: 'repo'; url: string } + | { kind: 'zip'; path: string } + | { kind: 'url'; url: string }; + +export interface IngestResult { + kind: IngestSource['kind']; + /** Human description of the source (path or URL). */ + source: string; + ok: boolean; + /** Workspace-relative subdir the source landed in (on success). */ + target?: string; + /** Approximate file count copied/cloned/extracted. */ + files?: number; + /** Failure reason (on error). */ + error?: string; +} + +function sourceLabel(s: IngestSource): string { + return s.kind === 'repo' || s.kind === 'url' ? s.url : s.path; +} + +/** A non-colliding subdir name inside the workspace for `base`. */ +function uniqueTarget(workspaceDir: string, base: string): string { + const safe = base.replace(/[^A-Za-z0-9._-]/g, '_').replace(/^\.+/, '') || 'source'; + let name = safe; + let n = 1; + while (existsSync(join(workspaceDir, name))) name = `${safe}-${n++}`; + return name; +} + +/** Bounded recursive file count — never walks more than COUNT_CAP entries. */ +function countFiles(dir: string): number { + let total = 0; + const stack = [dir]; + while (stack.length && total < COUNT_CAP) { + const cur = stack.pop()!; + let entries; + try { entries = readdirSync(cur, { withFileTypes: true }); } + catch { continue; } + for (const e of entries) { + if (e.isDirectory()) stack.push(join(cur, e.name)); + else total++; + if (total >= COUNT_CAP) break; + } + } + return total; +} + +/** Derive a sane subdir name from a clone/download URL. */ +function nameFromUrl(url: string): string { + try { + const u = new URL(url); + const last = u.pathname.split('/').filter(Boolean).pop() || u.hostname; + return last.replace(/\.git$/i, ''); + } catch { + return 'source'; + } +} + +async function ingestFolder(workspaceDir: string, path: string): Promise<IngestResult> { + // validateMountRoot resolves symlinks AND refuses secret-bearing roots + // (~/.ssh, .env, credentials, …) — same gate Work Mode uses for mounts. + const real = validateMountRoot(path); + if (!statSync(real).isDirectory()) throw new Error('not a folder'); + const target = uniqueTarget(workspaceDir, basename(real)); + const dest = join(workspaceDir, target); + // Copy a COPY — the original is never touched. Skip the VCS/dep noise that + // bloats the sandbox without adding signal. + cpSync(real, dest, { + recursive: true, + filter: (src) => !/(^|[\\/])(node_modules|\.git)([\\/]|$)/.test(src.slice(real.length)), + }); + return { kind: 'folder', source: path, ok: true, target, files: countFiles(dest) }; +} + +async function ingestRepo(workspaceDir: string, url: string): Promise<IngestResult> { + if (!/^(https?|git):\/\//i.test(url)) { + throw new Error('only http(s):// or git:// repo URLs are allowed'); + } + const target = uniqueTarget(workspaceDir, nameFromUrl(url)); + const dest = join(workspaceDir, target); + // --depth 1, no checkout of submodules — we want a working copy, fast. git + // clone does not execute repo code, so cloning untrusted repos on the host is + // safe; the agent only ever RUNS the code later inside the sandbox. + await execFileAsync('git', ['clone', '--depth', '1', '--', url, dest], { timeout: CLONE_TIMEOUT }); + return { kind: 'repo', source: url, ok: true, target, files: countFiles(dest) }; +} + +async function ingestZip(workspaceDir: string, path: string): Promise<IngestResult> { + const real = realpathSync(path); + if (!statSync(real).isFile()) throw new Error('not a file'); + const target = uniqueTarget(workspaceDir, basename(real).replace(/\.zip$/i, '')); + const dest = join(workspaceDir, target); + mkdirSync(dest, { recursive: true }); + // `unzip` is present on macOS and most Linux; surfaces a clear error if not. + await execFileAsync('unzip', ['-q', '-o', real, '-d', dest], { timeout: DOWNLOAD_TIMEOUT }); + return { kind: 'zip', source: path, ok: true, target, files: countFiles(dest) }; +} + +async function ingestUrl(workspaceDir: string, url: string): Promise<IngestResult> { + if (!/^https?:\/\//i.test(url)) throw new Error('only http(s):// URLs are allowed'); + const dir = join(workspaceDir, 'downloads'); + mkdirSync(dir, { recursive: true }); + const file = uniqueTarget(dir, nameFromUrl(url) || 'download'); + const res = await fetch(url, { signal: AbortSignal.timeout(DOWNLOAD_TIMEOUT) }); + if (!res.ok) throw new Error(`download failed (HTTP ${res.status})`); + const buf = Buffer.from(await res.arrayBuffer()); + writeFileSync(join(dir, file), buf); + return { kind: 'url', source: url, ok: true, target: `downloads/${file}`, files: 1 }; +} + +async function ingestOne(workspaceDir: string, s: IngestSource): Promise<IngestResult> { + switch (s.kind) { + case 'folder': return ingestFolder(workspaceDir, s.path); + case 'repo': return ingestRepo(workspaceDir, s.url); + case 'zip': return ingestZip(workspaceDir, s.path); + case 'url': return ingestUrl(workspaceDir, s.url); + } +} + +/** + * Ingest every source into the sandbox workspace, never throwing — a failed + * source becomes a `{ ok: false, error }` result so the others still land. Writes + * a `.iclaw-ingest.json` log into the workspace for the report / audit trail. + */ +export async function ingestSources( + workspaceDir: string, + sources: IngestSource[], +): Promise<IngestResult[]> { + const results: IngestResult[] = []; + for (const s of sources) { + try { + results.push(await ingestOne(workspaceDir, s)); + } catch (err) { + const error = err instanceof Error ? err.message : String(err); + log.warn('Safe-mode ingest failed', { source: sourceLabel(s), error }); + results.push({ kind: s.kind, source: sourceLabel(s), ok: false, error }); + } + } + try { + writeFileSync( + join(workspaceDir, '.iclaw-ingest.json'), + JSON.stringify({ at: new Date().toISOString(), results }, null, 2), + 'utf-8', + ); + } catch { + /* best-effort log */ + } + return results; +} + +/** One-line user-facing summary of an ingest pass (prepended to the turn). */ +export function describeIngest(results: IngestResult[]): string { + if (results.length === 0) return ''; + const ok = results.filter((r) => r.ok); + const failed = results.filter((r) => !r.ok); + const lines = ['[Safe sandbox] Working on a COPY — your original files are unchanged.']; + for (const r of ok) { + lines.push(` • copied ${r.source} → /workspace/${r.target} (${r.files ?? '?'} files)`); + } + for (const r of failed) { + lines.push(` • could not ingest ${r.source}: ${r.error}`); + } + return lines.join('\n'); +} diff --git a/packages/iclaw-runtime/src/secure-runner.ts b/packages/iclaw-runtime/src/secure-runner.ts new file mode 100644 index 0000000..c9baaf5 --- /dev/null +++ b/packages/iclaw-runtime/src/secure-runner.ts @@ -0,0 +1,471 @@ +/** + * Secure Mode runner. + * + * Agent loop runs on the HOST (can reach OpenRouter). + * Tool execution happens INSIDE a Docker container (isolated). + * + * Per-turn container lifecycle: + * start container → execute turn → stop container → (keep workspace dir) + * Next turn: start NEW container with SAME workspace volume. + * This allows changing network settings per message while preserving files. + */ +import { execFile } from 'node:child_process'; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync, readFileSync, existsSync, chmodSync, readdirSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join, basename } from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { promisify } from 'node:util'; + +import type OpenAI from 'openai'; +import type { AgentEvent, AgentOptions, Message } from './agent/loop.js'; +import type { SavingsNote } from './agent/tools.js'; +import { shrinkOldToolOutputs, withPromptCaching, makeToolGuard, HOST_INSTALL_POLICY } from './agent/loop.js'; +import { INSTALL_LABEL } from './install-id.js'; +import { dumpPrompt, newTurnId } from './agent/prompt-dump.js'; + +const execFileAsync = promisify(execFile); + +// Slim secure sandbox image: a small CLI toolset, no browser (build-secure.sh). +// The agent reaches the web via `curl` and can self-install more tools into +// /workspace/.tools (no root). +const CONTAINER_IMAGE = process.env.ICLAW_SECURE_IMAGE || 'iclaw-secure:latest'; +const CONTAINER_TIMEOUT = 30_000; +/** Max tool-call rounds per turn (env-tunable for long multi-step tasks). */ +const MAX_ROUNDS = Math.max(1, Number(process.env.ICLAW_MAX_ROUNDS) || 40); + +export type SecureEvent = AgentEvent; + +const SECURE_PREFIX = 'iclaw-secure-'; + +// Persistent root for Secure workspaces. NOT tmpdir(): macOS prunes /var/folders +// after a few idle days, which is shorter than our 7-day TTL. A stable dir means +// workspaces (and their TTL) genuinely survive runtime restarts. +const SECURE_DATA_DIR = process.env.ICLAW_SECURE_DATA_DIR || join(homedir(), '.iclaw', 'secure'); + +/** Metadata persisted alongside a workspace so sessions survive restarts. */ +export interface SessionMeta { + key?: string; // stable identity (e.g. "chat:156") for reconnection + lastActivity: number; // ms; TTL counts from here + ttlMs: number; // 0 = never + secure: boolean; + model?: string; +} + +function metaPathFor(workspaceDir: string): string { + return `${workspaceDir}.meta.json`; +} + +/** + * Create a persistent workspace directory. Lives under SECURE_DATA_DIR so it + * survives restarts (reaped only by TTL). + */ +export function createSecureWorkspace(): string { + mkdirSync(SECURE_DATA_DIR, { recursive: true }); + const dir = mkdtempSync(join(SECURE_DATA_DIR, `${SECURE_PREFIX}`)); + // The sandbox runs as the non-root `node` user; make the bind-mounted dir + // writable to it (mkdtemp defaults to 0700 owned by the host user). + try { chmodSync(dir, 0o777); } catch {} + // Pre-create the runtime tool dirs the container puts on PATH (.tools/bin for + // downloaded static binaries, .tools/npm for `npm i -g`). They live inside the + // workspace, so self-installed tools persist across container restarts and are + // auto-deleted with the workspace when its TTL expires. World-writable so the + // non-root container user can install into them. + for (const sub of ['.tools', '.tools/bin', '.tools/npm']) { + const p = join(dir, sub); + try { mkdirSync(p, { recursive: true }); chmodSync(p, 0o777); } catch {} + } + return dir; +} + +/** Destroy a workspace directory and its metadata sidecar. */ +export function destroySecureWorkspace(workspaceDir: string): void { + try { rmSync(workspaceDir, { recursive: true, force: true }); } catch {} + try { rmSync(metaPathFor(workspaceDir), { force: true }); } catch {} +} + +/** Persist session metadata next to (not inside) the workspace. */ +export function writeSessionMeta(workspaceDir: string, meta: SessionMeta): void { + try { writeFileSync(metaPathFor(workspaceDir), JSON.stringify(meta), 'utf-8'); } catch {} +} + +/** List every persisted workspace dir with its metadata (for startup reload). */ +export function listPersistedWorkspaces(): { dir: string; meta: SessionMeta | null }[] { + const out: { dir: string; meta: SessionMeta | null }[] = []; + let entries; + try { entries = readdirSync(SECURE_DATA_DIR, { withFileTypes: true }); } + catch { return out; } + for (const entry of entries) { + if (!entry.isDirectory() || !entry.name.startsWith(SECURE_PREFIX)) continue; + const dir = join(SECURE_DATA_DIR, entry.name); + let meta: SessionMeta | null = null; + try { meta = JSON.parse(readFileSync(metaPathFor(dir), 'utf-8')) as SessionMeta; } catch {} + out.push({ dir, meta }); + } + return out; +} + +/** + * Kill containers left running by a previous runtime process. Containers are + * pure runtime (no data), so this is always safe — unlike workspace dirs, which + * carry the user's files and are reaped only by TTL (see sessions.ts reload). + */ +export async function killOrphanContainers(): Promise<number> { + try { + // Scope to THIS install's Secure containers (name AND install label) so a + // second iClaw install can't reap ours. + const { stdout } = await execFileAsync( + 'docker', ['ps', '-aq', '--filter', `name=${SECURE_PREFIX}`, '--filter', `label=${INSTALL_LABEL}`], + { timeout: 10_000 }, + ); + const ids = stdout.split('\n').map((s) => s.trim()).filter(Boolean); + if (ids.length === 0) return 0; + await execFileAsync('docker', ['rm', '-f', ...ids], { timeout: 30_000 }).catch(() => {}); + return ids.length; + } catch { + return 0; // Docker not installed/running — nothing to clean. + } +} + +/** + * Start a container with the given workspace. Returns containerName. + * Fail-closed: if Docker can't start the sandbox we throw, so the caller + * aborts the turn instead of silently running file ops outside isolation. + * + * The container is long-lived (`sleep`) and reused across turns by the session + * manager — see sessions.ts. It is recreated only when the network setting + * changes or after an idle period. + */ +export async function startContainer(workspaceDir: string, networkEnabled: boolean): Promise<string> { + const containerName = `iclaw-secure-${randomUUID().slice(0, 8)}`; + + const networkArgs = networkEnabled ? [] : ['--network', 'none']; + + try { + // `-d` returns once the container is up (and pulls the image on first run), + // so we await the real result instead of guessing with a fixed sleep. + await execFileAsync('docker', [ + 'run', '--rm', '-d', + '--name', containerName, + '--label', INSTALL_LABEL, + ...networkArgs, + // No browser any more, so the default 64MB /dev/shm is fine and 512MB is + // plenty of headroom for shell/node tasks. Tunable via env. + '--memory', process.env.ICLAW_SECURE_MEMORY || '512m', + '--cpus', process.env.ICLAW_SECURE_CPUS || '1', + '-v', `${workspaceDir}:/workspace:rw`, + '--workdir', '/workspace', + CONTAINER_IMAGE, + 'sleep', '86400', + ], { timeout: 60_000 }); + } catch (err) { + const e = err as { stderr?: string; message?: string }; + throw new Error( + `Secure sandbox failed to start — is Docker running? ${(e.stderr || e.message || '').slice(0, 200)}`, + ); + } + return containerName; +} + +/** Stop a container (workspace is preserved on host). */ +export function stopContainer(containerName: string): void { + try { execFile('docker', ['rm', '-f', containerName], () => {}); } catch {} +} + +/** True if a container with this name is up. Used for warm-reuse liveness. */ +export async function isContainerRunning(containerName: string): Promise<boolean> { + try { + const { stdout } = await execFileAsync( + 'docker', ['inspect', '-f', '{{.State.Running}}', containerName], + { timeout: 5_000 }, + ); + return stdout.trim() === 'true'; + } catch { + return false; + } +} + +/** Execute a command inside the sandbox container. */ +async function execInContainer(containerName: string, command: string): Promise<string> { + try { + const { stdout, stderr } = await execFileAsync( + 'docker', ['exec', containerName, 'bash', '-c', command], + { timeout: CONTAINER_TIMEOUT }, + ); + return [stdout, stderr].filter(Boolean).join('\n').trim() || '(no output)'; + } catch (err: unknown) { + const e = err as { stdout?: string; stderr?: string; message?: string }; + return [e.stdout, e.stderr, e.message].filter(Boolean).join('\n').trim() || 'Error'; + } +} + +function readFromWorkspace(workspaceDir: string, filePath: string): string { + const safe = basename(filePath); + const full = join(workspaceDir, safe); + if (!existsSync(full)) return `File not found: ${safe}`; + return readFileSync(full, 'utf-8'); +} + +function writeToWorkspace(workspaceDir: string, filePath: string, content: string): string { + const safe = basename(filePath); + const full = join(workspaceDir, safe); + writeFileSync(full, content, 'utf-8'); + return `Written: /workspace/${safe}`; +} + +function editInWorkspace(workspaceDir: string, filePath: string, oldStr: string, newStr: string): string { + if (!oldStr) return 'edit_file requires old_string — the exact text to replace.'; + const safe = basename(filePath); + const full = join(workspaceDir, safe); + if (!existsSync(full)) return `File not found: ${safe}`; + const content = readFileSync(full, 'utf-8'); + const first = content.indexOf(oldStr); + if (first === -1) return 'old_string not found — copy the exact text (including whitespace) from the file.'; + if (content.indexOf(oldStr, first + oldStr.length) !== -1) { + return 'old_string is not unique — add more surrounding context so it matches one place.'; + } + writeFileSync(full, content.slice(0, first) + newStr + content.slice(first + oldStr.length), 'utf-8'); + return `Edited: /workspace/${safe}`; +} + +/** + * Run one turn in Secure Mode. + * + * The container is owned and reused by the session manager (warm reuse): it is + * passed in here, and NOT torn down at the end of the turn. Lifecycle (create + * on network change, reap on idle) lives in sessions.ts. + */ +export async function* runSecureTurn( + history: { role: string; content: string }[], + userMessage: string, + opts: { + apiKey: string; + model: string; + workspaceDir: string; + containerName: string; + networkEnabled?: boolean; + systemPrompt?: string; + signal?: AbortSignal; + /** Image data URLs for dropped files — shown to the model as vision blocks. */ + images?: string[]; + }, +): AsyncGenerator<SecureEvent> { + const networkEnabled = opts.networkEnabled ?? false; + + // The sandbox base prompt (identity + operating rules) is ALWAYS included; any + // caller-supplied prompt (e.g. project context) is APPENDED, not replaced — + // otherwise a project chat would lose the identity and the sandbox rules. + const secureBasePrompt = `You are iClaw, a private AI assistant${opts.model ? `, powered by ${opts.model}` : ''}. If asked what you are, say exactly that — never claim to be ChatGPT, Claude, Gemini or another product. +You are running in a secure isolated sandbox. This is the "work on a COPY" mode: the user does NOT trust the source, so everything happens inside /workspace and their real computer is never touched. +You can run commands and read/write files in /workspace only. +What's already here: any folders the user selected are COPIED into /workspace (originals untouched), and any files they dropped are saved here too. To examine an untrusted repo, archive, or URL, bring it in yourself — \`git clone <url>\`, \`unzip <file>\`, or \`curl -O <url>\` — into /workspace. Never expect or use host paths; only /workspace exists. +Work efficiently — each tool call is one step and steps are limited. Chain related +shell commands into a single run_command with && (e.g. \`git clone <url> repo && cd repo && pip install -r requirements.txt\`) instead of one command per step, and don't re-run exploratory commands you've already seen. +Preinstalled CLIs: git, rg (ripgrep), jq, curl, node, unzip/zip, less, tree. +You can install more tools yourself, no root needed${networkEnabled ? '' : ' (requires network, which is currently OFF — ask the user to enable it)'}: download a static binary into /workspace/.tools/bin (already on PATH) with curl, or run \`npm i -g <pkg>\`. Self-installed tools live in the workspace, so they persist across turns and are removed automatically when the workspace expires. +Network is ${networkEnabled ? 'enabled' : 'disabled'}.${ + networkEnabled + ? '\nTo fetch web pages or APIs, use run_command with `curl -s <url>` (there is no browser in this sandbox).' + + '\nFor a link\'s actual content, prefer the analyze_link tool (YouTube videos: subtitles/transcript). ' + + 'Use mode:"summary" with a short purpose by default to save tokens; mode:"full" only when you need exact wording.' + : '' + } +Be concise. +${HOST_INSTALL_POLICY}`; + const systemPrompt = opts.systemPrompt?.trim() + ? `${secureBasePrompt}\n\n${opts.systemPrompt.trim()}` + : secureBasePrompt; + + const gen = runSecureAgentLoop( + history.map((m) => ({ role: m.role as 'user' | 'assistant', content: m.content })), + userMessage, + { + apiKey: opts.apiKey, + model: opts.model, + allowedFolders: [opts.workspaceDir], + signal: opts.signal, + images: opts.images, + systemPrompt, + onWriteApproval: async () => true, + }, + opts.containerName, + opts.workspaceDir, + networkEnabled, + ); + + for await (const event of gen) { + yield event; + } +} + +async function* runSecureAgentLoop( + history: Message[], + userMessage: string, + opts: AgentOptions, + containerName: string, + workspaceDir: string, + networkEnabled: boolean, +): AsyncGenerator<SecureEvent> { + const OpenAI = (await import('openai')).default; + const { TOOL_DEFINITIONS, ANALYZE_LINK_TOOL, analyzeLink, clampMiddle, compressCommandOutput, TOOL_OUTPUT_MAX_CHARS } = + await import('./agent/tools.js'); + + // analyze_link is appended (not part of core TOOL_DEFINITIONS) and runs its + // network work INSIDE this container, so it respects the same network gate. + const tools = [...TOOL_DEFINITIONS, ANALYZE_LINK_TOOL]; + + // Buffer for analyze_link savings notes, flushed as `note` events per tool call. + const savingsNotes: SavingsNote[] = []; + + const client = new OpenAI({ + baseURL: 'https://openrouter.ai/api/v1', + apiKey: opts.apiKey, + }); + + // Dropped images → multimodal user message (text + vision blocks), seen once. + const userMsg: OpenAI.Chat.ChatCompletionUserMessageParam = opts.images?.length + ? { + role: 'user', + content: [ + { type: 'text', text: userMessage }, + ...opts.images.map((url) => ({ type: 'image_url' as const, image_url: { url } })), + ], + } + : { role: 'user', content: userMessage }; + + const messages: Message[] = [ + { role: 'system', content: opts.systemPrompt ?? 'You are a helpful assistant in a secure sandbox.' }, + ...history, + userMsg, + ]; + + let turnTokens = 0; + let turnCached = 0; + const dumpTurnId = newTurnId(); + const guard = makeToolGuard(); + for (let round = 0; round < MAX_ROUNDS; round++) { + // User pressed Stop between rounds → end cleanly (partial text already sent). + if (opts.signal?.aborted) { + yield { type: 'done', tokens: turnTokens || undefined, cached: turnCached || undefined }; + return; + } + let textBuffer = ''; + const toolCallBuffers: Record<string, { name: string; arguments: string }> = {}; + + dumpPrompt({ turnId: dumpTurnId, mode: 'secure', model: opts.model, round, messages, tools }); + + // eslint-disable-next-line @typescript-eslint/no-explicit-any + let stream: AsyncIterable<any>; + try { + stream = await client.chat.completions.create( + { + model: opts.model, + messages: withPromptCaching(messages), + // eslint-disable-next-line @typescript-eslint/no-explicit-any + tools: tools as any, + tool_choice: 'auto', + stream: true, + stream_options: { include_usage: true }, + }, + { signal: opts.signal }, + ); + } catch (err) { + if (opts.signal?.aborted) { + yield { type: 'done', tokens: turnTokens || undefined, cached: turnCached || undefined }; + return; + } + yield { type: 'error', message: err instanceof Error ? err.message : String(err) }; + return; + } + + let finishReason: string | null = null; + try { + for await (const chunk of stream) { + if (chunk.usage?.total_tokens) turnTokens += chunk.usage.total_tokens; + if (chunk.usage?.prompt_tokens_details?.cached_tokens) turnCached += chunk.usage.prompt_tokens_details.cached_tokens; + const choice = chunk.choices[0]; + if (!choice) continue; + finishReason = choice.finish_reason ?? finishReason; + const delta = choice.delta; + if (delta.content) { + textBuffer += delta.content; + yield { type: 'text', content: delta.content }; + } + if (delta.tool_calls) { + for (const tc of delta.tool_calls) { + const idx = String(tc.index ?? 0); + if (!toolCallBuffers[idx]) toolCallBuffers[idx] = { name: '', arguments: '' }; + if (tc.function?.name) toolCallBuffers[idx].name += tc.function.name; + if (tc.function?.arguments) toolCallBuffers[idx].arguments += tc.function.arguments; + } + } + } + } catch (err) { + // Stop pressed mid-stream → clean end; otherwise surface the error. + if (opts.signal?.aborted) { + yield { type: 'done', tokens: turnTokens || undefined, cached: turnCached || undefined }; + return; + } + yield { type: 'error', message: err instanceof Error ? err.message : String(err) }; + return; + } + + const toolCalls = Object.values(toolCallBuffers); + if (toolCalls.length === 0 || finishReason === 'stop') { + if (textBuffer) messages.push({ role: 'assistant', content: textBuffer }); + yield { type: 'done', tokens: turnTokens || undefined, cached: turnCached || undefined }; + return; + } + + messages.push({ + role: 'assistant', + content: textBuffer || null, + tool_calls: toolCalls.map((tc, i) => ({ + id: `call_${i}`, + type: 'function' as const, + function: { name: tc.name, arguments: tc.arguments }, + })), + } as Message); + + for (let i = 0; i < toolCalls.length; i++) { + const tc = toolCalls[i]; + let args: Record<string, unknown> = {}; + try { args = JSON.parse(tc.arguments); } catch {} + + yield { type: 'tool_start', name: tc.name, input: args }; + + let result: string; + const blocked = guard.check(tc.name, tc.arguments); + if (blocked) { + result = blocked; + } else if (tc.name === 'run_command') { + result = clampMiddle(compressCommandOutput(await execInContainer(containerName, String(args.command ?? ''))), TOOL_OUTPUT_MAX_CHARS); + } else if (tc.name === 'write_file') { + result = writeToWorkspace(workspaceDir, String(args.path ?? 'file.txt'), String(args.content ?? '')); + } else if (tc.name === 'edit_file') { + result = editInWorkspace(workspaceDir, String(args.path ?? ''), String(args.old_string ?? ''), String(args.new_string ?? '')); + } else if (tc.name === 'read_file') { + result = readFromWorkspace(workspaceDir, String(args.path ?? '')); + } else if (tc.name === 'list_files') { + result = await execInContainer(containerName, 'ls -la /workspace'); + } else if (tc.name === 'search_files') { + result = await execInContainer(containerName, `grep -r ${JSON.stringify(String(args.query ?? ''))} /workspace 2>/dev/null | head -20`); + } else if (tc.name === 'analyze_link') { + // Runs its fetch inside this same container (network honours the gate). + result = await analyzeLink(args, { + runInSandbox: (command) => execInContainer(containerName, command), + networkEnabled, + onNote: (note) => savingsNotes.push(note), + }); + } else { + result = `Tool not available in secure mode: ${tc.name}`; + } + + yield { type: 'tool_result', name: tc.name, result }; + while (savingsNotes.length) yield { type: 'note', note: savingsNotes.shift()! }; + messages.push({ role: 'tool', tool_call_id: `call_${i}`, content: result }); + } + shrinkOldToolOutputs(messages); // mid-turn compaction (see loop.ts) + } + + yield { type: 'error', message: `Reached the step limit (${MAX_ROUNDS} tool rounds). Send "continue" to keep going, or raise ICLAW_MAX_ROUNDS.` }; +} diff --git a/packages/iclaw-runtime/src/sessions.ts b/packages/iclaw-runtime/src/sessions.ts new file mode 100644 index 0000000..4617a0b --- /dev/null +++ b/packages/iclaw-runtime/src/sessions.ts @@ -0,0 +1,799 @@ +/** + * In-memory session manager for Work and Secure modes. + */ +import http from 'node:http'; +import { randomUUID } from 'node:crypto'; +import { readdirSync, statSync, readFileSync, copyFileSync } from 'node:fs'; +import { basename, dirname, join } from 'node:path'; + +import { runAgentTurn, type Message } from './agent/loop.js'; +import type { AgentEvent } from './agent/loop.js'; +import { validateMountRoot } from './agent/security.js'; +import { + runSecureTurn, createSecureWorkspace, destroySecureWorkspace, + startContainer, stopContainer, isContainerRunning, + writeSessionMeta, listPersistedWorkspaces, +} from './secure-runner.js'; +import { + resolveWorkImage, startWorkContainer, execInWorkContainer, + toWorkMounts, type WorkMount, +} from './work-container.js'; +import { ingestSources, describeIngest, type IngestSource } from './secure-ingest.js'; +import { + exportWorkspace, applyChanges, type ExportResult, type ApplyResult, +} from './secure-export.js'; +import { ensureDockerForTask, markDockerUse } from './docker-lifecycle.js'; + +export interface SessionOptions { + allowedFolders: string[]; + /** + * Per-folder access levels (path + readonly flag), parallel to allowedFolders. + * When omitted, every allowed folder is treated as writable. Used by Work Mode + * to enforce read-only folders; Secure Mode leaves it unset (workspace is RW). + */ + folderAccess?: { path: string; readonly: boolean }[]; + /** + * Safe Mode only: host folders to COPY into the sandbox workspace on the first + * turn (originals never touched). Realizes "I added a folder and it got copied + * into the sandbox" — distinct from Work Mode's live bind mounts. + */ + copyFolders?: string[]; + model: string; + apiKey: string; + secure?: boolean; + /** + * Incognito (read-only, ephemeral): runs the host loop like Work, but writes + * are denied, reads are unrestricted, the shell sandbox forces every folder + * to :ro, and web_fetch is available. Mutually exclusive with `secure`. + */ + incognito?: boolean; + networkEnabled?: boolean; + systemPrompt?: string; + /** Stable identity (e.g. "chat:156") for reconnecting to a workspace. */ + key?: string; + /** + * Optional compacted prior history (from the host's DB) used to seed context + * after a restart. Only applied when the session has no history yet — never + * clobbers a live session's accumulated context. + */ + history?: { role: string; content: string }[]; +} + +/** + * A file the user dropped into the chat, forwarded from the host. `path` is the + * absolute on-disk location of the persisted upload (the runtime shares the + * host filesystem), so the runtime can read it directly — to stage it into a + * Secure workspace, grant Work read access, or base64 an image for vision. + */ +export interface RuntimeAttachment { + path: string; + mimeType: string; + fileName: string; +} + +interface Session { + id: string; + opts: SessionOptions; + history: Message[]; + sseClient: http.ServerResponse | null; + pending: AgentEvent[]; + /** Persistent workspace dir for Secure Mode (survives container restarts). */ + secureWorkspaceDir?: string; + /** Safe Mode: folder paths already copied into the workspace (ingest once each). */ + ingestedFolders?: Set<string>; + /** + * Warm Secure-Mode sandbox container, reused across turns. Recreated only + * when the network setting changes or after the container is reaped for idle; + * the workspace dir outlives it either way. + */ + secureContainer?: { name: string; networkEnabled: boolean; lastUsed: number; inUse: boolean }; + /** + * Warm Work-Mode command sandbox, reused across turns (parallel to + * secureContainer — a session is exactly one mode). Holds the user's chosen + * folders bind-mounted :ro/:rw; only run_command runs inside it. + */ + workContainer?: { name: string; lastUsed: number; inUse: boolean }; + /** Stable identity for reconnection across restarts. */ + key?: string; + /** Last activity timestamp (ms). Updated on each message. */ + lastActivity: number; + /** TTL in ms after last activity before cleanup. 0 = never. Default 7 days. */ + ttlMs: number; + /** Controller for the in-flight turn — abort() stops the model stream and + * ends the agent loop (user pressed Stop). Set per turn, cleared after. */ + abort?: AbortController; +} + +const DEFAULT_TTL_MS = 7 * 86400_000; +// Warm container idle reap: many chats → keep this short so idle sandboxes +// don't pile up RAM. (The workspace dir persists; only the container is freed.) +const CONTAINER_IDLE_MS = 2 * 60_000; +// Cap concurrent warm sandbox containers; LRU-evict the rest. +const MAX_WARM_CONTAINERS = 4; + +// Live-session re-compaction: once in-memory history exceeds the trigger, fold +// the older turns into a summary so a long-running session (no restart) keeps a +// bounded context instead of growing unbounded. Tunable via env. +const HISTORY_COMPACT_TRIGGER = Number(process.env.ICLAW_HISTORY_COMPACT_TRIGGER) || 40; +const HISTORY_KEEP_RECENT = Number(process.env.ICLAW_HISTORY_KEEP_RECENT) || 16; +// Also compact by SIZE, not just message count: a few big messages bloat the +// resent context as much as many small ones. ~24k chars ≈ ~6k tokens. +const HISTORY_COMPACT_CHARS = Number(process.env.ICLAW_HISTORY_COMPACT_CHARS) || 24_000; +const SUMMARY_MODEL = process.env.ICLAW_SUMMARY_MODEL || 'google/gemini-2.5-flash-lite'; +const OPENROUTER_BASE = process.env.OPENROUTER_BASE_URL?.replace(/\/+$/, '') || 'https://openrouter.ai/api/v1'; +const SUMMARY_SYSTEM = + 'You compress conversation history into a concise, information-dense summary. ' + + 'Preserve facts, decisions, user preferences, names, numbers, file/work state, and open threads. ' + + 'Merge any existing summary with the new messages. Output only the summary.'; +const sessions = new Map<string, Session>(); +// Maps a stable key (e.g. "chat:156") to the current in-memory session id, so a +// chat reconnects to the same persisted workspace after a restart. +const keyToId = new Map<string, string>(); + +/** Persist TTL/activity metadata so the session survives a runtime restart. */ +function persistMeta(session: Session): void { + if (!session.opts.secure || !session.secureWorkspaceDir) return; + writeSessionMeta(session.secureWorkspaceDir, { + key: session.key, + lastActivity: session.lastActivity, + ttlMs: session.ttlMs, + secure: true, + model: session.opts.model, + }); +} + +/** + * Create a session, or — if a stable `key` is given and a live session already + * exists for it — reuse it (refreshing runtime params like model/systemPrompt/ + * apiKey while preserving the workspace, history, TTL and lastActivity). + */ +export function createSession(opts: SessionOptions): string { + if (opts.key) { + const existingId = keyToId.get(opts.key); + const existing = existingId ? sessions.get(existingId) : undefined; + if (existing) { + // Reconnect: keep state, refresh the volatile runtime params. + existing.opts = { ...existing.opts, ...opts }; + // Seed context only if empty (e.g. restored-from-disk after a restart) — + // never clobber a live session's accumulated history. + if (existing.history.length === 0) seedHistory(existing, opts.history); + return existing.id; + } + } + + const id = randomUUID(); + const session: Session = { + id, opts, history: [], sseClient: null, pending: [], + key: opts.key, lastActivity: Date.now(), ttlMs: DEFAULT_TTL_MS, + }; + seedHistory(session, opts.history); + if (opts.secure) { + session.secureWorkspaceDir = createSecureWorkspace(); + } + sessions.set(id, session); + if (opts.key) keyToId.set(opts.key, id); + persistMeta(session); + return id; +} + +/** Seed a session's history from compacted prior turns (host-supplied). */ +function seedHistory(session: Session, history?: { role: string; content: string }[]): void { + if (!history?.length) return; + session.history = history.map((m) => ({ role: m.role, content: m.content }) as Message); +} + +/** Set TTL (days) for a session. 0 = never expire. */ +export function setSessionTtl(id: string, ttlDays: number): void { + const session = sessions.get(id); + if (session) { + session.ttlMs = ttlDays <= 0 ? 0 : ttlDays * 86400_000; + persistMeta(session); + } +} + +/** + * Reload persisted Secure sessions at startup so workspaces (and their TTL) + * survive a runtime restart. Expired ones are deleted; the rest become live + * shell sessions that a chat reconnects to via its key. apiKey comes from env; + * model/systemPrompt are refreshed when the host next calls createSession. + */ +export function loadPersistedSessions(): { restored: number; expired: number } { + const now = Date.now(); + let restored = 0; + let expired = 0; + const apiKey = process.env.ICLAW_OPENROUTER_API_KEY || ''; + + for (const { dir, meta } of listPersistedWorkspaces()) { + if (!meta || (meta.ttlMs > 0 && now - meta.lastActivity > meta.ttlMs)) { + destroySecureWorkspace(dir); + expired++; + continue; + } + const id = randomUUID(); + const session: Session = { + id, + opts: { + allowedFolders: [dir], + model: meta.model || (process.env.ICLAW_MODEL || ''), + apiKey, + secure: true, + }, + history: [], + sseClient: null, + pending: [], + secureWorkspaceDir: dir, + key: meta.key, + lastActivity: meta.lastActivity, + ttlMs: meta.ttlMs, + }; + sessions.set(id, session); + if (meta.key) keyToId.set(meta.key, id); + restored++; + } + return { restored, expired }; +} + +/** Sweep expired sessions — destroys workspace + frees memory. Runs periodically. */ +export function sweepExpiredSessions(): number { + const now = Date.now(); + let removed = 0; + for (const [id, session] of sessions) { + if (session.ttlMs > 0 && now - session.lastActivity > session.ttlMs) { + if (session.secureContainer) stopContainer(session.secureContainer.name); + if (session.workContainer) stopContainer(session.workContainer.name); + if (session.secureWorkspaceDir) destroySecureWorkspace(session.secureWorkspaceDir); + if (session.key) keyToId.delete(session.key); + session.sseClient?.end(); + sessions.delete(id); + removed++; + } + } + return removed; +} + +/** + * Stop warm Secure containers that have been idle longer than CONTAINER_IDLE_MS. + * The session and its workspace dir survive — the next message re-creates the + * container. Driven by a short interval (startContainerReaper). + */ +export function reapIdleContainers(): number { + const now = Date.now(); + let reaped = 0; + for (const session of sessions.values()) { + const c = session.secureContainer; + if (c && !c.inUse && now - c.lastUsed > CONTAINER_IDLE_MS) { + stopContainer(c.name); + session.secureContainer = undefined; + reaped++; + } + const w = session.workContainer; + if (w && !w.inUse && now - w.lastUsed > CONTAINER_IDLE_MS) { + stopContainer(w.name); + session.workContainer = undefined; + reaped++; + } + } + return reaped; +} + +/** Start the idle-container reaper. Returns a stop handle. */ +export function startContainerReaper(intervalMs = 30_000): () => void { + const t = setInterval(reapIdleContainers, intervalMs); + t.unref?.(); + return () => clearInterval(t); +} + +/** + * Ensure the session has a running sandbox container matching `networkEnabled`, + * reusing the existing one when possible (warm reuse). Recreates it when the + * network setting changed or the old container died. Enforces a global cap on + * concurrent warm containers by LRU-evicting the least recently used. + * + * Throws (fail-closed) if Docker can't start the sandbox. + */ +async function ensureSecureContainer(session: Session, networkEnabled: boolean): Promise<string> { + const existing = session.secureContainer; + if (existing && existing.networkEnabled === networkEnabled && await isContainerRunning(existing.name)) { + existing.lastUsed = Date.now(); + return existing.name; + } + // Stale, dead, or network setting changed → drop the old one. + if (existing) { + stopContainer(existing.name); + session.secureContainer = undefined; + } + + evictWarmContainersIfNeeded(session); + + const name = await startContainer(session.secureWorkspaceDir!, networkEnabled); + session.secureContainer = { name, networkEnabled, lastUsed: Date.now(), inUse: false }; + return name; +} + +/** + * LRU-evict warm containers (excluding `keep`) until under the cap. Counts both + * Secure and Work sandboxes — a session has at most one, and both consume RAM. + */ +function evictWarmContainersIfNeeded(keep: Session): void { + type Warm = { session: Session; kind: 'secure' | 'work'; name: string; lastUsed: number }; + const warm: Warm[] = []; + for (const s of sessions.values()) { + if (s === keep) continue; + if (s.secureContainer && !s.secureContainer.inUse) { + warm.push({ session: s, kind: 'secure', name: s.secureContainer.name, lastUsed: s.secureContainer.lastUsed }); + } + if (s.workContainer && !s.workContainer.inUse) { + warm.push({ session: s, kind: 'work', name: s.workContainer.name, lastUsed: s.workContainer.lastUsed }); + } + } + while (warm.length >= MAX_WARM_CONTAINERS) { + warm.sort((a, b) => a.lastUsed - b.lastUsed); + const victim = warm.shift(); + if (!victim) break; + stopContainer(victim.name); + if (victim.kind === 'secure') victim.session.secureContainer = undefined; + else victim.session.workContainer = undefined; + } +} + +/** + * Ensure the Work session has a running command sandbox with `mounts`. Reuses + * the warm container when alive; recreates it if it died. Throws (fail-closed) + * if Docker can't start it, so run_command surfaces the error instead of + * leaking to the host. + */ +async function ensureWorkContainer(session: Session, mounts: WorkMount[], image: string): Promise<string> { + const existing = session.workContainer; + if (existing && await isContainerRunning(existing.name)) { + existing.lastUsed = Date.now(); + existing.inUse = true; + return existing.name; + } + if (existing) { + stopContainer(existing.name); + session.workContainer = undefined; + } + evictWarmContainersIfNeeded(session); + const name = await startWorkContainer(mounts, image); + session.workContainer = { name, lastUsed: Date.now(), inUse: true }; + return name; +} + +export function getSession(id: string): Session | undefined { + return sessions.get(id); +} + +/** + * Abort the in-flight turn for a session WITHOUT tearing the session down + * (unlike deleteSession, the workspace/container survive). The agent loop stops + * its model stream and ends cleanly. No-op if nothing is running. + */ +export function abortSession(id: string): boolean { + const session = sessions.get(id); + if (!session?.abort) return false; + session.abort.abort(); + return true; +} + +export function deleteSession(id: string): void { + const session = sessions.get(id); + if (session?.secureContainer) stopContainer(session.secureContainer.name); + if (session?.workContainer) stopContainer(session.workContainer.name); + if (session?.secureWorkspaceDir) { + destroySecureWorkspace(session.secureWorkspaceDir); + } + if (session?.key) keyToId.delete(session.key); + sessions.delete(id); +} + +/** Get workspace info for a session. */ +export function getSessionInfo(id: string): { workspaceSize: number; secure: boolean } | null { + const session = sessions.get(id); + if (!session) return null; + let workspaceSize = 0; + if (session.secureWorkspaceDir) { + workspaceSize = getDirSize(session.secureWorkspaceDir); + } + return { workspaceSize, secure: session.opts.secure ?? false }; +} + +/** Export a Safe session's sandbox to a host folder. Null if not a Safe session. */ +export function exportSessionWorkspace(id: string, destDir?: string): ExportResult | null { + const session = sessions.get(id); + if (!session?.secureWorkspaceDir) return null; + return exportWorkspace(session.secureWorkspaceDir, destDir); +} + +/** Apply a Safe session's changes back to the original ingested folders. */ +export function applySessionChanges(id: string): ApplyResult[] | null { + const session = sessions.get(id); + if (!session?.secureWorkspaceDir) return null; + return applyChanges(session.secureWorkspaceDir); +} + +function getDirSize(dir: string): number { + try { + let total = 0; + const walk = (d: string) => { + for (const entry of readdirSync(d, { withFileTypes: true })) { + const full = `${d}/${entry.name}`; + if (entry.isDirectory()) walk(full); + else total += statSync(full).size; + } + }; + walk(dir); + return total; + } catch { return 0; } +} + +/** Update network setting for a secure session (takes effect on next turn). */ +export function setNetworkEnabled(id: string, enabled: boolean): void { + const session = sessions.get(id); + if (session) session.opts.networkEnabled = enabled; +} + +export function attachSseClient(id: string, res: http.ServerResponse): void { + const session = sessions.get(id); + if (!session) return; + session.sseClient = res; + for (const event of session.pending) writeSse(res, event); + session.pending = []; +} + +export function detachSseClient(id: string): void { + const session = sessions.get(id); + if (session) session.sseClient = null; +} + +/** Summarize messages with the cheap model. Best-effort — null on any failure. */ +async function summarizeMessages(apiKey: string, msgs: Message[]): Promise<string | null> { + if (!apiKey || msgs.length === 0) return null; + const text = msgs + .map((m) => `${m.role}: ${String(m.content).slice(0, 2000)}`) + .join('\n\n'); + try { + const res = await fetch(`${OPENROUTER_BASE}/chat/completions`, { + method: 'POST', + headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, + body: JSON.stringify({ + model: SUMMARY_MODEL, + stream: false, + temperature: 0, + max_tokens: 900, + messages: [ + { role: 'system', content: SUMMARY_SYSTEM }, + { role: 'user', content: `Messages to summarize:\n${text}\n\nReturn the updated summary.` }, + ], + }), + }); + if (!res.ok) return null; + const j = await res.json() as { choices?: { message?: { content?: string } }[] }; + return j?.choices?.[0]?.message?.content?.trim() || null; + } catch { + return null; + } +} + +/** + * Re-compact a long-running session's in-memory history: fold older turns into + * one summary message, keep the most recent verbatim. Bounds context for + * sessions that live for hours without a restart. Best-effort — leaves history + * untouched if summarization fails. + */ +function historyChars(h: Message[]): number { + let n = 0; + for (const m of h) { + n += typeof m.content === 'string' ? m.content.length : JSON.stringify(m.content ?? '').length; + } + return n; +} + +async function compactHistoryIfNeeded(session: Session): Promise<void> { + const h = session.history; + // Trigger on message count OR total size — whichever hits first. + const over = h.length > HISTORY_COMPACT_TRIGGER || historyChars(h) > HISTORY_COMPACT_CHARS; + if (!over) return; + // Need enough older messages to make folding worthwhile. + if (h.length <= HISTORY_KEEP_RECENT + 1) return; + const recent = h.slice(-HISTORY_KEEP_RECENT); + const older = h.slice(0, -HISTORY_KEEP_RECENT); + const summary = await summarizeMessages(session.opts.apiKey, older); + if (!summary) return; + session.history = [ + { role: 'system', content: `Summary of earlier conversation (compacted):\n${summary}` } as Message, + ...recent, + ]; +} + +/** + * Make dropped files usable by this turn's agent, and describe them to the model. + * + * - Images → base64 data URLs returned in `images`, shown as vision blocks so + * the model literally sees them (e.g. a screenshot). + * - Other files → made readable, then pointed at by path in `noticeText`: + * · Secure: copied into the workspace dir (appears at /workspace/<name>). + * · Work/Incognito: the file already lives on the host; we grant the + * session read access to its folder and give the model the absolute path. + * + * `noticeText` is appended to the user's message so the model KNOWS a file was + * attached (the bug being fixed: previously the agent had no idea) and how to + * reach it. Best-effort per file — a failure to stage one is reported inline, + * never throws. + */ +function stageAttachments( + session: Session, + attachments: RuntimeAttachment[] | undefined, +): { noticeText: string; images: string[] } { + if (!attachments?.length) return { noticeText: '', images: [] }; + const images: string[] = []; + const lines: string[] = []; + + for (const att of attachments) { + if (att.mimeType.startsWith('image/')) { + try { + const b64 = readFileSync(att.path).toString('base64'); + images.push(`data:${att.mimeType};base64,${b64}`); + lines.push(`- "${att.fileName}" — image, shown to you directly in this message.`); + continue; + } catch { + // Fall through and treat it as a regular file (path notice only). + } + } + + if (session.opts.secure && session.secureWorkspaceDir) { + const safe = basename(att.fileName) || basename(att.path); + try { + copyFileSync(att.path, join(session.secureWorkspaceDir, safe)); + lines.push(`- "${att.fileName}" (${att.mimeType}) — in /workspace; read_file (path "${safe}") only if you need its contents.`); + } catch (err) { + lines.push(`- "${att.fileName}" — could not be staged (${err instanceof Error ? err.message : String(err)}).`); + } + } else { + // Work / Incognito read the host filesystem directly. Grant this session + // read access to the upload's folder so validatePath lets read_file in. + const dir = dirname(att.path); + if (!session.opts.allowedFolders.includes(dir)) { + session.opts.allowedFolders = [...session.opts.allowedFolders, dir]; + } + lines.push(`- "${att.fileName}" (${att.mimeType}) — available at "${att.path}"; read_file it only if you need its contents.`); + } + } + + const noticeText = lines.length + ? `\n\n[The user attached ${attachments.length} file(s) to THIS message:\n${lines.join('\n')}\nUse them if relevant to the request.]` + : ''; + return { noticeText, images }; +} + +export async function sendMessage(sessionId: string, content: string, networkEnabled?: boolean, ttlDays?: number, attachments?: RuntimeAttachment[], copyFolders?: string[]): Promise<void> { + const session = sessions.get(sessionId); + if (!session) throw new Error(`Session not found: ${sessionId}`); + + // Reset TTL countdown on activity + session.lastActivity = Date.now(); + if (ttlDays !== undefined) session.ttlMs = ttlDays <= 0 ? 0 : ttlDays * 86400_000; + + // Safe Mode: merge in any folders the user added since the session started, so + // a mid-chat addition gets copied into the sandbox on this turn (not silently + // dropped). The ingest step below copies only the not-yet-ingested ones. + if (copyFolders?.length && session.opts.secure) { + const merged = new Set([...(session.opts.copyFolders ?? []), ...copyFolders]); + session.opts.copyFolders = [...merged]; + } + + // Per-message network override + if (networkEnabled !== undefined && session.opts.secure) { + session.opts.networkEnabled = networkEnabled; + } + + // Persist the refreshed activity/TTL so it survives a runtime restart. + persistMeta(session); + + // Keep a long-running session's context bounded (summarize old turns). + await compactHistoryIfNeeded(session); + + // Fresh abort controller for this turn (user Stop → abortSession()). + const abort = new AbortController(); + session.abort = abort; + + // Make any dropped files reachable + tell the model about them. `messageText` + // (content + notice) is what goes to the model AND into history, so follow-up + // turns remember a file was shared; `images` ride as vision blocks for one turn. + const { noticeText, images } = stageAttachments(session, attachments); + const messageText = content + noticeText; + + if (session.opts.secure) { + const workspaceDir = session.secureWorkspaceDir!; + const netEnabled = session.opts.networkEnabled ?? false; + + // COPY the user's chosen folders into the isolated workspace — Safe Mode + // works on a copy, so the originals are never touched. Each folder is copied + // once (tracked in ingestedFolders), so folders added mid-chat are picked up + // on the next turn. The summary is prepended so the model knows what's in + // /workspace and tells the user their files are unchanged. + let turnText = messageText; + const ingested = (session.ingestedFolders ??= new Set<string>()); + const pending = (session.opts.copyFolders ?? []).filter((p) => !ingested.has(p)); + if (pending.length) { + // Mark up front so a failed copy isn't retried every turn. + for (const p of pending) ingested.add(p); + const sources: IngestSource[] = pending.map((p) => ({ kind: 'folder', path: p })); + const results = await ingestSources(workspaceDir, sources); + for (const r of results) { + if (!r.ok) emit(session, { type: 'error', message: `Sandbox ingest: ${r.source} — ${r.error}` }); + } + const summary = describeIngest(results); + if (summary) turnText = `${summary}\n\n${messageText}`; + } + + // A Safe turn IS the sandbox, so it needs Docker now — start it ourselves if + // it's down (it'll auto-stop when idle). Only fail if it's missing/unstartable. + if (!(await ensureDockerForTask())) { + emit(session, { + type: 'error', + message: 'Safe work needs Docker, which isn’t running and couldn’t be started automatically. Install or start Docker, then try again.', + }); + return; + } + markDockerUse(); + + // Warm reuse: get (or lazily start) the session's sandbox container. + // Fail-closed — if Docker can't start it, surface an error, don't run. + let containerName: string; + try { + containerName = await ensureSecureContainer(session, netEnabled); + } catch (err) { + emit(session, { type: 'error', message: err instanceof Error ? err.message : String(err) }); + return; + } + + const secureGen = runSecureTurn( + session.history.map((m) => ({ role: m.role as string, content: String(m.content) })), + turnText, + { + apiKey: session.opts.apiKey, + model: session.opts.model, + workspaceDir, + containerName, + networkEnabled: netEnabled, + systemPrompt: session.opts.systemPrompt, + signal: abort.signal, + images, + }, + ); + let assistantText = ''; + // Pin the container so the idle reaper / LRU never frees it mid-turn. + if (session.secureContainer) session.secureContainer.inUse = true; + try { + for await (const event of secureGen) { + emit(session, event as AgentEvent); + if (event.type === 'text') assistantText += event.content; + } + } finally { + if (session.secureContainer) { + session.secureContainer.inUse = false; + session.secureContainer.lastUsed = Date.now(); + } + if (session.abort === abort) session.abort = undefined; + } + if (assistantText) { + session.history.push({ role: 'user', content: turnText }); + session.history.push({ role: 'assistant', content: assistantText }); + } + return; + } + + // run_command runs in a Docker sandbox with per-folder :ro/:rw mounts so the + // kernel enforces read-only. Without Docker we leave runShell undefined → + // run_command is disabled (file tools still work). The container is created + // lazily on first command, so chat/read-only turns never spin one up. + // + // Only EXPLICITLY chosen folders are mounted — never the broad $HOME fallback + // (mounting all of home would expose ~/.ssh etc. to the shell). Each folder is + // validated against the secret deny-list and dropped (with a notice) if it + // names a sensitive root. + // Incognito forces every mounted folder to :ro — the shell may read but the + // kernel rejects all writes, matching the read-only contract. + const incognito = !!session.opts.incognito; + let runShell: ((command: string, cwd: string) => Promise<string>) | undefined; + let linkSandbox: ((command: string) => Promise<string>) | undefined; + + // Validate the explicitly-chosen folders into bind mounts (empty when none + // selected). This is host-only (no Docker needed), so it always runs. + const validated: { path: string; readonly: boolean }[] = []; + for (const f of session.opts.folderAccess ?? []) { + try { + validated.push({ path: validateMountRoot(f.path), readonly: incognito ? true : f.readonly }); + } catch (err) { + emit(session, { + type: 'error', + message: `Folder excluded from commands: ${err instanceof Error ? err.message : String(err)}`, + }); + } + } + const mounts: WorkMount[] = toWorkMounts(validated); + + // We DON'T gate on Docker being up. Docker is started lazily inside the tool, + // the first moment a task actually needs it (ensureDockerForTask) — so pure + // file-edit / read-only turns never touch Docker, and the user is never nagged + // to start it up front. If Docker is missing or can't be started, the tool + // returns a guidance message and file tools still work. + const ensureSandbox = async (): Promise<boolean> => { + if (!(await ensureDockerForTask())) return false; + markDockerUse(); + return true; + }; + // analyze_link runs yt-dlp inside the session's container (warm-reused, so + // yt-dlp self-installs once) — never on the host. Offered even with no folders + // (network-only container). Runs at "/" (it writes to its own scratch paths). + linkSandbox = async (command) => { + if (!(await ensureSandbox())) { + return 'analyze_link needs a Docker sandbox, which isn’t running and couldn’t be started. Use web_fetch/web_search instead.'; + } + const image = await resolveWorkImage(); + const name = await ensureWorkContainer(session, mounts, image); + return execInWorkContainer(name, command, '/', { mounts }); + }; + // run_command needs a real folder to operate in, so it stays gated on mounts. + if (mounts.length) { + runShell = async (command, cwd) => { + if (!(await ensureSandbox())) { + return 'run_command needs a Docker sandbox. Docker isn’t running and couldn’t be started automatically — start (or install) Docker and try again. File read/write tools still work without it.'; + } + const image = await resolveWorkImage(); + const name = await ensureWorkContainer(session, mounts, image); + // scan: report created/modified/deleted files after the command so the + // user sees exactly what changed inside their allowed folders. + return execInWorkContainer(name, command, cwd, { mounts, scan: true }); + }; + } + + const gen = runAgentTurn(session.history, messageText, { + apiKey: session.opts.apiKey, + model: session.opts.model, + allowedFolders: session.opts.allowedFolders, + folderAccess: session.opts.folderAccess, + runShell, + linkSandbox, + incognito, + systemPrompt: session.opts.systemPrompt, + signal: abort.signal, + images, + onWriteApproval: async (filePath, fileContent) => { + emit(session, { type: 'approval_request', changeId: randomUUID(), path: filePath, content: fileContent }); + return true; + }, + }); + + let assistantText = ''; + try { + for await (const event of gen) { + emit(session, event); + if (event.type === 'text') assistantText += event.content; + } + } finally { + // Release the warm container so the idle reaper / LRU can free it. + if (session.workContainer) { + session.workContainer.inUse = false; + session.workContainer.lastUsed = Date.now(); + } + if (session.abort === abort) session.abort = undefined; + } + + if (assistantText) { + session.history.push({ role: 'user', content: messageText }); + session.history.push({ role: 'assistant', content: assistantText }); + } +} + +function emit(session: Session, event: AgentEvent): void { + if (session.sseClient && !session.sseClient.writableEnded) { + writeSse(session.sseClient, event); + } else { + session.pending.push(event); + } +} + +function writeSse(res: http.ServerResponse, event: AgentEvent): void { + try { res.write(`data: ${JSON.stringify(event)}\n\n`); } catch {} +} diff --git a/packages/iclaw-runtime/src/work-container.ts b/packages/iclaw-runtime/src/work-container.ts new file mode 100644 index 0000000..7a99f6f --- /dev/null +++ b/packages/iclaw-runtime/src/work-container.ts @@ -0,0 +1,432 @@ +/** + * Work Mode command sandbox. + * + * Work Mode = "I trust this folder. Work on the REAL files, but only inside it." + * The agent loop and the file tools (read/write/list/search) run on the host — + * they're path-checked (validatePath) and gated by per-write approval. The one + * tool that can't be safely contained on the host is `run_command`: arbitrary + * `bash -c` can write/delete anywhere, ignoring our per-folder flags. + * + * So `run_command` is routed into a Docker container. Each allowed folder is + * bind-mounted to a NORMALIZED container path — `/work/0`, `/work/1`, … — with + * `:ro` or `:rw` per its access level. This scheme is identical on macOS, Linux + * and Windows (no same-path mounts, which can't work on Windows where the host + * path is `C:\…`). The kernel enforces the flags: a write into a `:ro` mount + * fails with "Read-only file system", and anything outside the mounts is simply + * invisible — the command literally cannot see the rest of the computer. + * + * The model speaks in HOST paths (the file tools and the folder list it sees use + * them), so run_command's host `cwd` and any host folder roots in the command + * are translated to their `/work/<n>` equivalents before exec — see + * hostToContainer / translateCommandPaths. + * + * After each command we scan the writable mounts and report which files were + * created / modified / deleted, reassuring the user that all changes stayed + * inside the allowed folders. + * + * When Docker is unavailable, `run_command` is disabled (strict fallback) — the + * host file tools still work, so read-only stays an honest guarantee. + */ +import { execFile } from 'node:child_process'; +import { randomUUID } from 'node:crypto'; +import { promisify } from 'node:util'; + +import { log } from './log.js'; +import { INSTALL_LABEL } from './install-id.js'; + +const execFileAsync = promisify(execFile); + +const WORK_PREFIX = 'iclaw-work-'; +const COMMAND_TIMEOUT = Number(process.env.ICLAW_WORK_COMMAND_TIMEOUT) || 60_000; +// `docker run` auto-pulls the image on first use; allow time for that. +const START_TIMEOUT = Number(process.env.ICLAW_WORK_START_TIMEOUT) || 300_000; +/** Cap the per-command file scan so a huge tree can't stall the turn. */ +const SCAN_MAX_FILES = Number(process.env.ICLAW_WORK_SCAN_MAX_FILES) || 20_000; +/** Cap how many changed paths we list back (the rest collapse to "+N more"). */ +const SCAN_MAX_REPORT = 50; + +// The shared iClaw sandbox image — ONE image for both Safe work and Work mode +// (container/secure-sandbox.Dockerfile, 80/20 toolset). Resolved lazily. +const SANDBOX_IMAGE = process.env.ICLAW_SECURE_IMAGE || 'iclaw-secure:latest'; +// Last-resort fallback when the shared image hasn't been built yet: a public +// base that already ships bash, git and node, so Work still runs (with a leaner +// toolset) on a fresh checkout. Auto-pulled by `docker run`. +const FALLBACK_IMAGE = process.env.ICLAW_WORK_SLIM_IMAGE || 'node:22'; + +/** + * One allowed folder, mapped from its host path to a normalized container path. + * `label` (the folder's basename) is only used to make reports readable. + */ +export interface WorkMount { + /** Validated absolute host path (the source of the bind mount). */ + path: string; + /** Normalized in-container mount point: `/work/<n>`. */ + containerPath: string; + /** Read-only (`:ro`) when true, read & write (`:rw`) otherwise. */ + readonly: boolean; + /** Folder basename, for human-readable change reports. */ + label: string; +} + +/** Folder basename for a host path (handles both separators). */ +function basenameOf(hostPath: string): string { + const parts = hostPath.replace(/[\\/]+$/, '').split(/[\\/]/); + return parts[parts.length - 1] || hostPath; +} + +/** + * Assign normalized `/work/<n>` container paths to validated allowed folders. + * Order is stable (index = position), so the same folder set always maps the + * same way within a session. + */ +export function toWorkMounts(folders: { path: string; readonly: boolean }[]): WorkMount[] { + return folders.map((f, i) => ({ + path: f.path, + containerPath: `/work/${i}`, + readonly: f.readonly, + label: basenameOf(f.path), + })); +} + +/** Path key for prefix comparison: forward slashes; lowercased on Windows. */ +function normKey(p: string, plat: NodeJS.Platform): string { + const slashed = p.replace(/\\/g, '/').replace(/\/+$/, ''); + return plat === 'win32' ? slashed.toLowerCase() : slashed; +} + +/** + * Map a host path (cwd or any path under an allowed folder) to its container + * path. Returns null when the path isn't inside any mount — the caller then + * refuses, so a command can never `cd` outside the allowed folders. + */ +export function hostToContainer( + hostPath: string, + mounts: WorkMount[], + plat: NodeJS.Platform = process.platform, +): string | null { + const target = normKey(hostPath, plat); + // Longest root first so a nested mount wins over its parent. + const sorted = [...mounts].sort((a, b) => b.path.length - a.path.length); + for (const m of sorted) { + const root = normKey(m.path, plat); + if (target === root) return m.containerPath; + if (target.startsWith(root + '/')) { + return m.containerPath + target.slice(root.length); + } + } + return null; +} + +/** Map a container path (`/work/<n>/…`) back to its host path, for reports. */ +export function containerToHost(containerPath: string, mounts: WorkMount[]): string { + for (const m of mounts) { + if (containerPath === m.containerPath) return m.path; + if (containerPath.startsWith(m.containerPath + '/')) { + const rel = containerPath.slice(m.containerPath.length + 1); + const sep = m.path.includes('\\') ? '\\' : '/'; + return m.path.replace(/[\\/]+$/, '') + sep + rel.replace(/\//g, sep); + } + } + return containerPath; +} + +function escapeRegex(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +/** + * Rewrite host folder roots inside a shell command to their container paths. + * + * The model emits host paths (that's all it sees), so a command may reference + * `~/Projects/app/...` or `C:\Users\me\app\...`. We replace each allowed-folder + * root (in both native and forward-slash form) with its `/work/<n>` mount and + * normalize the tail to forward slashes, stopping at whitespace, quotes, and + * shell metacharacters so a path can't swallow the rest of the command. The + * structured `cwd` is mapped separately (hostToContainer); this catches inline + * absolute paths in the command body. + */ +export function translateCommandPaths( + command: string, + mounts: WorkMount[], + plat: NodeJS.Platform = process.platform, +): string { + let out = command; + const flags = plat === 'win32' ? 'gi' : 'g'; + // Tail must start with a separator (or be empty) so the root only matches at a + // path boundary — `/a` must not match inside `/abc`. The lookahead then + // asserts the token really ends (whitespace, quote, shell metachar, or EOL). + const TAIL = "((?:[/\\\\][^\\s\"'`&|;<>()]*)?)"; + const BOUND = "(?=[\\s\"'`&|;<>()]|$)"; + // Longest root first so a nested mount wins over its parent. + const sorted = [...mounts].sort((a, b) => b.path.length - a.path.length); + for (const m of sorted) { + for (const variant of new Set([m.path, m.path.replace(/\\/g, '/')])) { + const esc = escapeRegex(variant); + out = out.replace( + new RegExp(esc + TAIL + BOUND, flags), + (_full, tail: string) => m.containerPath + (tail || '').replace(/\\/g, '/'), + ); + } + } + return out; +} + +let dockerOk: boolean | null = null; +let resolvedImage: string | null = null; + +/** True if a usable Docker daemon is reachable. Cached after the first probe. */ +export async function dockerAvailable(): Promise<boolean> { + if (dockerOk !== null) return dockerOk; + try { + await execFileAsync('docker', ['info'], { timeout: 8_000 }); + dockerOk = true; + } catch { + dockerOk = false; + } + return dockerOk; +} + +async function imageExists(image: string): Promise<boolean> { + try { + await execFileAsync('docker', ['image', 'inspect', image], { timeout: 8_000 }); + return true; + } catch { + return false; + } +} + +/** + * Pick the image for Work Mode commands. Order: + * 1. ICLAW_WORK_IMAGE — explicit override (e.g. a heavier python/go toolchain). + * 2. The shared sandbox image (iclaw-secure:latest) — the canonical choice, + * identical to what Safe work uses, so one build serves both modes. + * 3. Fallback public base (node:22) only when the shared image isn't built — + * logged as a warning so the leaner toolset isn't a silent surprise. + * Cached after the first resolve. + */ +export async function resolveWorkImage(): Promise<string> { + if (resolvedImage) return resolvedImage; + if (process.env.ICLAW_WORK_IMAGE) { + resolvedImage = process.env.ICLAW_WORK_IMAGE; + } else if (await imageExists(SANDBOX_IMAGE)) { + resolvedImage = SANDBOX_IMAGE; + } else { + log.warn( + 'Shared sandbox image not built — Work Mode falling back to a leaner base. ' + + 'Build it with `npm run build:secure-image` for the full toolset.', + { fallback: FALLBACK_IMAGE, sandbox: SANDBOX_IMAGE }, + ); + resolvedImage = FALLBACK_IMAGE; + } + return resolvedImage; +} + +/** + * Start a long-lived Work container with the given folder mounts. Returns the + * container name. Fail-closed: throws if Docker can't start it, so the caller + * keeps run_command disabled instead of silently running unsandboxed. + */ +export async function startWorkContainer(mounts: WorkMount[], image: string): Promise<string> { + const name = `${WORK_PREFIX}${randomUUID().slice(0, 8)}`; + const mountArgs: string[] = []; + for (const m of mounts) { + // Source = native host path (the Docker CLI handles the `X:\` drive prefix + // on Windows); target = normalized `/work/<n>` mount point. + mountArgs.push('-v', `${m.path}:${m.containerPath}:${m.readonly ? 'ro' : 'rw'}`); + } + // Default working dir: first writable folder, else the first mount. + const home = mounts.find((m) => !m.readonly) ?? mounts[0]; + + // Run the command process as the HOST user, so files it creates in the user's + // real (bind-mounted) folders are owned by the user — not root or the image's + // uid 1000. Without this, run_command output in a :rw folder lands root-owned + // and the user can't edit/delete it (visible on Linux; macOS Docker Desktop + // remaps ownership so it's hidden there). Skip when we'd add no value or can't: + // running as root already (uid 0), already matching the image user (1000), or + // Windows (getuid undefined). HOME=/tmp gives the now-nameless uid a writable + // home for tool caches (npm, git) since /home/node isn't writable by it. + const uid = process.getuid?.(); + const gid = process.getgid?.(); + const userArgs = + uid != null && uid !== 0 && uid !== 1000 + ? ['--user', `${uid}:${gid ?? uid}`, '-e', 'HOME=/tmp'] + : []; + + try { + await execFileAsync('docker', [ + 'run', '--rm', '-d', + '--name', name, + '--label', INSTALL_LABEL, + '--memory', process.env.ICLAW_WORK_MEMORY || '768m', + '--cpus', process.env.ICLAW_WORK_CPUS || '1', + ...userArgs, + ...mountArgs, + ...(home ? ['--workdir', home.containerPath] : []), + image, + 'sleep', '86400', + ], { timeout: START_TIMEOUT }); + } catch (err) { + const e = err as { stderr?: string; message?: string }; + throw new Error( + `Work sandbox failed to start — is Docker running? ${(e.stderr || e.message || '').slice(0, 200)}`, + ); + } + return name; +} + +export interface ExecOptions { + /** Folder mappings — used to translate the host cwd + command paths. */ + mounts?: WorkMount[]; + /** Run a post-command change scan and append a report (Work run_command). */ + scan?: boolean; +} + +/** + * Execute a command inside a Work container at the given HOST cwd. The cwd and + * any host folder roots in the command are translated to their `/work/<n>` + * container paths first; a cwd outside every mount is refused (the command can + * never operate outside the allowed folders). With `scan`, a before/after diff + * of the writable mounts is appended so the user sees exactly what changed. + */ +export async function execInWorkContainer( + name: string, + command: string, + hostCwd: string, + opts: ExecOptions = {}, +): Promise<string> { + const mounts = opts.mounts ?? []; + // Translate the host cwd → container path. '/' (linkSandbox) and already- + // container paths fall through unchanged. + let containerCwd = hostCwd; + if (mounts.length) { + const mapped = hostToContainer(hostCwd, mounts); + if (mapped) { + containerCwd = mapped; + } else if (!hostCwd.startsWith('/work/') && hostCwd !== '/') { + return `Refused: working directory "${hostCwd}" is outside the folders you allowed for this chat.`; + } + } + const containerCmd = translateCommandPaths(command, mounts); + + const before = opts.scan ? await snapshotWritable(name, mounts) : null; + const output = await rawExec(name, containerCmd, containerCwd); + if (!before) return output; + + const after = await snapshotWritable(name, mounts); + const report = renderChangeReport(before, after, mounts); + return report ? `${output}\n\n${report}` : output; +} + +/** Bare `docker exec` — returns combined stdout/stderr (or the error text). */ +async function rawExec(name: string, command: string, cwd: string): Promise<string> { + try { + const { stdout, stderr } = await execFileAsync( + 'docker', ['exec', '--workdir', cwd, name, 'bash', '-lc', command], + { timeout: COMMAND_TIMEOUT }, + ); + return [stdout, stderr].filter(Boolean).join('\n').trim() || '(no output)'; + } catch (err: unknown) { + const e = err as { stdout?: string; stderr?: string; message?: string }; + return [e.stdout, e.stderr, e.message].filter(Boolean).join('\n').trim() || 'Error'; + } +} + +type FileSnapshot = Map<string, string>; // container path → mtime (epoch float) + +/** + * Snapshot files in the WRITABLE mounts (read-only mounts can't change, so we + * skip them). Excludes .git / node_modules churn so the report stays meaningful. + * Best-effort: returns an empty snapshot if `find` fails or the tree is huge. + */ +async function snapshotWritable(name: string, mounts: WorkMount[]): Promise<FileSnapshot> { + const snap: FileSnapshot = new Map(); + const writable = mounts.filter((m) => !m.readonly); + if (writable.length === 0) return snap; + const roots = writable.map((m) => m.containerPath); + // GNU find (the container is always Linux): print "<mtime>\t<path>" per file. + const findCmd = + `find ${roots.map((r) => `'${r}'`).join(' ')} -type f ` + + `-not -path '*/.git/*' -not -path '*/node_modules/*' ` + + `-printf '%T@\\t%p\\n' 2>/dev/null | head -n ${SCAN_MAX_FILES}`; + try { + const { stdout } = await execFileAsync( + 'docker', ['exec', name, 'bash', '-lc', findCmd], + { timeout: COMMAND_TIMEOUT }, + ); + for (const line of stdout.split('\n')) { + const tab = line.indexOf('\t'); + if (tab === -1) continue; + snap.set(line.slice(tab + 1), line.slice(0, tab)); + } + } catch { + /* best-effort — an empty/partial snapshot just yields a sparser report. */ + } + return snap; +} + +interface ChangeSet { + created: string[]; + modified: string[]; + deleted: string[]; +} + +function diffSnapshots(before: FileSnapshot, after: FileSnapshot): ChangeSet { + const created: string[] = []; + const modified: string[] = []; + const deleted: string[] = []; + for (const [p, mtime] of after) { + const prev = before.get(p); + if (prev === undefined) created.push(p); + else if (prev !== mtime) modified.push(p); + } + for (const p of before.keys()) if (!after.has(p)) deleted.push(p); + return { created, modified, deleted }; +} + +/** Human-readable change block appended to run_command output (host paths). */ +function renderChangeReport( + before: FileSnapshot, + after: FileSnapshot, + mounts: WorkMount[], +): string { + const { created, modified, deleted } = diffSnapshots(before, after); + const total = created.length + modified.length + deleted.length; + const lines: string[] = ['── File changes (Work sandbox) ──']; + if (total === 0) { + lines.push('No files were created, modified, or deleted.'); + return lines.join('\n'); + } + const section = (mark: string, label: string, paths: string[]): void => { + if (paths.length === 0) return; + const shown = paths.slice(0, SCAN_MAX_REPORT); + for (const p of shown) lines.push(`${mark} ${label}: ${containerToHost(p, mounts)}`); + if (paths.length > shown.length) { + lines.push(` …and ${paths.length - shown.length} more ${label} file(s)`); + } + }; + section('+', 'created', created); + section('~', 'modified', modified); + section('-', 'deleted', deleted); + // By construction the shell only sees mounted folders, so every change above + // is inside an allowed folder — state it plainly for reassurance. + lines.push('All changes were inside the folders you allowed.'); + return lines.join('\n'); +} + +/** Kill Work containers orphaned by a previous runtime process. */ +export async function killOrphanWorkContainers(): Promise<number> { + try { + // Scope to THIS install's Work containers: name prefix AND install label are + // ANDed, so a second iClaw install never reaps ours (nor we theirs). + const { stdout } = await execFileAsync( + 'docker', ['ps', '-aq', '--filter', `name=${WORK_PREFIX}`, '--filter', `label=${INSTALL_LABEL}`], + { timeout: 10_000 }, + ); + const ids = stdout.split('\n').map((s) => s.trim()).filter(Boolean); + if (ids.length === 0) return 0; + await execFileAsync('docker', ['rm', '-f', ...ids], { timeout: 30_000 }).catch(() => {}); + return ids.length; + } catch { + return 0; + } +} diff --git a/packages/iclaw-runtime/tsconfig.json b/packages/iclaw-runtime/tsconfig.json new file mode 100644 index 0000000..c83c533 --- /dev/null +++ b/packages/iclaw-runtime/tsconfig.json @@ -0,0 +1,20 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "NodeNext", + "moduleResolution": "NodeNext", + "lib": ["ES2022"], + "outDir": "./dist", + "rootDir": "./src", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": true, + "declaration": true, + "declarationMap": true, + "sourceMap": true + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist"] +} diff --git a/public/css/style.css b/public/css/style.css index 538de1d..c6d8489 100644 --- a/public/css/style.css +++ b/public/css/style.css @@ -1067,28 +1067,6 @@ code { --_btn-bg-hover: color-mix(in srgb, var(--danger) 8%, transparent); } -.gateway { - font-size: 0.72rem; - padding: 4px 9px; - border-radius: 6px; - background: var(--msg-user-fill); - border: 1px solid var(--msg-user-stroke); - font-weight: 500; - letter-spacing: -0.01em; -} -.gateway.ok { color: var(--ok); border-color: var(--ok-border); } -.gateway.down { color: var(--down); border-color: var(--down-soft-border); } -.gateway.degraded { - color: var(--warn); - border-color: color-mix(in srgb, var(--warn) 40%, var(--border)); - background: color-mix(in srgb, var(--warn) 8%, var(--surface)); -} -.gateway.shutdown { - color: var(--down); - border-color: color-mix(in srgb, var(--down) 55%, var(--border)); - background: color-mix(in srgb, var(--down) 8%, var(--surface)); - font-weight: 600; -} /* ---------- Welcome / empty ---------- */ @@ -1101,12 +1079,13 @@ code { .empty-state { text-align: center; - margin: 48px auto 32px; - max-width: 22rem; + margin: 80px auto 32px; + max-width: 32rem; padding: 0 20px; font-size: 0.95rem; line-height: 1.5; letter-spacing: -0.01em; + white-space: nowrap; } /* ---------- Messages (ChatGPT-like thread) ---------- */ @@ -1223,6 +1202,31 @@ code { } /* ChatGPT hides role chrome; keep labels for screen readers only */ +/* Dev-mode token-usage badge (a subtle trailing line under a message). */ +.msg-tokens { + display: block; + margin-top: 4px; + font-size: 11px; + font-variant-numeric: tabular-nums; + letter-spacing: 0.01em; + color: color-mix(in srgb, var(--text) 42%, transparent); + user-select: none; +} +.msg.user .msg-tokens { text-align: right; } + +/* Dev-mode chat-wide token total (chat header). */ +.chat-token-total { + align-self: center; + flex: none; + padding: 2px 8px; + border-radius: 999px; + font-size: 11px; + font-variant-numeric: tabular-nums; + white-space: nowrap; + color: color-mix(in srgb, var(--text) 55%, transparent); + background: color-mix(in srgb, var(--text) 8%, transparent); +} + .msg .role { position: absolute; width: 1px; @@ -1731,56 +1735,102 @@ code { --composer-secret-strip-gap: var(--space-1); } +/* ── Composer field — new layout: textarea top, toolbar bottom ── */ .composer-field { - margin-top: -8px; margin-bottom: var(--space-4); display: flex; - align-items: flex-end; - gap: 6px; - padding: 6px 6px 6px 16px; + flex-direction: column; + padding: 12px 12px 8px 16px; background: var(--surface); border: 1px solid var(--border); - border-radius: 22px; - box-shadow: 0 1px 2px rgba(0, 0, 0, 0.04); - transition: border-color 0.15s ease, box-shadow 0.15s ease; + border-radius: 18px; + box-shadow: 0 1px 2px rgba(0,0,0,0.04); + transition: border-color 0.15s, box-shadow 0.15s; } .composer-field:focus-within { border-color: color-mix(in srgb, var(--text) 22%, var(--border)); - box-shadow: - 0 1px 2px rgba(0, 0, 0, 0.04), - 0 0 0 3px color-mix(in srgb, var(--text) 8%, transparent); + box-shadow: 0 1px 2px rgba(0,0,0,0.04), 0 0 0 3px color-mix(in srgb, var(--text) 8%, transparent); } +/* Textarea */ .composer-field .composer-input-stack { - flex: 1; - min-width: 0; - display: flex; - flex-direction: column; - align-items: stretch; + width: 100%; } - .composer-field .composer-input-stack textarea { - flex: 1; - min-width: 0; + width: 100%; margin: 0; - padding: 8px 0; + padding: 0 0 10px; border: none; background: transparent; - border-radius: 0; font-family: inherit; font-size: 0.95em; - line-height: 1.45; + line-height: 1.5; resize: none; min-height: 24px; max-height: 200px; field-sizing: content; } -.composer-field .composer-input-stack textarea:focus { - outline: none; +.composer-field .composer-input-stack textarea:focus { outline: none; } +.composer-field .composer-input-stack textarea::placeholder { color: var(--muted); opacity: 0.55; } + +/* Toolbar row */ +.composer-toolbar { + display: flex; + align-items: center; + justify-content: space-between; + gap: 4px; +} +.composer-toolbar__left { + display: flex; + align-items: center; + gap: 2px; +} +.composer-toolbar__right { + display: flex; + align-items: center; + gap: 4px; +} + +/* Generic toolbar button */ +.composer-toolbar-btn { + display: inline-flex; + align-items: center; + gap: 5px; + padding: 5px 8px; + border: none; + border-radius: 8px; + background: transparent; + color: var(--muted); + font: inherit; + font-size: var(--text-sm); + cursor: pointer; + transition: background 0.12s, color 0.12s; + white-space: nowrap; +} +.composer-toolbar-btn:hover, +.composer-toolbar-btn[aria-expanded='true'] { + background: color-mix(in srgb, var(--text) 8%, transparent); + color: var(--text); +} + +/* Mode selector in toolbar */ +.composer-toolbar-modes { position: relative; } +.composer-mode-btn { + font-weight: 500; +} +.composer-mode-btn svg { opacity: 0.6; } +.composer-mode-menu { + bottom: calc(100% + 6px); + left: 0; + min-width: 264px; } -.composer-field .composer-input-stack textarea::placeholder { +.composer-mode-menu-item[aria-checked='true'] { + background: color-mix(in srgb, var(--accent) 10%, transparent); +} +.composer-mode-menu-item__desc { + font-size: var(--text-xs); color: var(--muted); - opacity: 0.5; + line-height: 1.35; } .composer-send { @@ -4692,7 +4742,141 @@ code { /* ---------- Composer attachments ---------- */ -.composer-attach { +/* attach button inherits .composer-toolbar-btn */ +.composer-attach { padding: 5px 7px; border-radius: 8px; } +.composer-attach:active { transform: scale(0.94); } + +/* Work Mode folders button */ +.composer-network-toggle[hidden] { display: none; } +.composer-network-toggle[data-network='on'] { color: var(--accent); } +.composer-work-folders[hidden] { display: none; } +.composer-work-folders__count { + font-size: 11px; + font-weight: 600; + min-width: 14px; + text-align: center; +} + +/* Work folders modal */ +.work-folders-modal { + position: fixed; + inset: 0; + z-index: 200; + display: flex; + align-items: flex-end; + justify-content: center; + padding-bottom: 80px; +} +.work-folders-modal[hidden] { display: none; } +.work-folders-modal__backdrop { + position: absolute; + inset: 0; + background: rgba(0,0,0,0.4); +} +.work-folders-modal__panel { + position: relative; + background: var(--surface); + border: 1px solid var(--border); + border-radius: 12px; + padding: 20px; + width: min(480px, 94vw); + display: flex; + flex-direction: column; + gap: 12px; + box-shadow: 0 8px 32px rgba(0,0,0,0.2); +} +.work-folders-modal__head { } +.work-folders-modal__head-row { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 12px; +} +.work-folders-modal__title { margin: 0; font-size: var(--text-base); font-weight: 600; } +.work-folders-modal__hint { margin: 4px 0 0; font-size: var(--text-sm); } +.work-folders-list { + list-style: none; + margin: 0; + padding: 0; + display: flex; + flex-direction: column; + gap: 6px; + max-height: 200px; + overflow-y: auto; +} +.work-folders-list:empty::before { + content: 'No folders added yet'; + color: var(--muted); + font-size: var(--text-sm); + display: block; + padding: 4px 0; +} +.work-folders-list__item { + display: flex; + align-items: center; + gap: 8px; + background: color-mix(in srgb, var(--text) 5%, transparent); + border-radius: 6px; + padding: 6px 10px; + font-size: var(--text-sm); + font-family: monospace; +} +.work-folders-list__path { flex: 1; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.work-folders-list__access { + flex-shrink: 0; + border: 1px solid var(--border, color-mix(in srgb, var(--text) 18%, transparent)); + background: transparent; + color: var(--muted); + cursor: pointer; + border-radius: 999px; + padding: 2px 10px; + font-family: inherit; + font-size: var(--text-sm); + white-space: nowrap; +} +.work-folders-list__access[data-write="1"] { + color: var(--accent); + border-color: color-mix(in srgb, var(--accent) 45%, transparent); + background: color-mix(in srgb, var(--accent) 10%, transparent); +} +.work-folders-list__access:hover { color: var(--text); } +.work-folders-list__remove { + flex-shrink: 0; + border: none; + background: transparent; + color: var(--muted); + cursor: pointer; + padding: 0 2px; + font-size: 16px; + line-height: 1; +} +.work-folders-list__remove:hover { color: var(--text); } +.work-folders-add { + display: flex; + align-items: center; + gap: 8px; + flex-wrap: wrap; +} +.work-folders-add__or { + font-size: var(--text-sm); + flex-shrink: 0; +} +.work-folders-input { + flex: 1; + padding: 6px 10px; + border: 1px solid var(--border); + border-radius: 6px; + background: var(--bg); + color: var(--text); + font-size: var(--text-sm); + font-family: monospace; +} +.work-folders-input:focus { outline: none; border-color: var(--accent); } +.work-folders-modal__actions { display: flex; justify-content: flex-end; } + +/* Speech-to-text mic button — mirrors .composer-attach. Recording = red pulse; + busy (transcribing) swaps the mic for a spinner. */ +.composer-mic { flex: 0 0 auto; align-self: center; display: inline-flex; @@ -4701,7 +4885,6 @@ code { width: 32px; height: 32px; margin: 0; - margin-left: -8px; padding: 0; border: none; border-radius: 50%; @@ -4710,11 +4893,34 @@ code { cursor: pointer; transition: background 0.15s ease, color 0.15s ease, transform 0.1s ease; } -.composer-attach:hover { +.composer-mic:hover { background: color-mix(in srgb, var(--text) 8%, transparent); color: var(--text); } -.composer-attach:active { transform: scale(0.94); } +.composer-mic:active { transform: scale(0.94); } +.composer-mic__busy { display: none; } +.composer-mic[data-state='recording'] { + color: var(--danger); + background: color-mix(in srgb, var(--danger) 12%, transparent); + animation: composer-mic-pulse 1.2s ease-in-out infinite; +} +.composer-mic[data-state='recording']:hover { + color: var(--danger); + background: color-mix(in srgb, var(--danger) 18%, transparent); +} +.composer-mic[data-state='busy'] { cursor: progress; } +.composer-mic[data-state='busy'] .composer-mic__idle { display: none; } +.composer-mic[data-state='busy'] .composer-mic__busy { + display: inline-flex; + animation: composer-mic-spin 0.8s linear infinite; +} +@keyframes composer-mic-pulse { + 0%, 100% { opacity: 1; } + 50% { opacity: 0.45; } +} +@keyframes composer-mic-spin { + to { transform: rotate(360deg); } +} .composer-attach-menu, .composer-secret-pick-menu { @@ -4862,6 +5068,75 @@ code { transition: opacity 0.12s ease; } +/* Full Power selected while OpenClaw is off: swap the textarea for a message, + * but keep the toolbar (and its mode switcher) live so the user can switch. */ +.composer-exec-msg { + display: none; + align-items: center; + justify-content: center; + text-align: center; + min-height: 24px; + padding: 0 0 10px; + color: var(--muted); + font-size: 0.95em; + font-weight: 500; +} +.composer.is-exec-disabled .composer-input-stack textarea { display: none; } +.composer.is-exec-disabled .composer-exec-msg { display: flex; } + +/* Docker-required mode (Safe work) selected while Docker is off: same overlay + * treatment as the Full Power / OpenClaw case. */ +.composer-docker-msg { + display: none; + align-items: center; + justify-content: center; + flex-wrap: wrap; + gap: 6px; + text-align: center; + min-height: 24px; + padding: 0 0 10px; + color: var(--muted); + font-size: 0.95em; + font-weight: 500; +} +.composer.is-docker-disabled .composer-input-stack textarea { display: none; } +.composer.is-docker-disabled .composer-docker-msg { display: flex; } +.composer-docker-size { color: var(--muted); font-weight: 400; opacity: 0.8; } +.composer-docker-or { color: var(--muted); font-weight: 400; } + +/* Inline call-to-action button for installing Docker (Safe overlay). */ +.composer-docker-btn { + appearance: none; + border: 1px solid var(--border); + background: var(--msg-user-fill); + color: var(--text); + font: inherit; + font-weight: 600; + font-size: 0.92em; + padding: 3px 10px; + border-radius: 6px; + cursor: pointer; +} +.composer-docker-btn:hover { border-color: var(--ok-border); } +.composer-docker-btn:disabled { opacity: 0.6; cursor: default; } + +/* Mode menu item muted while its backend is unreachable: still clickable + * (selecting it surfaces the input overlay that explains why). The suffix is + * backend-specific so Execute reads "needs OpenClaw" and Safe "needs Docker". */ +.composer-mode-menu-item.is-unavailable { + opacity: 0.5; +} +.composer-mode-menu-item.is-unavailable[data-mode="execute"] .menu-item__title::after { + content: ' · needs OpenClaw'; + font-weight: 400; + color: var(--muted); +} +.composer-mode-menu-item.is-unavailable[data-requires-docker="1"] .menu-item__title::after { + content: ' · needs Docker'; + font-weight: 400; + color: var(--muted); +} + /* ---------- Message attachments (persisted user uploads) ---------- */ .msg-attachments { @@ -6944,13 +7219,6 @@ a.project-tab.project-tab--tasks:hover { border-radius: 12px; } - .draft-header .gateway { - flex-shrink: 0; - max-width: none; - white-space: nowrap; - font-size: 0.72rem; - } - .project-header { display: grid; grid-template-columns: 44px minmax(0, 1fr); @@ -7222,3 +7490,715 @@ a.project-tab.project-tab--tasks:hover { touch-action: auto; } } + +/* ── Secure workspace bar ── */ +.secure-workspace-bar { + /* Mirror .composer-secret-ui exactly so the two share identical insets — they + never show at once (refreshSecureBar hides the bar while the secret UI is + up). Absolute (not in-flow) so the bottom gap matches the secret strip's + `bottom: var(--space-4)` instead of sitting above the composer's padding. */ + position: absolute; + left: calc(20px + var(--space-2)); + right: calc(20px + var(--space-2)); + bottom: var(--space-4); + margin: 0; + padding: 0 var(--space-2); + display: flex; + align-items: center; + gap: 6px; + min-height: var(--composer-secret-row-h, 22px); + font-size: var(--text-xs); + color: var(--muted); +} +.secure-workspace-bar[hidden] { display: none; } +.secure-workspace-bar__size[hidden] { display: none; } +.secure-workspace-bar__sep { opacity: 0.4; } +.secure-workspace-bar__change { + border: none; + background: transparent; + color: var(--muted); + font: inherit; + font-size: var(--text-xs); + cursor: pointer; + text-decoration: underline; + padding: 0; +} +.secure-workspace-bar__change:hover { color: var(--text); } +.secure-workspace-bar__action { + border: none; + background: transparent; + color: var(--muted); + font: inherit; + font-size: var(--text-xs); + cursor: pointer; + text-decoration: underline; + padding: 0; + margin-left: 8px; +} +.secure-workspace-bar__action:hover { color: var(--text); } +.secure-workspace-bar__action:disabled { opacity: 0.6; cursor: default; text-decoration: none; } +.secure-workspace-bar__destroy { + border: none; + background: transparent; + color: var(--muted); + font: inherit; + font-size: var(--text-xs); + cursor: pointer; + text-decoration: underline; + padding: 0; + margin-left: 8px; +} +.secure-workspace-bar__destroy:hover { color: var(--down, #d9534f); } +.secure-workspace-bar__destroy:disabled { opacity: 0.6; cursor: default; text-decoration: none; } + +/* ── Incognito (ephemeral, read-only) ─────────────────────────────────────── */ +/* "Blacked out" identity — neutral/dark, no colour. */ + +/* Divider that sets Incognito apart at the bottom of the mode menu. */ +.composer-mode-menu-sep { + height: 1px; + margin: 6px 10px; + background: color-mix(in srgb, var(--text) 14%, transparent); +} + +/* Immersive surface: hide the chat list entirely — exit Incognito (the × at + top-left, or pick another mode) to see your chats again. */ +body.incognito-mode .app { grid-template-columns: 1fr; } +body.incognito-mode #app-sidebar { display: none; } + +/* No delete and no cloud-share while incognito. */ +body.incognito-mode #share-btn, +body.incognito-mode .delete-form { display: none !important; } + +/* Darken the conversation column (overlay only — keeps the base bg). */ +body.incognito-mode .col-main { + background-image: + radial-gradient(130% 60% at 50% 0%, rgba(0, 0, 0, 0.30), transparent 70%); +} + +/* Exit button — top-left, only while incognito. */ +.incognito-exit { + position: fixed; + top: 14px; + left: 14px; + z-index: 60; + display: none; + align-items: center; + justify-content: center; + width: 34px; + height: 34px; + padding: 0; + border-radius: 50%; + border: 1px solid color-mix(in srgb, var(--text) 18%, transparent); + background: var(--sidebar-bg, color-mix(in srgb, var(--text) 10%, transparent)); + color: var(--text); + cursor: pointer; + line-height: 0; +} +.incognito-exit:hover { background: color-mix(in srgb, var(--text) 20%, var(--sidebar-bg, transparent)); } +body.incognito-mode .incognito-exit { display: inline-flex; } +/* Make room for the × so it doesn't sit on top of the chat title. */ +body.incognito-mode .chat-header { padding-left: 58px; } + +.incognito-banner { + display: flex; + align-items: center; + gap: 10px; + width: min(var(--thread-max-width), 100%); + margin: 14px auto 6px; + padding: 10px 16px; + border-radius: 12px; + font-size: var(--text-sm); + line-height: 1.4; + color: var(--muted); + background: color-mix(in srgb, var(--text) 7%, transparent); + border: 1px solid color-mix(in srgb, var(--text) 16%, transparent); +} +.incognito-banner__icon { + flex: none; + width: 18px; + height: 18px; + color: var(--text); + opacity: 0.75; +} +.incognito-banner__dot { display: none; } /* superseded by the icon */ + +/* Composer: neutral emphasis, no colour. */ +body.incognito-mode .composer-field { + border-color: color-mix(in srgb, var(--text) 26%, var(--border, transparent)); + background: rgba(0, 0, 0, 0.10); +} +body.incognito-mode .composer-field:focus-within { + border-color: color-mix(in srgb, var(--text) 42%, transparent); +} +body.incognito-mode .composer-mode-btn, +body.incognito-mode .composer-mode-btn__label { + color: var(--text); + font-weight: 600; +} +/* Pop up above the bar, anchored to it — never reflow the page. */ +.secure-ttl-menu { + top: auto; + bottom: calc(100% + 4px); + right: calc(var(--space-2) * 2); + left: auto; + min-width: 120px; +} + +/* ---------------- project skills (procedural memory) ---------------- */ + +/* Chat inbox card — mirrors .fact-suggestions-card. */ +.msg.system.skill-suggestions-card { + align-self: stretch; + max-width: min(var(--thread-max-width), 100%); + width: 100%; + margin: 0 auto; + padding: 0; + text-align: left; + background: transparent; + border: none; + border-radius: 0; +} + +.skill-suggestions-shell { + border-radius: 14px; + border: 1px solid color-mix(in srgb, var(--border) 88%, transparent); + background: color-mix(in srgb, var(--surface) 55%, var(--chat-pane-bg)); + box-shadow: 0 1px 3px rgba(0, 0, 0, 0.05); + overflow: hidden; +} + +@media (prefers-color-scheme: dark) { + .skill-suggestions-shell { + box-shadow: 0 1px 4px rgba(0, 0, 0, 0.35); + } +} + +.skill-suggestions-lead { + margin: 0; + padding: 10px 14px 8px; + font-size: 0.8125rem; + font-weight: 500; + letter-spacing: -0.01em; + color: var(--muted); + line-height: 1.4; +} + +.skill-suggestions-list { + list-style: none; + margin: 0; + padding: 0 8px 8px; + display: flex; + flex-direction: column; + gap: 8px; +} + +.skill-suggestion-row { + display: flex; + flex-direction: column; + gap: 8px; + padding: 12px; + border-radius: 10px; + background: color-mix(in srgb, var(--surface) 40%, transparent); +} + +.skill-suggestion-head { + display: flex; + flex-wrap: wrap; + gap: 6px; + align-items: center; +} + +.skill-suggestion-badge, +.skill-scope-badge { + display: inline-flex; + align-items: center; + font-size: 0.6875rem; + font-weight: 600; + letter-spacing: 0.01em; + padding: 2px 8px; + border-radius: 999px; + line-height: 1.5; +} + +.skill-suggestion-badge--new, +.skill-scope-badge { + background: color-mix(in srgb, var(--accent, #4a8) 18%, transparent); + color: color-mix(in srgb, var(--accent, #4a8) 88%, var(--text)); +} + +.skill-suggestion-badge--patch { + background: color-mix(in srgb, #c9a227 22%, transparent); + color: color-mix(in srgb, #9a7b10 90%, var(--text)); +} + +.skill-suggestion-badge--untrusted { + background: color-mix(in srgb, var(--danger, #d35) 16%, transparent); + color: color-mix(in srgb, var(--danger, #d35) 90%, var(--text)); +} + +.skill-suggestion-name { + width: 100%; + box-sizing: border-box; + font-size: 0.9375rem; + font-weight: 600; + font-family: var(--mono-font, ui-monospace, monospace); + padding: 6px 8px; + border: 1px solid color-mix(in srgb, var(--border) 80%, transparent); + border-radius: 8px; + background: var(--chat-pane-bg); + color: var(--text); +} + +.skill-suggestion-desc, +.skill-suggestion-body { + width: 100%; + box-sizing: border-box; + font-size: 0.875rem; + line-height: 1.5; + padding: 6px 8px; + border: 1px solid color-mix(in srgb, var(--border) 80%, transparent); + border-radius: 8px; + background: var(--chat-pane-bg); + color: var(--text); + resize: vertical; +} + +.skill-suggestion-body { + font-family: var(--mono-font, ui-monospace, monospace); + font-size: 0.8125rem; +} + +.skill-suggestion-bodywrap > summary { + cursor: pointer; + font-size: 0.8125rem; + color: var(--muted); + padding: 2px 0; + user-select: none; +} + +.skill-suggestion-controls { + display: flex; + flex-wrap: wrap; + align-items: center; + justify-content: space-between; + gap: 10px; +} + +.skill-suggestion-scope { + display: inline-flex; + align-items: center; + gap: 6px; + font-size: 0.8125rem; + color: var(--muted); + cursor: pointer; +} + +.skill-suggestion-actions { + display: flex; + gap: 8px; + margin-left: auto; +} + +.skill-suggestion-btn { + font-size: 0.8125rem; + font-weight: 600; + padding: 6px 14px; + border-radius: 8px; + border: 1px solid transparent; + cursor: pointer; +} + +.skill-suggestion-accept { + background: var(--accent, #4a8); + color: #fff; +} + +.skill-suggestion-accept:disabled { + opacity: 0.6; + cursor: default; +} + +.skill-suggestion-reject { + background: transparent; + border-color: color-mix(in srgb, var(--border) 80%, transparent); + color: var(--muted); +} + +.skill-suggestion-reject:hover { + color: var(--text); +} + +/* Project page — skills list (mirrors .fact). */ +.project-chats > li.skill { + margin: 0; + padding: 12px 16px 14px; + background: transparent; +} + +.project-chats > li.skill .project-row-head { + margin-bottom: 6px; + display: flex; + align-items: center; + gap: 8px; +} + +.skill-name { + display: block; + width: 100%; + box-sizing: border-box; + margin: 0 0 6px; + border: none; + background: transparent; + padding: 2px 0; + font-size: 0.9375rem; + font-weight: 600; + font-family: var(--mono-font, ui-monospace, monospace); + color: var(--text); +} + +.skill-description { + display: block; + width: 100%; + box-sizing: border-box; + margin: 0 0 6px; + border: none; + background: transparent; + padding: 2px 0; + font-size: 0.9375rem; + line-height: 1.5; + resize: vertical; + min-height: 2.5rem; + color: var(--text); +} + +.skill-name:focus, +.skill-description:focus, +.skill-body:focus { + outline: none; +} + +.skill-body-details > summary { + cursor: pointer; + font-size: 0.8125rem; + color: var(--muted); + padding: 4px 0; + user-select: none; +} + +.skill-body { + display: block; + width: 100%; + box-sizing: border-box; + margin: 6px 0 0; + padding: 8px; + border: 1px solid color-mix(in srgb, var(--border) 80%, transparent); + border-radius: 8px; + background: var(--chat-pane-bg); + color: var(--text); + font-family: var(--mono-font, ui-monospace, monospace); + font-size: 0.8125rem; + line-height: 1.5; + resize: vertical; +} + +/* -------------------------------------------------------------------------- */ +/* First-run welcome / onboarding (views/welcome.ejs) */ +/* Apple "setup sheet" feel: one centered card, large title, calm monochrome. */ +/* -------------------------------------------------------------------------- */ + +.onb { + min-height: 100dvh; + display: flex; + align-items: center; + justify-content: center; + padding: var(--space-9); + background: var(--bg); +} + +.onb-card-shell { + position: relative; + width: 100%; + max-width: 30rem; + background: var(--surface); + border: 1px solid var(--border); + border-radius: var(--radius-xl); + box-shadow: var(--shadow-lg); + padding: var(--space-10); + text-align: center; +} + +.onb-step { animation: onb-fade 0.18s ease; } + +@keyframes onb-fade { + from { opacity: 0; transform: translateY(4px); } + to { opacity: 1; transform: none; } +} + +.onb-logo { + font-weight: 700; + letter-spacing: -0.02em; + font-size: var(--text-lg); + color: var(--muted); + margin-bottom: var(--space-7); +} + +.onb-title { + font-size: 1.6rem; + font-weight: 650; + letter-spacing: -0.02em; + margin: 0 0 var(--space-4); + color: var(--text); +} + +.onb-sub { + font-size: var(--text-base); + line-height: 1.5; + color: var(--muted); + margin: 0 auto var(--space-9); + max-width: 26rem; +} + +/* Big tappable option rows (Apple list-row style) */ +.onb-option { + display: flex; + align-items: center; + gap: var(--space-6); + width: 100%; + text-align: left; + background: var(--surface); + border: 1px solid var(--border); + border-radius: var(--radius-lg); + padding: var(--space-7); + margin-bottom: var(--space-5); + cursor: pointer; + transition: border-color 0.12s ease, background 0.12s ease, box-shadow 0.12s ease; +} +.onb-option:hover:not(:disabled) { + border-color: color-mix(in srgb, var(--text) 28%, var(--border)); + box-shadow: var(--shadow-md); +} +.onb-option:focus-visible { outline: none; box-shadow: var(--focus-ring); } + +.onb-option--disabled, +.onb-option:disabled { + cursor: default; + opacity: 0.65; +} + +.onb-option-main { display: flex; flex-direction: column; gap: var(--space-2); flex: 1; } +.onb-option-name { + font-size: var(--text-md); + font-weight: 600; + color: var(--text); + display: flex; + align-items: center; + gap: var(--space-4); +} +.onb-option-desc { font-size: var(--text-sm); line-height: 1.45; color: var(--muted); } + +.onb-option-chevron { + font-size: 1.5rem; + color: var(--muted); + line-height: 1; +} + +.onb-badge { + font-size: var(--text-xs); + font-weight: 600; + color: var(--muted); + background: color-mix(in srgb, var(--text) 7%, var(--surface)); + border: 1px solid var(--border); + border-radius: var(--radius-pill); + padding: 1px var(--space-4); +} + +.onb-skip { + display: inline-block; + margin-top: var(--space-5); + background: none; + border: none; + color: var(--muted); + font-size: var(--text-sm); + cursor: pointer; + padding: var(--space-4); +} +.onb-skip:hover { color: var(--text); text-decoration: underline; } + +.onb-back { + position: absolute; + top: var(--space-7); + left: var(--space-7); + background: none; + border: none; + color: var(--muted); + font-size: var(--text-sm); + cursor: pointer; +} +.onb-back:hover { color: var(--text); } + +/* Key step */ +.onb-steps { + text-align: left; + margin: var(--space-8) auto var(--space-9); + padding-left: 1.25rem; + max-width: 26rem; + color: var(--text); + font-size: var(--text-sm); + line-height: 1.6; +} +.onb-steps li { margin-bottom: var(--space-4); } +.onb-steps a { color: var(--info); } + +.onb-key-form { + display: flex; + gap: var(--space-4); + margin-bottom: var(--space-4); +} +.onb-key-input { + flex: 1; + font-family: var(--mono-font, ui-monospace, monospace); + font-size: var(--text-sm); + padding: var(--space-6); + border: 1px solid var(--border); + border-radius: var(--radius-md); + background: var(--bg); + color: var(--text); +} +.onb-key-input:focus-visible { outline: none; box-shadow: var(--focus-ring); border-color: color-mix(in srgb, var(--text) 28%, var(--border)); } +.onb-key-submit { white-space: nowrap; } + +.onb-key-error { color: var(--danger); font-size: var(--text-sm); margin: 0 0 var(--space-4); line-height: 1.45; } +.onb-key-error.is-warn { color: var(--warn); } +.onb-key-note { color: var(--muted); font-size: var(--text-xs); margin: 0; } + +/* Honest background-prep line */ +.onb-prep[hidden] { display: none; } +.onb-prep { + display: flex; + align-items: center; + justify-content: center; + gap: var(--space-4); + margin-top: var(--space-9); + padding-top: var(--space-7); + border-top: 1px solid var(--border); + font-size: var(--text-xs); + color: var(--muted); +} +.onb-prep-dot { + width: 7px; + height: 7px; + border-radius: var(--radius-circle); + background: var(--warn); + animation: onb-pulse 1.2s ease-in-out infinite; +} +.onb-prep-dot.is-done { + background: var(--ok); + animation: none; +} +@keyframes onb-pulse { + 0%, 100% { opacity: 0.35; } + 50% { opacity: 1; } +} + +@media (max-width: 32rem) { + .onb { padding: 0; align-items: stretch; } + .onb-card-shell { + border: none; + border-radius: 0; + box-shadow: none; + max-width: none; + padding: var(--space-10) var(--space-8); + display: flex; + flex-direction: column; + justify-content: center; + min-height: 100dvh; + } +} + +/* -------------------------------------------------------------------------- */ +/* First-chat conversational welcome (views/index.ejs draft thread) */ +/* Reads like the assistant's own first message: avatar + bubble + chips. */ +/* -------------------------------------------------------------------------- */ + +.welcome-card { + /* Overrides .empty-state (center + nowrap + narrow) — this is a left-aligned, + wrapping message bubble, not a one-line hint. */ + display: flex; + gap: var(--space-6); + align-items: flex-start; + width: 100%; + max-width: var(--thread-max-width); + box-sizing: border-box; + margin: var(--space-10) auto var(--space-9); + padding: 0 var(--space-7); + text-align: left; + white-space: normal; + animation: onb-fade 0.2s ease; +} + +.welcome-avatar { + flex: 0 0 auto; + width: 34px; + height: 34px; + border-radius: var(--radius-circle); + display: flex; + align-items: center; + justify-content: center; + font-size: var(--text-sm); + font-weight: 700; + letter-spacing: -0.02em; + color: var(--surface); + background: var(--accent); +} + +.welcome-bubble { + flex: 1; + min-width: 0; + color: var(--msg-prose, var(--text)); + font-size: var(--text-base); + line-height: 1.55; +} +.welcome-bubble p { margin: 0 0 var(--space-5); } +.welcome-greeting { font-weight: 650; font-size: var(--text-lg); color: var(--text); } +/* Extra breathing room below the lead — reads like a blank line before the list. */ +.welcome-lead { color: var(--text); margin-bottom: var(--space-9); } +.welcome-reassure { color: var(--muted); font-size: var(--text-sm); } + +.welcome-list { + list-style: disc; + margin: 0 0 var(--space-6); + padding-left: 1.25em; + display: flex; + flex-direction: column; + gap: var(--space-5); +} +.welcome-list li { + /* Same size as the lead line above it. */ + line-height: 1.5; + color: var(--text); + padding-left: 0.25em; +} + +.welcome-chips { + display: flex; + flex-wrap: wrap; + gap: var(--space-4); + margin-top: var(--space-6); +} +.welcome-chip { + border: 1px solid var(--border); + border-radius: var(--radius-pill); + background: var(--surface); + color: var(--text); + font-size: var(--text-sm); + padding: var(--space-4) var(--space-6); + cursor: pointer; + transition: border-color 0.12s ease, background 0.12s ease, box-shadow 0.12s ease; +} +.welcome-chip:hover { + border-color: color-mix(in srgb, var(--text) 28%, var(--border)); + background: var(--hover); + box-shadow: var(--shadow-sm); +} +.welcome-chip:focus-visible { outline: none; box-shadow: var(--focus-ring); } diff --git a/public/js/iclaw.js b/public/js/iclaw.js index 2bbeea9..d262c2b 100644 --- a/public/js/iclaw.js +++ b/public/js/iclaw.js @@ -42,6 +42,914 @@ const rawChatId = messagesEl?.dataset.chatId; const startedOnDraft = messagesEl?.dataset.draft === '1' || !rawChatId; let activeChatId = startedOnDraft ? null : Number(rawChatId); + + // ------------------------------------------------------------------------- + // composer mode (Ask / Execute). Mode rides along with each sent message. + // The set of selectable modes is rendered server-side from the config in + // services/chatModes.ts, so adding a mode there surfaces it here with no + // client change. Default + back-compat fallback is 'execute'. + // ------------------------------------------------------------------------- + const MODE_STORAGE_KEY = rawChatId ? `iclaw:composer-mode:${rawChatId}` : 'iclaw:composer-mode'; + const composerModesEl = document.getElementById('composer-modes'); + const composerModeBtn = document.getElementById('composer-mode-btn'); + const composerModeMenu = document.getElementById('composer-mode-menu'); + const composerModeLabel = document.getElementById('composer-mode-label'); + const composerModeDefault = + composerModesEl?.dataset.defaultMode || 'execute'; + const composerModeIds = composerModeMenu + ? Array.from(composerModeMenu.querySelectorAll('.composer-mode-menu-item')).map( + (el) => el.dataset.mode, + ) + : [composerModeDefault]; + let selectedComposerMode = composerModeDefault; + + /** Currently selected send mode (always one of the rendered, enabled ids). */ + function getComposerMode() { + return composerModeIds.includes(selectedComposerMode) + ? selectedComposerMode + : composerModeDefault; + } + + function setComposerMode(mode, opts) { + const next = composerModeIds.includes(mode) ? mode : composerModeDefault; + // Remember the mode we leave when entering Incognito, so the × can restore it. + if (next === 'incognito' && selectedComposerMode !== 'incognito') { + incognitoReturnMode = selectedComposerMode; + } + selectedComposerMode = next; + if (composerModeBtn) composerModeBtn.dataset.mode = next; + if (composerModeMenu) { + composerModeMenu.querySelectorAll('.composer-mode-menu-item').forEach((el) => { + const on = el.dataset.mode === next; + el.setAttribute('aria-checked', on ? 'true' : 'false'); + if (composerModeLabel && on) { + const t = el.querySelector('.menu-item__title'); + composerModeLabel.textContent = t ? t.textContent : next; + } + if (on) { + const desc = el.querySelector('.composer-mode-menu-item__desc'); + if (composerModeBtn && desc) composerModeBtn.title = desc.textContent || ''; + } + }); + } + // Never persist incognito as a chat's default mode — it's a transient, + // explicitly-entered surface, not a sticky preference. + if ((!opts || opts.persist !== false) && next !== 'incognito') { + try { localStorage.setItem(MODE_STORAGE_KEY, next); } catch (_) {} + } + // Incognito: tint the surface + show the "nothing saved" banner, and start a + // fresh ephemeral session each time the mode is (re)entered. + if (typeof syncIncognitoSurface === 'function') syncIncognitoSurface(next); + syncExecuteAvailability(); + if (typeof syncDockerAvailability === 'function') syncDockerAvailability(); + } + + // ── Full Power (Execute) gating ─────────────────────────────────────────── + // Execute routes to the OpenClaw gateway; the runtime modes don't. When the + // gateway is off we mute the Full Power option and, if it's the selected mode, + // cover the input with an explanation (same treatment as the drag-drop overlay) + // so the user switches mode instead of typing into a dead end. + const composerExecMsg = document.getElementById('composer-exec-msg'); + + // ── Docker gating (Safe-work sandbox) ───────────────────────────────────── + // We only block Safe work when Docker is NOT INSTALLED ('missing'). When it's + // merely stopped, the runtime starts it on demand the moment a task needs it + // (and auto-stops it when idle), so we don't block or nag. Work/Incognito are + // never blocked — file tools run on the host, run_command starts Docker lazily. + const composerDockerMsg = document.getElementById('composer-docker-msg'); + let dockerState = 'unknown'; + let dockerSizeHint = ''; + let dockerPollTimer = null; + + // Live "is the gateway usable for Full Power" flag. Seeded from the server via + // the composer form's data-gateway-ok, then kept current by applyGatewayStatus + // on every status change. + let gatewayOk = (function seedGatewayOk() { + if (form && form.dataset.gatewayOk != null) return form.dataset.gatewayOk !== '0'; + return true; // no signal → don't block + })(); + + /** True only when OpenClaw is reachable for Full Power (Execute). */ + function isExecuteAvailable() { + return gatewayOk; + } + + /** Reflect gateway availability on the Full Power option + the input overlay. */ + function syncExecuteAvailability() { + const avail = isExecuteAvailable(); + if (composerModeMenu) { + const execItem = composerModeMenu.querySelector( + '.composer-mode-menu-item[data-mode="execute"]', + ); + if (execItem) execItem.classList.toggle('is-unavailable', !avail); + } + const blocked = !avail && getComposerMode() === 'execute'; + if (form) form.classList.toggle('is-exec-disabled', blocked); + if (composerExecMsg) { + composerExecMsg.setAttribute('aria-hidden', blocked ? 'false' : 'true'); + } + refreshComposerInputDisabled(); + } + + /** + * The composer input is disabled while ANY blocking overlay is up — Full + * Power with OpenClaw off, or a Docker-required mode with Docker off. Both + * overlays cover the textarea, so the send target would be hidden anyway; + * disabling input + send keeps keyboard submit from firing into nothing. + */ + function refreshComposerInputDisabled() { + const blocked = !!( + form && + (form.classList.contains('is-exec-disabled') || + form.classList.contains('is-docker-disabled')) + ); + if (input) input.disabled = blocked; + const sb = document.getElementById('composer-send-btn'); + if (sb) sb.disabled = blocked; + } + + /** True when the given mode can't run without a Docker daemon (Safe work). */ + function modeRequiresDocker(mode) { + if (!composerModeMenu) return false; + const item = composerModeMenu.querySelector( + '.composer-mode-menu-item[data-mode="' + mode + '"]', + ); + return !!item && item.dataset.requiresDocker === '1'; + } + + /** + * Reflect Docker state. We only BLOCK when Docker is genuinely unusable — + * i.e. not installed ('missing') — and only for a mode that can't run without + * it (Safe work). When Docker is merely stopped we don't block or nag: the + * runtime starts it on demand the moment a task needs it (and auto-stops it + * when idle). Work/Incognito are never blocked — their file tools run on the + * host and run_command starts Docker lazily. No-op until the first poll. + */ + function syncDockerAvailability() { + const missing = dockerState === 'missing'; + if (composerModeMenu) { + composerModeMenu.querySelectorAll('.composer-mode-menu-item').forEach((el) => { + if (el.dataset.requiresDocker === '1') { + el.classList.toggle('is-unavailable', missing); + } + }); + } + const mode = getComposerMode(); + const blocked = missing && modeRequiresDocker(mode); + if (form) form.classList.toggle('is-docker-disabled', blocked); + if (composerDockerMsg) { + composerDockerMsg.setAttribute('aria-hidden', blocked ? 'false' : 'true'); + } + applyDockerActionLabels(); + refreshComposerInputDisabled(); + } + + /** Button copy reflects whether Docker is missing (Install) or just idle (Start). */ + function applyDockerActionLabels() { + const installing = dockerState === 'installing'; + const starting = dockerState === 'starting'; + const needsInstall = dockerState === 'missing'; + let label = needsInstall ? 'Install Docker' : 'Start Docker'; + if (installing) label = 'Installing Docker…'; + else if (starting) label = 'Starting Docker…'; + const busy = installing || starting; + + const mainBtn = document.getElementById('composer-docker-action'); + if (mainBtn) { + mainBtn.textContent = label; + mainBtn.disabled = busy; + } + const sizeEl = document.getElementById('composer-docker-size'); + if (sizeEl) sizeEl.textContent = needsInstall && !busy && dockerSizeHint ? dockerSizeHint : ''; + } + + function applyDockerState(next, sizeHint) { + dockerState = next || 'unknown'; + if (typeof sizeHint === 'string' && sizeHint) dockerSizeHint = sizeHint; + syncDockerAvailability(); + } + + function pollDockerStatus(opts) { + fetch('/api/docker/status', { headers: { Accept: 'application/json' } }) + .then((r) => (r.ok ? r.json() : null)) + .then((data) => { + if (!data) return; + applyDockerState(data.state, data.sizeHint); + // Keep polling only while a start/install is in flight; otherwise the + // state is settled and the next change is user-driven (button click). + if (dockerState === 'installing' || dockerState === 'starting') { + dockerPollTimer = setTimeout(() => pollDockerStatus(opts), 2500); + } + }) + .catch(() => { + if (opts && opts.keepAlive) dockerPollTimer = setTimeout(() => pollDockerStatus(opts), 4000); + }); + } + + function triggerDockerAction() { + // Missing → install (which also starts); installed-but-idle → just start. + const endpoint = dockerState === 'missing' ? '/api/docker/install' : '/api/docker/start'; + // Optimistic transient state so the button shows progress immediately. + applyDockerState(dockerState === 'missing' ? 'installing' : 'starting', dockerSizeHint); + fetch(endpoint, { method: 'POST', headers: { Accept: 'application/json' } }) + .then((r) => (r.ok ? r.json() : null)) + .then((data) => { + if (data && data.state) applyDockerState(data.state, dockerSizeHint); + if (dockerPollTimer) clearTimeout(dockerPollTimer); + pollDockerStatus({ keepAlive: true }); + }) + .catch(() => { + // Endpoint failed (e.g. non-localhost) — re-probe so the UI is honest. + if (dockerPollTimer) clearTimeout(dockerPollTimer); + pollDockerStatus({}); + }); + } + + (function initDockerGate() { + const mainBtn = document.getElementById('composer-docker-action'); + if (mainBtn) mainBtn.addEventListener('click', triggerDockerAction); + syncDockerAvailability(); + // First real status read; thereafter polling only runs during an action. + pollDockerStatus({}); + })(); + + // ── Incognito (ephemeral, never persisted) ──────────────────────────────── + // The conversation lives only in this tab: messages render locally, the turn + // streams over `incognito-*` WS events keyed by `activeIncognitoKey`, and + // nothing is written to the DB. Reloading the page clears it. + let activeIncognitoKey = null; + let incognitoReturnMode = composerModeDefault; + + function newIncognitoKey() { + return 'inc-' + Date.now().toString(36) + '-' + Math.random().toString(36).slice(2, 8); + } + + /** + * Leave Incognito → return to wherever we came from (that chat + the sidebar + * come back). Incognito ran on a fresh ephemeral surface, so we just navigate + * back; the ephemeral conversation is discarded. + */ + function exitIncognito() { + let origin = '/'; + try { + origin = sessionStorage.getItem('iclaw:incognito-origin') || '/'; + sessionStorage.removeItem('iclaw:incognito-origin'); + } catch (_) {} + window.location.assign(origin); + } + + /** The fixed top-left × shown only in incognito. Created once, CSS toggles it. */ + function ensureIncognitoExitButton() { + if (document.getElementById('incognito-exit')) return; + const btn = document.createElement('button'); + btn.id = 'incognito-exit'; + btn.type = 'button'; + btn.className = 'incognito-exit'; + btn.setAttribute('aria-label', 'Exit Incognito'); + btn.title = 'Exit Incognito'; + btn.innerHTML = + '<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" ' + + 'stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">' + + '<path d="M18 6 6 18"/><path d="m6 6 12 12"/></svg>'; + btn.addEventListener('click', exitIncognito); + document.body.appendChild(btn); + } + + function syncIncognitoSurface(mode) { + const on = mode === 'incognito'; + document.body.classList.toggle('incognito-mode', on); + if (on) ensureIncognitoExitButton(); + let banner = document.getElementById('incognito-banner'); + if (on) { + if (!banner) { + banner = document.createElement('div'); + banner.id = 'incognito-banner'; + banner.className = 'incognito-banner'; + banner.innerHTML = + '<svg class="incognito-banner__icon" viewBox="0 0 24 24" fill="none" ' + + 'stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">' + + '<path d="M2 12s3.5-7 10-7 10 7 10 7"/><path d="m4 4 16 16"/>' + + '<path d="M9.9 9.9a3 3 0 0 0 4.2 4.2"/></svg>' + + '<span>Incognito — read-only research. Your chat list is hidden, and this ' + + 'conversation isn’t saved or added to project memory.</span>'; + const root = typeof messagesAppendRoot === 'function' ? messagesAppendRoot() : messagesEl; + (root || messagesEl)?.prepend(banner); + } + // Fresh session whenever incognito is (re)selected. + activeIncognitoKey = null; + } else if (banner) { + banner.remove(); + } + } + + /** + * Finalize the streamed incognito reply (no message-appended for ephemeral + * turns): render the full markdown, strip the streaming chrome, and release + * the composer. Shared by the `incognito-turn-ended` event and the stop button. + */ + function finalizeIncognitoStream() { + setStopVisible(false); + cancelStreamRender(); + if (currentStreamEl) { + const body = currentStreamEl.querySelector('.stream-body, .msg-body'); + if (body && currentStreamFullText.trim()) { + body.classList.remove('stream-body'); + body.innerHTML = renderMarkdown(currentStreamFullText); + decorateMessageBody(body); + currentStreamEl.classList.remove('streaming', 'stream-waiting', 'stream-tool', 'stream-generating'); + const st = currentStreamEl.querySelector('.stream-status'); + if (st) { stopStreamStatusDotAnim(st); st.remove(); } + } else { + currentStreamEl.remove(); + } + currentStreamEl = null; + } + currentStreamFullText = ''; + streamShownLen = 0; + if (inFlight) { inFlight = false; if (waitingItems[0]) flushNextQueued(); } + } + + function closeComposerModeMenu() { + if (!composerModeMenu) return; + composerModeMenu.hidden = true; + if (composerModeBtn) composerModeBtn.setAttribute('aria-expanded', 'false'); + } + + if (composerModeBtn && composerModeMenu) { + // Initial mode precedence: + // 1. per-chat remembered choice (localStorage, existing chats only) + // 2. the mode this chat was last used in (server-derived, data-chat-mode) + // 3. the UI default (Work) — for new chats we ignore any stale GLOBAL + // localStorage value so a fresh chat always starts on the default. + let stored = null; + if (rawChatId) { + try { stored = localStorage.getItem(MODE_STORAGE_KEY); } catch (_) {} + } + const chatMode = composerModesEl ? composerModesEl.dataset.chatMode : ''; + setComposerMode(stored || chatMode || composerModeDefault, { persist: false }); + // Entered via "Incognito" elsewhere → ?mode=incognito on a fresh surface. + try { + const _qp = new URLSearchParams(window.location.search); + if (_qp.get('mode') === 'incognito' && composerModeIds.includes('incognito')) { + setComposerMode('incognito', { persist: false }); + window.history.replaceState({}, '', window.location.pathname); // tidy the URL + } + } catch (_) {} + + composerModeBtn.addEventListener('click', (e) => { + e.stopPropagation(); + const open = composerModeMenu.hidden; + composerModeMenu.hidden = !open; + composerModeBtn.setAttribute('aria-expanded', open ? 'true' : 'false'); + }); + const MODE_PLACEHOLDERS = { + work: 'Work inside selected folders', + secure: 'Run risky tasks in isolation', + incognito: 'Private read-only research — nothing saved', + execute: 'Use full iClaw power', + }; + + function updateComposerPlaceholder(mode) { + if (input) input.placeholder = MODE_PLACEHOLDERS[mode] || 'Ask anything'; + } + + composerModeMenu.addEventListener('click', (e) => { + const item = e.target.closest('.composer-mode-menu-item'); + if (!item) return; + // Incognito is a separate ephemeral surface, not a flag on the current + // chat: open a fresh blank chat instead of converting this one. Remember + // where we came from so the × can bring us back. + if (item.dataset.mode === 'incognito' && !document.body.classList.contains('incognito-mode')) { + try { sessionStorage.setItem('iclaw:incognito-origin', window.location.pathname + window.location.search); } catch (_) {} + window.location.assign('/?mode=incognito'); + return; + } + setComposerMode(item.dataset.mode); + closeComposerModeMenu(); + updateComposerPlaceholder(item.dataset.mode); + if (typeof updateWorkFoldersButton === 'function') updateWorkFoldersButton(); + input?.focus(); + }); + + // Set placeholder on initial load + updateComposerPlaceholder(getComposerMode()); + document.addEventListener('click', (e) => { + if (composerModeMenu.hidden) return; + if (composerModesEl && !composerModesEl.contains(e.target)) closeComposerModeMenu(); + }); + document.addEventListener('keydown', (e) => { + if (e.key === 'Escape' && !composerModeMenu.hidden) closeComposerModeMenu(); + }); + } + + // ------------------------------------------------------------------------- + // Work Mode — folders picker + // ------------------------------------------------------------------------- + const workFoldersBtn = document.getElementById('composer-work-folders-btn'); + const workFoldersModal = document.getElementById('work-folders-modal'); + const workFoldersList = document.getElementById('work-folders-list'); + const workFoldersInput = document.getElementById('work-folders-input'); + const workFoldersAddBtn = document.getElementById('work-folders-add-btn'); + const workFoldersBrowseBtn = document.getElementById('work-folders-browse-btn'); + const workFoldersClose = document.getElementById('work-folders-close'); + const workFoldersBackdrop = document.getElementById('work-folders-backdrop'); + const workFoldersCount = document.getElementById('composer-work-folders-count'); + + function workFoldersKey() { + const pid = messagesEl?.dataset.projectId; + if (pid) return `iclaw:work-folders:project:${pid}`; + return 'iclaw:work-folders:no-project'; + } + + // Folders are stored as { path, write }. Older clients stored bare path + // strings — migrate those to writable (their effective behavior at the time) + // so upgrading doesn't silently revoke access on existing folders. Newly + // added folders default to read-only (see addFolder / browse below). + function getWorkFolders() { + let raw; + try { raw = JSON.parse(localStorage.getItem(workFoldersKey()) || '[]'); } + catch { return []; } + if (!Array.isArray(raw)) return []; + return raw + .map((f) => (typeof f === 'string' + ? { path: f, write: true } + : (f && typeof f === 'object' && typeof f.path === 'string' + ? { path: f.path, write: f.write === true } + : null))) + .filter(Boolean); + } + + function saveWorkFolders(folders) { + try { localStorage.setItem(workFoldersKey(), JSON.stringify(folders)); } catch {} + } + + function escAttr(s) { + return String(s).replace(/&/g, '&').replace(/"/g, '"').replace(/</g, '<').replace(/>/g, '>'); + } + + function renderWorkFoldersList() { + if (!workFoldersList) return; + const folders = getWorkFolders(); + workFoldersList.innerHTML = ''; + for (const f of folders) { + const p = escAttr(f.path); + const li = document.createElement('li'); + li.className = 'work-folders-list__item'; + const label = f.write ? 'Read & write' : 'Read-only'; + li.innerHTML = + `<span class="work-folders-list__path" title="${p}">${p}</span>` + + `<button type="button" class="work-folders-list__access" data-path="${p}" data-write="${f.write ? '1' : '0'}" ` + + `title="Click to toggle access">${label}</button>` + + `<button type="button" class="work-folders-list__remove" data-path="${p}" aria-label="Remove">×</button>`; + workFoldersList.appendChild(li); + } + if (workFoldersCount) { + workFoldersCount.textContent = folders.length > 0 ? String(folders.length) : ''; + } + } + + // ── Network toggle (Secure Mode) ────────────────────────────────────────── + const networkToggleBtn = document.getElementById('composer-network-toggle-btn'); + + function networkKey() { + const pid = messagesEl?.dataset.projectId; + return pid ? `iclaw:secure-network:project:${pid}` : 'iclaw:secure-network:no-project'; + } + + function getNetworkEnabled() { + try { return localStorage.getItem(networkKey()) === 'on'; } catch { return false; } + } + + function setNetworkEnabledStorage(on) { + try { localStorage.setItem(networkKey(), on ? 'on' : 'off'); } catch {} + } + + function updateNetworkToggle() { + if (!networkToggleBtn) return; + const mode = getComposerMode(); + networkToggleBtn.hidden = (mode !== 'secure'); + const on = getNetworkEnabled(); + networkToggleBtn.dataset.network = on ? 'on' : 'off'; + networkToggleBtn.title = on ? 'Network is ON - click to disable' : 'Network is OFF - click to enable'; + networkToggleBtn.style.color = on ? 'var(--accent)' : ''; + } + + networkToggleBtn?.addEventListener('click', () => { + const on = !getNetworkEnabled(); + setNetworkEnabledStorage(on); + updateNetworkToggle(); + }); + + function updateWorkFoldersButton() { + if (!workFoldersBtn) return; + const mode = getComposerMode(); + // Incognito also uses folders — as read-only roots for its sandboxed shell. + workFoldersBtn.hidden = (mode !== 'work' && mode !== 'incognito'); + renderWorkFoldersList(); + updateNetworkToggle(); + } + + if (workFoldersBtn && workFoldersModal) { + workFoldersBtn.addEventListener('click', () => { + renderWorkFoldersList(); + workFoldersModal.hidden = false; + workFoldersInput?.focus(); + }); + + workFoldersList?.addEventListener('click', (e) => { + const removeBtn = e.target.closest('.work-folders-list__remove'); + if (removeBtn) { + const path = removeBtn.dataset.path; + saveWorkFolders(getWorkFolders().filter((f) => f.path !== path)); + renderWorkFoldersList(); + return; + } + const accessBtn = e.target.closest('.work-folders-list__access'); + if (accessBtn) { + const path = accessBtn.dataset.path; + const folders = getWorkFolders().map((f) => + f.path === path ? { ...f, write: !f.write } : f); + saveWorkFolders(folders); + renderWorkFoldersList(); + } + }); + + function addFolderPath(val) { + if (!val) return; + const folders = getWorkFolders(); + if (!folders.some((f) => f.path === val)) { + // New folders default to read-only; user opts into write explicitly. + folders.push({ path: val, write: false }); + saveWorkFolders(folders); + renderWorkFoldersList(); + } + } + + function addFolder() { + addFolderPath(workFoldersInput?.value.trim()); + if (workFoldersInput) workFoldersInput.value = ''; + } + + workFoldersAddBtn?.addEventListener('click', addFolder); + workFoldersInput?.addEventListener('keydown', (e) => { + if (e.key === 'Enter') { e.preventDefault(); addFolder(); } + }); + + workFoldersBrowseBtn?.addEventListener('click', async () => { + workFoldersBrowseBtn.disabled = true; + try { + const res = await fetch('/api/pick-folder', { method: 'POST' }); + if (res.status === 204) return; // user cancelled + const data = await res.json(); + if (data.path) addFolderPath(data.path); + } catch (e) { + console.error('pick-folder failed', e); + } finally { + workFoldersBrowseBtn.disabled = false; + } + }); + + const closeModal = () => { workFoldersModal.hidden = true; }; + workFoldersClose?.addEventListener('click', closeModal); + workFoldersBackdrop?.addEventListener('click', closeModal); + document.addEventListener('keydown', (e) => { + if (e.key === 'Escape' && !workFoldersModal.hidden) closeModal(); + }); + } + + updateWorkFoldersButton(); + + // ── Secure workspace bar ───────────────────────────────────────────────── + const secureBar = document.getElementById('secure-workspace-bar'); + const secureSizeEl = document.getElementById('secure-workspace-size'); + const secureTtlEl = document.getElementById('secure-workspace-ttl'); + const secureChangeBtn = document.getElementById('secure-workspace-change'); + const secureTtlMenu = document.getElementById('secure-ttl-menu'); + + function secureTtlKey() { + const pid = messagesEl?.dataset.projectId; + return `iclaw:secure-ttl:${pid ? 'project:' + pid : 'chat:' + rawChatId}`; + } + + function getSecureTtl() { + try { + const raw = localStorage.getItem(secureTtlKey()); + return raw ? JSON.parse(raw) : { ttlDays: 7, lastActivity: Date.now() }; + } catch { return { ttlDays: 7, lastActivity: Date.now() }; } + } + + function saveSecureTtl(ttlDays) { + try { + localStorage.setItem(secureTtlKey(), JSON.stringify({ ttlDays, lastActivity: Date.now() })); + } catch {} + } + + function formatBytes(bytes) { + if (bytes < 1024) return bytes + ' B'; + if (bytes < 1024 * 1024) return (bytes / 1024).toFixed(1) + ' KB'; + return (bytes / (1024 * 1024)).toFixed(1) + ' MB'; + } + + function formatTtlRemaining(ttlDays, lastActivity) { + if (ttlDays === 0) return 'never expires'; + // +15s demo buffer: holds the full day count for ~15s after each reset so + // users can see the countdown tick down (proof the TTL resets on activity). + const expiresAt = lastActivity + ttlDays * 86400_000 + 15_000; + const remaining = expiresAt - Date.now(); + if (remaining <= 0) return 'expired'; + const days = Math.floor(remaining / 86400_000); + const hours = Math.floor((remaining % 86400_000) / 3600_000); + const mins = Math.floor((remaining % 3600_000) / 60_000); + const secs = Math.floor((remaining % 60_000) / 1000); + // More than 2 days: show only days (no hours) + if (days > 2) return `expires in ${days}d`; + if (days > 0) return `expires in ${days}d ${hours}h`; + if (hours > 0) return `expires in ${hours}h ${mins}m`; + if (mins > 0) return `expires in ${mins}m ${secs}s`; + return `expires in ${secs}s`; + } + + // True when the secret hint UI is visible — it takes priority over the secure bar. + function secretUiVisible() { + const el = document.getElementById('composer-secret-ui'); + return el && !el.hidden; + } + + // Update only the TTL text (cheap, local — ticks every second). + function tickSecureTtl() { + if (!secureBar || secureBar.hidden) return; + const ttl = getSecureTtl(); + if (secureTtlEl) secureTtlEl.textContent = formatTtlRemaining(ttl.ttlDays, ttl.lastActivity); + } + + async function refreshSecureBar() { + if (!secureBar) return; + const mode = getComposerMode(); + // Hide if not in Secure Mode, or if the secret UI is currently showing. + if (mode !== 'secure' || secretUiVisible() || !rawChatId) { secureBar.hidden = true; return; } + + try { + const res = await fetch(`/chats/${rawChatId}/workspace-info`); + const data = await res.json(); + // Only surface the bar once a secure session actually exists — i.e. after + // the first message. Nothing is created on the host until then. + if (!data.active) { secureBar.hidden = true; return; } + secureBar.hidden = false; + tickSecureTtl(); + if (secureSizeEl) { + // Only show a size once there's actually something in the workspace — + // "0 B" is noise. + const size = data.workspaceSize != null ? data.workspaceSize : 0; + if (size > 0) { + secureSizeEl.textContent = formatBytes(size); + secureSizeEl.hidden = false; + } else { + secureSizeEl.textContent = ''; + secureSizeEl.hidden = true; + } + } + } catch { secureBar.hidden = true; } + } + + // Live countdown — updates the TTL text every second. + setInterval(tickSecureTtl, 1000); + + secureChangeBtn?.addEventListener('click', (e) => { + e.stopPropagation(); + if (secureTtlMenu) { + secureTtlMenu.hidden = !secureTtlMenu.hidden; + } + }); + + secureTtlMenu?.addEventListener('click', (e) => { + const item = e.target.closest('[data-ttl]'); + if (!item) return; + saveSecureTtl(Number(item.dataset.ttl)); + secureTtlMenu.hidden = true; + refreshSecureBar(); + }); + + // Export the sandbox out to a host folder (default ~/Downloads). Read-only — + // copies the sandbox contents to a fresh place, touches nothing of the user's. + const secureExportBtn = document.getElementById('secure-workspace-export'); + secureExportBtn?.addEventListener('click', async (e) => { + e.stopPropagation(); + if (!rawChatId) return; + secureExportBtn.disabled = true; + const prev = secureExportBtn.textContent; + secureExportBtn.textContent = 'Exporting…'; + try { + const res = await fetch(`/chats/${rawChatId}/export-sandbox`, { + method: 'POST', + headers: { 'Content-Type': 'application/json', Accept: 'application/json' }, + body: '{}', + }); + const data = await res.json(); + if (data && data.ok) alert(`Exported ${data.files != null ? data.files + ' files' : 'sandbox'} to:\n${data.path}`); + else alert(`Export failed: ${(data && data.error) || 'unknown error'}`); + } catch { alert('Export failed.'); } + secureExportBtn.disabled = false; + secureExportBtn.textContent = prev || 'Export'; + }); + + // Apply: copy the sandbox's new/changed files back to the ORIGINAL folders. + // This writes to the user's real files, so it's confirmed and additive only. + const secureApplyBtn = document.getElementById('secure-workspace-apply'); + secureApplyBtn?.addEventListener('click', async (e) => { + e.stopPropagation(); + if (!rawChatId) return; + if (!confirm('Apply the sandbox\'s new and changed files back to your original folders? Existing files may be overwritten; nothing is deleted.')) return; + secureApplyBtn.disabled = true; + const prev = secureApplyBtn.textContent; + secureApplyBtn.textContent = 'Applying…'; + try { + const res = await fetch(`/chats/${rawChatId}/apply-sandbox`, { + method: 'POST', + headers: { Accept: 'application/json' }, + }); + const data = await res.json(); + const results = (data && data.results) || []; + if (results.length === 0) { + alert('Nothing to apply — no folders were copied into this sandbox.'); + } else { + const lines = results.map((r) => r.ok + ? `• ${r.source}: ${(r.applied && r.applied.length) || 0} file(s)` + : `✗ ${r.source}: ${r.error}`); + alert('Applied changes:\n' + lines.join('\n')); + } + } catch { alert('Apply failed.'); } + secureApplyBtn.disabled = false; + secureApplyBtn.textContent = prev || 'Apply changes'; + }); + + // Destroy the sandbox: deletes the copied workspace + container. The next + // message starts a fresh sandbox (re-copying any selected folders). + const secureDestroyBtn = document.getElementById('secure-workspace-destroy'); + secureDestroyBtn?.addEventListener('click', async (e) => { + e.stopPropagation(); + if (!rawChatId) return; + if (!confirm('Destroy this sandbox? The copied files and anything created in it are deleted. Your original files are untouched.')) return; + secureDestroyBtn.disabled = true; + const prev = secureDestroyBtn.textContent; + secureDestroyBtn.textContent = 'Destroying…'; + try { + await fetch(`/chats/${rawChatId}/destroy-workspace`, { + method: 'POST', + headers: { Accept: 'application/json' }, + }); + } catch { /* best-effort */ } + secureDestroyBtn.disabled = false; + secureDestroyBtn.textContent = prev || 'Destroy'; + refreshSecureBar(); + }); + + document.addEventListener('click', (e) => { + if (secureTtlMenu && !secureTtlMenu.hidden && !secureChangeBtn?.contains(e.target) && !secureTtlMenu.contains(e.target)) { + secureTtlMenu.hidden = true; + } + }); + + // Refresh bar when mode changes + composerModeMenu?.addEventListener('click', () => setTimeout(refreshSecureBar, 50)); + refreshSecureBar(); + + // ------------------------------------------------------------------------- + // Speech-to-text (mic). Records via MediaRecorder, POSTs the clip to + // /api/stt, and inserts the returned transcript into the composer textarea. + // Only wired when the server rendered the button (OPENROUTER_API_KEY set). + // Click to start, click again to stop. + // ------------------------------------------------------------------------- + const micBtn = document.getElementById('composer-mic-btn'); + if (micBtn && navigator.mediaDevices && window.MediaRecorder) { + let mediaRecorder = null; + let mediaStream = null; + let micChunks = []; + let micRecording = false; + + function setMicState(state) { + micBtn.dataset.state = state; + micBtn.setAttribute('aria-pressed', state === 'recording' ? 'true' : 'false'); + micBtn.title = + state === 'recording' + ? 'Stop recording' + : state === 'busy' + ? 'Transcribing…' + : 'Record voice'; + } + + function pickMicMime() { + if (!window.MediaRecorder || !MediaRecorder.isTypeSupported) return ''; + const cands = [ + 'audio/webm;codecs=opus', + 'audio/webm', + 'audio/ogg;codecs=opus', + 'audio/mp4', + 'audio/mpeg', + ]; + return cands.find((t) => MediaRecorder.isTypeSupported(t)) || ''; + } + + function stopMicStream() { + if (mediaStream) { + mediaStream.getTracks().forEach((t) => t.stop()); + mediaStream = null; + } + } + + function insertTranscript(text) { + if (!input || !text) return; + const cur = input.value || ''; + const sep = cur && !/\s$/.test(cur) ? ' ' : ''; + input.value = cur + sep + text; + input.dispatchEvent(new Event('input', { bubbles: true })); + input.focus(); + try { + input.setSelectionRange(input.value.length, input.value.length); + } catch (_) {} + } + + async function transcribeBlob(blob) { + const res = await fetch('/api/stt', { + method: 'POST', + headers: { 'Content-Type': blob.type || 'audio/webm' }, + body: blob, + }); + if (!res.ok) { + let msg = 'HTTP ' + res.status; + try { + const j = await res.json(); + if (j && j.error) msg = j.error; + } catch (_) {} + throw new Error(msg); + } + const j = await res.json(); + return j && typeof j.text === 'string' ? j.text.trim() : ''; + } + + async function onMicStop() { + stopMicStream(); + const type = (mediaRecorder && mediaRecorder.mimeType) || 'audio/webm'; + const blob = new Blob(micChunks, { type }); + micChunks = []; + if (!blob.size) { + setMicState('idle'); + return; + } + setMicState('busy'); + try { + const text = await transcribeBlob(blob); + if (text) insertTranscript(text); + } catch (err) { + window.alert( + 'Transcription failed: ' + (err && err.message ? err.message : 'unknown error'), + ); + } finally { + setMicState('idle'); + } + } + + async function startMicRecording() { + try { + mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true }); + } catch (_) { + window.alert('Microphone access was blocked. Allow it in your browser to use voice input.'); + return; + } + micChunks = []; + const mimeType = pickMicMime(); + try { + mediaRecorder = mimeType + ? new MediaRecorder(mediaStream, { mimeType }) + : new MediaRecorder(mediaStream); + } catch (_) { + mediaRecorder = new MediaRecorder(mediaStream); + } + mediaRecorder.addEventListener('dataavailable', (e) => { + if (e.data && e.data.size) micChunks.push(e.data); + }); + mediaRecorder.addEventListener('stop', onMicStop); + mediaRecorder.start(); + micRecording = true; + setMicState('recording'); + } + + function stopMicRecording() { + micRecording = false; + if (mediaRecorder && mediaRecorder.state !== 'inactive') { + try { + mediaRecorder.stop(); + } catch (_) {} + } + } + + micBtn.addEventListener('click', (e) => { + e.preventDefault(); + if (micBtn.dataset.state === 'busy') return; + if (micRecording) stopMicRecording(); + else startMicRecording(); + }); + } + /** Clears timed reply-quote highlights in the transcript. */ let replyJumpHighlightTimer = null; let replyJumpHighlightFadeTimer = null; @@ -129,8 +1037,14 @@ } else { messagesEl.dataset.projectId = ''; } + if (typeof updateWorkFoldersButton === 'function') updateWorkFoldersButton(); } history.replaceState(null, '', '/chats/' + id); + // Migrate mode from the draft fallback key to the per-chat key + try { + const draftMode = localStorage.getItem('iclaw:composer-mode'); + if (draftMode) localStorage.setItem(`iclaw:composer-mode:${id}`, draftMode); + } catch (_) {} applyTitleForActive(payload.title || 'New chat'); if (ws && ws.readyState === WebSocket.OPEN) { wsSend({ type: 'subscribe', chatId: id }); @@ -334,9 +1248,35 @@ : initSel.replace(/\\/g, '\\\\').replace(/"/g, '\\"'); const card = projectPickEl.querySelector('.project-pick-card[data-project-id="' + esc + '"]'); if (card) queueMicrotask(() => void commitDraftFromCard(card)); + } else if (draftBody?.dataset.autoNone === '1') { + // First-ever chat: skip the "Choose a project" step so a new user lands + // straight on the welcome greeting + composer (No project). + const card = projectPickEl.querySelector('.project-pick-card--none'); + if (card) queueMicrotask(() => void commitDraftFromCard(card)); } } + // Welcome-card suggestion chips: drop the text into the composer and send it, + // so a non-technical user gets going with one click. + function initWelcomeChips() { + const card = document.getElementById('welcome-card'); + if (!card) return; + card.addEventListener('click', (e) => { + const chip = e.target.closest('.welcome-chip'); + if (!chip) return; + const prompt = chip.getAttribute('data-prompt') || chip.textContent || ''; + const ta = document.getElementById('composer-input'); + const form = document.getElementById('send-form'); + if (!ta || !form) return; + ta.value = prompt; + ta.dispatchEvent(new Event('input', { bubbles: true })); + if (typeof form.requestSubmit === 'function') form.requestSubmit(); + else form.dispatchEvent(new Event('submit', { cancelable: true, bubbles: true })); + }); + } + + initWelcomeChips(); + initDraftProjectPick(); // serializes turns per chat too, so this is just for the visible label const waitingItems = []; @@ -379,6 +1319,10 @@ /** the assistant DOM node we're streaming into right now */ let currentStreamEl = null; let currentStreamFullText = ''; + /** Pending requestAnimationFrame id for the streaming typewriter (0 = none). */ + let streamRenderRaf = 0; + /** How many chars of currentStreamFullText the typewriter has revealed. */ + let streamShownLen = 0; /** Debounce hljs while `turn-delta` re-renders markdown (innerHTML each chunk). */ let streamSyntaxHlTimer = null; @@ -629,6 +1573,63 @@ refreshProjectTabLabels(active || 'chats'); } + function syncProjectSkillsTabCountFromDom() { + const root = document.querySelector('main.project-page[data-project-id]'); + const btn = root?.querySelector('[data-project-tab="skills"]'); + const ul = document.getElementById('skills-list'); + if (!btn || !ul) return; + btn.setAttribute('data-tab-count', String(ul.querySelectorAll('li.skill').length)); + const active = root.querySelector('.project-tab.is-active')?.getAttribute('data-project-tab'); + refreshProjectTabLabels(active || 'chats'); + } + + /** Build a skill row matching `views/project.ejs` (WS-driven updates on project page). */ + function buildSkillLi(s) { + const li = document.createElement('li'); + li.className = 'skill'; + li.dataset.skillId = String(s.id); + li.dataset.skillVersion = String(s.version != null ? s.version : 1); + const isGlobal = s.project_id == null; + const titleRaw = + s.source_chat_title != null && String(s.source_chat_title).trim() !== '' + ? String(s.source_chat_title).trim() + : 'Chat'; + const head = + '<div class="project-row-head muted">' + + (isGlobal + ? '<span class="skill-scope-badge" title="Available to every project">Global</span>' + : '') + + (s.source_chat_id != null + ? '<a href="/chats/' + + s.source_chat_id + + '" class="project-chat-source">' + + escapeHtml(titleRaw) + + '</a>' + : '<span>—</span>') + + '</div>'; + li.innerHTML = + head + + '<input class="skill-name" aria-label="Skill name" spellcheck="false" value="' + + escapeHtml(s.name || '') + + '" />' + + '<textarea class="skill-description" aria-label="Skill summary" rows="2">' + + escapeHtml(s.description || '') + + '</textarea>' + + '<details class="skill-body-details"><summary>View / edit procedure</summary>' + + '<textarea class="skill-body" aria-label="Skill procedure (SKILL.md)" rows="10">' + + escapeHtml(s.body || '') + + '</textarea></details>' + + '<div class="fact-meta">' + + '<button type="button" class="fact-delete skill-delete" aria-label="Remove skill">Remove</button></div>'; + const nameEl = li.querySelector('.skill-name'); + if (nameEl) nameEl.dataset.saved = String(s.name || '').trim(); + const descEl = li.querySelector('.skill-description'); + if (descEl) descEl.dataset.saved = String(s.description || '').trim(); + const bodyEl = li.querySelector('.skill-body'); + if (bodyEl) bodyEl.dataset.saved = String(s.body || '').trim(); + return li; + } + /** Build a secrets row matching `views/project.ejs` (WS-driven updates on project page). */ function buildProjectSecretRowLi(secret) { const label = String(secret?.label ?? ''); @@ -746,10 +1747,26 @@ if (!pre || pre.tagName !== 'PRE') return; if (pre.closest('.exec-approval-card')) return; if (code.classList.contains('hljs')) return; - try { - hl.highlightElement(code); - } catch (_) { - /* unknown language / empty */ + // Only highlight when the fence declares a language hljs actually knows. + // On a bare/unknown fence, highlightElement() falls back to auto-detection, + // which mis-guesses prose as Ruby/Perl (an apostrophe opens a "string" and + // the github-dark theme dims whole paragraphs). Leave those as plain text. + const langClass = [...code.classList].find((c) => c.startsWith('language-')); + const lang = langClass && langClass.slice(9); + const known = + lang && + lang !== 'text' && + lang !== 'plaintext' && + typeof hl.getLanguage === 'function' && + hl.getLanguage(lang); + if (known) { + try { + hl.highlightElement(code); + } catch (_) { + code.classList.add('hljs'); /* keep block styling even if hljs throws */ + } + } else { + code.classList.add('hljs'); /* plaintext — styled, no auto-detect */ } }); } @@ -923,6 +1940,63 @@ function scrollToBottom() { if (messagesEl) messagesEl.scrollTop = messagesEl.scrollHeight; } + + /** True when the messages list is at/near the bottom (within `threshold` px). */ + function isNearBottom(threshold = 120) { + if (!messagesEl) return true; + return messagesEl.scrollHeight - messagesEl.scrollTop - messagesEl.clientHeight <= threshold; + } + + /** + * Smooth "typewriter" streaming. Backend deltas arrive in bursts and can stall + * for ~a second; we decouple what's shown from what's arrived. `currentStream + * FullText` is the target (grows as deltas arrive); `streamShownLen` is how much + * we've revealed. Each animation frame we reveal a few more characters, so the + * user sees a steady flow no matter how lumpy the source is. + * + * The reveal is adaptive — a larger backlog drains faster (so we never fall + * seconds behind on a big burst) while a small backlog types out gently. Re- + * rendering happens at most once per frame (re-parsing the whole markdown + + * replacing innerHTML per token would be O(n²) and thrash layout). Auto-scroll + * fires only when the user is already at the bottom, so scrolling up mid-stream + * no longer yanks them back down. + */ + function renderStreamFrame() { + streamRenderRaf = 0; + const el = currentStreamEl; + if (!el || !messagesEl?.contains(el)) return; + const body = el.querySelector('.stream-body, .msg-body'); + if (!body) return; + + const target = currentStreamFullText; + if (streamShownLen < target.length) { + const remaining = target.length - streamShownLen; + // ~120 chars/s floor (2 per frame), faster as the backlog grows. + const step = Math.max(2, Math.ceil(remaining / 8)); + streamShownLen = Math.min(target.length, streamShownLen + step); + const stick = isNearBottom(); + body.innerHTML = renderMarkdown(target.slice(0, streamShownLen)); + decorateMessageBody(body, { deferSyntaxHighlight: true }); + if (stick) scrollToBottom(); + } + + // Keep going while there's still backlog; otherwise idle until the next + // delta re-arms the loop via ensureTyping(). + if (streamShownLen < currentStreamFullText.length) { + streamRenderRaf = requestAnimationFrame(renderStreamFrame); + } + } + + /** Start the typewriter loop if it isn't already running. */ + function ensureTyping() { + if (!streamRenderRaf) streamRenderRaf = requestAnimationFrame(renderStreamFrame); + } + + /** Stop the typewriter and reset the revealed position (call when finalizing). */ + function cancelStreamRender() { + if (streamRenderRaf) { cancelAnimationFrame(streamRenderRaf); streamRenderRaf = 0; } + streamShownLen = 0; + } /** Build the HTML block for persisted attachments (image inline / file as link). */ function attachmentsHtml(attachments) { if (!Array.isArray(attachments) || attachments.length === 0) return ''; @@ -962,6 +2036,30 @@ return '<div class="msg-attachments">' + items + '</div>'; } + /** Dev mode: add a live message's tokens to the chat-wide running total. */ + function bumpChatTokenTotal(tokens) { + if (!window.__ICLAW_DEV__ || !tokens) return; + const el = document.getElementById('chat-token-total'); + if (!el) return; + const cur = (Number(el.dataset.total) || 0) + Number(tokens); + el.dataset.total = String(cur); + el.textContent = cur.toLocaleString() + ' tok total'; + el.hidden = false; + } + + /** Dev mode: show token usage (+cache hits) on a message bubble + chat total. */ + function applyTokenBadge(el, tokens, cached) { + if (!window.__ICLAW_DEV__ || !tokens || !el) return; + if (el.querySelector(':scope > .msg-tokens')) return; + const c = Number(cached) || 0; + const span = document.createElement('span'); + span.className = 'msg-tokens'; + span.title = 'Tokens spent on this reply' + (c ? ' (' + c.toLocaleString() + ' served from cache)' : ''); + span.textContent = Number(tokens).toLocaleString() + ' tok' + (c ? ' · ' + c.toLocaleString() + ' cached' : ''); + el.appendChild(span); + bumpChatTokenTotal(tokens); + } + function appendMessage(msg, opts) { if (!messagesEl) return null; clearEmptyState(); @@ -999,6 +2097,7 @@ '<div class="msg-body">' + renderMessageHtml(msg.content || '') + '</div>' + attachmentsHtml(msg.attachments); decorateMessageBody(div); + applyTokenBadge(div, msg.tokens, msg.cached_tokens); messagesAppendRoot().appendChild(div); scrollToBottom(); return div; @@ -1006,6 +2105,10 @@ function appendStreamingAssistant() { if (!messagesEl) return null; clearEmptyState(); + // First turn of a chat can be a cold start (model/Docker warming up, 20–60s). + // Say "Warming up…" instead of "Thinking…" so the wait reads as honest setup, + // not a hang. Detected by the absence of any prior assistant message. + const isFirstTurn = !messagesAppendRoot()?.querySelector('.msg.assistant'); const div = document.createElement('div'); div.className = 'msg assistant streaming stream-waiting'; div.innerHTML = @@ -1014,7 +2117,7 @@ '<div class="msg-body stream-body"></div>'; messagesAppendRoot().appendChild(div); const st = div.querySelector('.stream-status'); - if (st) setStreamStatusLabel(st, 'Thinking…'); + if (st) setStreamStatusLabel(st, isFirstTurn ? 'Warming up…' : 'Thinking…'); scrollToBottom(); return div; } @@ -1360,6 +2463,166 @@ } } + // ------------------------------------------------------------------------- + // skill suggestions (inbox-gated procedural memory) — chat cards + // Mirrors fact suggestions, but with no auto-reject: a skill is a standing + // instruction, so acceptance is always a deliberate user action. + // ------------------------------------------------------------------------- + + function existingSkillSuggestionIds() { + const ids = new Set(); + if (!messagesEl) return ids; + messagesEl.querySelectorAll('.skill-suggestion-row[data-suggestion-id]').forEach((el) => { + const n = Number(el.dataset.suggestionId); + if (Number.isFinite(n)) ids.add(n); + }); + return ids; + } + + function removeSkillSuggestionRow(chatId, sid) { + if (!messagesEl || chatId !== activeChatId) return; + const row = messagesEl.querySelector( + '.skill-suggestion-row[data-suggestion-id="' + sid + '"]', + ); + if (!row) return; + const card = row.closest('.skill-suggestions-card'); + row.remove(); + if (card && !card.querySelector('.skill-suggestion-row')) card.remove(); + } + + function buildSkillSuggestionRowHtml(s) { + const id = Number(s.id); + if (!Number.isFinite(id)) return ''; + const kind = s.kind === 'patch' ? 'patch' : 'new'; + const kindBadge = + kind === 'patch' + ? '<span class="skill-suggestion-badge skill-suggestion-badge--patch">Updates “' + + escapeHtml(s.targetName || s.name || 'skill') + + '”</span>' + : '<span class="skill-suggestion-badge skill-suggestion-badge--new">New skill</span>'; + const untrustedBadge = s.untrusted + ? '<span class="skill-suggestion-badge skill-suggestion-badge--untrusted" title="Distilled from a turn that may have ingested untrusted external content — review carefully.">From untrusted content</span>' + : ''; + return ( + '<li class="skill-suggestion-row" data-suggestion-id="' + + id + + '" data-kind="' + + kind + + '" role="listitem">' + + '<div class="skill-suggestion-head">' + + kindBadge + + untrustedBadge + + '</div>' + + '<input class="skill-suggestion-name" aria-label="Skill name" spellcheck="false" value="' + + escapeHtml(s.name || '') + + '" />' + + '<textarea class="skill-suggestion-desc" aria-label="Skill summary" rows="2">' + + escapeHtml(s.description || '') + + '</textarea>' + + '<details class="skill-suggestion-bodywrap"><summary>Preview / edit procedure</summary>' + + '<textarea class="skill-suggestion-body" aria-label="Skill procedure (SKILL.md)" rows="10">' + + escapeHtml(s.body || '') + + '</textarea></details>' + + '<div class="skill-suggestion-controls">' + + '<label class="skill-suggestion-scope"><input type="checkbox" class="skill-suggestion-global" /> Save as global (all projects)</label>' + + '<div class="skill-suggestion-actions">' + + '<button type="button" class="skill-suggestion-btn skill-suggestion-reject" data-suggestion-id="' + + id + + '">Dismiss</button>' + + '<button type="button" class="skill-suggestion-btn skill-suggestion-accept" data-suggestion-id="' + + id + + '">Save skill</button>' + + '</div></div></li>' + ); + } + + function appendSkillSuggestionsCard(opts) { + if (!messagesEl) return; + const { projectId, chatId, suggestions, projectName } = opts; + if (!suggestions || suggestions.length === 0) return; + clearEmptyState(); + const safeName = escapeHtml((projectName || '').trim() || 'project'); + const rowsHtml = suggestions.map(buildSkillSuggestionRowHtml).filter(Boolean).join(''); + if (!rowsHtml) return; + + const pidEsc = String(projectId); + const cidEsc = String(chatId); + const existing = messagesEl.querySelector( + '.skill-suggestions-card[data-project-id="' + pidEsc + '"][data-chat-id="' + cidEsc + '"]', + ); + if (existing) { + const ul = existing.querySelector('.skill-suggestions-list'); + if (!ul) return; + const tpl = document.createElement('template'); + tpl.innerHTML = rowsHtml.trim(); + tpl.content.childNodes.forEach((n) => { + if (n.nodeType !== 1) return; + const el = n; + const sid = el.dataset.suggestionId; + if (!sid || ul.querySelector('.skill-suggestion-row[data-suggestion-id="' + sid + '"]')) + return; + ul.appendChild(el); + }); + scrollToBottom(); + return; + } + + const card = document.createElement('div'); + card.className = 'msg system skill-suggestions-card'; + card.dataset.projectId = pidEsc; + card.dataset.chatId = cidEsc; + card.innerHTML = + '<div class="skill-suggestions-shell">' + + '<p class="skill-suggestions-lead">Learned a skill for «' + + safeName + + '». Save it to project memory?</p>' + + '<ul class="skill-suggestions-list" role="list">' + + rowsHtml + + '</ul></div>'; + messagesAppendRoot().appendChild(card); + scrollToBottom(); + } + + async function loadPendingSkillSuggestions() { + if (activeChatId == null || !messagesEl) return; + try { + const res = await fetch( + '/chats/' + encodeURIComponent(activeChatId) + '/skill-suggestions', + { headers: { Accept: 'application/json' } }, + ); + if (!res.ok) return; + const data = await res.json(); + const list = Array.isArray(data.suggestions) ? data.suggestions : []; + if (list.length === 0) return; + const first = list[0]; + const pid = first ? Number(first.project_id) : NaN; + if (!Number.isFinite(pid)) return; + const have = existingSkillSuggestionIds(); + const fresh = list + .filter((s) => s && Number.isFinite(Number(s.id)) && !have.has(Number(s.id))) + .map((s) => ({ + id: Number(s.id), + kind: s.kind === 'patch' ? 'patch' : 'new', + name: String(s.name ?? ''), + description: String(s.description ?? ''), + body: String(s.body ?? ''), + untrusted: !!s.untrusted, + })); + const pname = + typeof data.projectName === 'string' && data.projectName.trim() + ? data.projectName.trim() + : 'project'; + appendSkillSuggestionsCard({ + projectId: pid, + chatId: activeChatId, + projectName: pname, + suggestions: fresh, + }); + } catch (_) { + /* ignore */ + } + } + // ------------------------------------------------------------------------- // queue widget — shows only WAITING items (not the in-flight one) // ------------------------------------------------------------------------- @@ -1369,6 +2632,7 @@ serverId: row.id, id: String(row.id), content: row.content, + mode: row.mode || undefined, }; if (row.reply_to_message_id != null && row.reply_quote) { item.replyTo = { @@ -1398,6 +2662,7 @@ async function enqueueQueueOnServer(chatId, draft) { const body = { content: draft.content }; + if (draft.mode) body.mode = draft.mode; if (draft.replyTo) body.replyTo = draft.replyTo; if (draft.inlineSecrets && draft.inlineSecrets.length > 0) { body.inlineSecrets = draft.inlineSecrets; @@ -1592,6 +2857,59 @@ .catch(() => {}); return; } + const sAcc = e.target.closest('.skill-suggestion-accept'); + const sRej = e.target.closest('.skill-suggestion-reject'); + if (sAcc || sRej) { + const row = (sAcc || sRej).closest('.skill-suggestion-row'); + const sid = Number((sAcc || sRej).dataset.suggestionId); + if (!Number.isFinite(sid) || activeChatId == null) return; + e.preventDefault(); + if (sRej) { + fetch( + '/chats/' + + encodeURIComponent(activeChatId) + + '/skill-suggestions/' + + encodeURIComponent(sid) + + '/reject', + { method: 'POST', headers: { Accept: 'application/json' } }, + ) + .then((res) => { + if (res.ok) removeSkillSuggestionRow(activeChatId, sid); + }) + .catch(() => {}); + return; + } + // Accept — submit any inline edits + the scope choice. + const name = row?.querySelector('.skill-suggestion-name')?.value?.trim() || ''; + const description = row?.querySelector('.skill-suggestion-desc')?.value?.trim() || ''; + const body = row?.querySelector('.skill-suggestion-body')?.value?.trim() || ''; + const global = !!row?.querySelector('.skill-suggestion-global')?.checked; + const payload = { scope: global ? 'global' : 'project' }; + if (name) payload.name = name; + if (description) payload.description = description; + if (body) payload.body = body; + if (sAcc) sAcc.disabled = true; + fetch( + '/chats/' + + encodeURIComponent(activeChatId) + + '/skill-suggestions/' + + encodeURIComponent(sid) + + '/accept', + { + method: 'POST', + headers: { 'content-type': 'application/json', Accept: 'application/json' }, + body: JSON.stringify(payload), + }, + ) + .then((res) => { + if (res.ok) removeSkillSuggestionRow(activeChatId, sid); + else if (sAcc) sAcc.disabled = false; + }) + .catch(() => { + if (sAcc) sAcc.disabled = false; + }); + return; + } const status = e.target.closest('.stream-status.has-detail'); if (!status) return; const expanded = status.classList.toggle('detail-expanded'); @@ -2544,6 +3862,7 @@ if ( el.classList.contains('streaming') || el.classList.contains('fact-suggestions-card') || + el.classList.contains('skill-suggestions-card') || el.classList.contains('reasoning-block') ) { return null; @@ -2753,6 +4072,7 @@ if (activeChatId != null) { wsSend({ type: 'subscribe', chatId: activeChatId }); loadPendingFactSuggestions(); + loadPendingSkillSuggestions(); } /* If the socket dropped while we had pending task-create records (or * was never up when the server emitted 'ready'), reconcile against the @@ -2885,6 +4205,7 @@ // rather than left orphaned above the final message. const target = ensureStreamEl(); if (target) { + cancelStreamRender(); // final render below is authoritative target.classList.remove( 'streaming', 'stream-waiting', 'stream-tool', 'stream-generating', ); @@ -2900,6 +4221,7 @@ body.innerHTML = renderMarkdown(msg.message.content || ''); decorateMessageBody(body); } + applyTokenBadge(target, msg.message.tokens, msg.message.cached_tokens); currentStreamEl = null; currentStreamFullText = ''; } else { @@ -2916,6 +4238,7 @@ setWorkingDot(msg.chatId, true); if (msg.chatId !== activeChatId) return; setStopVisible(true); + streamShownLen = 0; // fresh typewriter for the new turn ensureStreamEl(); { const status = currentStreamEl?.querySelector('.stream-status'); @@ -2945,12 +4268,9 @@ status.hidden = true; } } - const body = el.querySelector('.stream-body, .msg-body'); - if (body) { - body.innerHTML = renderMarkdown(currentStreamFullText); - decorateMessageBody(body, { deferSyntaxHighlight: true }); - } - scrollToBottom(); + // Feed the typewriter — it reveals the accumulated text smoothly, + // a few chars per frame, instead of dumping each lumpy delta at once. + ensureTyping(); return; } @@ -3015,6 +4335,7 @@ // turn ends without an assistant message. Without this, the // "Finishing…" / "Thinking…" status would sit on the page until // the user reloaded. + cancelStreamRender(); if (currentStreamEl && currentStreamEl.classList.contains('streaming')) { const body = currentStreamEl.querySelector('.stream-body, .msg-body'); const hasContent = !!(body && body.textContent && body.textContent.trim()); @@ -3072,6 +4393,44 @@ return; } + /* ---- incognito (ephemeral; keyed, never persisted) ---- */ + case 'incognito-turn-delta': { + if (msg.key !== activeIncognitoKey) return; + const el = ensureStreamEl(); + currentStreamFullText += msg.text; + if (el.classList.contains('stream-waiting') || el.classList.contains('stream-tool')) { + el.classList.remove('stream-waiting', 'stream-tool'); + el.classList.add('stream-generating'); + const status = el.querySelector('.stream-status'); + if (status) { stopStreamStatusDotAnim(status); status.hidden = true; } + } + ensureTyping(); + return; + } + + case 'incognito-turn-tool': + // Tool activity indicator could go here; ignored for now. + return; + + case 'incognito-error': { + if (msg.key !== activeIncognitoKey) return; + const div = document.createElement('div'); + div.className = 'msg system error'; + div.innerHTML = + '<div class="role">error</div>' + + '<div class="msg-body">' + escapeHtml('Error: ' + msg.message) + '</div>'; + messagesAppendRoot()?.appendChild(div); + return; + } + + case 'incognito-turn-ended': { + if (msg.key !== activeIncognitoKey) return; + const finishedEl = currentStreamEl; + finalizeIncognitoStream(); + applyTokenBadge(finishedEl, msg.tokens, msg.cached); + return; + } + case 'project-fact-suggestions': { if (msg.chatId !== activeChatId) return; const have = existingFactSuggestionIds(); @@ -3090,6 +4449,68 @@ removeFactSuggestionRow(msg.chatId, msg.suggestionId); return; + case 'project-skill-suggestions': { + if (msg.chatId !== activeChatId) return; + const have = existingSkillSuggestionIds(); + const fresh = (msg.suggestions || []).filter((s) => s && !have.has(Number(s.id))); + if (fresh.length === 0) return; + appendSkillSuggestionsCard({ + projectId: msg.projectId, + chatId: msg.chatId, + projectName: typeof msg.projectName === 'string' ? msg.projectName : 'project', + suggestions: fresh, + }); + return; + } + + case 'project-skill-suggestion-removed': + removeSkillSuggestionRow(msg.chatId, msg.suggestionId); + return; + + case 'project-skill-added': { + if (currentProjectPageId() !== msg.projectId) return; + const ul = document.getElementById('skills-list'); + if (!ul) return; + ul.querySelector('li.project-chats-empty')?.remove(); + // Replace if a row with this id already exists (e.g. accept-as-update). + ul.querySelector('li.skill[data-skill-id="' + msg.skill.id + '"]')?.remove(); + ul.insertBefore(buildSkillLi(msg.skill), ul.firstChild); + syncProjectSkillsTabCountFromDom(); + return; + } + + case 'project-skill-updated': { + if (currentProjectPageId() !== msg.projectId) return; + const ul = document.getElementById('skills-list'); + if (!ul) return; + const existing = ul.querySelector('li.skill[data-skill-id="' + msg.skill.id + '"]'); + const fresh = buildSkillLi(msg.skill); + if (existing) existing.replaceWith(fresh); + else { + ul.querySelector('li.project-chats-empty')?.remove(); + ul.insertBefore(fresh, ul.firstChild); + } + syncProjectSkillsTabCountFromDom(); + return; + } + + case 'project-skill-deleted': { + if (currentProjectPageId() !== msg.projectId) return; + document + .querySelector('#skills-list li.skill[data-skill-id="' + msg.skillId + '"]') + ?.remove(); + const ul = document.getElementById('skills-list'); + if (ul && !ul.querySelector('li.skill')) { + const empty = document.createElement('li'); + empty.className = 'project-chats-empty muted'; + empty.textContent = + 'No skills yet. Accept a skill suggestion in a chat for this project.'; + ul.appendChild(empty); + } + syncProjectSkillsTabCountFromDom(); + return; + } + case 'project-created': { const arr = window.__ICLAW_PROJECTS__; if (Array.isArray(arr)) { @@ -3381,7 +4802,6 @@ } function applyGatewayStatus(status, detail) { - const badge = document.getElementById('gateway-badge'); // "degraded" (gateway answered /health but the WS RPC can't get through) // gets its own in-page banner on the home/projects pages, so the sidebar // "Start OpenClaw" banner is reserved for a genuinely-offline gateway — @@ -3389,33 +4809,14 @@ const offline = status === 'down' || status === 'shutdown'; setGatewayOfflineBannerVisible(offline); - if (!badge) return; - badge.classList.remove('ok', 'down', 'degraded', 'shutdown'); - if (status === 'ok') { - badge.classList.add('ok'); - badge.textContent = 'OpenClaw: connected'; - } else if (status === 'degraded') { - badge.classList.add('degraded'); - badge.textContent = 'OpenClaw: unreachable'; - } else if (status === 'shutdown') { - badge.classList.add('shutdown'); - badge.textContent = 'OpenClaw: shutting down'; - } else { - badge.classList.add('down'); - badge.textContent = 'OpenClaw: off'; - } - const baseUrl = badge.dataset.baseUrl || ''; - badge.title = baseUrl; + // Keep Full Power gating current on every page. + gatewayOk = status === 'ok'; + if (typeof syncExecuteAvailability === 'function') syncExecuteAvailability(); } (function initGatewayOfflineBanner() { - const badge = document.getElementById('gateway-badge'); // Only a genuinely-offline gateway shows the sidebar "Start OpenClaw" banner. // "degraded" is handled by the in-page banner instead. - if (badge && badge.classList.contains('down')) { - setGatewayOfflineBannerVisible(true); - return; - } if (!gatewayBanner) return; void fetch('/api/gateway/status', { headers: { Accept: 'application/json' } }) .then((res) => (res.ok ? res.json() : null)) @@ -3517,10 +4918,38 @@ if (item.serverId != null) ownQueueIds.delete(item.serverId); renderQueue(); inFlight = true; + + // Incognito: ephemeral turn — render locally, stream over `incognito-*` + // events, never persist. No pending-id (no message-appended adopts it) and + // no server chat is created. + if (item.mode === 'incognito') { + if (!activeIncognitoKey) activeIncognitoKey = newIncognitoKey(); + appendMessage({ role: 'user', content: item.content, mode: 'incognito' }); + currentStreamFullText = ''; + streamShownLen = 0; + currentStreamEl = ensureStreamEl(); + setStopVisible(true); + const wf = (typeof getWorkFolders === 'function' ? getWorkFolders() : []) + .map((f) => ({ path: f.path, readonly: true })); + const ok = wsSend({ + type: 'incognito-send', + key: activeIncognitoKey, + requestId: 'r-' + Date.now() + '-' + Math.random().toString(36).slice(2, 6), + content: item.content, + workFolders: wf.length ? wf : undefined, + }); + if (!ok) { + inFlight = false; + addWaitingItem(item, { at: 'front' }); + renderQueue(); + } + return; + } + // Optimistically append user msg. Mark it as pending-id so the // upcoming `message-appended` for the same user msg adopts this node // instead of duplicating. - const optimistic = { role: 'user', content: item.content }; + const optimistic = { role: 'user', content: item.content, mode: item.mode }; if (item.replyTo) { optimistic.reply_to_message_id = item.replyTo.messageId; optimistic.reply_quote = item.replyTo.quote; @@ -3539,12 +4968,32 @@ } appendMessage(optimistic, { pendingId: true }); currentStreamFullText = ''; + streamShownLen = 0; currentStreamEl = ensureStreamEl(); const payload = { type: 'send', requestId: 'r-' + Date.now() + '-' + Math.random().toString(36).slice(2, 6), content: item.content, }; + if (item.mode) payload.mode = item.mode; + if (item.mode === 'work') { + const wf = getWorkFolders ? getWorkFolders() : []; + if (wf.length > 0) { + payload.workFolders = wf.map((f) => ({ path: f.path, readonly: !f.write })); + } + } + if (item.mode === 'secure') { + payload.networkEnabled = typeof getNetworkEnabled === 'function' ? getNetworkEnabled() : false; + // Reset TTL countdown on each message + send TTL to runtime + if (typeof saveSecureTtl === 'function') { + const cur = getSecureTtl(); + payload.ttlDays = cur.ttlDays; + saveSecureTtl(cur.ttlDays); + // Session is created host-side during this turn; re-check a few times + // so the bar appears as soon as it exists (not before the first message). + [250, 800, 1600, 3000].forEach((d) => setTimeout(refreshSecureBar, d)); + } + } if (item.replyTo) { payload.replyTo = { messageId: item.replyTo.messageId, @@ -4140,6 +5589,14 @@ * selection-only changes are correct without waiting for debounce. */ function applyComposerSecretStripLayout() { + applyComposerSecretStripLayoutInner(); + // Secret strip and the secure-workspace bar share the bottom row; whenever + // the strip's visibility changes, re-evaluate the bar so it yields to the + // strip ("Selection in message") and reappears once the strip is gone. + if (typeof refreshSecureBar === 'function') refreshSecureBar(); + } + + function applyComposerSecretStripLayoutInner() { if (!composerSecretUi) return; if (composerSecretModal && !composerSecretModal.hidden) { cancelComposerSelectionHintReveal(); @@ -4617,6 +6074,12 @@ form.addEventListener('submit', async (e) => { e.preventDefault(); if (startedOnDraft && !draftProjectLocked) return; + // Full Power needs OpenClaw — don't send into a dead gateway. The input is + // already disabled in this state; this is the belt-and-braces guard. + if (getComposerMode() === 'execute' && !isExecuteAvailable()) { + syncExecuteAvailability(); + return; + } // If the schedule menu was just opened by a long-press, the bubbling // click on the send button would otherwise submit a regular message. if (scheduleMenuJustOpened || isScheduleMenuOpen()) { @@ -4651,6 +6114,7 @@ replyTo: replySnap || undefined, attachments: attachmentsSnap.length > 0 ? attachmentsSnap : undefined, inlineSecrets, + mode: getComposerMode(), }; let queued; const persistOnServer = @@ -6128,6 +7592,14 @@ } if (stopBtn) { stopBtn.addEventListener('click', () => { + // Incognito has no DB chat — abort by ephemeral key and finalize locally. + if (getComposerMode() === 'incognito' && activeIncognitoKey) { + wsSend({ type: 'incognito-abort', key: activeIncognitoKey }); + finalizeIncognitoStream(); + stopBtn.disabled = true; + setTimeout(() => { stopBtn.disabled = false; }, 3000); + return; + } if (activeChatId == null) return; wsSend({ type: 'abort', chatId: activeChatId }); // Optimistically disable until server confirms via turn-error/ended, @@ -6845,6 +8317,70 @@ ); } + // ------------------------------------------------------------------------- + // project page — skills list (fetch + WS sync from other tabs) + // ------------------------------------------------------------------------- + const skillsListEl = document.getElementById('skills-list'); + if (skillsListEl && projectPageId != null) { + skillsListEl + .querySelectorAll('.skill-name, .skill-description, .skill-body') + .forEach((el) => { + el.dataset.saved = el.value.trim(); + }); + skillsListEl.addEventListener('click', (e) => { + const btn = e.target.closest('.skill-delete'); + if (!btn) return; + const li = btn.closest('li.skill'); + const skillId = li?.dataset.skillId; + if (!skillId) return; + e.preventDefault(); + fetch( + '/projects/' + + encodeURIComponent(projectPageId) + + '/skills/' + + encodeURIComponent(skillId) + + '/delete', + { method: 'POST', headers: { Accept: 'application/json' } }, + ).catch(() => {}); + }); + skillsListEl.addEventListener( + 'blur', + (e) => { + const el = e.target.closest('.skill-name, .skill-description, .skill-body'); + if (!el || !skillsListEl.contains(el)) return; + const li = el.closest('li.skill'); + const skillId = li?.dataset.skillId; + if (!skillId) return; + const next = el.value.trim(); + if (!next) return; + if (next === (el.dataset.saved || '').trim()) return; + const field = el.classList.contains('skill-name') + ? 'name' + : el.classList.contains('skill-description') + ? 'description' + : 'body'; + const payload = {}; + payload[field] = next; + fetch( + '/projects/' + + encodeURIComponent(projectPageId) + + '/skills/' + + encodeURIComponent(skillId), + { + method: 'PATCH', + headers: { 'content-type': 'application/json', Accept: 'application/json' }, + body: JSON.stringify(payload), + }, + ) + .then((res) => { + if (res.ok) el.dataset.saved = next; + }) + .catch(() => {}); + }, + true, + ); + } + function collapseProjectSecretRow(li) { li.classList.remove('project-secret-row--revealed'); const preview = li.querySelector('.project-secret-reveal'); @@ -6984,6 +8520,7 @@ const panels = { chats: document.getElementById('project-panel-chats'), memory: document.getElementById('project-panel-memory'), + skills: document.getElementById('project-panel-skills'), links: document.getElementById('project-panel-links'), files: document.getElementById('project-panel-files'), secrets: document.getElementById('project-panel-secrets'), @@ -6992,6 +8529,7 @@ !tabs.length || !panels.chats || !panels.memory || + !panels.skills || !panels.links || !panels.files || !panels.secrets diff --git a/scripts/install-docker.sh b/scripts/install-docker.sh new file mode 100755 index 0000000..7d9af9a --- /dev/null +++ b/scripts/install-docker.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# install-docker.sh — best-effort Docker engine install for iClaw Work/Safe modes. +# +# Triggered by the composer's "Install Docker" button via POST /api/docker/install +# (localhost only). Idempotent: a no-op when Docker is already on PATH. The +# caller starts the daemon and polls readiness afterwards, so this script only +# has to land the binary. +# +# macOS → Homebrew cask (errors out with guidance if brew is missing) +# Linux → official get.docker.com convenience script + docker group +set -euo pipefail + +if command -v docker >/dev/null 2>&1; then + echo "docker already installed: $(docker --version 2>/dev/null || echo unknown)" + exit 0 +fi + +case "$(uname -s)" in + Darwin) + if ! command -v brew >/dev/null 2>&1; then + echo "ERROR: Homebrew not found. Install it from https://brew.sh then retry, or install Docker Desktop manually." >&2 + exit 1 + fi + echo "Installing Docker Desktop via Homebrew…" + brew install --cask docker + ;; + Linux) + echo "Installing Docker Engine via get.docker.com…" + curl -fsSL https://get.docker.com | sh + if command -v sudo >/dev/null 2>&1; then + sudo usermod -aG docker "$USER" || true + echo "NOTE: log out and back in for docker group membership to take effect." + fi + ;; + *) + echo "ERROR: unsupported platform $(uname -s). Install Docker Desktop manually from https://docker.com/products/docker-desktop" >&2 + exit 1 + ;; +esac + +echo "docker install step done." diff --git a/src/app.ts b/src/app.ts index 493f8aa..29b82f6 100644 --- a/src/app.ts +++ b/src/app.ts @@ -8,6 +8,7 @@ import { projectsRouter } from './routes/projects'; import { agentsRouter } from './routes/sessions'; import { mediaRouter } from './routes/media'; import { gatewayRouter } from './routes/gateway'; +import { dockerRouter } from './routes/docker'; import { updateRouter } from './routes/update'; import { tasksRouter } from './routes/tasks'; import { projects, scheduledMessages } from './services/store'; @@ -71,6 +72,7 @@ export function createApp(): express.Express { })); res.locals.scheduledChatIds = scheduledMessages.chatIdsWithPending(); res.locals.scheduledChatCounts = scheduledMessages.pendingCountByChatId(); + res.locals.devMode = process.env.ICLAW_DEV_MODE === 'true'; next(); }); @@ -125,6 +127,7 @@ export function createApp(): express.Express { app.use('/', settingsRouter); app.use('/api/agents', agentsRouter); app.use('/api/gateway', gatewayRouter); + app.use('/api/docker', dockerRouter); app.use('/api/update', updateRouter); app.use('/api/remote-access', remoteAccessApiRouter); app.use('/media', mediaRouter); diff --git a/src/db/database.ts b/src/db/database.ts index 6b0401d..5fab637 100644 --- a/src/db/database.ts +++ b/src/db/database.ts @@ -44,6 +44,8 @@ CREATE TABLE IF NOT EXISTS messages ( reply_to_role TEXT, /** JSON array of {url, mimeType, fileName, sizeBytes} for user-attached files. NULL when no attachments. */ attachments TEXT, + /** Send mode: 'ask' | 'execute' (see services/chatModes.ts). Legacy rows default to 'execute'. */ + mode TEXT NOT NULL DEFAULT 'execute', created_at TEXT NOT NULL DEFAULT (datetime('now')) ); @@ -74,6 +76,45 @@ CREATE TABLE IF NOT EXISTS project_fact_suggestions ( CREATE INDEX IF NOT EXISTS idx_fact_suggestions_chat ON project_fact_suggestions(chat_id, id); +-- Active, accepted project skills (procedural memory). Stored as SKILL.md. +-- The declarative half is project_facts; this is the procedural half. +CREATE TABLE IF NOT EXISTS project_skills ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + project_id INTEGER REFERENCES projects(id) ON DELETE CASCADE, -- NULL = global skill + name TEXT NOT NULL, -- kebab-case, unique within scope + description TEXT NOT NULL, -- one-line summary (shown to user + prompt index) + body TEXT NOT NULL, -- full SKILL.md (frontmatter + procedure) + tags TEXT, -- JSON array of strings (optional) + source_chat_id INTEGER REFERENCES chats(id) ON DELETE SET NULL, + usage_count INTEGER NOT NULL DEFAULT 0, + version INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) +); +CREATE INDEX IF NOT EXISTS idx_skills_project ON project_skills(project_id, id); +-- Uniqueness within a scope (project_id may be NULL for global). SQLite treats +-- NULLs as distinct, so global uniqueness is also enforced in the store layer. +CREATE UNIQUE INDEX IF NOT EXISTS idx_skills_scope_name + ON project_skills(project_id, name); + +-- Inbox: proposed skills awaiting user acceptance. Never active until accepted. +CREATE TABLE IF NOT EXISTS project_skill_suggestions ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + project_id INTEGER NOT NULL REFERENCES projects(id) ON DELETE CASCADE, + chat_id INTEGER NOT NULL REFERENCES chats(id) ON DELETE CASCADE, + kind TEXT NOT NULL DEFAULT 'new', -- 'new' | 'patch' + target_skill_id INTEGER REFERENCES project_skills(id) ON DELETE CASCADE, -- for 'patch' + name TEXT NOT NULL, + description TEXT NOT NULL, + body TEXT NOT NULL, -- full proposed SKILL.md + tags TEXT, -- JSON array (optional) + untrusted INTEGER NOT NULL DEFAULT 0, -- 1 if source turn ingested untrusted content + assistant_message_id INTEGER REFERENCES messages(id) ON DELETE SET NULL, + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); +CREATE INDEX IF NOT EXISTS idx_skill_suggestions_chat ON project_skill_suggestions(chat_id, id); +CREATE INDEX IF NOT EXISTS idx_skill_suggestions_project ON project_skill_suggestions(project_id, id); + CREATE TABLE IF NOT EXISTS project_secrets ( id INTEGER PRIMARY KEY AUTOINCREMENT, project_id INTEGER REFERENCES projects(id) ON DELETE CASCADE, @@ -111,6 +152,8 @@ CREATE TABLE IF NOT EXISTS queued_messages ( attachments TEXT, /** JSON array of {slot, label, plain} for [[iclaw:sN]] markers; resolved on flush. */ inline_secrets TEXT, + /** Send mode chosen at enqueue time; preserved so flush sends with it. */ + mode TEXT NOT NULL DEFAULT 'execute', /** Lower sorts first; promote-to-front uses values below the current min. */ position INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL DEFAULT (datetime('now')) @@ -232,6 +275,17 @@ CREATE TABLE IF NOT EXISTS remote_access_state ( updated_at INTEGER NOT NULL DEFAULT (CAST(strftime('%s','now') AS INTEGER) * 1000) ); +-- Context compaction cache. One row per chat: a rolling summary of all turns +-- up to (and including) up_to_message_id. Lets us feed the model +-- [summary] + [recent verbatim] so a chat's context is never cleared, only +-- compressed. Rebuilt incrementally as the chat grows. +CREATE TABLE IF NOT EXISTS chat_summaries ( + chat_id INTEGER PRIMARY KEY REFERENCES chats(id) ON DELETE CASCADE, + up_to_message_id INTEGER NOT NULL, + summary TEXT NOT NULL, + updated_at TEXT NOT NULL DEFAULT (datetime('now')) +); + -- Robustness: any new message bumps the parent chat's updated_at so sidebar -- sorting is always correct even if a caller forgets the manual chats.touch(). CREATE TRIGGER IF NOT EXISTS trg_chats_touch_on_message @@ -257,6 +311,10 @@ function ensureColumn(table: string, column: string, ddl: string): void { } } ensureColumn('messages', 'attachments', 'TEXT'); +ensureColumn('messages', 'mode', "TEXT NOT NULL DEFAULT 'execute'"); +ensureColumn('messages', 'tokens', 'INTEGER'); // dev-mode token usage (runtime modes) +ensureColumn('messages', 'cached_tokens', 'INTEGER'); // dev-mode: prompt tokens served from cache +ensureColumn('queued_messages', 'mode', "TEXT NOT NULL DEFAULT 'execute'"); ensureColumn('chats', 'chat_kind', "TEXT NOT NULL DEFAULT 'normal'"); ensureColumn('remote_access_tunnels', 'access_token', 'TEXT'); ensureColumn('remote_access_tunnels', 'opaque_registration_record', 'TEXT'); diff --git a/src/db/kv.ts b/src/db/kv.ts new file mode 100644 index 0000000..7f061eb --- /dev/null +++ b/src/db/kv.ts @@ -0,0 +1,31 @@ +/** + * Tiny typed accessor over the `iclaw_kv` table — a plain string key/value + * store for app-level settings that don't deserve their own table (e.g. the + * OpenRouter API key entered in Settings). + * + * Deliberately low-level: imports ONLY `db`, so modules like `config.ts` can + * read settings without pulling in the higher-level `store.ts` (which would + * create an import cycle: config → store → chatModes → openRouter → config). + */ + +import { db } from './database'; + +const getStmt = db.prepare('SELECT value FROM iclaw_kv WHERE key = ?'); +const setStmt = db.prepare( + `INSERT INTO iclaw_kv (key, value) VALUES (?, ?) + ON CONFLICT(key) DO UPDATE SET value = excluded.value`, +); +const delStmt = db.prepare('DELETE FROM iclaw_kv WHERE key = ?'); + +export function kvGet(key: string): string | null { + const row = getStmt.get(key) as { value: string } | undefined; + return row?.value ?? null; +} + +export function kvSet(key: string, value: string): void { + setStmt.run(key, value); +} + +export function kvDelete(key: string): void { + delStmt.run(key); +} diff --git a/src/index.ts b/src/index.ts index 2482111..8e11a4d 100644 --- a/src/index.ts +++ b/src/index.ts @@ -6,6 +6,7 @@ import { openclaw } from './services/openclaw'; import { scheduler } from './services/scheduler'; import { gatewayEvents } from './services/gatewayEvents'; import { remoteAccess, setRemoteAccessQuiet } from './services/remoteAccess'; +import { runtimeProcess } from './services/runtimeProcess'; import { setBoundLocalAddress } from './services/localAddress'; import { findAvailablePort, @@ -44,6 +45,7 @@ function gracefulShutdown( } removeLockFileIfOwned(); scheduler.stop(); + runtimeProcess.stop(); remoteAccess.shutdown(); const exitCode = signal === 'SIGINT' ? 130 : 0; @@ -108,6 +110,7 @@ async function main(): Promise<void> { server = createServer(app); attachWsServer(server); scheduler.start(); + runtimeProcess.start(); gatewayEvents.start(); const stop = () => gracefulShutdown(server, 'SIGINT'); diff --git a/src/routes/chats.ts b/src/routes/chats.ts index 07c7c39..9915983 100644 --- a/src/routes/chats.ts +++ b/src/routes/chats.ts @@ -6,12 +6,15 @@ import { projects, projectFactSuggestions, projectFacts, + projectSkills, + projectSkillSuggestions, projectSecrets, secretUsableInChat, scheduledMessages, queuedMessages, tasks, enrichFactWithSourceChatTitle, + enrichSkillWithSourceChatTitle, } from '../services/store'; import { persistIncomingAttachments, type IncomingAttachment } from '../services/uploads'; import { @@ -24,8 +27,18 @@ import { openclawWs } from '../services/openclawWs'; import { openclaw, cloudShareBaseUrl } from '../services/openclaw'; import { chatStatus } from '../services/chatStatus'; import { wsHub } from '../services/wsHub'; -import { sendMessage } from '../services/chatRunner'; +import { sendMessage, getWorkSessionId, destroyWorkSession, exportChatSandbox, applyChatSandboxChanges } from '../services/chatRunner'; +import { getWorkspaceInfo } from '../services/workRuntime'; +import { + defaultComposerMode, + isSelectableMode, + isEphemeralMode, + listSelectableModes, + normalizeChatMode, +} from '../services/chatModes'; +import type { ChatMode } from '../types'; import { shouldShowSendHint } from '../services/sendHint'; +import { openRouterEnabled } from '../services/openRouter'; export const chatsRouter: Router = Router(); @@ -152,23 +165,160 @@ chatsRouter.post('/:id/fact-suggestions/:suggestionId/reject', (req, res) => { res.type('application/json').json({ ok: true }); }); +/* ---------------- project skill suggestions (inbox-gated) ---------------- */ + +function parseTags(raw: unknown): string[] | null { + if (!Array.isArray(raw)) return null; + const tags = raw.filter((t): t is string => typeof t === 'string' && t.trim().length > 0); + return tags.length > 0 ? tags : null; +} + +/** Pending project-skill suggestions for this chat (JSON, includes full bodies). */ +chatsRouter.get('/:id/skill-suggestions', (req, res) => { + const id = Number(req.params.id); + if (!Number.isFinite(id) || !chats.get(id)) { + res.status(404).json({ error: 'chat not found' }); + return; + } + const suggestions = projectSkillSuggestions.listByChat(id); + const first = suggestions[0]; + const projectName = + first != null ? (projects.get(first.project_id)?.name?.trim() ?? 'project') : null; + res.type('application/json').json({ suggestions, projectName }); +}); + +chatsRouter.post('/:id/skill-suggestions/:suggestionId/accept', (req, res) => { + const chatId = Number(req.params.id); + const sid = Number(req.params.suggestionId); + if (!Number.isFinite(chatId) || !Number.isFinite(sid)) { + res.status(400).json({ error: 'invalid id' }); + return; + } + const chat = chats.get(chatId); + const sug = projectSkillSuggestions.get(sid); + if (!chat || !sug || sug.chat_id !== chatId) { + res.status(404).json({ error: 'not found' }); + return; + } + if (!chat.project_id || chat.project_id !== sug.project_id) { + res.status(400).json({ error: 'project mismatch' }); + return; + } + + // Optional user edits submitted alongside the accept. + const name = typeof req.body?.name === 'string' && req.body.name.trim() ? req.body.name : sug.name; + const description = + typeof req.body?.description === 'string' && req.body.description.trim() + ? req.body.description + : sug.description; + const body = typeof req.body?.body === 'string' && req.body.body.trim() ? req.body.body : sug.body; + const tags = parseTags(req.body?.tags) ?? (sug.tags ? (JSON.parse(sug.tags) as string[]) : null); + const scope = req.body?.scope === 'global' ? 'global' : 'project'; + const projectId = scope === 'global' ? null : sug.project_id; + + try { + if (sug.kind === 'patch' && sug.target_skill_id != null && projectSkills.get(sug.target_skill_id)) { + projectSkills.update(sug.target_skill_id, { name, description, body, tags }); + const updated = projectSkills.get(sug.target_skill_id)!; + projectSkillSuggestions.remove(sid); + wsHub.broadcastAll({ + type: 'project-skill-updated', + projectId: updated.project_id ?? sug.project_id, + skill: enrichSkillWithSourceChatTitle(updated), + }); + wsHub.broadcastAll({ type: 'project-skill-suggestion-removed', chatId, suggestionId: sid }); + res.type('application/json').json({ skill: updated }); + return; + } + + // 'new' (or a patch whose target vanished). Same-scope name collision → + // treat as an update of the existing skill rather than a duplicate insert. + const existing = projectSkills.getByName(projectId, name); + if (existing) { + projectSkills.update(existing.id, { name, description, body, tags }); + const updated = projectSkills.get(existing.id)!; + projectSkillSuggestions.remove(sid); + wsHub.broadcastAll({ + type: 'project-skill-updated', + projectId: updated.project_id ?? sug.project_id, + skill: enrichSkillWithSourceChatTitle(updated), + }); + wsHub.broadcastAll({ type: 'project-skill-suggestion-removed', chatId, suggestionId: sid }); + res.type('application/json').json({ skill: updated }); + return; + } + + const skill = projectSkills.create({ + projectId, + name, + description, + body, + tags, + sourceChatId: chatId, + }); + projectSkillSuggestions.remove(sid); + wsHub.broadcastAll({ + type: 'project-skill-added', + projectId: skill.project_id ?? sug.project_id, + skill: enrichSkillWithSourceChatTitle(skill), + }); + wsHub.broadcastAll({ type: 'project-skill-suggestion-removed', chatId, suggestionId: sid }); + res.type('application/json').json({ skill }); + } catch (err) { + res.status(400).json({ error: err instanceof Error ? err.message : 'accept failed' }); + } +}); + +chatsRouter.post('/:id/skill-suggestions/:suggestionId/reject', (req, res) => { + const chatId = Number(req.params.id); + const sid = Number(req.params.suggestionId); + if (!Number.isFinite(chatId) || !Number.isFinite(sid)) { + res.status(400).json({ error: 'invalid id' }); + return; + } + const sug = projectSkillSuggestions.get(sid); + if (!sug || sug.chat_id !== chatId) { + res.status(404).json({ error: 'not found' }); + return; + } + projectSkillSuggestions.remove(sid); + wsHub.broadcastAll({ type: 'project-skill-suggestion-removed', chatId, suggestionId: sid }); + res.type('application/json').json({ ok: true }); +}); + chatsRouter.get('/:id', async (req, res, next) => { try { const id = Number(req.params.id); const chat = chats.get(id); if (!chat) { - res.status(404).send('chat not found'); + // Stale link / deleted chat → send the user home instead of a dead-end + // 404 page. Keeps navigation friendly for non-technical users. + res.redirect('/'); return; } if (chats.markRead(id)) wsHub.broadcastAll({ type: 'chat-read', chatId: id }); const { agents, error: agentsError } = await getAgentsSafe(); + const chatMessages = messages.listByChat(id); + // The chat's "current" mode = the most recent message's (non-ephemeral, + // still-selectable) mode, so reopening a chat restores the mode it was last + // used in instead of snapping back to the global default. Empty string when + // the chat has no usable mode yet (brand-new chat) — the client then falls + // back to the UI default. + let chatCurrentMode = ''; + for (let i = chatMessages.length - 1; i >= 0; i--) { + const m = chatMessages[i].mode; + if (m && isSelectableMode(m) && !isEphemeralMode(m as ChatMode)) { + chatCurrentMode = m; + break; + } + } res.render('chat', { chats: chats.list(), allProjects: projects.list(), hasAnyTasks: tasks.hasAny(), taskStatusSignals: tasks.statusSignals(), activeChat: chat, - chatMessages: messages.listByChat(id), + chatMessages, agents, agentsError, defaultAgent: DEFAULT_AGENT, @@ -180,6 +330,13 @@ chatsRouter.get('/:id', async (req, res, next) => { scheduledList: scheduledMessages.listByChat(id), queueList: queuedMessages.listByChat(id), sendHintShow: shouldShowSendHint(), + chatModes: listSelectableModes(), + defaultChatMode: defaultComposerMode(), + chatCurrentMode, + sttEnabled: openRouterEnabled(), + // Full Power (Execute) needs the gateway; agents.list succeeding implies it's + // reachable. Seeds the composer's Full Power gating (no badge on this page). + gatewayOk: !agentsError, }); } catch (err) { next(err); @@ -619,6 +776,7 @@ chatsRouter.post('/:id/queue', (req, res) => { replyTo: replyTo ?? null, attachments: persistedAttachments, inlineSecrets: inlineSecrets ?? null, + mode: normalizeChatMode(req.body?.mode), }); wsHub.broadcastAll({ type: 'queue-added', chatId: id, item: row }); res.json({ item: row }); @@ -696,6 +854,7 @@ chatsRouter.post('/:id/queue/:queueId/flush', async (req, res) => { prePersistedAttachments: row.attachments && row.attachments.length > 0 ? row.attachments : undefined, requestId: String(req.body?.requestId ?? '').trim() || undefined, + mode: row.mode, }); res.json({ ok: true }); } catch (err) { @@ -934,3 +1093,49 @@ chatsRouter.patch('/:id/scheduled/:scheduledId', (req, res) => { .json({ error: err instanceof Error ? err.message : 'failed to update' }); } }); + +/** GET /chats/:id/workspace-info — workspace size for Work/Secure Mode. */ +chatsRouter.get('/:id/workspace-info', async (req, res) => { + const chatId = Number(req.params.id); + const sessionId = getWorkSessionId(chatId); + if (!sessionId) return res.json({ active: false }); + const info = await getWorkspaceInfo(sessionId); + res.json({ active: true, sessionId, ...info }); +}); + +/** + * POST /chats/:id/destroy-workspace — tear down the chat's Safe/Work sandbox: + * stops the container and deletes the workspace (the Safe-mode copy and all of + * its contents). The next message starts a fresh sandbox. + */ +chatsRouter.post('/:id/destroy-workspace', async (req, res) => { + const chatId = Number(req.params.id); + if (!Number.isFinite(chatId)) return res.status(400).json({ error: 'bad chat id' }); + const destroyed = await destroyWorkSession(chatId); + res.json({ destroyed }); +}); + +/** POST /chats/:id/export-sandbox — copy the Safe sandbox out to a host folder. */ +chatsRouter.post('/:id/export-sandbox', async (req, res) => { + const chatId = Number(req.params.id); + if (!Number.isFinite(chatId)) return res.status(400).json({ error: 'bad chat id' }); + const destDir = typeof req.body?.destDir === 'string' ? req.body.destDir : undefined; + try { + const result = await exportChatSandbox(chatId, destDir); + res.json(result); + } catch (err) { + res.status(502).json({ ok: false, error: err instanceof Error ? err.message : String(err) }); + } +}); + +/** POST /chats/:id/apply-sandbox — apply the sandbox's changes to the originals. */ +chatsRouter.post('/:id/apply-sandbox', async (req, res) => { + const chatId = Number(req.params.id); + if (!Number.isFinite(chatId)) return res.status(400).json({ error: 'bad chat id' }); + try { + const results = await applyChatSandboxChanges(chatId); + res.json({ results }); + } catch (err) { + res.status(502).json({ error: err instanceof Error ? err.message : String(err) }); + } +}); diff --git a/src/routes/docker.ts b/src/routes/docker.ts new file mode 100644 index 0000000..c766997 --- /dev/null +++ b/src/routes/docker.ts @@ -0,0 +1,41 @@ +/** + * Docker status + lifecycle for the chat composer's Docker gate. + * + * GET /api/docker/status — current daemon readiness (cached probe) + * POST /api/docker/start — launch an installed-but-idle daemon (localhost) + * POST /api/docker/install — install the engine, then start it (localhost) + * + * The actions run host commands, so they're localhost-only. They return at once + * with a `starting`/`installing` state; the composer polls /status until ready. + */ + +import { Router } from 'express'; +import { + DOCKER_SIZE_HINT, + getDockerState, + installDocker, + startDocker, +} from '../services/docker'; +import { isLocalhostRequest } from '../services/gatewayStart'; + +export const dockerRouter: Router = Router(); + +dockerRouter.get('/status', async (_req, res) => { + res.json({ state: await getDockerState(), sizeHint: DOCKER_SIZE_HINT }); +}); + +dockerRouter.post('/start', (req, res) => { + if (!isLocalhostRequest(req)) { + res.status(403).json({ error: 'forbidden' }); + return; + } + res.json({ state: startDocker() }); +}); + +dockerRouter.post('/install', (req, res) => { + if (!isLocalhostRequest(req)) { + res.status(403).json({ error: 'forbidden' }); + return; + } + res.json({ state: installDocker() }); +}); diff --git a/src/routes/index.ts b/src/routes/index.ts index bb1fbc2..e2fb6c7 100644 --- a/src/routes/index.ts +++ b/src/routes/index.ts @@ -1,18 +1,142 @@ -import { Router } from 'express'; +import express, { Router } from 'express'; +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; import { chats, projectSecrets, projects, tasks } from '../services/store'; import { openclaw } from '../services/openclaw'; import { probeGateway } from '../services/gatewayProbe'; import { chatStatus } from '../services/chatStatus'; import { shouldShowSendHint } from '../services/sendHint'; +import { defaultComposerMode, listSelectableModes } from '../services/chatModes'; +import { openRouterEnabled, transcribeAudio, isOpenRouterFailure } from '../services/openRouter'; +import { isOnboardingDone, setOnboardingDone } from '../services/config'; +import { startOnboardingPrep, getOnboardingEnv } from '../services/onboardingEnv'; + +const execFileAsync = promisify(execFile); export const indexRouter: Router = Router(); +/** + * First-run welcome screen. Shown until the user picks a power source (or skips) + * — gated solely on the `onboarding.done` flag so it never reappears after. + * Starts the background environment prep (Docker probe + image pre-pull) so the + * slow download happens while the user reads the copy / pastes their key. + */ +indexRouter.get('/welcome', (_req, res) => { + startOnboardingPrep(); + res.render('welcome', { + title: 'Welcome to iClaw', + hasKey: openRouterEnabled(), + }); +}); + +/** Honest background-prep status for the welcome progress line. */ +indexRouter.get('/api/onboarding/status', (_req, res) => { + res.json(getOnboardingEnv()); +}); + +/** Finish (or skip) onboarding — flips the flag so /welcome never shows again. */ +indexRouter.post('/api/onboarding/complete', (_req, res) => { + setOnboardingDone(); + res.json({ ok: true }); +}); + +/** Native OS folder picker — opens system dialog, returns selected path. */ +indexRouter.post('/api/pick-folder', async (_req, res) => { + try { + let folderPath: string; + if (process.platform === 'darwin') { + const { stdout } = await execFileAsync('osascript', [ + '-e', 'POSIX path of (choose folder with prompt "Select a folder for Work Mode")', + ]); + folderPath = stdout.trim().replace(/\/$/, ''); + } else if (process.platform === 'linux') { + const { stdout } = await execFileAsync('zenity', ['--file-selection', '--directory', '--title=Select folder for Work Mode']).catch(() => + execFileAsync('kdialog', ['--getexistingdirectory', process.env.HOME ?? '/']), + ); + folderPath = stdout.trim(); + } else { + return res.status(400).json({ error: 'Folder picker not supported on this platform' }); + } + if (!folderPath) return res.status(400).json({ error: 'No folder selected' }); + res.json({ path: folderPath }); + } catch (err: unknown) { + // User cancelled dialog — not an error + const msg = err instanceof Error ? err.message : String(err); + if (msg.includes('User canceled') || msg.includes('cancelled') || msg.includes('-128')) { + return res.status(204).end(); + } + res.status(500).json({ error: msg }); + } +}); + /** Draft composer — secret name check before the chat row exists. */ indexRouter.get('/api/secrets/check-label', (req, res) => { res.json({ available: projectSecrets.isLabelAvailable(String(req.query.label ?? '')) }); }); +/** Map an upload mime to the container hint OpenRouter expects for input_audio. */ +function audioFormatFromMime(mime: string): string { + const m = mime.toLowerCase(); + if (m.includes('webm')) return 'webm'; + if (m.includes('ogg')) return 'ogg'; + if (m.includes('wav')) return 'wav'; + if (m.includes('mpeg') || m.includes('mp3')) return 'mp3'; + if (m.includes('mp4') || m.includes('m4a') || m.includes('aac')) return 'm4a'; + if (m.includes('flac')) return 'flac'; + return 'webm'; +} + +/** + * Speech-to-text. The composer mic POSTs the recorded clip as a raw audio body + * (Content-Type carries the container). We transcribe it via OpenRouter and + * return `{ text }`. Gated on OPENROUTER_API_KEY — the mic button is only + * rendered when that's set; this 503 covers a stale client. + */ +indexRouter.post( + '/api/stt', + express.raw({ type: () => true, limit: '25mb' }), + async (req, res) => { + if (!openRouterEnabled()) { + res.status(503).json({ error: 'Speech-to-text is unavailable: set OPENROUTER_API_KEY.' }); + return; + } + const buf = req.body; + if (!Buffer.isBuffer(buf) || buf.length === 0) { + res.status(400).json({ error: 'No audio received.' }); + return; + } + const mime = String(req.headers['content-type'] || 'audio/webm'); + try { + const text = await transcribeAudio({ + audioBase64: buf.toString('base64'), + format: audioFormatFromMime(mime), + }); + res.json({ text }); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + console.warn('[stt] transcription failed:', msg); + res + .status(502) + .json({ error: isOpenRouterFailure(err) ? 'Transcription failed (OpenRouter).' : msg }); + } + }, +); + indexRouter.get('/', async (req, res) => { + // First run: render the welcome flow in place of the empty chat. Rendered + // (not redirected) so `/` stays 200 — the CLI/instance readiness probe in + // startup.ts checks for a 200 that contains "iClaw". + if (!isOnboardingDone()) { + // Power users who already configured a key (env/prior run) shouldn't see + // the welcome at all — mark it done and fall through to the app. + if (openRouterEnabled()) { + setOnboardingDone(); + } else { + startOnboardingPrep(); + return res.render('welcome', { title: 'Welcome to iClaw', hasKey: openRouterEnabled() }); + } + } + const list = chats.list(); const allProjects = projects.list(); @@ -27,6 +151,9 @@ indexRouter.get('/', async (req, res) => { res.render('index', { chats: list, + // First-ever empty state → show the conversational welcome (and skip the + // project picker) instead of dropping a non-technical user into a blank chat. + isFirstChat: list.length === 0, allProjects, hasAnyTasks: tasks.hasAny(), taskStatusSignals: tasks.statusSignals(), @@ -41,5 +168,13 @@ indexRouter.get('/', async (req, res) => { openclawBaseUrl: openclaw.baseUrl, workingIds: chatStatus.workingIds(), sendHintShow: shouldShowSendHint(), + chatModes: listSelectableModes(), + defaultChatMode: defaultComposerMode(), + sttEnabled: openRouterEnabled(), + // With an OpenRouter key the runtime modes (Work / Safe work / Incognito) + // work without OpenClaw — so a missing gateway must NOT block starting a chat. + openRouterReady: openRouterEnabled(), + // Full Power (Execute) needs a reachable gateway; seeds the composer gating. + gatewayOk: gatewayStatus === 'ok', }); }); diff --git a/src/routes/projects.ts b/src/routes/projects.ts index 8e7998a..68e2470 100644 --- a/src/routes/projects.ts +++ b/src/routes/projects.ts @@ -10,7 +10,7 @@ import { Router } from 'express'; import { listProjectLinkGroups } from '../services/projectLinks'; -import { chats, projects, projectFacts, projectSecrets, tasks, enrichFactsWithSourceChatTitles, enrichFactWithSourceChatTitle } from '../services/store'; +import { chats, projects, projectFacts, projectSkills, projectSecrets, tasks, enrichFactsWithSourceChatTitles, enrichFactWithSourceChatTitle, enrichSkillsWithSourceChatTitles, enrichSkillWithSourceChatTitle } from '../services/store'; import { chatStatus } from '../services/chatStatus'; import { wsHub } from '../services/wsHub'; import { openclaw } from '../services/openclaw'; @@ -77,7 +77,8 @@ projectsRouter.get('/:id', (req, res) => { const id = Number(req.params.id); const project = projects.get(id); if (!project) { - res.status(404).send('project not found'); + // Stale link / deleted project → home, not a dead-end 404 page. + res.redirect('/'); return; } const linkGroups = listProjectLinkGroups(id); @@ -89,6 +90,7 @@ projectsRouter.get('/:id', (req, res) => { project, projectChats: chats.listByProject(id), facts: enrichFactsWithSourceChatTitles(projectFacts.listByProject(id)), + skills: enrichSkillsWithSourceChatTitles(projectSkills.listForProject(id)), projectSecrets: projectSecrets.listMetaByProject(id), projectWebLinks: linkGroups.web, projectFileLinks: linkGroups.files, @@ -186,7 +188,8 @@ projectsRouter.get('/:id/secrets/:secretId/value', (req, res) => { projectsRouter.post('/:id/delete', (req, res) => { const id = Number(req.params.id); if (!projects.get(id)) { - res.status(404).send('project not found'); + // Already gone → mirror the success path back to the projects list. + res.redirect('/projects'); return; } const detachedChatIds = chats.listByProject(id).map((c) => c.id); @@ -247,3 +250,80 @@ projectsRouter.post('/:id/facts/:factId/delete', (req, res) => { if (wantsJson(req)) res.json({ id: factId, deleted: true }); else res.redirect(`/projects/${projectId}`); }); + +/* ---------------- skills (procedural memory; edit/delete here) ---------------- */ + +/** Skill belongs to the project when project-scoped, or is global (project_id null). */ +function skillVisibleToProject( + skill: { project_id: number | null } | undefined, + projectId: number, +): boolean { + return !!skill && (skill.project_id === projectId || skill.project_id === null); +} + +/** Active skills (project + global), index info only. */ +projectsRouter.get('/:id/skills', (req, res) => { + const id = Number(req.params.id); + if (!projects.get(id)) { + res.status(404).json({ error: 'project not found' }); + return; + } + res.json({ skills: enrichSkillsWithSourceChatTitles(projectSkills.listForProject(id)) }); +}); + +/** Full skill body (for the view/edit modal). */ +projectsRouter.get('/:id/skills/:skillId', (req, res) => { + const projectId = Number(req.params.id); + const skillId = Number(req.params.skillId); + const skill = projectSkills.get(skillId); + if (!projects.get(projectId) || !skillVisibleToProject(skill, projectId)) { + res.status(404).json({ error: 'not found' }); + return; + } + res.json({ skill: enrichSkillWithSourceChatTitle(skill!) }); +}); + +projectsRouter.patch('/:id/skills/:skillId', (req, res) => { + const projectId = Number(req.params.id); + const skillId = Number(req.params.skillId); + const skill = projectSkills.get(skillId); + if (!projects.get(projectId) || !skillVisibleToProject(skill, projectId)) { + res.status(404).json({ error: 'not found' }); + return; + } + const patch: { name?: string; description?: string; body?: string; tags?: string[] | null } = {}; + if (typeof req.body?.name === 'string' && req.body.name.trim()) patch.name = req.body.name; + if (typeof req.body?.description === 'string' && req.body.description.trim()) + patch.description = req.body.description; + if (typeof req.body?.body === 'string' && req.body.body.trim()) patch.body = req.body.body; + if (Array.isArray(req.body?.tags)) { + const tags = req.body.tags.filter((t: unknown): t is string => typeof t === 'string'); + patch.tags = tags.length > 0 ? tags : null; + } + try { + projectSkills.update(skillId, patch); + const updated = projectSkills.get(skillId)!; + wsHub.broadcastAll({ + type: 'project-skill-updated', + projectId, + skill: enrichSkillWithSourceChatTitle(updated), + }); + res.json(updated); + } catch (err) { + res.status(400).json({ error: err instanceof Error ? err.message : 'update failed' }); + } +}); + +projectsRouter.post('/:id/skills/:skillId/delete', (req, res) => { + const projectId = Number(req.params.id); + const skillId = Number(req.params.skillId); + const skill = projectSkills.get(skillId); + if (!skillVisibleToProject(skill, projectId)) { + res.status(404).json({ error: 'not found' }); + return; + } + projectSkills.remove(skillId); + wsHub.broadcastAll({ type: 'project-skill-deleted', projectId, skillId }); + if (wantsJson(req)) res.json({ id: skillId, deleted: true }); + else res.redirect(`/projects/${projectId}`); +}); diff --git a/src/routes/settings.ts b/src/routes/settings.ts index 480b82c..624ca25 100644 --- a/src/routes/settings.ts +++ b/src/routes/settings.ts @@ -1,9 +1,9 @@ /** * GET /settings — Settings page. * - * For now the only section is Remote Access, but the page is built as a - * sectioned scaffold so future settings can land alongside without a - * separate URL. + * Sectioned scaffold: Remote Access + OpenRouter (the key that unlocks voice + * messages, Ask mode, and smart titles). The key is stored in the local DB via + * Settings — not an env var — and takes effect on the next page load. */ import { Router } from 'express'; @@ -12,12 +12,20 @@ import { chats, projects, tasks } from '../services/store'; import { chatStatus } from '../services/chatStatus'; import { openclaw } from '../services/openclaw'; import { remoteAccess, ALLOWED_DURATIONS_MS } from '../services/remoteAccess'; +import { + maskOpenRouterApiKey, + setOpenRouterApiKey, + clearOpenRouterApiKey, +} from '../services/config'; +import { openRouterEnabled, fetchUsage, isOpenRouterFailure, validateKey } from '../services/openRouter'; +import { runtimeProcess } from '../services/runtimeProcess'; + export const settingsRouter = Router(); -settingsRouter.get('/settings', (_req, res) => { - res.render('settings', { +/** Locals every settings sub-page needs for the shared sidebar + shell. */ +function sidebarLocals() { + return { title: 'Settings — iClaw', - // Sidebar locals. chats: chats.list(), workingIds: chatStatus.workingIds(), allProjects: projects.list(), @@ -29,8 +37,95 @@ settingsRouter.get('/settings', (_req, res) => { activeTasksList: false, activeSettings: true, openclawBaseUrl: openclaw.baseUrl, - // Page-specific. + }; +} + +/** /settings → first sub-page. Each section is now its own page. */ +settingsRouter.get('/settings', (_req, res) => { + res.redirect('/settings/voice-ask'); +}); + +settingsRouter.get('/settings/voice-ask', (_req, res) => { + res.render('settings', { + ...sidebarLocals(), + settingsTab: 'voice-ask', + openRouter: { + hasKey: openRouterEnabled(), + maskedKey: maskOpenRouterApiKey(), + }, + }); +}); + +settingsRouter.get('/settings/remote-access', (_req, res) => { + res.render('settings', { + ...sidebarLocals(), + settingsTab: 'remote-access', tunnels: remoteAccess.list(), allowedDurationsMs: ALLOWED_DURATIONS_MS, }); }); + +/** + * Validate a key WITHOUT saving it — onboarding (and Settings) calls this before + * committing, so a dead/zero-balance key is caught up front instead of in chat. + */ +settingsRouter.post('/api/openrouter/validate', async (req, res) => { + const key = typeof req.body?.key === 'string' ? req.body.key.trim() : ''; + if (!key) { + res.status(400).json({ valid: false, reason: 'empty' }); + return; + } + try { + const result = await validateKey(key); + res.json(result); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + res.status(502).json({ valid: false, reason: 'network', error: msg }); + } +}); + +/** Save / update the OpenRouter API key. Takes effect on the next page load. */ +settingsRouter.post('/api/openrouter/key', (req, res) => { + const key = typeof req.body?.key === 'string' ? req.body.key.trim() : ''; + if (!key) { + res.status(400).json({ error: 'Paste your OpenRouter API key first.' }); + return; + } + // OpenRouter keys look like `sk-or-...`. Be lenient (warn-not-block) so a + // future key format still works, but catch obvious paste mistakes. + if (!/^sk-or-/.test(key)) { + res.status(400).json({ error: 'That doesn’t look like an OpenRouter key (expected to start with “sk-or-”).' }); + return; + } + setOpenRouterApiKey(key); + // The runtime reads the key from the DB only at spawn — restart it so Work / + // Safe / Incognito modes work immediately after onboarding, without an app + // restart. + runtimeProcess.restart(); + res.json({ ok: true, maskedKey: maskOpenRouterApiKey() }); +}); + +/** Disconnect — remove the stored key. */ +settingsRouter.delete('/api/openrouter/key', (_req, res) => { + clearOpenRouterApiKey(); + // Drop the key from the running runtime too. + runtimeProcess.restart(); + res.json({ ok: true }); +}); + +/** Spend / credits readout for the connected key. */ +settingsRouter.get('/api/openrouter/usage', async (_req, res) => { + if (!openRouterEnabled()) { + res.status(404).json({ error: 'No OpenRouter key connected.' }); + return; + } + try { + const usage = await fetchUsage(); + res.json({ ok: true, usage }); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + res + .status(502) + .json({ error: isOpenRouterFailure(err) ? 'Could not load usage from OpenRouter.' : msg }); + } +}); diff --git a/src/routes/ws.ts b/src/routes/ws.ts index 68fad60..20e7fab 100644 --- a/src/routes/ws.ts +++ b/src/routes/ws.ts @@ -8,13 +8,36 @@ import type { Server as HttpServer, IncomingMessage } from 'node:http'; import { WebSocketServer, type WebSocket } from 'ws'; import { chats } from '../services/store'; import { wsHub } from '../services/wsHub'; -import { sendMessage, abortChatRun } from '../services/chatRunner'; +import { sendMessage, abortChatRun, runIncognitoTurn, abortIncognito, type WorkFolder } from '../services/chatRunner'; import { openclawWs } from '../services/openclawWs'; import type { ClientMsg, ServerMsg } from '../types/protocol'; import type { InlineSecretWire } from '../services/inlineSecrets'; +import { normalizeChatMode } from '../services/chatModes'; const PATH = '/ws'; +/** + * Coerce the untrusted `workFolders` wire field into typed WorkFolder[]. Accepts + * either objects ({ path, readonly }) from the current client or bare path + * strings from older clients. Missing/unknown readonly defaults to true + * (read-only) — the safe default. Returns undefined when nothing valid is sent. + */ +function parseWorkFolders(raw: unknown): WorkFolder[] | undefined { + if (!Array.isArray(raw)) return undefined; + const out: WorkFolder[] = []; + for (const entry of raw) { + if (typeof entry === 'string') { + if (entry) out.push({ path: entry, readonly: true }); + } else if (entry && typeof entry === 'object') { + const o = entry as Record<string, unknown>; + if (typeof o.path === 'string' && o.path) { + out.push({ path: o.path, readonly: o.readonly !== false }); + } + } + } + return out.length > 0 ? out : undefined; +} + function send(socket: WebSocket, msg: ServerMsg): void { wsHub.send(socket, msg); } @@ -90,6 +113,12 @@ async function handleClientMsg(socket: WebSocket, msg: ClientMsg): Promise<void> replyTo: msg.replyTo, incomingAttachments: msg.attachments, inlineSecrets, + mode: normalizeChatMode((msg as { mode?: unknown }).mode), + networkEnabled: (msg as Record<string, unknown>).networkEnabled === true, + ttlDays: typeof (msg as Record<string, unknown>).ttlDays === 'number' + ? ((msg as Record<string, unknown>).ttlDays as number) + : undefined, + workFolders: parseWorkFolders((msg as Record<string, unknown>).workFolders), }); } catch (err) { // Errors are already broadcast via chatRunner; nothing more to do. @@ -106,6 +135,39 @@ async function handleClientMsg(socket: WebSocket, msg: ClientMsg): Promise<void> } return; + case 'incognito-send': { + const key = String((msg as { key?: unknown }).key ?? '').trim(); + const content = String(msg.content ?? '').trim(); + if (!key || !content) return; + let endedTokens: number | undefined; + let endedCached: number | undefined; + try { + const res = await runIncognitoTurn({ + key, + content, + workFolders: parseWorkFolders((msg as Record<string, unknown>).workFolders), + onEvent: (e) => { + if (e.type === 'text-delta') send(socket, { type: 'incognito-turn-delta', key, text: e.text }); + else if (e.type === 'tool') send(socket, { type: 'incognito-turn-tool', key, name: e.name }); + else if (e.type === 'error') send(socket, { type: 'incognito-error', key, message: e.message }); + }, + }); + endedTokens = res.tokens; + endedCached = res.cached; + } catch (err) { + send(socket, { type: 'incognito-error', key, message: err instanceof Error ? err.message : String(err) }); + } + // Always close the turn so the client can re-enable the composer. + send(socket, { type: 'incognito-turn-ended', key, tokens: endedTokens, cached: endedCached }); + return; + } + + case 'incognito-abort': { + const key = String((msg as { key?: unknown }).key ?? '').trim(); + if (key) await abortIncognito(key).catch((err) => console.error('[ws] incognito-abort failed', err)); + return; + } + case 'exec-approval': { const approvalId = String(msg.approvalId ?? '').trim(); const decision = msg.decision === 'denied' ? 'denied' : 'approved'; diff --git a/src/services/chatModes.ts b/src/services/chatModes.ts new file mode 100644 index 0000000..466368f --- /dev/null +++ b/src/services/chatModes.ts @@ -0,0 +1,184 @@ +/** + * Single source of truth for chat "modes" — the distinctions the composer + * offers, plus a place to grow future modes without hardcoding them everywhere. + * + * Design notes + * ------------ + * - The wire/DB value is a plain string. The TS union (`ChatMode` in ../types) + * names the live modes; the catalog below can list `enabled: false` + * placeholders for modes we haven't built yet. Adding a live mode later = + * flip `enabled` and (if it needs a new union member) widen `ChatMode`. No DB + * migration is needed because the column is TEXT. + * - `DEFAULT_MODE` is 'execute' so anything that doesn't specify a mode — + * legacy rows, older clients, scheduled messages, task runs — behaves exactly + * as before. `normalizeChatMode()` enforces this fallback. + * + * Backends + * -------- + * - execute → the chat's main OpenClaw agent session (full tools). + * - work / secure / incognito → iclaw-runtime (our runtime), which requires + * OPENROUTER_API_KEY to be configured. + * + * Incognito (read-only, ephemeral) + * -------------------------------- + * A privacy mode: read files in ANY folder, read-only shell in a sandbox over + * folders the user selects, and web research — but NO writes and NO other + * actions. The conversation is ephemeral (kept only in the browser tab, never + * persisted) and contributes nothing to project memory. Enforcement lives in + * the runtime; the host treats incognito turns as non-persistent. + */ + +import type { ChatMode } from '../types'; +import { openRouterEnabled } from './openRouter'; + +export interface ChatModeDef { + /** Wire/DB value. */ + id: string; + /** Short label for the selector chip. */ + label: string; + /** One-liner shown in the selector / tooltip. */ + description: string; + /** Whether the mode is selectable today. Placeholders are `false`. */ + enabled: boolean; + /** Runs on iclaw-runtime (requires OPENROUTER_API_KEY), not the OpenClaw agent. */ + runtimeBacked: boolean; + /** Ephemeral, never-persisted conversation (incognito). */ + ephemeral: boolean; + /** + * Mode is unusable without a running Docker daemon. Safe work IS the Docker + * sandbox, so it fails closed without it. Work/Incognito only LOSE shell + * commands (run_command) — their file tools run on the host — so they stay + * usable and are NOT marked here; the UI shows a softer "shell needs Docker" + * hint for them instead of disabling the mode. + */ + requiresDocker: boolean; +} + +/** + * The full catalog. Order here is the order shown in the UI. Disabled entries + * are intentionally kept so the selector and any future router can enumerate + * the roadmap without scattering string literals across the codebase. + */ +export const CHAT_MODES: readonly ChatModeDef[] = [ + { + id: 'work', + label: 'Work', + description: 'AI edits files in folders you choose - you approve every change', + enabled: true, + runtimeBacked: true, + ephemeral: false, + requiresDocker: false, + }, + { + id: 'secure', + label: 'Safe work & Internet research', + description: 'Locked sandbox - safely run untrusted code and research the web, isolated from your system', + enabled: true, + runtimeBacked: true, + ephemeral: false, + requiresDocker: true, + }, + { + id: 'execute', + label: 'Full Power', + description: 'Full access via OpenClaw - for complex tasks that need more power', + enabled: true, + runtimeBacked: false, + ephemeral: false, + requiresDocker: false, + }, + // Set apart at the bottom (the composer renders a divider before any + // `ephemeral` mode) — it's a distinct, off-the-record surface. + { + id: 'incognito', + label: 'Incognito', + description: 'Private, read-only research - reads files & the web, never writes, nothing saved', + enabled: true, + runtimeBacked: true, + ephemeral: true, + requiresDocker: false, + }, + // --- Planned modes (not selectable yet) ------------------------------- + { + id: 'image', + label: 'Image', + description: 'Generate or edit images. (Coming soon.)', + enabled: false, + runtimeBacked: false, + ephemeral: false, + requiresDocker: false, + }, +] as const; + +/** Mode used whenever none is supplied or the supplied one is unknown/disabled. */ +export const DEFAULT_MODE: ChatMode = 'execute'; + +/** + * UI default for the composer's mode selector on a NEW/empty chat. Deliberately + * separate from DEFAULT_MODE: the latter is the backend normalization fallback + * (mode-less posted/API messages keep routing to the OpenClaw agent), while this + * only seeds what the picker shows when there's nothing else to go on. + */ +export const DEFAULT_COMPOSER_MODE: ChatMode = 'work'; + +/** Modes a client is allowed to select right now. */ +export const ENABLED_MODE_IDS: readonly string[] = CHAT_MODES.filter( + (m) => m.enabled, +).map((m) => m.id); + +/** + * A mode is available when it's enabled AND its backend is reachable. The + * runtime-backed modes (Work / Secure / Incognito) run on iclaw-runtime, which + * requires OpenRouter — so they're hidden (and posted values coerced to + * Execute) when no key is configured. + */ +function modeAvailable(def: ChatModeDef): boolean { + if (!def.enabled) return false; + if (def.runtimeBacked && !openRouterEnabled()) return false; + return true; +} + +/** Available modes only — feeds the composer selector (EJS locals / client). */ +export function listSelectableModes(): ChatModeDef[] { + return CHAT_MODES.filter(modeAvailable); +} + +function findMode(id: string): ChatModeDef | undefined { + return CHAT_MODES.find((m) => m.id === id); +} + +/** True only for an available (enabled + backend-reachable) known mode id. */ +export function isSelectableMode(id: string): boolean { + const def = findMode(id); + return Boolean(def && modeAvailable(def)); +} + +/** + * UI default for the composer, with a graceful fallback: Work is runtime-backed, + * so when it isn't currently selectable (e.g. no OpenRouter key) we fall back to + * the always-available DEFAULT_MODE instead of pre-selecting a hidden mode. + */ +export function defaultComposerMode(): ChatMode { + return isSelectableMode(DEFAULT_COMPOSER_MODE) ? DEFAULT_COMPOSER_MODE : DEFAULT_MODE; +} + +/** + * Coerce any untrusted input (query body, WS frame, DB row) into a valid, + * currently-selectable `ChatMode`. Unknown, disabled, or missing → DEFAULT_MODE. + */ +export function normalizeChatMode(raw: unknown): ChatMode { + if (typeof raw !== 'string') return DEFAULT_MODE; + const id = raw.trim().toLowerCase(); + if (isSelectableMode(id)) return id as ChatMode; + return DEFAULT_MODE; +} + +/** Definition for a (normalized) mode — always defined for selectable modes. */ +export function getModeDef(mode: ChatMode): ChatModeDef { + return findMode(mode) ?? findMode(DEFAULT_MODE)!; +} + +/** True for the ephemeral, never-persisted incognito mode. */ +export function isEphemeralMode(mode: ChatMode): boolean { + return getModeDef(mode).ephemeral; +} diff --git a/src/services/chatRunner.ts b/src/services/chatRunner.ts index b233895..c1d2491 100644 --- a/src/services/chatRunner.ts +++ b/src/services/chatRunner.ts @@ -7,8 +7,10 @@ */ import { randomUUID } from 'node:crypto'; -import { chats, messages, projects, projectSecrets } from './store'; +import { chats, messages, projects, projectSecrets, projectFacts } from './store'; import { buildGatewayUserMessage, scheduleProjectFactExtraction } from './projectMemory'; +import { buildSkillsPromptBlock, scheduleProjectSkillReview } from './projectSkills'; +import { projectSkills } from './store'; import { chatStatus } from './chatStatus'; import { openclawWs, type TurnEvent } from './openclawWs'; import { deriveTitle, suggestChatTitleWithTimeout } from './chatTitle'; @@ -17,10 +19,14 @@ import { wsHub } from './wsHub'; import { gatewayAttachmentsFromPersisted, persistIncomingAttachments, + runtimeAttachmentsFromPersisted, type IncomingAttachment, type ProcessedAttachment, } from './uploads'; -import type { Message, MessageAttachment } from '../types'; +import type { ChatMode, Message, MessageAttachment } from '../types'; +import { DEFAULT_MODE } from './chatModes'; +import { buildCompactedHistory } from './contextCompaction'; +import { createWorkSession, sendWorkMessage, subscribeWorkEvents, stopWorkSession, abortWorkSession, exportSandbox, applySandboxChanges, type ExportResult, type ApplyResult } from './workRuntime'; import { expandStoredSecretPlaceholdersForGateway, resolveInlineSecretMarkersInContent, @@ -30,6 +36,22 @@ import { const DEFAULT_AGENT = 'openclaw/default'; +/** + * A folder granted to a Work Mode chat, with its access level. `readonly: true` + * means the agent may read/list/search but not write_file or run_command under + * it; `readonly: false` grants read & write. New folders default to read-only. + */ +export interface WorkFolder { + path: string; + readonly: boolean; +} + +/** + * OpenClaw session key currently executing an Execute turn for a chat. Lets + * `abortChatRun` stop the right gateway run. Cleared in the runTurn `finally`. + */ +const activeRunSessionKeys = new Map<number, string>(); + /** * Map an iClaw agent label ("openclaw/default", "openclaw/code", ...) to the * raw OpenClaw agentId ("main", "code", ...). @@ -211,6 +233,11 @@ async function runTurnLocked(opts: { /** Attachments already saved under data/uploads (queued-message flush). */ prePersistedAttachments?: MessageAttachment[]; inlineSecrets?: InlineSecretWire[]; + /** 'execute' | 'work' | 'secure' | 'incognito'. Defaults to 'execute'. */ + mode?: ChatMode; + workFolders?: WorkFolder[]; + networkEnabled?: boolean; + ttlDays?: number; }): Promise<void> { const { chatId, @@ -221,8 +248,8 @@ async function runTurnLocked(opts: { prePersistedAttachments, inlineSecrets, } = opts; + const mode: ChatMode = opts.mode ?? DEFAULT_MODE; const chat = chats.get(chatId)!; - const sessionKey = await ensureSession(chatId); const projectId = chat.project_id ?? null; let storedUserContent = content; @@ -268,10 +295,12 @@ async function runTurnLocked(opts: { gatewayBody = formatReplyGatewayBlock(refExpanded, reply.quote) + gatewayBody; } - const gatewayMessage = + const gatewayMessageBase = chat.project_id != null && projects.get(chat.project_id) ? buildGatewayUserMessage(gatewayBody, chat.project_id) : gatewayBody; + // The turn is dispatched AFTER the user row is persisted (Ask needs + // userMsg.id to seed prior-thread context). See the Ask/Execute branch below. // Persist user message + broadcast (stored text keeps placeholders only). const replyToRole = @@ -289,6 +318,7 @@ async function runTurnLocked(opts: { } : null, persistedAttachments.length > 0 ? persistedAttachments : null, + mode, ); for (const sid of newSecretIds) { projectSecrets.setSourceMessage(sid, userMsg.id); @@ -350,6 +380,9 @@ async function runTurnLocked(opts: { let switchedToGenerating = false; let assistantText = ''; + // Whether this turn invoked any tool — used to throttle skill review to + // "substantive" turns (procedural learning comes from tool use, not chit-chat). + let usedTools = false; const onEvent = (ev: TurnEvent): void => { if (ev.type === 'text-delta') { @@ -360,6 +393,7 @@ async function runTurnLocked(opts: { assistantText += ev.text; wsHub.broadcastToChat(chatId, { type: 'turn-delta', chatId, text: ev.text }); } else if (ev.type === 'tool-start') { + usedTools = true; chatStatus.setActivity(chatId, { kind: 'tool', name: ev.name, @@ -421,16 +455,42 @@ async function runTurnLocked(opts: { } }; - const { - text: gatewayAccumulated, - aborted, - authoritativeText, - } = await openclawWs.runTurn({ - sessionKey, - message: gatewayMessage, - onEvent, - attachments: gatewayAttachments.length > 0 ? gatewayAttachments : undefined, - }); + // Work / Secure Mode — routes to iclaw-runtime, returns early. Forward dropped + // files so the runtime agent can read them (Work) / stage them (Secure) and + // see images — otherwise the model has no idea a file was attached. + if (mode === 'work' || mode === 'secure') { + const runtimeAttachments = runtimeAttachmentsFromPersisted(chatId, persistedAttachments); + await runWorkModeTurn({ chatId, content: gatewayMessageBase, onEvent, workFolders: opts.workFolders, secure: mode === 'secure', networkEnabled: opts.networkEnabled, ttlDays: opts.ttlDays, beforeMsgId: userMsg.id, reviewUserMessage: storedUserContent, attachments: runtimeAttachments }); + wsHub.broadcastAll({ type: 'turn-ended', chatId, title: chats.get(chatId)?.title ?? '', aborted: false }); + return; + } + + // Execute: the chat's own main-agent OpenClaw session, full tools. + // Provision the gateway session lazily, HERE — so Work / Safe work / Incognito + // chats (which use iclaw-runtime, not the gateway) can start and run even when + // OpenClaw is unreachable, as long as an OpenRouter key is configured. + const sessionKey = await ensureSession(chatId); + let gatewayAccumulated = ''; + let aborted = false; + let authoritativeText: string | null = null; + let turnUsage: { tokens: number | null; cached: number | null } = { tokens: null, cached: null }; + + activeRunSessionKeys.set(chatId, sessionKey); + try { + const turn = await openclawWs.runTurn({ + sessionKey, + message: gatewayMessageBase, + onEvent, + attachments: gatewayAttachments.length > 0 ? gatewayAttachments : undefined, + }); + gatewayAccumulated = turn.text; + aborted = turn.aborted; + authoritativeText = turn.authoritativeText; + // `usage` is absent from older stubs/mocks — keep the null default then. + if (turn.usage) turnUsage = turn.usage; + } finally { + activeRunSessionKeys.delete(chatId); + } // Picking the assistant text in priority order: // @@ -469,7 +529,12 @@ async function runTurnLocked(opts: { const skipAssistant = aborted && finalText.trim().length === 0; const assistantMsg = skipAssistant ? null - : messages.append(chatId, 'assistant', finalText, aborted ? 'aborted' : null); + : messages.append( + chatId, 'assistant', finalText, aborted ? 'aborted' : null, null, null, + // Stamp the execute (OpenClaw) mode + dev-mode token usage resolved from + // the gateway history slice, mirroring the Work/Secure path. + 'execute', turnUsage.tokens, turnUsage.cached, + ); if (assistantMsg) { wsHub.broadcastToChat(chatId, { type: 'message-appended', @@ -479,6 +544,7 @@ async function runTurnLocked(opts: { syncSidebarUnread(chatId); } + // Persistent "Stopped by user" marker. Lives in `messages` so it // survives page reload + lands in the iClaw-cloud share payload. // Rendered exactly like the existing "Task done: …" / "Task created: @@ -511,6 +577,22 @@ async function runTurnLocked(opts: { assistantText: finalText, assistantMessageId: assistantMsg.id, }); + + // Procedural memory: throttled skill review (heavier than fact extraction, + // so only every Nth substantive turn). MVP untrusted heuristic: a tool-using + // turn with network reach may have ingested external content (web/email/etc.), + // so its distilled skills are flagged for extra scrutiny in the inbox. + // (This site handles Ask/Execute; Work/Secure returns earlier — see above.) + scheduleProjectSkillReview({ + chatId, + projectId: chatAfter.project_id, + sharesToProject: Boolean(chatAfter.shares_to_project), + substantive: usedTools, + userMessage: storedUserContent, + assistantText: finalText, + assistantMessageId: assistantMsg.id, + untrusted: usedTools && Boolean(opts.networkEnabled), + }); } // Wait for the title task (if any) so the final `turn-ended` reflects it. @@ -584,6 +666,14 @@ export async function sendMessage(opts: { incomingAttachments?: IncomingAttachment[]; /** Files already on disk (queued-message flush). */ prePersistedAttachments?: MessageAttachment[]; + /** 'ask' | 'execute'. Defaults to 'execute' when omitted. */ + mode?: ChatMode; + /** Allowed folders for Work Mode, each with a read-only / read&write flag. */ + workFolders?: WorkFolder[]; + /** Network toggle for Secure Mode. */ + networkEnabled?: boolean; + /** TTL in days for Secure Mode workspace. */ + ttlDays?: number; }): Promise<{ chatId: number }> { let chatId = opts.chatId; let isFirstTurn = false; @@ -629,6 +719,10 @@ export async function sendMessage(opts: { incomingAttachments: opts.incomingAttachments, prePersistedAttachments: opts.prePersistedAttachments, inlineSecrets: opts.inlineSecrets, + mode: opts.mode, + workFolders: opts.workFolders, + networkEnabled: opts.networkEnabled, + ttlDays: opts.ttlDays, }), ); } catch (err) { @@ -663,7 +757,420 @@ export async function sendMessage(opts: { return { chatId }; } +/** Build system prompt for Work/Secure Mode including project context. */ +function buildWorkSystemPrompt(chatId: number): string { + const chat = chats.get(chatId); + const lines: string[] = []; + + if (chat?.project_id) { + const project = projects.get(chat.project_id); + if (project?.name) lines.push(`Project: ${project.name}`); + if (project?.description) lines.push(`Description: ${project.description}`); + + const facts = projectFacts.listByProject(chat.project_id, 20); + if (facts.length > 0) { + lines.push('\nProject context:'); + facts.forEach((f) => lines.push(`- ${f.content}`)); + } + + const skillsBlock = buildSkillsPromptBlock(chat.project_id); + if (skillsBlock) { + lines.push('\n' + skillsBlock); + } + } + + return lines.length > 0 ? lines.join('\n') : ''; +} + +/** + * Persistent Work Mode sessions — one per chat, reused across turns. We store + * the folder signature alongside the sessionId so a mid-chat change to the + * folder set or a folder's read-only/read&write flag forces the session to be + * recreated with the new access (folderAccess is fixed at session creation). + */ +const workSessions = new Map< + number, + { sessionId: string; foldersKey: string; secure: boolean; skillsKey: string } +>(); + +/** + * Stable signature of the folders granted to a chat. Order-independent so the + * key only changes when a path or its readonly flag actually changes. + */ +function foldersSignature(folders?: WorkFolder[]): string { + if (!folders?.length) return ''; + return JSON.stringify( + folders + .map((f) => ({ p: f.path, r: f.readonly })) + .sort((a, b) => a.p.localeCompare(b.p)), + ); +} + +/** + * Stable signature of the active skills injected into a chat's session. Changes + * when a skill is accepted/edited/deleted (id+version), so the work session is + * recreated with the new skill set without a manual restart — same class of fix + * as folder-access / Work<->Secure mode changes. + */ +function skillsSignature(chatId: number): string { + const chat = chats.get(chatId); + if (!chat?.project_id) return ''; + const idx = projectSkills.listForProject(chat.project_id); + if (idx.length === 0) return ''; + return idx + .map((s) => `${s.id}:${s.version}`) + .sort() + .join(','); +} + +/** + * Skill-review cadence for Work/Secure. Smaller than the Ask/Execute interval: + * these turns are agentic and tool-heavy, so each one is more likely to contain + * a reusable procedure worth distilling. + */ +const WORK_SKILL_REVIEW_INTERVAL = 4; + +/** Get the runtime sessionId for a chat (if active). */ +export function getWorkSessionId(chatId: number): string | undefined { + return workSessions.get(chatId)?.sessionId; +} + +/** + * Destroy a chat's Work/Safe runtime session — stops the container AND deletes + * the workspace (the Safe sandbox copy and anything in it). Backs the + * "Destroy sandbox" button. The map entry is cleared so the next turn starts a + * fresh session (re-ingesting any selected folders for Safe Mode). + */ +export async function destroyWorkSession(chatId: number): Promise<boolean> { + const sessionId = workSessions.get(chatId)?.sessionId; + workSessions.delete(chatId); + if (!sessionId) return false; + try { + await stopWorkSession(sessionId); + } catch { + /* best-effort — the map is already cleared, so a stale session just TTLs out. */ + } + return true; +} + +/** Export a chat's Safe sandbox to a host folder. */ +export async function exportChatSandbox(chatId: number, destDir?: string): Promise<ExportResult> { + const sessionId = workSessions.get(chatId)?.sessionId; + if (!sessionId) return { ok: false, error: 'no active sandbox for this chat' }; + return exportSandbox(sessionId, destDir); +} + +/** Apply a chat's Safe sandbox changes back to the original folders. */ +export async function applyChatSandboxChanges(chatId: number): Promise<ApplyResult[]> { + const sessionId = workSessions.get(chatId)?.sessionId; + if (!sessionId) return []; + return applySandboxChanges(sessionId); +} + +/** + * Route a Work Mode turn to iclaw-runtime. + * Reuses the session across turns to preserve conversation history. + */ +async function runWorkModeTurn(opts: { + chatId: number; + content: string; + onEvent: (event: TurnEvent) => void; + workFolders?: WorkFolder[]; + secure?: boolean; + networkEnabled?: boolean; + ttlDays?: number; + /** Current user message id — history before it seeds the session context. */ + beforeMsgId?: number; + /** Original (stored) user text for skill review — without injected project prefix. */ + reviewUserMessage?: string; + /** Dropped files (absolute host paths) to forward to the runtime agent. */ + attachments?: { path: string; mimeType: string; fileName: string }[]; +}): Promise<void> { + const { chatId, content, onEvent } = opts; + + // Mode and folder access are baked in at session creation. Tear the session + // down (so it's recreated below) when either changed since it was created: + // - the mode switched Work↔Secure, or + // - (Work only) the folder set or a read-only/read&write flag changed. + // Without this, switching mode or toggling a folder in the UI is silently + // ignored — the chat keeps running on the stale session. + const wantSecure = !!opts.secure; + const foldersKey = wantSecure ? '' : foldersSignature(opts.workFolders); + const skillsKey = skillsSignature(chatId); + const existing = workSessions.get(chatId); + if ( + existing && + (existing.secure !== wantSecure || + (!wantSecure && existing.foldersKey !== foldersKey) || + existing.skillsKey !== skillsKey) + ) { + await stopWorkSession(existing.sessionId).catch(() => {}); + workSessions.delete(chatId); + const note = + existing.secure !== wantSecure + ? `Mode changed — restarted the session in ${wantSecure ? 'Secure' : 'Work'} mode.` + : existing.skillsKey !== skillsKey && + existing.foldersKey === foldersKey + ? 'Project skills changed — restarted the work session with the updated skills.' + : 'Folder access changed — restarted the work session with the new permissions.'; + const sys = messages.append(chatId, 'system', note, null); + wsHub.broadcastToChat(chatId, { type: 'message-appended', chatId, message: sys }); + } + + // Reuse existing session for this chat, or create a new one + let sessionId = workSessions.get(chatId)?.sessionId; + if (!sessionId) { + try { + // Per-folder access drives both the allowed-path list and read-only + // enforcement. Secure Mode ignores it (the sandbox workspace is the only + // mount); when no folders are chosen we fall back to HOME with write + // access, preserving prior behavior. + const hasFolders = !opts.secure && !!opts.workFolders?.length; + const folderAccess = hasFolders ? opts.workFolders : undefined; + const allowedFolders = hasFolders + ? opts.workFolders!.map((f) => f.path) + : [process.env.HOME ?? ''].filter(Boolean); + // Safe Mode: the folders the user picked are COPIED into the isolated + // sandbox (originals untouched), not bind-mounted live like Work Mode. + const copyFolders = + opts.secure && opts.workFolders?.length + ? opts.workFolders.map((f) => f.path) + : undefined; + // Seed the (possibly restored) session with compacted prior history from + // our DB, so context survives runtime restarts (older turns summarized). + const history = opts.beforeMsgId + ? await buildCompactedHistory(chatId, opts.beforeMsgId) + : undefined; + sessionId = await createWorkSession({ + allowedFolders, + folderAccess, + copyFolders, + secure: opts.secure, + systemPrompt: buildWorkSystemPrompt(chatId), + // Stable key → the chat reconnects to its persisted Secure workspace + // (and its running TTL) after a runtime restart. + key: `chat:${chatId}`, + history, + }); + workSessions.set(chatId, { sessionId, foldersKey, secure: wantSecure, skillsKey }); + } catch (err) { + const note = `Work Mode runtime unavailable. (${err instanceof Error ? err.message : String(err)})`; + const sys = messages.append(chatId, 'system', note, 'work-unavailable'); + wsHub.broadcastToChat(chatId, { type: 'message-appended', chatId, message: sys }); + return; + } + } + + // Safe Mode: forward the current folder selection every turn so folders added + // mid-chat get copied into the live sandbox (the runtime ingests new ones only). + const turnCopyFolders = + opts.secure && opts.workFolders?.length + ? opts.workFolders.map((f) => f.path) + : undefined; + try { + await sendWorkMessage(sessionId, content, opts.networkEnabled, opts.ttlDays, opts.attachments, turnCopyFolders); + } catch (err) { + // Session may have expired — retry with a fresh one + workSessions.delete(chatId); + const note = `Work Mode session lost, please resend. (${err instanceof Error ? err.message : String(err)})`; + const sys = messages.append(chatId, 'system', note, null); + wsHub.broadcastToChat(chatId, { type: 'message-appended', chatId, message: sys }); + return; + } + + await new Promise<void>((resolve) => { + let accumulated = ''; + // Savings notes arrive mid-stream (when a tool truncates), but we want them + // rendered BELOW the assistant's reply, not above it. So buffer them and + // flush as system rows only after the assistant row is appended on `done`. + const savingsTexts: string[] = []; + const unsubscribe = subscribeWorkEvents( + sessionId!, + (event) => { + if (event.type === 'text') { + accumulated += event.content; + onEvent({ type: 'text-delta', text: event.content }); + } else if (event.type === 'note') { + // A tool gave the model less than the full content — surface the saving + // as a friendly, plain-language chat note (no jargon: no "tokens", + // "output", "command"). When we have a real measured percentage we show + // it; otherwise (search line-trimming, where ripgrep never reports how + // much it dropped) we stay quantity-free rather than invent a number. + // The "ніж звичайні помічники" framing is a category claim anchored to + // the naive approach (load the whole file) — never a per-competitor + // benchmark, which we don't measure. + const n = event.note; + const text = + typeof n.savedPct === 'number' + ? `iClaw обробив це на ${n.savedPct}% економніше, ніж звичайні помічники 💚` + : `iClaw обробив це економніше, ніж звичайні помічники 💚`; + if (!savingsTexts.includes(text)) savingsTexts.push(text); // dedupe within a turn + } else if (event.type === 'done') { + if (accumulated.trim()) { + // Stamp the assistant row with the actual turn mode (secure/work), + // not the append() default of 'execute'. Keeps history + UI honest. + // `tokens` powers the dev-mode usage badge (null when not reported). + const assistantMsg = messages.append( + chatId, 'assistant', accumulated, null, null, null, + opts.secure ? 'secure' : 'work', event.tokens ?? null, event.cached ?? null, + ); + wsHub.broadcastToChat(chatId, { type: 'message-appended', chatId, message: assistantMsg }); + + // Procedural memory: review Work/Secure turns too. These are agentic + // and tool-capable by nature, so every completed turn counts as + // "substantive". Untrusted heuristic: a turn with network reach may + // have ingested external content (web/email/etc.). Throttled with a + // smaller interval than Ask/Execute. Fire-and-forget. + const chatNow = chats.get(chatId); + if (chatNow?.project_id != null && projects.get(chatNow.project_id)) { + scheduleProjectSkillReview({ + chatId, + projectId: chatNow.project_id, + sharesToProject: Boolean(chatNow.shares_to_project), + substantive: true, + userMessage: opts.reviewUserMessage ?? content, + assistantText: accumulated, + assistantMessageId: assistantMsg.id, + untrusted: Boolean(opts.networkEnabled), + interval: WORK_SKILL_REVIEW_INTERVAL, + }); + } + } + // Now that the reply row exists, drop the savings note(s) UNDER it. + for (const text of savingsTexts) { + const sys = messages.append(chatId, 'system', text, 'savings'); + wsHub.broadcastToChat(chatId, { type: 'message-appended', chatId, message: sys }); + } + unsubscribe(); + resolve(); + } else if (event.type === 'error') { + const sys = messages.append(chatId, 'system', `Work Mode error: ${event.message}`, null); + wsHub.broadcastToChat(chatId, { type: 'message-appended', chatId, message: sys }); + unsubscribe(); + resolve(); + } + }, + (err) => { + // Ignore "aborted" — it just means the SSE stream closed normally + if (err.message !== 'aborted') { + const sys = messages.append(chatId, 'system', `Work Mode connection error: ${err.message}`, null); + wsHub.broadcastToChat(chatId, { type: 'message-appended', chatId, message: sys }); + } + resolve(); + }, + ); + }); +} + +/** + * Incognito turns — fully ephemeral. Keyed by a client-generated string (the + * browser's in-RAM chat id), NOT a DB chat. Nothing is persisted: no message + * rows, no chat row, no facts/skills review. Output streams to the caller, who + * forwards it to the originating socket only. The runtime session enforces + * read-only + read-anywhere + web_fetch (incognito:true). + */ +const incognitoSessions = new Map<string, { sessionId: string; foldersKey: string }>(); + +export type IncognitoEvent = + | { type: 'text-delta'; text: string } + | { type: 'tool'; name: string } + | { type: 'error'; message: string }; + +export async function runIncognitoTurn(opts: { + key: string; + content: string; + workFolders?: WorkFolder[]; + onEvent: (event: IncognitoEvent) => void; +}): Promise<{ tokens?: number; cached?: number }> { + const { key, content, onEvent } = opts; + + // Recreate the runtime session if the selected folders changed (the shell's + // :ro mounts are baked in at creation), mirroring the Work path. + const foldersKey = foldersSignature(opts.workFolders); + const existing = incognitoSessions.get(key); + if (existing && existing.foldersKey !== foldersKey) { + await stopWorkSession(existing.sessionId).catch(() => {}); + incognitoSessions.delete(key); + } + + let sessionId = incognitoSessions.get(key)?.sessionId; + if (!sessionId) { + try { + const folderAccess = opts.workFolders?.length ? opts.workFolders : undefined; + // No HOME fallback: incognito reads anywhere via the runtime, and with no + // folders the read-only shell is simply unavailable (file/web tools work). + const allowedFolders = folderAccess ? folderAccess.map((f) => f.path) : []; + sessionId = await createWorkSession({ + allowedFolders, + folderAccess, + incognito: true, + key: `incognito:${key}`, + }); + incognitoSessions.set(key, { sessionId, foldersKey }); + } catch (err) { + onEvent({ type: 'error', message: `Incognito runtime unavailable. (${err instanceof Error ? err.message : String(err)})` }); + return {}; + } + } + + try { + await sendWorkMessage(sessionId, content); + } catch (err) { + incognitoSessions.delete(key); + onEvent({ type: 'error', message: `Incognito session lost, please resend. (${err instanceof Error ? err.message : String(err)})` }); + return {}; + } + + return await new Promise<{ tokens?: number; cached?: number }>((resolve) => { + const unsubscribe = subscribeWorkEvents( + sessionId!, + (event) => { + if (event.type === 'text') { + onEvent({ type: 'text-delta', text: event.content }); + } else if (event.type === 'tool') { + onEvent({ type: 'tool', name: event.name }); + } else if (event.type === 'done') { + unsubscribe(); + resolve({ tokens: event.tokens, cached: event.cached }); + } else if (event.type === 'error') { + onEvent({ type: 'error', message: event.message }); + unsubscribe(); + resolve({}); + } + }, + (err) => { + if (err.message !== 'aborted') onEvent({ type: 'error', message: err.message }); + resolve({}); + }, + ); + }); +} + +/** Stop and forget an incognito session (abort / tab close). */ +export async function abortIncognito(key: string): Promise<void> { + const e = incognitoSessions.get(key); + if (!e) return; + incognitoSessions.delete(key); + await stopWorkSession(e.sessionId).catch(() => {}); +} + export async function abortChatRun(chatId: number): Promise<void> { + // Execute: abort the OpenClaw run on the session actually executing this turn. + const active = activeRunSessionKeys.get(chatId); + if (active && active.startsWith('agent:')) { + await openclawWs.abortRun(active); + return; + } + // Work / Secure: a runtime turn. Abort it without destroying the session — + // the workspace and warm container survive (unlike stopWorkSession). This is + // what makes the header Stop button work for Work and Sandbox, not just + // OpenClaw/Execute. + const workSessionId = getWorkSessionId(chatId); + if (workSessionId) { + await abortWorkSession(workSessionId).catch(() => {}); + return; + } + // Fallback: abort the chat's main OpenClaw session if it's mid-run. const chat = chats.get(chatId); if (!chat) return; if (chat.openclaw_session_id?.startsWith('agent:')) { diff --git a/src/services/chatTitle.ts b/src/services/chatTitle.ts index aa86e34..d574cbc 100644 --- a/src/services/chatTitle.ts +++ b/src/services/chatTitle.ts @@ -1,4 +1,6 @@ import { openclawWs } from './openclawWs'; +import { complete, openRouterEnabled } from './openRouter'; +import { loadOpenRouterConfig } from './config'; export const TITLE_LIMIT = 60; /** @@ -87,7 +89,39 @@ function modelToAgentId(model: string): string { return model.startsWith('openclaw/') ? model.slice('openclaw/'.length) : model; } -export async function suggestChatTitle(opts: { +/** + * Cheap single-shot title via OpenRouter. Returns null on failure or a rejected + * suggestion so the caller can fall back to the OpenClaw path. + */ +async function suggestChatTitleViaOpenRouter(userMessage: string): Promise<string | null> { + try { + const cfg = loadOpenRouterConfig(); + const text = await complete({ + model: cfg.titleModel, + messages: [{ role: 'user', content: buildTitlePrompt(userMessage) }], + temperature: 0.3, + maxTokens: 32, + }); + const cleaned = normalizeSuggestedTitle(text); + if (!cleaned) { + console.warn('[chatTitle] openrouter rejected suggestion:', JSON.stringify(text.slice(0, 120))); + return null; + } + return cleaned; + } catch (err) { + console.warn( + '[chatTitle] openrouter failed, will fall back to OpenClaw:', + err instanceof Error ? err.message : err, + ); + return null; + } +} + +/** + * Title via a throw-away OpenClaw agent session — the original path, kept as + * the fallback when OpenRouter is unconfigured (or its call failed). + */ +async function suggestChatTitleViaOpenClaw(opts: { model: string; userMessage: string; }): Promise<string | null> { @@ -125,6 +159,23 @@ export async function suggestChatTitle(opts: { } } +/** + * Suggest a chat title. Prefers OpenRouter (cheap, no agent run) when a key is + * configured; falls back to a throw-away OpenClaw session when OpenRouter is + * unconfigured or its call fails/returns an unusable suggestion. + */ +export async function suggestChatTitle(opts: { + model: string; + userMessage: string; +}): Promise<string | null> { + if (openRouterEnabled()) { + const viaOpenRouter = await suggestChatTitleViaOpenRouter(opts.userMessage); + if (viaOpenRouter) return viaOpenRouter; + // null → OpenRouter unusable this time; fall through to OpenClaw. + } + return suggestChatTitleViaOpenClaw(opts); +} + /** Race title generation against a hard budget; null on timeout. */ export async function suggestChatTitleWithTimeout( opts: Parameters<typeof suggestChatTitle>[0], diff --git a/src/services/config.ts b/src/services/config.ts index b1556a9..226d207 100644 --- a/src/services/config.ts +++ b/src/services/config.ts @@ -1,6 +1,7 @@ import { readFileSync, existsSync } from 'node:fs'; import { homedir } from 'node:os'; import { join } from 'node:path'; +import { kvGet, kvSet, kvDelete } from '../db/kv'; export interface OpenClawConfig { baseUrl: string; @@ -44,6 +45,90 @@ export function loadCloudShareBaseUrl(): string { return base; } +export interface OpenRouterConfig { + /** Empty when unconfigured — gates Ask/STT and title routing. */ + apiKey: string; + baseUrl: string; + /** Model for Ask turns. */ + askModel: string; + /** Model for chat-title generation (cheapest sensible default). */ + titleModel: string; + /** Cheap model for context compaction (summarizing old turns). */ + summaryModel: string; + /** Multimodal model used for speech-to-text transcription. */ + sttModel: string; + /** OpenRouter app-attribution headers (optional, for rankings). */ + referer: string; + appTitle: string; +} + +/** KV key under which the user's OpenRouter API key is stored (set in Settings). */ +const OPENROUTER_API_KEY_KV = 'openrouter.api_key'; + +/** + * Direct OpenRouter access for the tool-less features (Ask, titles, STT). + * + * The API key is entered by the user in Settings and stored in the local DB + * (`iclaw_kv`) — NOT an env var. Models default to a cheap, fast, multimodal + * flash model; advanced users can still override per-feature via the optional + * `ICLAW_*_MODEL` env vars (undocumented). + */ +export function loadOpenRouterConfig(): OpenRouterConfig { + const apiKey = (kvGet(OPENROUTER_API_KEY_KV) ?? '').trim(); + const baseUrl = ( + process.env.OPENROUTER_BASE_URL?.trim() || 'https://openrouter.ai/api/v1' + ).replace(/\/+$/, ''); + const askModel = process.env.ICLAW_ASK_MODEL?.trim() || 'google/gemini-2.5-flash'; + const titleModel = process.env.ICLAW_TITLE_MODEL?.trim() || 'google/gemini-2.5-flash'; + // Cheap/fast model for compaction; overridable. Falls back to truncation if + // the call fails, so an invalid slug degrades gracefully. + const summaryModel = process.env.ICLAW_SUMMARY_MODEL?.trim() || 'google/gemini-2.5-flash-lite'; + const sttModel = process.env.ICLAW_STT_MODEL?.trim() || 'google/gemini-2.5-flash'; + const referer = process.env.OPENROUTER_REFERER?.trim() || 'https://iclaw.digital'; + const appTitle = process.env.OPENROUTER_APP_TITLE?.trim() || 'iClaw'; + return { apiKey, baseUrl, askModel, titleModel, summaryModel, sttModel, referer, appTitle }; +} + +/** Persist the user's OpenRouter API key (from Settings). Empty/blank clears it. */ +export function setOpenRouterApiKey(key: string): void { + const trimmed = key.trim(); + if (trimmed) kvSet(OPENROUTER_API_KEY_KV, trimmed); + else kvDelete(OPENROUTER_API_KEY_KV); +} + +/** Remove the stored OpenRouter API key (disconnect). */ +export function clearOpenRouterApiKey(): void { + kvDelete(OPENROUTER_API_KEY_KV); +} + +/** KV key recording that the user has been through (or skipped) the welcome flow. */ +const ONBOARDING_DONE_KV = 'onboarding.done'; + +/** + * Has the first-run welcome flow been completed (or explicitly skipped)? + * This is the sole gate for showing /welcome — independent of whether a key is + * set, so a user can dismiss the screen and still configure things later. + */ +export function isOnboardingDone(): boolean { + return kvGet(ONBOARDING_DONE_KV) === '1'; +} + +/** Mark the welcome flow complete so it never shows again (unless reset). */ +export function setOnboardingDone(): void { + kvSet(ONBOARDING_DONE_KV, '1'); +} + +/** + * A privacy-preserving display form of the stored key: keeps the `sk-or-` + * prefix and the last 4 chars, masks the middle. Empty string when unset. + */ +export function maskOpenRouterApiKey(): string { + const key = (kvGet(OPENROUTER_API_KEY_KV) ?? '').trim(); + if (!key) return ''; + if (key.length <= 12) return '••••' + key.slice(-2); + return key.slice(0, 7) + '••••••••' + key.slice(-4); +} + export function loadOpenClawConfig(): OpenClawConfig { const envToken = process.env.OPENCLAW_API_KEY?.trim(); const envUrl = process.env.OPENCLAW_BASE_URL?.trim(); diff --git a/src/services/contextCompaction.ts b/src/services/contextCompaction.ts new file mode 100644 index 0000000..2c36b89 --- /dev/null +++ b/src/services/contextCompaction.ts @@ -0,0 +1,140 @@ +/** + * Context compaction — "never clear the chat, just compress old turns". + * + * Each chat's full transcript lives permanently in the `messages` table. For + * the model's context window we build a bounded view: + * + * [rolling summary of older turns] + [last N turns verbatim] + [new message] + * + * The summary is produced by a cheap model (config.summaryModel) and cached in + * `chat_summaries`, rebuilt incrementally as the chat grows. Old context is + * therefore compressed, never dropped — full detail is always recoverable from + * the DB. Used by Ask directly, and to seed Work/Secure runtime sessions. + */ +import { db } from '../db/database'; +import { messages } from './store'; +import { loadOpenRouterConfig } from './config'; +import { complete, type OpenRouterMessage } from './openRouter'; + +/** Keep this many most-recent messages verbatim; older ones get summarized. */ +const RECENT_VERBATIM_COUNT = 16; +/** Don't compact until there are clearly "old" turns beyond the verbatim window. */ +const MIN_OLDER_TO_SUMMARIZE = 4; +/** Clip each old message to this many chars when feeding the summarizer. */ +const SUMMARIZE_PER_MSG_CHARS = 2000; +/** Cap the generated summary length. */ +const SUMMARY_MAX_TOKENS = 900; + +interface SummaryRow { + up_to_message_id: number; + summary: string; +} + +function getSummaryRow(chatId: number): SummaryRow | undefined { + return db + .prepare('SELECT up_to_message_id, summary FROM chat_summaries WHERE chat_id = ?') + .get(chatId) as SummaryRow | undefined; +} + +function upsertSummaryRow(chatId: number, upToId: number, summary: string): void { + db.prepare( + `INSERT INTO chat_summaries (chat_id, up_to_message_id, summary, updated_at) + VALUES (?, ?, ?, datetime('now')) + ON CONFLICT(chat_id) DO UPDATE SET + up_to_message_id = excluded.up_to_message_id, + summary = excluded.summary, + updated_at = excluded.updated_at`, + ).run(chatId, upToId, summary); +} + +interface TurnMsg { id: number; role: string; content: string } + +function toModelMsg(m: TurnMsg): OpenRouterMessage { + return { role: m.role === 'assistant' ? 'assistant' : 'user', content: m.content.replace(/\s+$/, '') }; +} + +function formatForSummary(msgs: TurnMsg[]): string { + return msgs + .map((m) => { + const who = m.role === 'assistant' ? 'Assistant' : 'User'; + let text = m.content.trim(); + if (text.length > SUMMARIZE_PER_MSG_CHARS) text = text.slice(0, SUMMARIZE_PER_MSG_CHARS) + '…'; + return `${who}: ${text}`; + }) + .join('\n\n'); +} + +const SUMMARY_SYSTEM_PROMPT = [ + 'You compress conversation history into a concise, information-dense summary.', + 'Preserve: facts, user preferences and decisions, established context, names,', + 'numbers, file/work state, and any open threads or unfinished tasks.', + 'Merge the existing summary with the new messages into one updated summary.', + 'Do not add commentary, greetings, or meta text — output only the summary.', +].join(' '); + +/** + * Produce (or extend) the summary covering `older`. Incremental: re-summarizes + * only the messages added since the cached summary. Best-effort — returns the + * previous (possibly stale) summary, or null, if the model call fails. + */ +async function getOrBuildSummary(chatId: number, older: TurnMsg[], boundaryId: number): Promise<string | null> { + const row = getSummaryRow(chatId); + if (row && row.up_to_message_id >= boundaryId) return row.summary; + + const prevSummary = row?.summary ?? ''; + const newOlder = row ? older.filter((m) => m.id > row.up_to_message_id) : older; + if (newOlder.length === 0) return prevSummary || null; + + const cfg = loadOpenRouterConfig(); + if (!cfg.apiKey) return prevSummary || null; + + try { + const summary = (await complete({ + model: cfg.summaryModel, + temperature: 0, + maxTokens: SUMMARY_MAX_TOKENS, + messages: [ + { role: 'system', content: SUMMARY_SYSTEM_PROMPT }, + { + role: 'user', + content: + `Existing summary:\n${prevSummary || '(none yet)'}\n\n` + + `New messages to fold in:\n${formatForSummary(newOlder)}\n\n` + + 'Return the updated summary.', + }, + ], + })).trim(); + if (!summary) return prevSummary || null; + upsertSummaryRow(chatId, boundaryId, summary); + return summary; + } catch { + return prevSummary || null; // degrade: keep stale summary rather than fail the turn + } +} + +/** + * Build a compacted history for a chat: messages with id < beforeMsgId, as + * [summary?] + [recent verbatim]. Small chats are returned verbatim (no call). + */ +export async function buildCompactedHistory(chatId: number, beforeMsgId: number): Promise<OpenRouterMessage[]> { + const all = messages + .listByChat(chatId) + .filter((m) => m.id < beforeMsgId && (m.role === 'user' || m.role === 'assistant') && m.content.trim()) + .map((m) => ({ id: m.id, role: m.role, content: m.content })); + + if (all.length <= RECENT_VERBATIM_COUNT + MIN_OLDER_TO_SUMMARIZE) { + return all.map(toModelMsg); + } + + const recent = all.slice(-RECENT_VERBATIM_COUNT); + const older = all.slice(0, -RECENT_VERBATIM_COUNT); + const boundaryId = older[older.length - 1].id; + + const summary = await getOrBuildSummary(chatId, older, boundaryId); + const out: OpenRouterMessage[] = []; + if (summary) { + out.push({ role: 'system', content: `Summary of earlier conversation (compacted):\n${summary}` }); + } + for (const m of recent) out.push(toModelMsg(m)); + return out; +} diff --git a/src/services/docker.ts b/src/services/docker.ts new file mode 100644 index 0000000..a1dd7e9 --- /dev/null +++ b/src/services/docker.ts @@ -0,0 +1,171 @@ +/** + * Docker readiness + lifecycle for the chat UI. + * + * Work mode's file tools run on the host (no Docker), but `run_command` and the + * whole Safe-work sandbox need a running daemon. This service backs the chat + * composer's Docker gate: a cached status probe plus best-effort "start" and + * "install" actions the user can trigger from the Install/Start button. + * + * It never blocks: the action endpoints kick the work off in the background and + * flip the cached state to `starting`/`installing`, which the composer polls + * via GET /api/docker/status until it settles on `ready` (or back to + * `stopped`/`missing` if it didn't take). + * + * Mirrors the onboarding probe in onboardingEnv.ts, but that one is a one-shot + * welcome-screen state machine; this is the live, re-pollable runtime service. + */ + +import { execFile } from 'node:child_process'; +import { platform as osPlatform } from 'node:os'; +import path from 'node:path'; +import { promisify } from 'node:util'; + +const execFileAsync = promisify(execFile); + +export type DockerState = + | 'ready' // daemon reachable (`docker info` ok) + | 'stopped' // binary present, daemon down + | 'missing' // binary not installed + | 'starting' // we're launching the daemon + | 'installing'; // we're installing the engine + +/** Human-readable footprint shown on the Install button. */ +export const DOCKER_SIZE_HINT = '~600 MB download · ~4 GB on disk'; + +const PROBE_TIMEOUT = 8_000; +const CACHE_MS = 4_000; +const POLL_ATTEMPTS = 30; +const POLL_INTERVAL = 2_000; +const INSTALL_TIMEOUT = 600_000; + +let cached: DockerState = 'missing'; +let cachedAt = 0; +let probeInFlight: Promise<DockerState> | null = null; +/** Single-flight for start/install — only one host action runs at a time. */ +let actionInFlight: Promise<DockerState> | null = null; + +function sleep(ms: number): Promise<void> { + return new Promise((r) => setTimeout(r, ms)); +} + +/** Daemon reachable (`docker info` exits 0). */ +async function reachable(): Promise<boolean> { + try { + await execFileAsync('docker', ['info'], { timeout: PROBE_TIMEOUT }); + return true; + } catch { + return false; + } +} + +/** Docker CLI present on PATH (daemon may be down). */ +async function installed(): Promise<boolean> { + try { + await execFileAsync('docker', ['--version'], { timeout: PROBE_TIMEOUT }); + return true; + } catch { + return false; + } +} + +async function probe(): Promise<DockerState> { + if (await reachable()) return 'ready'; + if (await installed()) return 'stopped'; + return 'missing'; +} + +/** Re-probe and cache the settled state (ready/stopped/missing). */ +async function settle(): Promise<DockerState> { + cached = await probe(); + cachedAt = Date.now(); + return cached; +} + +/** + * Cached status. Never overrides an in-flight action (`starting`/`installing`) + * and dedupes concurrent probes. Refreshes once the cache is stale. + */ +export async function getDockerState(): Promise<DockerState> { + if (cached === 'starting' || cached === 'installing') return cached; + if (Date.now() - cachedAt < CACHE_MS) return cached; + if (!probeInFlight) { + probeInFlight = settle().finally(() => { + probeInFlight = null; + }); + } + return probeInFlight; +} + +/** Best-effort daemon launch for the current platform. */ +async function launchDaemon(): Promise<void> { + const plat = osPlatform(); + try { + if (plat === 'darwin') { + await execFileAsync('open', ['-a', 'Docker'], { timeout: PROBE_TIMEOUT }); + } else if (plat === 'win32') { + await execFileAsync('cmd', ['/c', 'start', '', 'Docker Desktop'], { timeout: PROBE_TIMEOUT }); + } + // Linux: starting dockerd needs sudo, which can't prompt from this web + // context — leave it to the user. The poll below still picks it up if they + // start it manually. + } catch { + /* Docker.app not found / launch failed — the poll reports the real state. */ + } +} + +/** Poll until the daemon accepts connections (Docker Desktop can take 20–40s). */ +async function pollReady(): Promise<boolean> { + for (let i = 0; i < POLL_ATTEMPTS; i++) { + await sleep(POLL_INTERVAL); + if (await reachable()) return true; + } + return false; +} + +/** + * Start an installed-but-idle Docker, in the background. Returns immediately + * with `starting`; the cached state settles to `ready`/`stopped` once the poll + * resolves. Single-flight: a concurrent call rides the same action. + */ +export function startDocker(): DockerState { + if (!actionInFlight) { + cached = 'starting'; + cachedAt = Date.now(); + actionInFlight = (async () => { + await launchDaemon(); + await pollReady(); + return settle(); + })().finally(() => { + actionInFlight = null; + }); + } + return cached; +} + +const INSTALL_SCRIPT = path.resolve(__dirname, '../../scripts/install-docker.sh'); + +/** + * Install the Docker engine (Homebrew cask on macOS, get.docker.com on Linux), + * then start it — all in the background. Returns immediately with `installing`; + * the cached state settles once the install + start + poll resolve. + */ +export function installDocker(): DockerState { + if (!actionInFlight) { + cached = 'installing'; + cachedAt = Date.now(); + actionInFlight = (async () => { + try { + await execFileAsync('bash', [INSTALL_SCRIPT], { timeout: INSTALL_TIMEOUT }); + } catch { + /* Install failed (no brew, offline, declined sudo) — settle() reports + whether anything landed; the UI keeps the Install button. */ + } + await launchDaemon(); + await pollReady(); + return settle(); + })().finally(() => { + actionInFlight = null; + }); + } + return cached; +} diff --git a/src/services/onboardingEnv.ts b/src/services/onboardingEnv.ts new file mode 100644 index 0000000..a011de9 --- /dev/null +++ b/src/services/onboardingEnv.ts @@ -0,0 +1,150 @@ +/** + * First-run environment detection for the welcome flow. + * + * The onboarding screen shows an HONEST background progress line while the user + * reads the welcome copy / pastes their OpenRouter key. We do NOT install + * anything here — Docker Desktop and the OpenClaw gateway are user-installed. + * What we CAN do without the user noticing: + * - probe whether Docker is reachable (`docker info`), + * - if it is and the Work/Safe sandbox base image is missing, pre-pull it in + * the background (the one genuinely slow step we can hide behind the key + * entry), and + * - probe whether the OpenClaw gateway is reachable (Full Power). + * + * Everything is best-effort and cached — the screen polls `getOnboardingEnv()` + * for a status line, never blocking the actual "Continue" action. + */ + +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; + +import { probeGateway } from './gatewayProbe'; + +const execFileAsync = promisify(execFile); + +// Mirror work-container.ts's image resolution so onboarding agrees with what +// Work/Safe actually use. The canonical image is the prebuilt sandbox; the +// public base is only a last-resort fallback we pre-pull when NOTHING usable is +// present yet. If the user already has either, there is nothing to download. +const SANDBOX_IMAGE = process.env.ICLAW_SECURE_IMAGE || 'iclaw-secure:latest'; +/** Base image Work/Safe modes fall back to — public, so `docker pull` works on a fresh box. */ +const FALLBACK_IMAGE = process.env.ICLAW_WORK_SLIM_IMAGE || 'node:22'; + +type DockerState = + | 'unknown' // not probed yet + | 'missing' // Docker binary not installed + | 'stopped' // installed but daemon not running (and we couldn't start it) + | 'starting' // installed, daemon down — we're trying to start it + | 'pulling' // daemon up, downloading the sandbox image + | 'ready'; // daemon up and an image is available + +export interface OnboardingEnv { + /** Docker engine + sandbox image readiness (gates Work / Safe work). */ + docker: DockerState; + /** OpenClaw gateway reachable (gates Full Power / Execute). */ + openclaw: 'unknown' | 'connected' | 'off'; +} + +let state: OnboardingEnv = { docker: 'unknown', openclaw: 'unknown' }; +let prepStarted = false; + +/** Daemon reachable (`docker info` exits 0). */ +async function dockerReachable(): Promise<boolean> { + try { + await execFileAsync('docker', ['info'], { timeout: 8_000 }); + return true; + } catch { + return false; + } +} + +/** Docker CLI present on PATH (binary installed, daemon may be down). */ +async function dockerInstalled(): Promise<boolean> { + try { + await execFileAsync('docker', ['--version'], { timeout: 8_000 }); + return true; + } catch { + return false; + } +} + +async function imageExists(image: string): Promise<boolean> { + try { + await execFileAsync('docker', ['image', 'inspect', image], { timeout: 8_000 }); + return true; + } catch { + return false; + } +} + +/** Pull the fallback base image in the background, flipping docker → ready when done. */ +function pullImageInBackground(image: string): void { + state.docker = 'pulling'; + // 10 min ceiling — a slow connection on a multi-hundred-MB image. + execFileAsync('docker', ['pull', image], { timeout: 600_000 }) + .then(() => { + state.docker = 'ready'; + }) + .catch(() => { + // Pull failed (offline, disk, daemon hiccup). `docker run` still + // auto-pulls on first real use, so this is non-fatal — leave it as + // 'pulling' is misleading, so report 'ready' (Docker is installed; the + // image just arrives lazily later). + state.docker = 'ready'; + }); +} + +/** + * Daemon is up: mark ready if a usable image is already present, otherwise + * pre-pull the public fallback in the background. + */ +async function ensureImage(): Promise<void> { + if ((await imageExists(SANDBOX_IMAGE)) || (await imageExists(FALLBACK_IMAGE))) { + state.docker = 'ready'; + } else { + pullImageInBackground(FALLBACK_IMAGE); + } +} + +/** + * Kick off detection + the (slow) image pre-pull exactly once. Safe to call on + * every /welcome render — it no-ops after the first call. + */ +export function startOnboardingPrep(): void { + if (prepStarted) return; + prepStarted = true; + + void (async () => { + // 1. Daemon already up (the user started it)? Pre-pull the sandbox image so + // the first Work/Safe use is fast. + if (await dockerReachable()) { + await ensureImage(); + return; + } + // 2. Installed but idle → DON'T start it here. Onboarding is informational; + // the runtime starts Docker on demand the moment a task needs it + // (docker-lifecycle.ts) and auto-stops it when idle. Starting it here + // instead would be a transparent start the runtime can't track → it would + // never get auto-stopped. So we just report the state. + if (await dockerInstalled()) { + state.docker = 'stopped'; + return; + } + // 3. Not installed at all. + state.docker = 'missing'; + })(); + + void (async () => { + try { + const { gatewayStatus } = await probeGateway('onboarding'); + state.openclaw = gatewayStatus === 'ok' ? 'connected' : 'off'; + } catch { + state.openclaw = 'off'; + } + })(); +} + +/** Current best-effort environment snapshot for the welcome progress line. */ +export function getOnboardingEnv(): OnboardingEnv { + return { ...state }; +} diff --git a/src/services/openRouter.ts b/src/services/openRouter.ts new file mode 100644 index 0000000..b31a13f --- /dev/null +++ b/src/services/openRouter.ts @@ -0,0 +1,370 @@ +/** + * Direct OpenRouter client — the FIRST non-OpenClaw LLM path in iClaw. + * + * Used for the lightweight, tool-less features that should NOT spin up a full + * OpenClaw agent run: + * - Chat titles → a cheap single-shot completion. + * - Background sub-tasks (fact extraction, fact compaction, skill review) via + * services/subtaskLlm.ts (preferred over a throwaway OpenClaw turn). + * - Speech-to-text → audio transcription via a multimodal model. + * + * Talks the OpenAI-compatible `/chat/completions` endpoint OpenRouter exposes. + * Streaming responses are parsed here (server-sent events over `fetch`) and the + * caller re-emits deltas through `wsHub` — the same shape `openclawWs.runTurn` + * uses, so the browser stream renderer is identical for both backends. + * + * No SDK / extra deps: Node 18+ (we run 25) ships global `fetch` + streams. + * + * Availability is config-driven: when `OPENROUTER_API_KEY` is unset, the + * features that REQUIRE OpenRouter (Ask, STT) are hidden/refused, and title + * generation falls back to the OpenClaw path. See `loadOpenRouterConfig`. + */ + +import { loadOpenRouterConfig } from './config'; + +export interface OpenRouterMessage { + role: 'system' | 'user' | 'assistant'; + content: string; +} + +/** True when an API key is configured — gates Ask/STT and title routing. */ +export function openRouterEnabled(): boolean { + return Boolean(loadOpenRouterConfig().apiKey); +} + +/** + * Failures from the OpenRouter bridge are tagged with this prefix so callers + * (chatRunner) can distinguish them from OpenClaw gateway failures and surface + * a sensible user message instead of leaking raw text. + */ +function fail(message: string): never { + throw new Error(`openrouter: ${message}`); +} + +export function isOpenRouterFailure(err: unknown): boolean { + const t = (err instanceof Error ? err.message : String(err)).trim(); + return /^openrouter:/i.test(t); +} + +function buildHeaders(apiKey: string, referer: string, appTitle: string): Record<string, string> { + const h: Record<string, string> = { + Authorization: `Bearer ${apiKey}`, + 'Content-Type': 'application/json', + }; + // OpenRouter uses these for app attribution / rankings. Optional, harmless. + if (referer) h['HTTP-Referer'] = referer; + if (appTitle) h['X-Title'] = appTitle; + return h; +} + +interface ChatCompletionOpts { + messages: OpenRouterMessage[]; + /** Defaults to the configured Ask model. */ + model?: string; + temperature?: number; + maxTokens?: number; + /** Abort the in-flight request (Stop button). */ + signal?: AbortSignal; +} + +/** + * Non-streaming completion. Returns the full assistant text. Used for titles + * and any short single-shot call. + */ +export async function complete(opts: ChatCompletionOpts): Promise<string> { + const cfg = loadOpenRouterConfig(); + if (!cfg.apiKey) fail('no API key configured (set OPENROUTER_API_KEY)'); + const model = opts.model ?? cfg.askModel; + + let res: Response; + try { + res = await fetch(`${cfg.baseUrl}/chat/completions`, { + method: 'POST', + headers: buildHeaders(cfg.apiKey, cfg.referer, cfg.appTitle), + body: JSON.stringify({ + model, + messages: opts.messages, + stream: false, + thinking: { type: 'disabled' }, + ...(opts.temperature != null ? { temperature: opts.temperature } : {}), + ...(opts.maxTokens != null ? { max_tokens: opts.maxTokens } : {}), + }), + signal: opts.signal, + }); + } catch (err) { + fail(`request failed: ${err instanceof Error ? err.message : String(err)}`); + } + if (!res.ok) fail(`HTTP ${res.status}: ${(await res.text().catch(() => '')).slice(0, 300)}`); + + const json = (await res.json().catch(() => null)) as { + choices?: Array<{ message?: { content?: string } }>; + } | null; + return json?.choices?.[0]?.message?.content ?? ''; +} + +interface StreamCompletionOpts extends ChatCompletionOpts { + /** Fires for every text delta as it streams in. */ + onDelta: (text: string) => void; +} + +/** + * Streaming completion. Calls `onDelta` for each token chunk and resolves with + * the full accumulated text. Parses OpenRouter's SSE stream (`data: {json}` + * lines, terminated by `data: [DONE]`). + */ +export async function streamComplete(opts: StreamCompletionOpts): Promise<string> { + const cfg = loadOpenRouterConfig(); + if (!cfg.apiKey) fail('no API key configured (set OPENROUTER_API_KEY)'); + const model = opts.model ?? cfg.askModel; + + let res: Response; + try { + res = await fetch(`${cfg.baseUrl}/chat/completions`, { + method: 'POST', + headers: buildHeaders(cfg.apiKey, cfg.referer, cfg.appTitle), + body: JSON.stringify({ + model, + messages: opts.messages, + stream: true, + // Disable extended thinking — Ask mode is for quick answers, not deep reasoning + thinking: { type: 'disabled' }, + ...(opts.temperature != null ? { temperature: opts.temperature } : {}), + ...(opts.maxTokens != null ? { max_tokens: opts.maxTokens } : {}), + }), + signal: opts.signal, + }); + } catch (err) { + fail(`request failed: ${err instanceof Error ? err.message : String(err)}`); + } + if (!res.ok) fail(`HTTP ${res.status}: ${(await res.text().catch(() => '')).slice(0, 300)}`); + if (!res.body) fail('no response body to stream'); + + const reader = res.body.getReader(); + const decoder = new TextDecoder(); + let buffer = ''; + let full = ''; + + // SSE frames are separated by a blank line. We accumulate across chunks + // because a single network chunk can split a `data:` line mid-JSON. + const consumeLine = (line: string): boolean => { + const trimmed = line.trim(); + if (!trimmed.startsWith('data:')) return false; + const payload = trimmed.slice('data:'.length).trim(); + if (payload === '[DONE]') return true; + try { + const json = JSON.parse(payload) as { + choices?: Array<{ delta?: { content?: string } }>; + }; + const delta = json.choices?.[0]?.delta?.content; + if (delta) { + full += delta; + opts.onDelta(delta); + } + } catch { + // OpenRouter sends `: OPENROUTER PROCESSING` keep-alive comments and the + // occasional partial line — ignore anything that isn't valid JSON. + } + return false; + }; + + try { + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + buffer += decoder.decode(value, { stream: true }); + let nl: number; + while ((nl = buffer.indexOf('\n')) !== -1) { + const line = buffer.slice(0, nl); + buffer = buffer.slice(nl + 1); + if (consumeLine(line)) { + await reader.cancel().catch(() => {}); + return full; + } + } + } + // Flush any trailing buffered line (stream ended without final newline). + if (buffer.trim()) consumeLine(buffer); + } finally { + reader.releaseLock?.(); + } + return full; +} + +/** + * Transcribe audio to text via a multimodal model. Sends the clip as an + * `input_audio` content part to /chat/completions and asks for a verbatim + * transcript. Returns the transcript (possibly empty). Throws `openrouter:` on + * failure. + * + * NOTE: accepted audio formats depend on the model. Gemini-class models take + * common containers, but the browser's MediaRecorder output varies (webm/opus + * on Chrome, mp4/aac on Safari). If a model rejects the upload, point + * `ICLAW_STT_MODEL` at one that accepts that container. + */ +export async function transcribeAudio(opts: { + audioBase64: string; + /** Container hint for the model: "webm" | "mp3" | "wav" | "m4a" | "ogg" | … */ + format: string; + model?: string; + signal?: AbortSignal; +}): Promise<string> { + const cfg = loadOpenRouterConfig(); + if (!cfg.apiKey) fail('no API key configured (set OPENROUTER_API_KEY)'); + const model = opts.model ?? cfg.sttModel; + + let res: Response; + try { + res = await fetch(`${cfg.baseUrl}/chat/completions`, { + method: 'POST', + headers: buildHeaders(cfg.apiKey, cfg.referer, cfg.appTitle), + body: JSON.stringify({ + model, + stream: false, + messages: [ + { + role: 'user', + content: [ + { + type: 'text', + text: + 'Transcribe this audio verbatim. Output ONLY the transcript text — ' + + 'no preamble, quotes, or commentary. If there is no speech, output nothing.', + }, + { type: 'input_audio', input_audio: { data: opts.audioBase64, format: opts.format } }, + ], + }, + ], + }), + signal: opts.signal, + }); + } catch (err) { + fail(`request failed: ${err instanceof Error ? err.message : String(err)}`); + } + if (!res.ok) fail(`HTTP ${res.status}: ${(await res.text().catch(() => '')).slice(0, 300)}`); + + const json = (await res.json().catch(() => null)) as { + choices?: Array<{ message?: { content?: string } }>; + } | null; + return (json?.choices?.[0]?.message?.content ?? '').trim(); +} + +export interface KeyValidation { + /** The key authenticates against OpenRouter. */ + valid: boolean; + /** USD remaining if known; null when unlimited/unknown. */ + remaining: number | null; + /** Optional key label from OpenRouter. */ + label?: string; + /** Machine-readable reason when invalid: 'empty' | 'unauthorized' | 'http' | 'network'. */ + reason?: string; +} + +/** + * Validate an OpenRouter key WITHOUT storing it — used by onboarding/Settings to + * catch a dead, mistyped, or zero-balance key before it silently breaks the + * first chat. Hits `/auth/key` (cheap, no completion spend) with the candidate + * key and reports validity + remaining credit. Network failures are reported as + * `valid: false, reason: 'network'` so the UI can say "couldn't verify" rather + * than wrongly rejecting a good key. + */ +export async function validateKey(key: string, signal?: AbortSignal): Promise<KeyValidation> { + const trimmed = key.trim(); + if (!trimmed) return { valid: false, remaining: null, reason: 'empty' }; + + const cfg = loadOpenRouterConfig(); + const headers = buildHeaders(trimmed, cfg.referer, cfg.appTitle); + + let res: Response; + try { + res = await fetch(`${cfg.baseUrl}/auth/key`, { headers, signal }); + } catch { + return { valid: false, remaining: null, reason: 'network' }; + } + if (res.status === 401 || res.status === 403) { + return { valid: false, remaining: null, reason: 'unauthorized' }; + } + if (!res.ok) return { valid: false, remaining: null, reason: 'http' }; + + const j = (await res.json().catch(() => null)) as { + data?: { label?: string; usage?: number; limit?: number | null; limit_remaining?: number | null }; + } | null; + const d = j?.data ?? {}; + const usage = Number(d.usage ?? 0); + const limit = d.limit == null ? null : Number(d.limit); + const remaining = + d.limit_remaining == null ? (limit != null ? Math.max(limit - usage, 0) : null) : Number(d.limit_remaining); + return { valid: true, remaining, label: d.label }; +} + +export interface OpenRouterUsage { + /** USD spent so far on this key/account. */ + usage: number; + /** USD credit limit, or null when there is none. */ + limit: number | null; + /** USD remaining, or null when unknown/unlimited. */ + remaining: number | null; + /** Optional key label from OpenRouter. */ + label?: string; + isFreeTier?: boolean; +} + +/** + * Fetch spend/credit info for the configured key, for the Settings usage + * readout. Tries `/credits` (purchased credits + total usage) first, then + * falls back to `/auth/key` (usage + limit + label). Throws `openrouter:` on + * failure so the caller can show a friendly "couldn't load usage" note. + */ +export async function fetchUsage(signal?: AbortSignal): Promise<OpenRouterUsage> { + const cfg = loadOpenRouterConfig(); + if (!cfg.apiKey) fail('no API key configured (add it in Settings)'); + const headers = buildHeaders(cfg.apiKey, cfg.referer, cfg.appTitle); + + // Preferred: /credits → { data: { total_credits, total_usage } }. + try { + const res = await fetch(`${cfg.baseUrl}/credits`, { headers, signal }); + if (res.ok) { + const j = (await res.json().catch(() => null)) as { + data?: { total_credits?: number; total_usage?: number }; + } | null; + const d = j?.data; + if (d && (typeof d.total_usage === 'number' || typeof d.total_credits === 'number')) { + const usage = Number(d.total_usage ?? 0); + const credits = Number(d.total_credits ?? 0); + const limit = credits > 0 ? credits : null; + const remaining = limit != null ? Math.max(limit - usage, 0) : null; + return { usage, limit, remaining }; + } + } + } catch (err) { + if (signal?.aborted) fail('aborted'); + // fall through to /auth/key + } + + // Fallback: /auth/key → { data: { label, usage, limit, limit_remaining, is_free_tier } }. + let res: Response; + try { + res = await fetch(`${cfg.baseUrl}/auth/key`, { headers, signal }); + } catch (err) { + fail(`request failed: ${err instanceof Error ? err.message : String(err)}`); + } + if (!res.ok) fail(`HTTP ${res.status}: ${(await res.text().catch(() => '')).slice(0, 200)}`); + const j = (await res.json().catch(() => null)) as { + data?: { + label?: string; + usage?: number; + limit?: number | null; + limit_remaining?: number | null; + is_free_tier?: boolean; + }; + } | null; + const d = j?.data ?? {}; + const usage = Number(d.usage ?? 0); + const limit = d.limit == null ? null : Number(d.limit); + const remaining = + d.limit_remaining == null + ? limit != null + ? Math.max(limit - usage, 0) + : null + : Number(d.limit_remaining); + return { usage, limit, remaining, label: d.label, isFreeTier: d.is_free_tier }; +} diff --git a/src/services/openclawWs.ts b/src/services/openclawWs.ts index a0c0d83..609dadd 100644 --- a/src/services/openclawWs.ts +++ b/src/services/openclawWs.ts @@ -14,10 +14,17 @@ import { gatewayWs, type RawGatewayFrame } from './gatewayWs'; import { toolActivityLabel, lifecycleActivityLabel } from './toolLabels'; import { extractAssistantText, + extractTurnUsage, resolveFromHistorySlice, sliceFromLastUser, } from './turnReply'; +/** Token usage for one turn, resolved from the gateway history slice. */ +export interface TurnUsage { + tokens: number | null; + cached: number | null; +} + // ---------- shapes --------------------------------------------------------- export interface OpenClawAgent { @@ -39,6 +46,8 @@ export interface HistoryMessage { timestamp?: number; toolName?: string; isError?: boolean; + /** Per-message token usage (assistant rows) — powers the dev-mode badge. */ + usage?: unknown; } export type TurnEvent = @@ -121,7 +130,9 @@ const HISTORY_FETCH_LIMIT = 1000; const HISTORY_FETCH_RETRIES = 3; const HISTORY_FETCH_RETRY_DELAY_MS = 150; -async function fetchAuthoritativeFromHistory(sessionKey: string): Promise<string | null> { +async function fetchAuthoritativeFromHistory( + sessionKey: string, +): Promise<{ text: string | null; usage: TurnUsage }> { let lastErr: unknown; for (let attempt = 0; attempt < HISTORY_FETCH_RETRIES; attempt++) { try { @@ -129,8 +140,12 @@ async function fetchAuthoritativeFromHistory(sessionKey: string): Promise<string 'chat.history', { sessionKey, limit: HISTORY_FETCH_LIMIT }, ); - const resolved = resolveFromHistorySlice(sliceFromLastUser(res.messages ?? [])); - return resolved.trim().length > 0 ? resolved : null; + const slice = sliceFromLastUser(res.messages ?? []); + const resolved = resolveFromHistorySlice(slice); + return { + text: resolved.trim().length > 0 ? resolved : null, + usage: extractTurnUsage(slice), + }; } catch (err) { lastErr = err; if (attempt < HISTORY_FETCH_RETRIES - 1) { @@ -144,7 +159,7 @@ async function fetchAuthoritativeFromHistory(sessionKey: string): Promise<string 'attempts:', lastErr instanceof Error ? lastErr.message : String(lastErr), ); - return null; + return { text: null, usage: { tokens: null, cached: null } }; } function pickAuthoritativeReplyText(opts: { @@ -195,12 +210,38 @@ const pendingAbortIntents = new Set<string>(); // ---------- public client ------------------------------------------------- +/** Short cache for agent-existence lookups so Ask routing doesn't hit agents.list every turn. */ +const AGENT_EXISTS_TTL_MS = 30_000; +const agentExistsCache = new Map<string, { at: number; exists: boolean }>(); + export const openclawWs = { async listAgents(): Promise<OpenClawAgent[]> { const res = await gatewayWs.request<{ agents: OpenClawAgent[] }>('agents.list', {}); return res.agents ?? []; }, + /** + * Whether an agent with this id is configured on the gateway. Cached for a + * few seconds. Returns false on any lookup failure (caller falls back to a + * non-agent path rather than erroring). Empty id is always false. + */ + async agentExists(agentId: string): Promise<boolean> { + const id = agentId.trim(); + if (!id) return false; + const hit = agentExistsCache.get(id); + const now = Date.now(); + if (hit && now - hit.at < AGENT_EXISTS_TTL_MS) return hit.exists; + try { + const agents = await this.listAgents(); + const exists = agents.some((a) => a.id === id); + agentExistsCache.set(id, { at: now, exists }); + return exists; + } catch { + // Gateway unreachable / RPC failed — don't cache, treat as absent. + return false; + } + }, + /** Create a new "dashboard" session for an agent. Empty params → default agent. */ async createSession(opts: { agentId?: string } = {}): Promise<OpenClawSession> { const params: Record<string, unknown> = {}; @@ -216,6 +257,25 @@ export const openclawWs = { await gatewayWs.request('sessions.delete', { key }); }, + /** + * Append a message into a session's transcript WITHOUT running the model + * (`chat.inject`). The gateway records it as an assistant note (zero cost, + * `model: "gateway-injected"`) and broadcasts a final `chat` event. Used to + * make the main Execute session aware of an out-of-band Ask exchange. + */ + async injectMessage(opts: { + sessionKey: string; + message: string; + label?: string; + }): Promise<void> { + if (!opts.sessionKey.startsWith('agent:')) return; + await gatewayWs.request('chat.inject', { + sessionKey: opts.sessionKey, + message: opts.message, + ...(opts.label ? { label: opts.label } : {}), + }); + }, + /** Fetch the canonical, UI-normalized transcript. */ async getHistory(sessionKey: string, limit = 200): Promise<HistoryMessage[]> { const res = await gatewayWs.request<{ messages: HistoryMessage[] }>('chat.history', { @@ -395,6 +455,12 @@ export const openclawWs = { * `null` on abort or when nothing usable was found. */ authoritativeText: string | null; + /** + * Token usage for this turn, summed across the assistant rows of the + * `chat.history` slice. `{ null, null }` on abort or when the gateway + * reported no usage (powers the dev-mode token badge — see chatRunner). + */ + usage: TurnUsage; }> { let runId: string | null = null; let accumulatedText = ''; @@ -716,10 +782,12 @@ export const openclawWs = { // Canonical reply: transcript/history first, then last `chat:final`, then stream. let authoritativeText: string | null = null; + let usage: TurnUsage = { tokens: null, cached: null }; if (!wasAborted) { const fromHistory = await fetchAuthoritativeFromHistory(opts.sessionKey); + usage = fromHistory.usage; authoritativeText = pickAuthoritativeReplyText({ - fromHistory, + fromHistory: fromHistory.text, fromTranscript: transcriptAssistantText, fromLastFinal: lastFinalText || accumulatedText, }); @@ -730,6 +798,7 @@ export const openclawWs = { text: lastFinalText || accumulatedText, aborted: wasAborted, authoritativeText, + usage, }; }, }; diff --git a/src/services/projectMemory.ts b/src/services/projectMemory.ts index 8c1c487..b082494 100644 --- a/src/services/projectMemory.ts +++ b/src/services/projectMemory.ts @@ -3,8 +3,9 @@ * facts after each turn (when enabled), and compact when the fact table grows. */ -import { openclawWs } from './openclawWs'; import { projectFacts, projects, chats, projectFactSuggestions, enrichFactsWithSourceChatTitles } from './store'; +import { buildSkillsPromptBlock } from './projectSkills'; +import { runSubtaskTurn } from './subtaskLlm'; import { wsHub } from './wsHub'; /** Rough token budget for the prepended project block (80/20 vs full window). */ @@ -19,14 +20,6 @@ const MAX_FACT_LINE_CHARS = 240; const compactingProjects = new Set<number>(); -/** - * Sub-tasks (extract + compact) always run against the gateway's default - * agent. Using whatever agent the chat is on (`openclaw/code`, etc.) made the - * two pipelines inconsistent and risked specialised agents underperforming on - * a plain text-extraction prompt. One agent, one behaviour. - */ -const SUBTASK_AGENT_ID = 'main'; - function approxTokens(s: string): number { return Math.ceil(s.length / CHARS_PER_TOKEN_EST); } @@ -55,11 +48,20 @@ export function buildGatewayUserMessage( ): string { if (!projects.get(projectId)) return storedUserContent; + const skillsBlock = buildSkillsPromptBlock(projectId); const all = projectFacts.listByProject(projectId, 200); - if (all.length === 0) return storedUserContent; + // Nothing to inject — neither facts nor skills. + if (all.length === 0 && !skillsBlock) return storedUserContent; + + const skillsPrefix = skillsBlock ? skillsBlock + '\n\n' : ''; + if (all.length === 0) { + // Skills-only injection (no facts yet). + return `[Project context for this workspace. Treat as background only; respond in direct conversation to the user (their message is after the separator).]\n${skillsPrefix}---\n[User message]\n${storedUserContent}`; + } const prefixBase = - '[Project context — shared facts for this workspace. Treat as background only; respond in direct conversation to the user (their message is after the separator).]\n'; + '[Project context — shared facts for this workspace. Treat as background only; respond in direct conversation to the user (their message is after the separator).]\n' + + skillsPrefix; const suffix = '\n---\n[User message]\n' + storedUserContent; const budget = Math.max( 200, @@ -159,33 +161,13 @@ function buildExtractPrompt(opts: { ].join('\n'); } -async function runThrowawayTurn(message: string): Promise<string> { - let sessionKey: string | null = null; - try { - const session = await openclawWs.createSession({ agentId: SUBTASK_AGENT_ID }); - sessionKey = session.key; - let acc = ''; - await openclawWs.runTurn({ - sessionKey: session.key, - message, - onEvent: (ev) => { - if (ev.type === 'text-delta') acc += ev.text; - else if (ev.type === 'text-final') acc = ev.text || acc; - }, - }); - return acc; - } finally { - if (sessionKey) openclawWs.deleteSession(sessionKey).catch(() => {}); - } -} - export async function extractFactsFromTurn(opts: { userMessage: string; assistantText: string; existingFacts: string[]; }): Promise<string[]> { const raw = await Promise.race([ - runThrowawayTurn(buildExtractPrompt(opts)), + runSubtaskTurn(buildExtractPrompt(opts), { maxTokens: 512 }), sleep<string>(EXTRACT_BUDGET_MS, ''), ]); if (raw == null || raw === '') return []; @@ -214,7 +196,7 @@ export async function compactProjectFacts(projectId: number): Promise<void> { if (contents.length <= FACT_COMPACTION_THRESHOLD) return; const raw = await Promise.race([ - runThrowawayTurn(buildCompactionPrompt(contents)), + runSubtaskTurn(buildCompactionPrompt(contents), { maxTokens: 1024 }), sleep<string>(EXTRACT_BUDGET_MS, ''), ]); if (raw == null || raw === '') return; diff --git a/src/services/projectSkills.ts b/src/services/projectSkills.ts new file mode 100644 index 0000000..2081446 --- /dev/null +++ b/src/services/projectSkills.ts @@ -0,0 +1,404 @@ +/** + * Per-project procedural memory (the "skills" half of memory). After a chat + * turn, a throttled background reviewer distills reusable *procedural skills* + * from what just happened. Distilled skills do NOT activate automatically — + * they land in a per-project inbox as suggestions (project_skill_suggestions). + * The user accepts/edits/rejects them, exactly like project fact suggestions. + * + * This mirrors src/services/projectMemory.ts (the declarative "facts" half) and + * layers Hermes's distillation discipline (class-level skills, patch-over-new) + * plus the SKILL.md format on top, with a human-acceptance gate. + * + * SECURITY: the reviewer only ever emits inbox suggestions — it never writes an + * active skill. Untrusted turns (web/email/Telegram ingestion, secure+network) + * are flagged so the UI can warn the user. See docs/project-skills-spec.md §10. + */ + +import { + projectSkills, + projectSkillSuggestions, + projects, + chats, +} from './store'; +import type { ProjectSkillIndexRow } from './store'; +import { runSubtaskTurn } from './subtaskLlm'; +import { wsHub } from './wsHub'; + +/** Local copy (mirrors projectMemory.normalizeForDedup) — avoids a circular import. */ +function normalizeForDedup(s: string): string { + return s.trim().toLowerCase().replace(/\s+/g, ' '); +} + +/** Rough token budget for the prepended skills index block. */ +const CHARS_PER_TOKEN_EST = 4; +const SKILLS_INDEX_TARGET_TOKENS = 600; +const MAX_INDEX_LINES = 30; +const MAX_DESC_CHARS = 200; +/** MVP shortcut (§7b): inline the full bodies of up to N small skills. */ +const INLINE_BODY_COUNT = 3; +const INLINE_BODY_MAX_CHARS = 1200; +/** + * Default review cadence (Ask/Execute). Work/Secure turns are agentic and + * information-rich, so they pass a smaller interval (see WORK_REVIEW_INTERVAL + * in chatRunner) and get reviewed more often. + */ +const SKILL_REVIEW_TURN_INTERVAL = 8; +const REVIEW_BUDGET_MS = 90_000; +/** Max suggestions accepted from a single review pass. */ +const MAX_SUGGESTIONS_PER_REVIEW = 3; + +/** Per-chat counter of substantive (tool-using) turns since the last review. */ +const turnCounters = new Map<number, number>(); +/** Chats with a review in flight — never run two concurrently for one chat. */ +const reviewingChats = new Set<number>(); + +function approxTokens(s: string): number { + return Math.ceil(s.length / CHARS_PER_TOKEN_EST); +} + +function sleep<T = null>(ms: number, value: T | null = null): Promise<T | null> { + return new Promise((resolve) => setTimeout(() => resolve(value), ms)); +} + +/** + * Build the "Skills available" block injected into the system prompt (Work / + * Secure) or the gateway user message (Execute). Returns '' when the project + * has no active skills. Index lines are token-bounded like facts; the bodies of + * up to INLINE_BODY_COUNT small, most-recently-updated skills are inlined as an + * MVP shortcut for `get_skill` (spec §7b). + */ +export function buildSkillsPromptBlock(projectId: number): string { + if (!projects.get(projectId)) return ''; + const skills = projectSkills.listForProject(projectId); + if (skills.length === 0) return ''; + + const header = 'Skills available (procedural memory for this project):'; + let used = approxTokens(header); + const lines: string[] = []; + for (const s of skills) { + if (lines.length >= MAX_INDEX_LINES) break; + const desc = + s.description.length > MAX_DESC_CHARS + ? s.description.slice(0, MAX_DESC_CHARS - 1) + '…' + : s.description; + const line = `- ${s.name}: ${desc}`; + const t = approxTokens(line + '\n'); + if (used + t > SKILLS_INDEX_TARGET_TOKENS && lines.length > 0) break; + used += t; + lines.push(line); + } + + // Inline the bodies of a few small skills so the agent can follow them now, + // without a round-trip. Larger skills stay index-only (request on demand). + const inlined: string[] = []; + let inlinedCount = 0; + for (const s of skills) { + if (inlinedCount >= INLINE_BODY_COUNT) break; + if (s.body.length > INLINE_BODY_MAX_CHARS) continue; + inlined.push(`\n--- skill: ${s.name} ---\n${s.body.trim()}`); + inlinedCount++; + } + + const parts = [header, ...lines]; + if (inlined.length > 0) { + parts.push( + '\nFull procedures for the most relevant skills follow. For any other skill above, ask the user or apply the one-line summary.', + ...inlined, + ); + } else if (lines.length > 0) { + parts.push('\nApply a skill when its summary matches the task at hand.'); + } + return parts.join('\n'); +} + +// --------------------------------------------------------------------------- +// Reviewer +// --------------------------------------------------------------------------- + +interface ReviewedSkill { + action: 'new' | 'patch'; + target?: string; + name: string; + description: string; + tags?: string[]; + body: string; +} + +/** Extract the first balanced JSON object from a possibly fenced model output. */ +export function extractJsonObject(raw: string): string | null { + const t = raw.trim(); + if (!t || /^none\.?$/i.test(t)) return null; + const start = t.indexOf('{'); + if (start < 0) return null; + let depth = 0; + let inStr = false; + let esc = false; + for (let i = start; i < t.length; i++) { + const ch = t[i]; + if (inStr) { + if (esc) esc = false; + else if (ch === '\\') esc = true; + else if (ch === '"') inStr = false; + continue; + } + if (ch === '"') inStr = true; + else if (ch === '{') depth++; + else if (ch === '}') { + depth--; + if (depth === 0) return t.slice(start, i + 1); + } + } + return null; +} + +const KEBAB_RE = /^[a-z0-9]+(?:-[a-z0-9]+)*$/; + +function toKebab(name: string): string { + return name + .trim() + .toLowerCase() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 64); +} + +/** Parse + sanitize the reviewer JSON into well-formed skill candidates. */ +export function parseReviewedSkills(raw: string): ReviewedSkill[] { + const json = extractJsonObject(raw); + if (!json) return []; + let parsed: unknown; + try { + parsed = JSON.parse(json); + } catch { + return []; + } + const arr = (parsed as { skills?: unknown })?.skills; + if (!Array.isArray(arr)) return []; + const out: ReviewedSkill[] = []; + for (const item of arr) { + if (!item || typeof item !== 'object') continue; + const o = item as Record<string, unknown>; + const name = typeof o.name === 'string' ? toKebab(o.name) : ''; + const description = typeof o.description === 'string' ? o.description.trim() : ''; + const body = typeof o.body === 'string' ? o.body.trim() : ''; + if (!name || !KEBAB_RE.test(name) || !description || !body) continue; + const action = o.action === 'patch' ? 'patch' : 'new'; + const target = typeof o.target === 'string' ? toKebab(o.target) : undefined; + const tags = Array.isArray(o.tags) + ? o.tags.filter((t): t is string => typeof t === 'string').slice(0, 8) + : undefined; + out.push({ action, target, name, description, tags, body }); + if (out.length >= MAX_SUGGESTIONS_PER_REVIEW) break; + } + return out; +} + +function buildReviewPrompt(opts: { + transcript: string; + existingIndex: ProjectSkillIndexRow[]; +}): string { + const existingBlock = + opts.existingIndex.length > 0 + ? opts.existingIndex.map((s) => `- ${s.name}: ${s.description}`).join('\n') + : '(none yet)'; + return [ + 'TASK: From the conversation below, distill 0–3 reusable PROCEDURAL skills', + 'that would help an agent do similar work next time in this project.', + '', + 'A skill captures a procedure / convention / workflow discovered this session:', + 'commands that worked, project conventions, tool quirks, gotchas, repeatable', + 'multi-step processes. It is CLASS-LEVEL and reusable — NOT a one-shot task log.', + '', + 'Strongly PREFER patching an existing skill (action "patch", set "target" to', + 'its name) over creating a near-duplicate. Only create a "new" skill for a', + 'genuinely distinct capability. Most sessions produce zero or one skill update.', + '', + 'Hard skip (never emit): transient task state ("user is debugging X"),', + 'secrets/tokens/credentials/.env values, SSH/IPs/hostnames, user-local', + 'filesystem paths, one-off facts (those belong to project facts, not skills).', + '', + 'Existing skills (patch these instead of duplicating):', + existingBlock, + '', + 'Conversation:', + opts.transcript.slice(0, 16000), + '', + 'OUTPUT: strict JSON, no prose, no code fences:', + '{"skills":[{"action":"new"|"patch","target":"<existing-name, patch only>",', + '"name":"<kebab-case>","description":"<one line summary>","tags":["..."],', + '"body":"<full SKILL.md markdown: frontmatter + procedure sections>"}]}', + 'If nothing qualifies, output exactly: {"skills":[]}', + 'Use the same language as the technical content.', + ].join('\n'); +} + +/** Build a compact transcript of the recent turn for the reviewer. */ +function buildTranscript(userMessage: string, assistantText: string): string { + const user = userMessage.replace(/\s+/g, ' ').trim().slice(0, 6000); + const assistant = assistantText.trim().slice(0, 12000); + return `User:\n${user}\n\nAssistant:\n${assistant}`; +} + +async function runProjectSkillReview(opts: { + chatId: number; + projectId: number; + userMessage: string; + assistantText: string; + assistantMessageId: number; + untrusted: boolean; +}): Promise<void> { + const chat = chats.get(opts.chatId); + if (!chat || chat.project_id !== opts.projectId) return; + if (!chat.shares_to_project) return; + if (!projects.get(opts.projectId)) return; + if (reviewingChats.has(opts.chatId)) return; + + reviewingChats.add(opts.chatId); + try { + const existingIndex = projectSkills.listIndex(opts.projectId); + const raw = await Promise.race([ + runSubtaskTurn( + buildReviewPrompt({ + transcript: buildTranscript(opts.userMessage, opts.assistantText), + existingIndex, + }), + { maxTokens: 4096 }, + ), + sleep<string>(REVIEW_BUDGET_MS, ''), + ]); + if (raw == null || raw === '') return; + + const reviewed = parseReviewedSkills(raw); + if (reviewed.length === 0) return; + + // Dedup pool: active skill names/descriptions + pending suggestion names. + const activeNames = new Set(existingIndex.map((s) => normalizeForDedup(s.name))); + const activeDescs = existingIndex.map((s) => normalizeForDedup(s.description)); + const pending = projectSkillSuggestions.listByProject(opts.projectId); + const pendingNames = new Set(pending.map((s) => normalizeForDedup(s.name))); + + const inserted: { + id: number; + kind: 'new' | 'patch'; + name: string; + description: string; + untrusted: boolean; + targetSkillId: number | null; + }[] = []; + + for (const r of reviewed) { + const normName = normalizeForDedup(r.name); + const normDesc = normalizeForDedup(r.description); + + // Resolve patch target; downgrade to 'new' if the target is unknown. + let kind: 'new' | 'patch' = r.action; + let targetSkillId: number | null = null; + if (kind === 'patch') { + const targetName = r.target || r.name; + const target = + projectSkills.getByName(opts.projectId, targetName) ?? + projectSkills.getByName(null, targetName); + if (target) { + targetSkillId = target.id; + } else { + kind = 'new'; + } + } + + // For 'new', skip if a same-named active/pending skill already exists, or + // if the description duplicates an existing skill's summary. + if (kind === 'new') { + if (activeNames.has(normName) || pendingNames.has(normName)) continue; + if (activeDescs.some((d) => d && (d === normDesc))) continue; + } + + const row = projectSkillSuggestions.insert({ + projectId: opts.projectId, + chatId: opts.chatId, + kind, + targetSkillId, + name: r.name, + description: r.description, + body: r.body, + tags: r.tags ?? null, + untrusted: opts.untrusted, + assistantMessageId: opts.assistantMessageId, + }); + pendingNames.add(normName); + inserted.push({ + id: row.id, + kind: row.kind, + name: row.name, + description: row.description, + untrusted: Boolean(row.untrusted), + targetSkillId: row.target_skill_id, + }); + } + + if (inserted.length > 0) { + const proj = projects.get(opts.projectId); + wsHub.broadcastAll({ + type: 'project-skill-suggestions', + chatId: opts.chatId, + projectId: opts.projectId, + projectName: proj?.name?.trim() || 'project', + suggestions: inserted, + }); + } + } catch (err) { + console.error( + '[projectSkills] review pipeline failed', + err instanceof Error ? err.message : err, + ); + } finally { + reviewingChats.delete(opts.chatId); + } +} + +/** + * Fire-and-forget skill review after a successful assistant reply. Throttled: + * only runs once every SKILL_REVIEW_TURN_INTERVAL substantive turns per chat, + * because skill review is heavier than fact extraction. Never blocks the turn. + */ +export function scheduleProjectSkillReview(opts: { + chatId: number; + projectId: number; + sharesToProject: boolean; + substantive: boolean; + userMessage: string; + assistantText: string; + assistantMessageId: number; + untrusted: boolean; + /** Turns between reviews for this chat. Defaults to the Ask/Execute cadence. */ + interval?: number; +}): void { + if (!opts.sharesToProject) return; + if (!opts.substantive) return; + + const interval = opts.interval && opts.interval > 0 ? opts.interval : SKILL_REVIEW_TURN_INTERVAL; + const next = (turnCounters.get(opts.chatId) ?? 0) + 1; + if (next < interval) { + turnCounters.set(opts.chatId, next); + return; + } + turnCounters.set(opts.chatId, 0); + + void runProjectSkillReview({ + chatId: opts.chatId, + projectId: opts.projectId, + userMessage: opts.userMessage, + assistantText: opts.assistantText, + assistantMessageId: opts.assistantMessageId, + untrusted: opts.untrusted, + }).catch((err) => { + console.error( + '[projectSkills] review failed', + err instanceof Error ? err.message : err, + ); + }); +} + +/** Test/maintenance helper: reset the in-memory turn counter for a chat. */ +export function resetSkillReviewCounter(chatId: number): void { + turnCounters.delete(chatId); +} diff --git a/src/services/runtimeProcess.ts b/src/services/runtimeProcess.ts new file mode 100644 index 0000000..b9ebf22 --- /dev/null +++ b/src/services/runtimeProcess.ts @@ -0,0 +1,111 @@ +/** + * Manages the iclaw-runtime sidecar process lifecycle. + * + * Spawns packages/iclaw-runtime/src/index.ts via tsx when Work Mode is + * needed. Restarts automatically on crash (up to MAX_RESTARTS times). + * Shuts down cleanly with the main iClaw process. + */ +import { spawn, type ChildProcess } from 'node:child_process'; +import path from 'node:path'; +import fs from 'node:fs'; +import { kvGet } from '../db/kv'; + +const RUNTIME_DIR = path.resolve(__dirname, '../../packages/iclaw-runtime'); +const RUNTIME_ENTRY = path.join(RUNTIME_DIR, 'src/index.ts'); +const MAX_RESTARTS = 5; +const RESTART_WINDOW_MS = 60_000; + +let child: ChildProcess | null = null; +let restartCount = 0; +let windowStart = Date.now(); +let stopping = false; + +function runtimeInstalled(): boolean { + return fs.existsSync(RUNTIME_ENTRY); +} + +function spawnRuntime(): void { + if (stopping || !runtimeInstalled()) return; + + // Pass OpenRouter key from iClaw's DB into the runtime environment + const openRouterKey = (kvGet('openrouter.api_key') ?? '').trim(); + const runtimeEnv: NodeJS.ProcessEnv = { + ...process.env, + ...(openRouterKey ? { ICLAW_OPENROUTER_API_KEY: openRouterKey } : {}), + }; + + child = spawn('npx', ['tsx', RUNTIME_ENTRY], { + cwd: RUNTIME_DIR, + stdio: ['ignore', 'pipe', 'pipe'], + env: runtimeEnv, + }); + + child.stdout?.on('data', (chunk: Buffer) => { + process.stdout.write(`[iclaw-runtime] ${chunk}`); + }); + child.stderr?.on('data', (chunk: Buffer) => { + process.stderr.write(`[iclaw-runtime] ${chunk}`); + }); + + child.on('exit', (code, signal) => { + child = null; + if (stopping) return; + + const now = Date.now(); + if (now - windowStart > RESTART_WINDOW_MS) { + restartCount = 0; + windowStart = now; + } + restartCount++; + + if (restartCount > MAX_RESTARTS) { + console.error( + `[iclaw-runtime] crashed ${MAX_RESTARTS} times in ${RESTART_WINDOW_MS / 1000}s — giving up`, + ); + return; + } + + const delay = Math.min(1000 * restartCount, 10_000); + console.warn( + `[iclaw-runtime] exited (code=${code ?? signal}) — restarting in ${delay}ms (attempt ${restartCount}/${MAX_RESTARTS})`, + ); + setTimeout(spawnRuntime, delay).unref(); + }); +} + +export const runtimeProcess = { + start(): void { + if (!runtimeInstalled()) return; + stopping = false; + spawnRuntime(); + }, + + stop(): void { + stopping = true; + if (child) { + child.kill('SIGTERM'); + child = null; + } + }, + + /** + * Restart so the runtime picks up a changed OpenRouter key (the key is read + * from the DB at spawn and passed via env — a running runtime can't see a key + * added/changed later, e.g. during onboarding). Safe to call even if the + * runtime isn't running yet: it just starts it. + */ + restart(): void { + if (!runtimeInstalled()) return; + this.stop(); + // Brief gap so port 7430 frees before respawn (the runtime also self-retries + // on a busy port, so this is just to avoid the noisy "port in use" log). + setTimeout(() => { + stopping = false; + spawnRuntime(); + }, 500).unref(); + }, + + get running(): boolean { + return child !== null && !stopping; + }, +}; diff --git a/src/services/store.ts b/src/services/store.ts index dbcc5fa..54f977e 100644 --- a/src/services/store.ts +++ b/src/services/store.ts @@ -4,11 +4,14 @@ import { deriveTitle } from './chatTitle'; import type { Chat, ChatKind, + ChatMode, Message, MessageAttachment, Project, ProjectFact, ProjectFactSuggestion, + ProjectSkill, + ProjectSkillSuggestion, ProjectSecret, QueuedMessage, ScheduledMessage, @@ -24,6 +27,7 @@ import type { TaskWithSteps, } from '../types'; import type { InlineSecretWire } from './inlineSecrets'; +import { DEFAULT_MODE } from './chatModes'; import { clampLogoColor, clampLogoEmoji } from '../constants/projectLogos'; // ---------- chats ---------- @@ -210,6 +214,12 @@ export const messages = { finishReason: string | null = null, reply?: { replyToMessageId: number; replyQuote: string; replyToRole: string } | null, attachments?: MessageAttachment[] | null, + /** Send mode for user rows. Defaults to 'execute' (back-compat). */ + mode: ChatMode = DEFAULT_MODE, + /** Total tokens spent producing this message (dev-mode; assistant rows). */ + tokens: number | null = null, + /** Of `tokens`, prompt tokens served from cache (dev-mode). */ + cachedTokens: number | null = null, ): Message { const rid = reply?.replyToMessageId ?? null; const rq = reply?.replyQuote ?? null; @@ -218,9 +228,9 @@ export const messages = { attachments && attachments.length > 0 ? JSON.stringify(attachments) : null; const info = db .prepare( - 'INSERT INTO messages (chat_id, role, content, finish_reason, reply_to_message_id, reply_quote, reply_to_role, attachments) VALUES (?, ?, ?, ?, ?, ?, ?, ?)', + 'INSERT INTO messages (chat_id, role, content, finish_reason, reply_to_message_id, reply_quote, reply_to_role, attachments, mode, tokens, cached_tokens) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', ) - .run(chatId, role, content, finishReason, rid, rq, rrole, att); + .run(chatId, role, content, finishReason, rid, rq, rrole, att, mode, tokens, cachedTokens); // chats.updated_at is bumped by the trg_chats_touch_on_message SQLite // trigger; no manual touch() needed here. We keep chats.touch() public // for callers that mutate parents without writing a message (e.g. @@ -479,6 +489,197 @@ export const projectFactSuggestions = { }, }; +// ---------- project skills (procedural memory; SKILL.md) ---------- + +/** name + description only — for prompt injection / panel list. */ +export interface ProjectSkillIndexRow { + id: number; + name: string; + description: string; +} + +export const projectSkills = { + /** Active skills for a project (does NOT include global; merge in the caller). */ + listByProject(projectId: number): ProjectSkill[] { + return db + .prepare( + 'SELECT * FROM project_skills WHERE project_id = ? ORDER BY updated_at DESC, id DESC', + ) + .all(projectId) as ProjectSkill[]; + }, + listGlobal(): ProjectSkill[] { + return db + .prepare( + 'SELECT * FROM project_skills WHERE project_id IS NULL ORDER BY updated_at DESC, id DESC', + ) + .all() as ProjectSkill[]; + }, + /** Active skills visible to a project: its own skills plus all global skills. */ + listForProject(projectId: number): ProjectSkill[] { + return [...this.listByProject(projectId), ...this.listGlobal()]; + }, + get(id: number): ProjectSkill | undefined { + return db.prepare('SELECT * FROM project_skills WHERE id = ?').get(id) as + | ProjectSkill + | undefined; + }, + getByName(projectId: number | null, name: string): ProjectSkill | undefined { + const trimmed = name.trim(); + if (projectId == null) { + return db + .prepare('SELECT * FROM project_skills WHERE project_id IS NULL AND name = ?') + .get(trimmed) as ProjectSkill | undefined; + } + return db + .prepare('SELECT * FROM project_skills WHERE project_id = ? AND name = ?') + .get(projectId, trimmed) as ProjectSkill | undefined; + }, + /** Index (id/name/description) of all skills visible to a project. */ + listIndex(projectId: number): ProjectSkillIndexRow[] { + return this.listForProject(projectId).map((s) => ({ + id: s.id, + name: s.name, + description: s.description, + })); + }, + create(opts: { + projectId: number | null; + name: string; + description: string; + body: string; + tags?: string[] | null; + sourceChatId?: number | null; + }): ProjectSkill { + const name = opts.name.trim(); + const description = opts.description.trim(); + const body = opts.body.trim(); + if (!name) throw new Error('skill name required'); + if (!description) throw new Error('skill description required'); + if (!body) throw new Error('skill body required'); + const tags = opts.tags && opts.tags.length > 0 ? JSON.stringify(opts.tags) : null; + const info = db + .prepare( + `INSERT INTO project_skills (project_id, name, description, body, tags, source_chat_id) + VALUES (?, ?, ?, ?, ?, ?)`, + ) + .run(opts.projectId, name, description, body, tags, opts.sourceChatId ?? null); + if (opts.projectId != null) projects.touch(opts.projectId); + return this.get(Number(info.lastInsertRowid))!; + }, + /** Patch description/body/tags/name; bumps version + updated_at. */ + update( + id: number, + patch: { description?: string; body?: string; tags?: string[] | null; name?: string }, + ): void { + const existing = this.get(id); + if (!existing) return; + const name = patch.name != null ? patch.name.trim() : existing.name; + const description = + patch.description != null ? patch.description.trim() : existing.description; + const body = patch.body != null ? patch.body.trim() : existing.body; + const tags = + patch.tags !== undefined + ? patch.tags && patch.tags.length > 0 + ? JSON.stringify(patch.tags) + : null + : existing.tags; + db.prepare( + `UPDATE project_skills + SET name = ?, description = ?, body = ?, tags = ?, + version = version + 1, updated_at = datetime('now') + WHERE id = ?`, + ).run(name, description, body, tags, id); + if (existing.project_id != null) projects.touch(existing.project_id); + }, + incrementUsage(id: number): void { + db.prepare('UPDATE project_skills SET usage_count = usage_count + 1 WHERE id = ?').run(id); + }, + remove(id: number): void { + const row = this.get(id); + db.prepare('DELETE FROM project_skills WHERE id = ?').run(id); + if (row?.project_id != null) projects.touch(row.project_id); + }, +}; + +// ---------- project skill suggestions (user confirm in chat; inbox-gated) ---------- + +export const projectSkillSuggestions = { + listByChat(chatId: number): ProjectSkillSuggestion[] { + return db + .prepare( + 'SELECT * FROM project_skill_suggestions WHERE chat_id = ? ORDER BY id ASC', + ) + .all(chatId) as ProjectSkillSuggestion[]; + }, + listByProject(projectId: number): ProjectSkillSuggestion[] { + return db + .prepare( + 'SELECT * FROM project_skill_suggestions WHERE project_id = ? ORDER BY id ASC', + ) + .all(projectId) as ProjectSkillSuggestion[]; + }, + get(id: number): ProjectSkillSuggestion | undefined { + return db.prepare('SELECT * FROM project_skill_suggestions WHERE id = ?').get(id) as + | ProjectSkillSuggestion + | undefined; + }, + insert(opts: { + projectId: number; + chatId: number; + kind: 'new' | 'patch'; + targetSkillId?: number | null; + name: string; + description: string; + body: string; + tags?: string[] | null; + untrusted?: boolean; + assistantMessageId: number | null; + }): ProjectSkillSuggestion { + const name = opts.name.trim(); + const description = opts.description.trim(); + const body = opts.body.trim(); + if (!name) throw new Error('suggestion name required'); + if (!description) throw new Error('suggestion description required'); + if (!body) throw new Error('suggestion body required'); + const tags = opts.tags && opts.tags.length > 0 ? JSON.stringify(opts.tags) : null; + const info = db + .prepare( + `INSERT INTO project_skill_suggestions + (project_id, chat_id, kind, target_skill_id, name, description, body, tags, untrusted, assistant_message_id) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + opts.projectId, + opts.chatId, + opts.kind, + opts.targetSkillId ?? null, + name, + description, + body, + tags, + opts.untrusted ? 1 : 0, + opts.assistantMessageId ?? null, + ); + return this.get(Number(info.lastInsertRowid))!; + }, + remove(id: number): void { + db.prepare('DELETE FROM project_skill_suggestions WHERE id = ?').run(id); + }, +}; + +/** UI: attach source chat title from `chats` (not persisted on `project_skills`). */ +export function enrichSkillWithSourceChatTitle(skill: ProjectSkill): ProjectSkill { + const sid = skill.source_chat_id; + if (sid == null) return { ...skill }; + const c = chats.get(sid); + const source_chat_title = (c?.title ?? '').trim() || 'Chat'; + return { ...skill, source_chat_title }; +} + +export function enrichSkillsWithSourceChatTitles(skills: ProjectSkill[]): ProjectSkill[] { + return skills.map(enrichSkillWithSourceChatTitle); +} + // ---------- project secrets (tokens / API keys; placeholders in messages) ---------- export type SecretPickerRow = Omit<ProjectSecret, 'value'> & { @@ -767,7 +968,7 @@ export const queuedMessages = { listByChat(chatId: number): QueuedMessage[] { const rows = db .prepare( - 'SELECT id, chat_id, content, reply_to_message_id, reply_quote, reply_to_role, attachments, created_at FROM queued_messages WHERE chat_id = ? ORDER BY position ASC, id ASC', + 'SELECT id, chat_id, content, reply_to_message_id, reply_quote, reply_to_role, attachments, mode, created_at FROM queued_messages WHERE chat_id = ? ORDER BY position ASC, id ASC', ) .all(chatId) as QueuedRow[]; return rows.map(parseQueuedRow); @@ -775,7 +976,7 @@ export const queuedMessages = { get(id: number): QueuedMessage | undefined { const row = db .prepare( - 'SELECT id, chat_id, content, reply_to_message_id, reply_quote, reply_to_role, attachments, created_at FROM queued_messages WHERE id = ?', + 'SELECT id, chat_id, content, reply_to_message_id, reply_quote, reply_to_role, attachments, mode, created_at FROM queued_messages WHERE id = ?', ) .get(id) as QueuedRow | undefined; return row ? parseQueuedRow(row) : undefined; @@ -786,7 +987,7 @@ export const queuedMessages = { | undefined { const row = db .prepare( - 'SELECT id, chat_id, content, reply_to_message_id, reply_quote, reply_to_role, attachments, inline_secrets, created_at FROM queued_messages WHERE id = ?', + 'SELECT id, chat_id, content, reply_to_message_id, reply_quote, reply_to_role, attachments, mode, inline_secrets, created_at FROM queued_messages WHERE id = ?', ) .get(id) as (QueuedRow & { inline_secrets: string | null }) | undefined; if (!row) return undefined; @@ -808,6 +1009,7 @@ export const queuedMessages = { replyTo?: { messageId: number; quote: string; role?: string } | null; attachments?: MessageAttachment[] | null; inlineSecrets?: InlineSecretWire[] | null; + mode?: ChatMode; }): QueuedMessage { const trimmed = opts.content.trim(); const hasAttachments = opts.attachments && opts.attachments.length > 0; @@ -826,8 +1028,8 @@ export const queuedMessages = { .prepare( `INSERT INTO queued_messages ( chat_id, content, reply_to_message_id, reply_quote, reply_to_role, - attachments, inline_secrets, position - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + attachments, inline_secrets, mode, position + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, ) .run( opts.chatId, @@ -837,6 +1039,7 @@ export const queuedMessages = { opts.replyTo?.role ?? null, attachmentsJson, inlineJson, + opts.mode ?? DEFAULT_MODE, position, ); return this.get(Number(info.lastInsertRowid))!; diff --git a/src/services/subtaskLlm.ts b/src/services/subtaskLlm.ts new file mode 100644 index 0000000..6762154 --- /dev/null +++ b/src/services/subtaskLlm.ts @@ -0,0 +1,88 @@ +/** + * Shared runner for the background "throwaway" LLM sub-tasks — fact extraction, + * fact compaction, and skill review. These are cheap, tool-less, single-shot + * prompts that should NOT spin up a full OpenClaw agent run when a direct, + * cheaper path is available. + * + * Routing: + * 1. If OpenRouter is configured, run on the cheap `summaryModel` via a direct + * /chat/completions call (much cheaper than an OpenClaw agent turn). + * 2. Otherwise — or if the OpenRouter call fails — fall back to a throwaway + * OpenClaw session on the default `main` agent (the original behaviour). + * + * Only the background sub-tasks use this. The user-facing turns (Ask, Execute, + * Work, Secure) keep their own backends. + */ + +import { openclawWs } from './openclawWs'; +import { complete, openRouterEnabled, isOpenRouterFailure } from './openRouter'; +import { loadOpenRouterConfig } from './config'; + +/** + * Sub-tasks always run against the gateway's default agent when falling back to + * OpenClaw. Using whatever agent the chat is on risked specialised agents + * underperforming on a plain text-extraction prompt. One agent, one behaviour. + */ +const SUBTASK_AGENT_ID = 'main'; + +export interface SubtaskTurnOptions { + /** Optional system prompt (OpenRouter path only; ignored on OpenClaw fallback). */ + system?: string; + temperature?: number; + maxTokens?: number; +} + +/** Throwaway OpenClaw session — original behaviour, used as the fallback path. */ +async function runViaOpenClaw(message: string): Promise<string> { + let sessionKey: string | null = null; + try { + const session = await openclawWs.createSession({ agentId: SUBTASK_AGENT_ID }); + sessionKey = session.key; + let acc = ''; + await openclawWs.runTurn({ + sessionKey: session.key, + message, + onEvent: (ev) => { + if (ev.type === 'text-delta') acc += ev.text; + else if (ev.type === 'text-final') acc = ev.text || acc; + }, + }); + return acc; + } finally { + if (sessionKey) openclawWs.deleteSession(sessionKey).catch(() => {}); + } +} + +/** + * Run a single-shot background sub-task prompt. Prefers OpenRouter (cheap model) + * and falls back to OpenClaw when OpenRouter is unconfigured or the call throws. + * Returns the assistant text (possibly empty — callers parse/validate it). + */ +export async function runSubtaskTurn( + message: string, + opts: SubtaskTurnOptions = {}, +): Promise<string> { + if (openRouterEnabled()) { + try { + const cfg = loadOpenRouterConfig(); + const messages = []; + if (opts.system) messages.push({ role: 'system' as const, content: opts.system }); + messages.push({ role: 'user' as const, content: message }); + // Cheapest configured model — these tasks don't need a frontier model. + return await complete({ + messages, + model: cfg.summaryModel, + temperature: opts.temperature, + maxTokens: opts.maxTokens, + }); + } catch (err) { + // OpenRouter unavailable / errored → fall back to OpenClaw rather than + // dropping the sub-task. Log once; the fallback path is best-effort too. + console.error( + '[subtaskLlm] OpenRouter failed, falling back to OpenClaw', + isOpenRouterFailure(err) ? (err as Error).message : err, + ); + } + } + return runViaOpenClaw(message); +} diff --git a/src/services/turnReply.ts b/src/services/turnReply.ts index de5e7c2..421c8bc 100644 --- a/src/services/turnReply.ts +++ b/src/services/turnReply.ts @@ -28,6 +28,71 @@ export interface HistoryMessageLike { toolName?: string; timestamp?: number; isError?: boolean; + /** + * Per-message token usage, present on assistant rows in OpenClaw's + * UI-normalized `chat.history`. The gateway sanitizes it to a flat object + * of numeric fields (field names vary by provider — see `pickUsageTotal`). + */ + usage?: unknown; +} + +/** Pull a finite number from a usage object under the first matching key. */ +function firstNumber(u: Record<string, unknown>, keys: string[]): number | null { + for (const k of keys) { + const v = u[k]; + if (typeof v === 'number' && Number.isFinite(v)) return v; + } + return null; +} + +/** + * Total tokens for one assistant row's `usage`. Prefers an explicit total; + * otherwise sums input + output. Field names cover OpenClaw's sanitized set + * (camelCase, snake_case, and Anthropic-style). Returns `null` when the row + * carries no usable usage numbers. + */ +function pickUsageTotal(usage: unknown): number | null { + if (!usage || typeof usage !== 'object') return null; + const u = usage as Record<string, unknown>; + const total = firstNumber(u, ['total_tokens', 'totalTokens', 'total']); + if (total != null) return total; + const input = firstNumber(u, ['input_tokens', 'inputTokens', 'input', 'prompt_tokens', 'promptTokens']); + const output = firstNumber(u, ['output_tokens', 'outputTokens', 'output', 'completion_tokens', 'completionTokens']); + if (input == null && output == null) return null; + return (input ?? 0) + (output ?? 0); +} + +/** Prompt tokens served from cache for one assistant row's `usage`, or null. */ +function pickUsageCached(usage: unknown): number | null { + if (!usage || typeof usage !== 'object') return null; + return firstNumber(usage as Record<string, unknown>, [ + 'cacheRead', + 'cache_read_input_tokens', + 'cachedTokens', + 'cached_tokens', + ]); +} + +/** + * Sum token usage across every assistant row in a turn-scoped slice. A native + * tool loop commits multiple assistant segments per turn (preamble + post-tool + * reply), each with its own `usage`, so the turn's true cost is their sum. + * Returns `null` for a field when no assistant row reported it. + */ +export function extractTurnUsage(slice: HistoryMessageLike[]): { + tokens: number | null; + cached: number | null; +} { + let tokens: number | null = null; + let cached: number | null = null; + for (const row of slice) { + if (row.role !== 'assistant') continue; + const t = pickUsageTotal(row.usage); + if (t != null) tokens = (tokens ?? 0) + t; + const c = pickUsageCached(row.usage); + if (c != null) cached = (cached ?? 0) + c; + } + return { tokens, cached }; } /** diff --git a/src/services/uploads.ts b/src/services/uploads.ts index bba9c58..eff1562 100644 --- a/src/services/uploads.ts +++ b/src/services/uploads.ts @@ -197,6 +197,32 @@ export function persistIncomingAttachments( return out; } +/** + * Resolve persisted attachments to absolute on-disk paths for the iclaw-runtime + * (Work/Secure/Incognito). The runtime shares the host filesystem, so it reads + * the file directly — no base64 roundtrip. Rows with an unexpected URL are + * skipped (rather than thrown) so one bad attachment can't sink the whole turn. + */ +export function runtimeAttachmentsFromPersisted( + chatId: number, + persisted: MessageAttachment[], +): { path: string; mimeType: string; fileName: string }[] { + if (!persisted.length) return []; + const chatDir = join(resolveUploadsRoot(), String(chatId)); + const prefix = `/uploads/${chatId}/`; + const out: { path: string; mimeType: string; fileName: string }[] = []; + for (const att of persisted) { + const url = att.url || ''; + if (!url.startsWith(prefix)) continue; + out.push({ + path: join(chatDir, basename(url)), + mimeType: att.mimeType || 'application/octet-stream', + fileName: att.fileName || 'attachment', + }); + } + return out; +} + /** * Rebuild OpenClaw gateway attachment payloads from rows already on disk * (queued messages persist files at enqueue time). diff --git a/src/services/workRuntime.ts b/src/services/workRuntime.ts new file mode 100644 index 0000000..e9ede62 --- /dev/null +++ b/src/services/workRuntime.ts @@ -0,0 +1,233 @@ +/** + * iClaw Work Mode runtime client. + * + * Communicates with the `@iclaw/runtime` service (packages/iclaw-runtime) + * over HTTP. The runtime runs as a sidecar process on the same host. + * + * The runtime exposes: + * POST /sessions → { sessionId } + * POST /sessions/:id/messages → 202 + * GET /sessions/:id/events → SSE stream + * DELETE /sessions/:id → stop + */ +import http from 'http'; + +const RUNTIME_PORT = parseInt(process.env.ICLAW_RUNTIME_PORT || '7430', 10); +const RUNTIME_SECRET = process.env.ICLAW_RUNTIME_SECRET || ''; + +function runtimeHeaders(): Record<string, string> { + const h: Record<string, string> = { 'Content-Type': 'application/json' }; + if (RUNTIME_SECRET) h['x-iclaw-token'] = RUNTIME_SECRET; + return h; +} + +function request( + method: string, + path: string, + body?: unknown, +): Promise<{ status: number; data: unknown }> { + return new Promise((resolve, reject) => { + const payload = body != null ? JSON.stringify(body) : undefined; + const headers = runtimeHeaders(); + if (payload) headers['Content-Length'] = Buffer.byteLength(payload).toString(); + + const req = http.request( + { hostname: '127.0.0.1', port: RUNTIME_PORT, path, method, headers }, + (res) => { + let raw = ''; + res.on('data', (chunk) => { raw += chunk; }); + res.on('end', () => { + try { + resolve({ status: res.statusCode ?? 0, data: JSON.parse(raw) }); + } catch { + resolve({ status: res.statusCode ?? 0, data: raw }); + } + }); + }, + ); + req.on('error', reject); + if (payload) req.write(payload); + req.end(); + }); +} + +export interface CreateSessionOptions { + allowedFolders?: string[]; + /** + * Per-folder access levels. When provided, the runtime enforces read-only + * folders (denies write_file / run_command under them) and derives the + * allowed-path list from it. + */ + folderAccess?: { path: string; readonly: boolean }[]; + /** + * Safe Mode only: host folders to COPY into the sandbox workspace (originals + * never touched). Ignored in Work Mode, which bind-mounts folders live. + */ + copyFolders?: string[]; + model?: string; + secure?: boolean; + /** Incognito: read-only, read-anywhere, web_fetch enabled. Mutually exclusive with secure. */ + incognito?: boolean; + systemPrompt?: string; + /** Stable identity (e.g. "chat:156") so a chat reconnects to its workspace. */ + key?: string; + /** Compacted prior history to seed context (used after a restart). */ + history?: { role: string; content: string }[]; +} + +/** Create a new Work Mode session. Returns sessionId. */ +export async function createWorkSession(opts: CreateSessionOptions = {}): Promise<string> { + const body: Record<string, unknown> = { allowedFolders: opts.allowedFolders, secure: opts.secure }; + if (opts.incognito) body.incognito = true; + if (opts.folderAccess?.length) body.folderAccess = opts.folderAccess; + if (opts.copyFolders?.length) body.copyFolders = opts.copyFolders; + if (opts.model) body.model = opts.model; + if (opts.systemPrompt) body.systemPrompt = opts.systemPrompt; + if (opts.key) body.key = opts.key; + if (opts.history?.length) body.history = opts.history; + const res = await request('POST', '/sessions', body); + if (res.status !== 201) { + throw new Error(`Failed to create work session: ${JSON.stringify(res.data)}`); + } + return (res.data as { sessionId: string }).sessionId; +} + +/** A dropped file forwarded to the runtime: absolute host path + metadata. */ +export interface RuntimeAttachmentInput { + path: string; + mimeType: string; + fileName: string; +} + +/** Send a user message to a work session. */ +export async function sendWorkMessage(sessionId: string, content: string, networkEnabled?: boolean, ttlDays?: number, attachments?: RuntimeAttachmentInput[], copyFolders?: string[]): Promise<void> { + const body: Record<string, unknown> = { content }; + if (networkEnabled !== undefined) body.networkEnabled = networkEnabled; + if (ttlDays !== undefined) body.ttlDays = ttlDays; + if (attachments?.length) body.attachments = attachments; + // Safe Mode: lets the runtime copy folders the user added mid-chat into the + // sandbox on this turn (ignored in Work Mode / by non-secure sessions). + if (copyFolders?.length) body.copyFolders = copyFolders; + const res = await request('POST', `/sessions/${sessionId}/messages`, body); + if (res.status !== 202) { + throw new Error(`Failed to send work message: ${JSON.stringify(res.data)}`); + } +} + +/** Stop a work session. */ +export async function stopWorkSession(sessionId: string): Promise<void> { + await request('DELETE', `/sessions/${sessionId}`); +} + +/** + * Abort the in-flight turn for a session WITHOUT destroying it (workspace + + * container survive — unlike stopWorkSession). Backs the Stop button for + * Work / Secure / Incognito turns. + */ +export async function abortWorkSession(sessionId: string): Promise<void> { + await request('POST', `/sessions/${sessionId}/abort`); +} + +/** A "saved N% cost" note from the runtime (e.g. analyze_link summary mode). */ +export interface RuntimeSavingsNote { + kind: string; + /** Short human label for the source, e.g. "video transcript". */ + source: string; + /** Whole-percent saved. Absent for quantity-free notes (search line-trimming). */ + savedPct?: number; + fullChars?: number; + deliveredChars?: number; +} + +export type WorkEvent = + | { type: 'text'; content: string } + | { type: 'tool'; name: string; input?: unknown } + | { type: 'note'; note: RuntimeSavingsNote } + | { type: 'done'; tokens?: number; cached?: number } + | { type: 'error'; message: string }; + +/** + * Subscribe to SSE events from a work session. + * Calls `onEvent` for each event, resolves when stream closes. + */ +export function subscribeWorkEvents( + sessionId: string, + onEvent: (event: WorkEvent) => void, + onError?: (err: Error) => void, +): () => void { + const headers = runtimeHeaders(); + const req = http.request( + { + hostname: '127.0.0.1', + port: RUNTIME_PORT, + path: `/sessions/${sessionId}/events`, + method: 'GET', + headers, + }, + (res) => { + let buf = ''; + res.on('data', (chunk: string) => { + buf += chunk; + const lines = buf.split('\n'); + buf = lines.pop() ?? ''; + for (const line of lines) { + if (line.startsWith('data: ')) { + try { + const event = JSON.parse(line.slice(6)) as WorkEvent; + onEvent(event); + } catch { + // ignore malformed SSE lines + } + } + } + }); + res.on('end', () => onEvent({ type: 'done' })); + res.on('error', (err) => onError?.(err)); + }, + ); + req.on('error', (err) => onError?.(err)); + req.end(); + + return () => req.destroy(); +} + +/** Get workspace info for a session. */ +export async function getWorkspaceInfo(sessionId: string): Promise<{ workspaceSize: number; secure: boolean } | null> { + try { + const res = await request('GET', `/sessions/${sessionId}/info`); + if (res.status !== 200) return null; + return res.data as { workspaceSize: number; secure: boolean }; + } catch { return null; } +} + +export interface ExportResult { ok: boolean; path?: string; files?: number; error?: string } +export interface ApplyResult { + ok: boolean; + source?: string; + applied?: { path: string; kind: 'created' | 'modified' }[]; + error?: string; +} + +/** Export a Safe sandbox to a host folder (default ~/Downloads). */ +export async function exportSandbox(sessionId: string, destDir?: string): Promise<ExportResult> { + const res = await request('POST', `/sessions/${sessionId}/export`, destDir ? { destDir } : {}); + if (res.status !== 200) throw new Error(`Export failed: ${JSON.stringify(res.data)}`); + return res.data as ExportResult; +} + +/** Apply a Safe sandbox's changes back to the originally-ingested folders. */ +export async function applySandboxChanges(sessionId: string): Promise<ApplyResult[]> { + const res = await request('POST', `/sessions/${sessionId}/apply`, {}); + if (res.status !== 200) throw new Error(`Apply failed: ${JSON.stringify(res.data)}`); + return (res.data as { results: ApplyResult[] }).results; +} + +/** True if the runtime service appears to be running. */ +export async function runtimeAvailable(): Promise<boolean> { + try { + const res = await request('GET', '/health'); + return res.status === 200; + } catch { + return false; + } +} diff --git a/src/types/index.ts b/src/types/index.ts index 7c74b97..94fb232 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -41,6 +41,8 @@ export interface QueuedMessage { reply_quote: string | null; reply_to_role: string | null; attachments: MessageAttachment[] | null; + /** Selected mode at enqueue time; preserved so the flush sends with it. */ + mode: ChatMode; created_at: string; } @@ -54,6 +56,42 @@ export interface ProjectFactSuggestion { created_at: string; } +/** + * Procedural memory: an accepted, active project skill stored as SKILL.md. + * `project_id === null` means a global skill (available to every project). + */ +export interface ProjectSkill { + id: number; + project_id: number | null; // null = global + name: string; + description: string; + body: string; + tags: string | null; // JSON array + source_chat_id: number | null; + usage_count: number; + version: number; + created_at: string; + updated_at: string; + /** Filled by server for UI (not a DB column). */ + source_chat_title?: string; +} + +/** LLM-proposed skill awaiting user accept/reject in the chat UI (inbox-gated). */ +export interface ProjectSkillSuggestion { + id: number; + project_id: number; + chat_id: number; + kind: 'new' | 'patch'; + target_skill_id: number | null; + name: string; + description: string; + body: string; + tags: string | null; + untrusted: number; // 0 | 1 + assistant_message_id: number | null; + created_at: string; +} + /** API key / token; message text uses `[[iclaw:secret:id|…]]` placeholders. */ export interface ProjectSecret { id: number; @@ -68,6 +106,24 @@ export interface ProjectSecret { export type ChatKind = 'normal' | 'draft' | 'task_execution'; +/** + * How a user message should be handled. + * + * - 'execute' — default: OpenClaw may use tools, files, shell, browser, etc. + * The back-compat fallback for any message whose `mode` is missing or + * unrecognized. + * - 'work' / 'secure' — run on iclaw-runtime (our runtime). + * - 'incognito' — read-only, ephemeral research on iclaw-runtime; never + * persisted (see services/chatModes.ts). + * + * Kept as a string union for the live modes, but the full catalog + * (incl. planned modes like image) lives in + * `services/chatModes.ts` so new modes can be added without touching this + * type everywhere. Storage columns are plain TEXT, so adding a mode later + * needs no DB migration. + */ +export type ChatMode = 'execute' | 'work' | 'secure' | 'incognito'; + export type TaskStatus = | 'planning' | 'ready' @@ -214,5 +270,15 @@ export interface Message { reply_to_role?: string | null; /** Persisted user-uploaded files (image / doc). `null` row column is parsed to undefined here. */ attachments?: MessageAttachment[] | null; + /** + * How this message was sent. Only meaningful on `user` rows; assistant / + * system rows default to 'execute'. Missing/legacy rows read back as + * 'execute' (DB column default), so old chats stay fully compatible. + */ + mode: ChatMode; + /** Total tokens spent producing this (assistant) message. Dev-mode only; null otherwise. */ + tokens?: number | null; + /** Of `tokens`, how many prompt tokens were served from the provider cache. Dev-mode. */ + cached_tokens?: number | null; created_at: string; } diff --git a/src/types/protocol.ts b/src/types/protocol.ts index 1ba4c9a..08b5635 100644 --- a/src/types/protocol.ts +++ b/src/types/protocol.ts @@ -10,9 +10,11 @@ */ import type { + ChatMode, Message, Project, ProjectFact, + ProjectSkill, QueuedMessage, ScheduledMessage, TaskWithSteps, @@ -39,6 +41,12 @@ export type ClientMsg = content: string; agent?: string; projectId?: number | null; + /** + * How to handle this message — 'execute' (full agent, default), + * 'work' / 'secure' / 'incognito' (iclaw-runtime). Omitted/unknown → + * server treats it as 'execute'. See services/chatModes.ts. + */ + mode?: ChatMode; /** Reply to an existing user/assistant row in this chat (quote ≤240 chars). */ replyTo?: { messageId: number; quote: string; role?: string }; /** @@ -62,6 +70,21 @@ export type ClientMsg = } /** Abort a running turn for this chat. */ | { type: 'abort'; chatId: number } + /** + * Incognito turn — ephemeral, never persisted. `key` is the browser's in-RAM + * chat id (not a DB chatId). The server streams `incognito-*` events back to + * THIS socket only. `workFolders` mirrors the `send` field (folders are forced + * read-only in incognito). + */ + | { + type: 'incognito-send'; + key: string; + content: string; + requestId?: string; + workFolders?: Array<{ path: string; readonly?: boolean } | string>; + } + /** Abort / forget an incognito session. */ + | { type: 'incognito-abort'; key: string } /** Resolve a pending exec approval — `decision` is 'approved' | 'denied'. */ | { type: 'exec-approval'; @@ -85,6 +108,12 @@ export type ServerMsg = | { type: 'hello'; serverStarted: number } | { type: 'pong' } + /* ---- incognito (ephemeral; keyed by the browser's in-RAM chat id) ---- */ + | { type: 'incognito-turn-delta'; key: string; text: string } + | { type: 'incognito-turn-tool'; key: string; name: string } + | { type: 'incognito-turn-ended'; key: string; tokens?: number; cached?: number } + | { type: 'incognito-error'; key: string; message: string } + /* ---- chat lifecycle ---- */ | { type: 'chat-created'; @@ -171,6 +200,31 @@ export type ServerMsg = } | { type: 'project-fact-suggestion-removed'; chatId: number; suggestionId: number } + /* ---- project skills (procedural memory) ---- */ + | { type: 'project-skill-added'; projectId: number; skill: ProjectSkill } + | { type: 'project-skill-updated'; projectId: number; skill: ProjectSkill } + | { type: 'project-skill-deleted'; projectId: number; skillId: number } + /** + * After a turn, proposed skills the user can accept into project memory + * (confirm in chat). Card list carries summary fields only; the full body is + * fetched via REST when the user expands/edits — keeps WS frames small. + */ + | { + type: 'project-skill-suggestions'; + chatId: number; + projectId: number; + projectName: string; + suggestions: { + id: number; + kind: 'new' | 'patch'; + name: string; + description: string; + untrusted: boolean; + targetSkillId: number | null; + }[]; + } + | { type: 'project-skill-suggestion-removed'; chatId: number; suggestionId: number } + /* ---- scheduled messages (Telegram-style send-later) ---- */ | { type: 'scheduled-added'; chatId: number; scheduled: ScheduledMessage } | { type: 'scheduled-updated'; chatId: number; scheduled: ScheduledMessage } diff --git a/test/helpers/db.ts b/test/helpers/db.ts index bfb5ddc..a01daff 100644 --- a/test/helpers/db.ts +++ b/test/helpers/db.ts @@ -32,6 +32,8 @@ export function resetTestDb(): void { DELETE FROM task_context_snapshots; DELETE FROM project_fact_suggestions; DELETE FROM project_facts; + DELETE FROM project_skill_suggestions; + DELETE FROM project_skills; DELETE FROM project_secrets; DELETE FROM scheduled_messages; DELETE FROM queued_messages; diff --git a/test/unit/secureExport.test.ts b/test/unit/secureExport.test.ts new file mode 100644 index 0000000..b3b5820 --- /dev/null +++ b/test/unit/secureExport.test.ts @@ -0,0 +1,90 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { + mkdtempSync, mkdirSync, writeFileSync, rmSync, readFileSync, existsSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { ingestSources } from '../../packages/iclaw-runtime/src/secure-ingest'; +import { + exportWorkspace, applyChanges, +} from '../../packages/iclaw-runtime/src/secure-export'; + +let root: string; +let workspace: string; + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'iclaw-export-test-')); + workspace = join(root, 'workspace'); + mkdirSync(workspace, { recursive: true }); +}); +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +describe('exportWorkspace', () => { + it('copies the sandbox out, excluding workspace internals', () => { + writeFileSync(join(workspace, 'result.txt'), 'output'); + mkdirSync(join(workspace, '.tools'), { recursive: true }); + writeFileSync(join(workspace, '.tools', 'bin'), 'junk'); + writeFileSync(join(workspace, '.iclaw-ingest.json'), '{}'); + + const dest = join(root, 'out'); + const r = exportWorkspace(workspace, dest); + expect(r.ok).toBe(true); + expect(r.path!.startsWith(dest)).toBe(true); + expect(readFileSync(join(r.path!, 'result.txt'), 'utf-8')).toBe('output'); + // Internals are not exported. + expect(existsSync(join(r.path!, '.tools'))).toBe(false); + expect(existsSync(join(r.path!, '.iclaw-ingest.json'))).toBe(false); + }); +}); + +describe('applyChanges', () => { + it('copies new and modified files back to the original, without deleting', async () => { + // Original project. + const src = join(root, 'project'); + mkdirSync(join(src, 'src'), { recursive: true }); + writeFileSync(join(src, 'src', 'a.ts'), 'A\n'); + writeFileSync(join(src, 'keep.txt'), 'KEEP\n'); + + // Ingest it (records the source→target mapping in .iclaw-ingest.json). + await ingestSources(workspace, [{ kind: 'folder', path: src }]); + const sandbox = join(workspace, 'project'); + + // Agent edits a.ts, adds b.ts, and "deletes" keep.txt inside the sandbox. + writeFileSync(join(sandbox, 'src', 'a.ts'), 'A changed\n'); + writeFileSync(join(sandbox, 'new.ts'), 'NEW\n'); + rmSync(join(sandbox, 'keep.txt'), { force: true }); + + const results = applyChanges(workspace); + expect(results).toHaveLength(1); + expect(results[0].ok).toBe(true); + const applied = results[0].applied!.map((c) => `${c.kind}:${c.path}`).sort(); + + // Modified + created are applied back to the original. + expect(readFileSync(join(src, 'src', 'a.ts'), 'utf-8')).toBe('A changed\n'); + expect(readFileSync(join(src, 'new.ts'), 'utf-8')).toBe('NEW\n'); + expect(applied.some((s) => s.includes('a.ts'))).toBe(true); + expect(applied.some((s) => s.includes('new.ts'))).toBe(true); + + // Deletion in the sandbox does NOT delete the user's original file. + expect(existsSync(join(src, 'keep.txt'))).toBe(true); + expect(readFileSync(join(src, 'keep.txt'), 'utf-8')).toBe('KEEP\n'); + }); + + it('applies nothing when the sandbox is unchanged', async () => { + const src = join(root, 'p2'); + mkdirSync(src, { recursive: true }); + writeFileSync(join(src, 'f.txt'), 'same\n'); + await ingestSources(workspace, [{ kind: 'folder', path: src }]); + + const results = applyChanges(workspace); + expect(results[0].ok).toBe(true); + expect(results[0].applied).toEqual([]); + }); + + it('returns empty when nothing was ingested', () => { + expect(applyChanges(workspace)).toEqual([]); + }); +}); diff --git a/test/unit/secureIngest.test.ts b/test/unit/secureIngest.test.ts new file mode 100644 index 0000000..03e39fb --- /dev/null +++ b/test/unit/secureIngest.test.ts @@ -0,0 +1,110 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync, readFileSync, existsSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { ingestSources, describeIngest } from '../../packages/iclaw-runtime/src/secure-ingest'; + +let root: string; +let workspace: string; + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'iclaw-ingest-test-')); + workspace = join(root, 'workspace'); + mkdirSync(workspace, { recursive: true }); +}); +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +function makeProject(name: string): string { + const dir = join(root, name); + mkdirSync(join(dir, 'src'), { recursive: true }); + writeFileSync(join(dir, 'src', 'index.ts'), 'export const x = 1;\n'); + writeFileSync(join(dir, 'README.md'), '# hi\n'); + // noise that should be skipped + mkdirSync(join(dir, 'node_modules', 'pkg'), { recursive: true }); + writeFileSync(join(dir, 'node_modules', 'pkg', 'big.js'), 'junk'); + mkdirSync(join(dir, '.git'), { recursive: true }); + writeFileSync(join(dir, '.git', 'HEAD'), 'ref: x'); + return dir; +} + +describe('ingestSources — folder', () => { + it('copies a folder into the workspace and leaves the original untouched', async () => { + const src = makeProject('my-app'); + const results = await ingestSources(workspace, [{ kind: 'folder', path: src }]); + + expect(results).toHaveLength(1); + expect(results[0]).toMatchObject({ kind: 'folder', ok: true, target: 'my-app' }); + + // Copy landed in the sandbox. + const copied = join(workspace, 'my-app'); + expect(readFileSync(join(copied, 'src', 'index.ts'), 'utf-8')).toContain('export const x'); + expect(existsSync(join(copied, 'README.md'))).toBe(true); + + // Original is byte-for-byte intact (Safe Mode never touches it). + expect(readFileSync(join(src, 'src', 'index.ts'), 'utf-8')).toBe('export const x = 1;\n'); + expect(existsSync(join(src, 'node_modules', 'pkg', 'big.js'))).toBe(true); + }); + + it('skips node_modules and .git when copying', async () => { + const src = makeProject('app2'); + await ingestSources(workspace, [{ kind: 'folder', path: src }]); + const copied = join(workspace, 'app2'); + expect(existsSync(join(copied, 'node_modules'))).toBe(false); + expect(existsSync(join(copied, '.git'))).toBe(false); + // file count reflects only the real source files (index.ts + README.md). + const log = JSON.parse(readFileSync(join(workspace, '.iclaw-ingest.json'), 'utf-8')); + expect(log.results[0].files).toBe(2); + }); + + it('gives colliding basenames distinct targets', async () => { + const a = join(root, 'a', 'project'); + const b = join(root, 'b', 'project'); + mkdirSync(a, { recursive: true }); + mkdirSync(b, { recursive: true }); + writeFileSync(join(a, 'f.txt'), 'A'); + writeFileSync(join(b, 'f.txt'), 'B'); + const results = await ingestSources(workspace, [ + { kind: 'folder', path: a }, + { kind: 'folder', path: b }, + ]); + expect(results[0].target).toBe('project'); + expect(results[1].target).toBe('project-1'); + expect(readFileSync(join(workspace, 'project', 'f.txt'), 'utf-8')).toBe('A'); + expect(readFileSync(join(workspace, 'project-1', 'f.txt'), 'utf-8')).toBe('B'); + }); + + it('refuses a secret-bearing folder root (e.g. .ssh) without throwing', async () => { + const secret = join(root, '.ssh'); + mkdirSync(secret, { recursive: true }); + writeFileSync(join(secret, 'id_rsa'), 'PRIVATE'); + const results = await ingestSources(workspace, [{ kind: 'folder', path: secret }]); + expect(results[0].ok).toBe(false); + expect(results[0].error).toBeTruthy(); + // Nothing leaked into the sandbox. + expect(existsSync(join(workspace, '.ssh'))).toBe(false); + }); + + it('reports a missing folder as a failed (not thrown) result', async () => { + const results = await ingestSources(workspace, [ + { kind: 'folder', path: join(root, 'does-not-exist') }, + ]); + expect(results[0].ok).toBe(false); + }); +}); + +describe('describeIngest', () => { + it('summarizes copies and reassures about originals', () => { + const text = describeIngest([ + { kind: 'folder', source: '/home/me/app', ok: true, target: 'app', files: 12 }, + ]); + expect(text).toContain('original files are unchanged'); + expect(text).toContain('/workspace/app'); + }); + + it('returns empty for no sources', () => { + expect(describeIngest([])).toBe(''); + }); +}); diff --git a/test/unit/store.projectSkills.test.ts b/test/unit/store.projectSkills.test.ts new file mode 100644 index 0000000..0d4bf13 --- /dev/null +++ b/test/unit/store.projectSkills.test.ts @@ -0,0 +1,174 @@ +import { afterEach, describe, expect, it } from 'vitest'; +import { resetTestDb } from '../helpers/db'; +import { + projects, + projectSkills, + projectSkillSuggestions, + chats, + enrichSkillWithSourceChatTitle, +} from '../../src/services/store'; +import { + extractJsonObject, + parseReviewedSkills, +} from '../../src/services/projectSkills'; + +afterEach(() => resetTestDb()); + +describe('store.projectSkills', () => { + it('create() persists a project skill and serializes tags', () => { + const p = projects.create('P'); + const s = projectSkills.create({ + projectId: p.id, + name: 'deploy-flow', + description: 'How to deploy the service', + body: '# Deploy\nSteps...', + tags: ['ops', 'deploy'], + }); + expect(s.name).toBe('deploy-flow'); + expect(s.project_id).toBe(p.id); + expect(s.version).toBe(1); + expect(JSON.parse(s.tags!)).toEqual(['ops', 'deploy']); + }); + + it('create() supports global skills (project_id null) and listForProject merges them', () => { + const p = projects.create('P'); + projectSkills.create({ projectId: p.id, name: 'local-skill', description: 'd', body: 'b' }); + projectSkills.create({ projectId: null, name: 'global-skill', description: 'd', body: 'b' }); + const forProject = projectSkills.listForProject(p.id); + const names = forProject.map((s) => s.name).sort(); + expect(names).toEqual(['global-skill', 'local-skill']); + expect(projectSkills.listGlobal().map((s) => s.name)).toEqual(['global-skill']); + }); + + it('getByName() distinguishes project scope from global scope', () => { + const p = projects.create('P'); + projectSkills.create({ projectId: p.id, name: 'dup', description: 'proj', body: 'b' }); + projectSkills.create({ projectId: null, name: 'dup', description: 'glob', body: 'b' }); + expect(projectSkills.getByName(p.id, 'dup')?.description).toBe('proj'); + expect(projectSkills.getByName(null, 'dup')?.description).toBe('glob'); + }); + + it('update() bumps version + updated_at and can rewrite tags', () => { + const p = projects.create('P'); + const s = projectSkills.create({ projectId: p.id, name: 'n', description: 'd', body: 'b' }); + projectSkills.update(s.id, { description: 'd2', body: 'b2', tags: ['x'] }); + const after = projectSkills.get(s.id)!; + expect(after.version).toBe(2); + expect(after.description).toBe('d2'); + expect(after.body).toBe('b2'); + expect(JSON.parse(after.tags!)).toEqual(['x']); + }); + + it('listIndex() returns id/name/description only', () => { + const p = projects.create('P'); + projectSkills.create({ projectId: p.id, name: 'n', description: 'summary', body: 'long body' }); + const idx = projectSkills.listIndex(p.id); + expect(idx).toHaveLength(1); + expect(idx[0]).toMatchObject({ name: 'n', description: 'summary' }); + expect(idx[0]).not.toHaveProperty('body'); + }); + + it('remove() deletes the skill; project removal cascades skills', () => { + const p = projects.create('P'); + const s = projectSkills.create({ projectId: p.id, name: 'n', description: 'd', body: 'b' }); + projectSkills.remove(s.id); + expect(projectSkills.get(s.id)).toBeUndefined(); + + const p2 = projects.create('P2'); + projectSkills.create({ projectId: p2.id, name: 'n', description: 'd', body: 'b' }); + projects.remove(p2.id); + expect(projectSkills.listByProject(p2.id)).toHaveLength(0); + }); + + it('enrichSkillWithSourceChatTitle pulls the source chat title', () => { + const p = projects.create('P'); + const c = chats.create('openclaw/default', p.id); + chats.rename(c.id, 'My Chat'); + const s = projectSkills.create({ + projectId: p.id, + name: 'n', + description: 'd', + body: 'b', + sourceChatId: c.id, + }); + expect(enrichSkillWithSourceChatTitle(s).source_chat_title).toBe('My Chat'); + }); +}); + +describe('store.projectSkillSuggestions', () => { + it('insert() persists a suggestion with kind + untrusted flag', () => { + const p = projects.create('P'); + const c = chats.create('openclaw/default', p.id); + const sug = projectSkillSuggestions.insert({ + projectId: p.id, + chatId: c.id, + kind: 'new', + name: 'n', + description: 'd', + body: 'b', + untrusted: true, + assistantMessageId: null, + }); + expect(sug.kind).toBe('new'); + expect(sug.untrusted).toBe(1); + expect(projectSkillSuggestions.listByChat(c.id)).toHaveLength(1); + expect(projectSkillSuggestions.listByProject(p.id)).toHaveLength(1); + projectSkillSuggestions.remove(sug.id); + expect(projectSkillSuggestions.get(sug.id)).toBeUndefined(); + }); +}); + +describe('projectSkills reviewer parsing', () => { + it('extractJsonObject pulls a balanced object from fenced output', () => { + const raw = 'Sure!\n```json\n{"skills":[{"name":"a"}]}\n```\nDone'; + expect(extractJsonObject(raw)).toBe('{"skills":[{"name":"a"}]}'); + }); + + it('extractJsonObject returns null for NONE / no object', () => { + expect(extractJsonObject('NONE')).toBeNull(); + expect(extractJsonObject('no json here')).toBeNull(); + }); + + it('parseReviewedSkills accepts well-formed skills and kebab-cases names', () => { + const raw = JSON.stringify({ + skills: [ + { + action: 'new', + name: 'Deploy Flow', + description: 'How to deploy', + tags: ['ops'], + body: '# Deploy\nstep', + }, + ], + }); + const out = parseReviewedSkills(raw); + expect(out).toHaveLength(1); + expect(out[0].name).toBe('deploy-flow'); + expect(out[0].action).toBe('new'); + expect(out[0].tags).toEqual(['ops']); + }); + + it('parseReviewedSkills drops entries missing name/description/body', () => { + const raw = JSON.stringify({ + skills: [ + { action: 'new', name: 'ok', description: '', body: 'b' }, + { action: 'new', name: 'good', description: 'd', body: 'b' }, + ], + }); + const out = parseReviewedSkills(raw); + expect(out.map((s) => s.name)).toEqual(['good']); + }); + + it('parseReviewedSkills returns [] for empty skills array', () => { + expect(parseReviewedSkills('{"skills":[]}')).toEqual([]); + }); + + it('parseReviewedSkills keeps patch action + target', () => { + const raw = JSON.stringify({ + skills: [{ action: 'patch', target: 'Existing Skill', name: 'existing-skill', description: 'd', body: 'b' }], + }); + const out = parseReviewedSkills(raw); + expect(out[0].action).toBe('patch'); + expect(out[0].target).toBe('existing-skill'); + }); +}); diff --git a/test/unit/subtaskLlm.test.ts b/test/unit/subtaskLlm.test.ts new file mode 100644 index 0000000..593dca9 --- /dev/null +++ b/test/unit/subtaskLlm.test.ts @@ -0,0 +1,84 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +// Mock the three collaborators so we can assert routing without real network. +vi.mock('../../src/services/openRouter', () => ({ + openRouterEnabled: vi.fn(), + complete: vi.fn(), + isOpenRouterFailure: (e: unknown) => + /^openrouter:/i.test(e instanceof Error ? e.message : String(e)), +})); +vi.mock('../../src/services/config', () => ({ + loadOpenRouterConfig: () => ({ summaryModel: 'cheap/model' }), +})); +vi.mock('../../src/services/openclawWs', () => { + const runTurn = vi.fn(async (opts: { onEvent: (ev: { type: string; text: string }) => void }) => { + opts.onEvent({ type: 'text-final', text: 'openclaw-result' }); + }); + return { + openclawWs: { + createSession: vi.fn(async () => ({ key: 'k1' })), + runTurn, + deleteSession: vi.fn(async () => {}), + }, + }; +}); + +import { runSubtaskTurn } from '../../src/services/subtaskLlm'; +import { openRouterEnabled, complete } from '../../src/services/openRouter'; +import { openclawWs } from '../../src/services/openclawWs'; + +const mockEnabled = openRouterEnabled as unknown as ReturnType<typeof vi.fn>; +const mockComplete = complete as unknown as ReturnType<typeof vi.fn>; + +afterEach(() => vi.clearAllMocks()); + +describe('runSubtaskTurn routing', () => { + it('uses OpenRouter (cheap model) when configured', async () => { + mockEnabled.mockReturnValue(true); + mockComplete.mockResolvedValue('openrouter-result'); + + const out = await runSubtaskTurn('hi', { maxTokens: 100 }); + + expect(out).toBe('openrouter-result'); + expect(mockComplete).toHaveBeenCalledTimes(1); + expect(mockComplete.mock.calls[0][0]).toMatchObject({ + model: 'cheap/model', + maxTokens: 100, + }); + expect(openclawWs.createSession).not.toHaveBeenCalled(); + }); + + it('passes an optional system prompt through to OpenRouter', async () => { + mockEnabled.mockReturnValue(true); + mockComplete.mockResolvedValue('ok'); + await runSubtaskTurn('user text', { system: 'be terse' }); + const messages = mockComplete.mock.calls[0][0].messages; + expect(messages[0]).toEqual({ role: 'system', content: 'be terse' }); + expect(messages[1]).toEqual({ role: 'user', content: 'user text' }); + }); + + it('falls back to OpenClaw when OpenRouter is not configured', async () => { + mockEnabled.mockReturnValue(false); + const out = await runSubtaskTurn('hi'); + expect(out).toBe('openclaw-result'); + expect(mockComplete).not.toHaveBeenCalled(); + expect(openclawWs.createSession).toHaveBeenCalledTimes(1); + expect(openclawWs.deleteSession).toHaveBeenCalledTimes(1); + }); + + it('falls back to OpenClaw when the OpenRouter call throws', async () => { + mockEnabled.mockReturnValue(true); + mockComplete.mockRejectedValue(new Error('openrouter: HTTP 500')); + const out = await runSubtaskTurn('hi'); + expect(out).toBe('openclaw-result'); + expect(openclawWs.createSession).toHaveBeenCalledTimes(1); + }); + + it('returns OpenRouter output verbatim even when empty (no double-spend fallback)', async () => { + mockEnabled.mockReturnValue(true); + mockComplete.mockResolvedValue(''); + const out = await runSubtaskTurn('hi'); + expect(out).toBe(''); + expect(openclawWs.createSession).not.toHaveBeenCalled(); + }); +}); diff --git a/test/unit/turnReply.test.ts b/test/unit/turnReply.test.ts index 933cdee..cb71cc8 100644 --- a/test/unit/turnReply.test.ts +++ b/test/unit/turnReply.test.ts @@ -10,6 +10,7 @@ import { describe, expect, it } from 'vitest'; import { extractAssistantText, extractSourceReplyFromMessageToolResult, + extractTurnUsage, resolveFromHistorySlice, sliceFromLastUser, type HistoryMessageLike, @@ -267,3 +268,45 @@ describe('resolveFromHistorySlice — chat #23 regression', () => { expect(resolveFromHistorySlice(slice)).toBe(samari); }); }); + +describe('extractTurnUsage', () => { + it('returns nulls when no assistant row carries usage', () => { + const slice: HistoryMessageLike[] = [ + { role: 'assistant', content: 'hi' }, + { role: 'toolResult', content: [] }, + ]; + expect(extractTurnUsage(slice)).toEqual({ tokens: null, cached: null }); + }); + + it('prefers an explicit total and reads cache-read tokens', () => { + const slice: HistoryMessageLike[] = [ + { role: 'user', content: 'q' }, + { role: 'assistant', content: 'a', usage: { total_tokens: 1234, cache_read_input_tokens: 200 } }, + ]; + expect(extractTurnUsage(slice)).toEqual({ tokens: 1234, cached: 200 }); + }); + + it('falls back to input + output when no total is present (camelCase)', () => { + const slice: HistoryMessageLike[] = [ + { role: 'assistant', content: 'a', usage: { inputTokens: 100, outputTokens: 23, cacheRead: 10 } }, + ]; + expect(extractTurnUsage(slice)).toEqual({ tokens: 123, cached: 10 }); + }); + + it('sums usage across the multiple assistant segments of a tool-loop turn', () => { + const slice: HistoryMessageLike[] = [ + { role: 'assistant', content: 'preamble', usage: { total_tokens: 100, cacheRead: 5 } }, + { role: 'toolResult', content: [] }, + { role: 'assistant', content: 'final', usage: { input_tokens: 40, output_tokens: 10 } }, + ]; + expect(extractTurnUsage(slice)).toEqual({ tokens: 150, cached: 5 }); + }); + + it('ignores non-object / malformed usage', () => { + const slice: HistoryMessageLike[] = [ + { role: 'assistant', content: 'a', usage: 'nope' }, + { role: 'assistant', content: 'b', usage: { foo: 'bar' } }, + ]; + expect(extractTurnUsage(slice)).toEqual({ tokens: null, cached: null }); + }); +}); diff --git a/test/unit/workContainerPaths.test.ts b/test/unit/workContainerPaths.test.ts new file mode 100644 index 0000000..77f359b --- /dev/null +++ b/test/unit/workContainerPaths.test.ts @@ -0,0 +1,119 @@ +import { describe, it, expect } from 'vitest'; + +import { + toWorkMounts, + hostToContainer, + containerToHost, + translateCommandPaths, +} from '../../packages/iclaw-runtime/src/work-container'; + +const WIN = 'win32' as NodeJS.Platform; +const POSIX = 'linux' as NodeJS.Platform; + +describe('toWorkMounts', () => { + it('assigns normalized /work/<n> paths and basename labels', () => { + const mounts = toWorkMounts([ + { path: '/Users/me/project', readonly: false }, + { path: '/Users/me/Downloads', readonly: true }, + ]); + expect(mounts[0]).toMatchObject({ + path: '/Users/me/project', + containerPath: '/work/0', + readonly: false, + label: 'project', + }); + expect(mounts[1]).toMatchObject({ containerPath: '/work/1', readonly: true, label: 'Downloads' }); + }); + + it('derives a label from a Windows path', () => { + const [m] = toWorkMounts([{ path: 'C:\\Users\\me\\app', readonly: false }]); + expect(m.label).toBe('app'); + expect(m.containerPath).toBe('/work/0'); + }); +}); + +describe('hostToContainer', () => { + const mounts = toWorkMounts([ + { path: '/Users/me/project', readonly: false }, + { path: '/Users/me/Downloads', readonly: true }, + ]); + + it('maps the folder root and nested paths', () => { + expect(hostToContainer('/Users/me/project', mounts, POSIX)).toBe('/work/0'); + expect(hostToContainer('/Users/me/project/src/a.ts', mounts, POSIX)).toBe('/work/0/src/a.ts'); + expect(hostToContainer('/Users/me/Downloads/x', mounts, POSIX)).toBe('/work/1/x'); + }); + + it('returns null for a path outside every mount', () => { + expect(hostToContainer('/etc/passwd', mounts, POSIX)).toBeNull(); + expect(hostToContainer('/Users/me/projectX/y', mounts, POSIX)).toBeNull(); + }); + + it('prefers the longest (nested) mount root', () => { + const nested = toWorkMounts([ + { path: '/a', readonly: false }, + { path: '/a/b', readonly: false }, + ]); + expect(hostToContainer('/a/b/c', nested, POSIX)).toBe('/work/1/c'); + }); + + it('maps Windows drive paths case-insensitively', () => { + const win = toWorkMounts([{ path: 'C:\\Users\\me\\app', readonly: false }]); + expect(hostToContainer('C:\\Users\\me\\app\\src\\x.ts', win, WIN)).toBe('/work/0/src/x.ts'); + expect(hostToContainer('c:/users/me/app/y', win, WIN)).toBe('/work/0/y'); + }); +}); + +describe('containerToHost', () => { + it('reverses the mapping with the host separator', () => { + const posix = toWorkMounts([{ path: '/Users/me/project', readonly: false }]); + expect(containerToHost('/work/0/src/a.ts', posix)).toBe('/Users/me/project/src/a.ts'); + expect(containerToHost('/work/0', posix)).toBe('/Users/me/project'); + + const win = toWorkMounts([{ path: 'C:\\Users\\me\\app', readonly: false }]); + expect(containerToHost('/work/0/src/x.ts', win)).toBe('C:\\Users\\me\\app\\src\\x.ts'); + }); +}); + +describe('translateCommandPaths', () => { + const mounts = toWorkMounts([{ path: '/Users/me/project', readonly: false }]); + + it('rewrites a host root + tail to its container path', () => { + expect(translateCommandPaths('cat /Users/me/project/a.txt', mounts, POSIX)).toBe( + 'cat /work/0/a.txt', + ); + expect(translateCommandPaths('cd /Users/me/project && ls', mounts, POSIX)).toBe( + 'cd /work/0 && ls', + ); + }); + + it('rewrites Windows roots whole, in both slash forms', () => { + const win = toWorkMounts([{ path: 'C:\\Users\\me\\app', readonly: false }]); + expect(translateCommandPaths('cat C:\\Users\\me\\app\\a.txt', win, WIN)).toBe('cat /work/0/a.txt'); + expect(translateCommandPaths('cat C:/Users/me/app/a.txt', win, WIN)).toBe('cat /work/0/a.txt'); + }); + + it('stops at shell metacharacters and leaves unrelated paths alone', () => { + expect(translateCommandPaths('cat /Users/me/project/a|wc -l', mounts, POSIX)).toBe( + 'cat /work/0/a|wc -l', + ); + expect(translateCommandPaths('ls /etc/hosts', mounts, POSIX)).toBe('ls /etc/hosts'); + }); + + it('does not match a root inside a longer sibling path', () => { + const ms = toWorkMounts([{ path: '/a', readonly: false }]); + // /abc is a different folder — must stay untouched. + expect(translateCommandPaths('ls /abc/x', ms, POSIX)).toBe('ls /abc/x'); + // The exact root (and nested) still map. + expect(translateCommandPaths('ls /a/x', ms, POSIX)).toBe('ls /work/0/x'); + expect(translateCommandPaths('cd /a && ls', ms, POSIX)).toBe('cd /work/0 && ls'); + }); + + it('prefers the longest mount root', () => { + const nested = toWorkMounts([ + { path: '/a', readonly: false }, + { path: '/a/b', readonly: false }, + ]); + expect(translateCommandPaths('cd /a/b/c', nested, POSIX)).toBe('cd /work/1/c'); + }); +}); diff --git a/views/chat.ejs b/views/chat.ejs index 4892503..73343a1 100644 --- a/views/chat.ejs +++ b/views/chat.ejs @@ -14,6 +14,10 @@ aria-label="Chat title" /> </form> + <% if (typeof devMode !== 'undefined' && devMode) { const _tokTotal = (chatMessages || []).reduce((a, m) => a + (Number(m.tokens) || 0), 0); %> + <span class="chat-token-total" id="chat-token-total" data-total="<%= _tokTotal %>" + title="Total tokens spent in this chat (dev mode)" <%= _tokTotal > 0 ? '' : 'hidden' %>><%= _tokTotal.toLocaleString() %> tok total</span> + <% } %> <div class="chat-header-tools"> <form class="agent-form header-field" method="post" action="/chats/<%= activeChat.id %>/agent"> @@ -169,7 +173,7 @@ > <div class="messages-thread"> <% if (chatMessages.length === 0) { %> - <p class="muted empty-state">No messages yet. Say something.</p> + <p class="muted empty-state">iClaw - the safer way to work with powerful AI</p> <% } else { %> <% chatMessages.forEach(function (m) { %> <div class="msg <%= m.role %>" data-msg-id="<%= m.id %>"> @@ -220,6 +224,9 @@ <% }); %> </div> <% } %> + <% if (typeof devMode !== 'undefined' && devMode && m.tokens) { %> + <span class="msg-tokens" title="Tokens spent on this reply"><%= Number(m.tokens).toLocaleString() %> tok<% if (m.cached_tokens) { %> · <%= Number(m.cached_tokens).toLocaleString() %> cached<% } %></span> + <% } %> </div> <% }); %> <% } %> diff --git a/views/index.ejs b/views/index.ejs index bdbd1d9..535fd46 100644 --- a/views/index.ejs +++ b/views/index.ejs @@ -44,23 +44,17 @@ <span class="header-tool-label">Stop</span> </button> - <span - class="gateway <%= gatewayStatus %>" - id="gateway-badge" - data-base-url="<%= openclawBaseUrl %>" - title="<%= openclawBaseUrl %>" - > - OpenClaw: <%= gatewayStatus === 'ok' ? 'connected' : (gatewayStatus === 'degraded' ? 'unreachable' : 'off') %> - </span> </div> </header> + <% const _firstChat = typeof isFirstChat !== 'undefined' && isFirstChat; %> <div class="draft-body is-picking" id="draft-body" + <%- _firstChat ? 'data-auto-none="1"' : '' %> > <div class="draft-pick-stage" id="draft-pick-stage"> - <% if (agentsError) { %> + <% if (agentsError && !openRouterReady) { %> <%- include('partials/gatewayError', { agentsError, gatewayUp, openclawBaseUrl }) %> <% } else { %> <section @@ -106,15 +100,36 @@ <section class="messages" id="messages" data-chat-id="" data-draft="1"> <div class="messages-thread"> + <% if (_firstChat) { %> + <div class="welcome-card empty-state" id="welcome-card"> + <div class="welcome-avatar" aria-hidden="true">iC</div> + <div class="welcome-bubble"> + <p class="welcome-greeting">Hi - I’m iClaw</p> + <p class="welcome-lead">From now on, you don’t have to choose between convenience, power and safety</p> + <ul class="welcome-list"> + <li>Unlike other agents, I go online inside an isolated space - so a dangerous site can’t do any harm</li> + <li>Other agents want access to your whole computer. I only work where you let me</li> + <li>And our developers are 70% coffee - so we built a really efficient system that saves your tokens and your money</li> + </ul> + <div class="welcome-chips" role="group" aria-label="Suggestions to get started"> + <button type="button" class="welcome-chip" data-prompt="Open an unknown website safely">Open an unknown website safely</button> + <button type="button" class="welcome-chip" data-prompt="Check a ZIP file safely, isolated from my computer">Check a ZIP file safely, isolated from my computer</button> + <button type="button" class="welcome-chip" data-prompt="Check this GitHub repo in a sandbox">Check this GitHub repo in a sandbox</button> + <button type="button" class="welcome-chip" data-prompt="Summarize this 2-hour YouTube video for $0.0005">Summarize this 2-hour YouTube video for $0.0005</button> + </div> + </div> + </div> + <% } else { %> <p class="muted empty-state" id="draft-empty-hint" hidden>Write your first message.</p> + <% } %> </div> </section> <section class="queue" id="queue" aria-live="polite"></section> - <% if (!agentsError) { %> + <% if (!agentsError || openRouterReady) { %> <div id="draft-composer-wrap" class="draft-composer-wrap" hidden> - <%- include('partials/composer') %> + <%- include('partials/composer', { gatewayOk: gatewayStatus === 'ok' }) %> </div> <% } %> </div> diff --git a/views/partials/composer.ejs b/views/partials/composer.ejs index dea5d7c..e24517f 100644 --- a/views/partials/composer.ejs +++ b/views/partials/composer.ejs @@ -37,7 +37,33 @@ }) %> <% }); %></div> -<form id="send-form" class="composer"> +<% + // Mode selector data. Server passes the enabled modes (config-driven) and + // the default. Fall back gracefully if a render path forgot the locals so + // the composer never breaks — the selector just hides. + const _modes = + typeof chatModes !== 'undefined' && Array.isArray(chatModes) ? chatModes : []; + const _defaultMode = + typeof defaultChatMode !== 'undefined' && defaultChatMode ? defaultChatMode : 'execute'; + const _defaultModeDef = + _modes.find(function (m) { return m.id === _defaultMode; }) || _modes[0] || null; + // The mode this chat was last used in (server-derived from its last message). + // Empty for new chats. Used to pre-select the picker so switching chats keeps + // each chat's own mode instead of resetting to the default. + const _chatCurrentMode = + typeof chatCurrentMode !== 'undefined' && chatCurrentMode ? chatCurrentMode : ''; + const _initialMode = + _modes.find(function (m) { return m.id === _chatCurrentMode; }) ? _chatCurrentMode : _defaultMode; + const _initialModeDef = + _modes.find(function (m) { return m.id === _initialMode; }) || _defaultModeDef; +%> +<% + // Whether Full Power (Execute → OpenClaw) is usable right now. Stamped here so + // the JS gating works on every composer surface (the gateway badge only exists + // on the new-chat header). Defaults to available when a render path omits it. + const _gatewayOk = typeof gatewayOk !== 'undefined' ? !!gatewayOk : true; +%> +<form id="send-form" class="composer" data-gateway-ok="<%= _gatewayOk ? '1' : '0' %>"> <div id="composer-reply-bar" class="composer-reply-bar" hidden> <div class="composer-reply-bar-inner"> <span class="composer-reply-accent" aria-hidden="true"></span> @@ -63,66 +89,144 @@ </svg> <span>Drop file here</span> </div> - <input - type="file" - id="composer-file-input" - class="composer-file-input" - multiple - hidden - /> - <div - id="composer-attach-menu" - class="menu composer-attach-menu" - role="menu" - aria-label="Attachments" - hidden - > - <button type="button" class="menu-item composer-attach-menu-item" data-attach-pick="file" role="menuitem"> - <span class="menu-item__title">File</span> - </button> - <button type="button" class="menu-item composer-attach-menu-item" data-attach-pick="secret" role="menuitem"> - <span class="menu-item__title">Secret</span> - </button> + <input type="file" id="composer-file-input" class="composer-file-input" multiple hidden /> + <!-- menus stay here so JS can position them relative to the field --> + <div id="composer-attach-menu" class="menu composer-attach-menu" role="menu" aria-label="Attachments" hidden> + <button type="button" class="menu-item composer-attach-menu-item" data-attach-pick="file" role="menuitem"><span class="menu-item__title">File</span></button> + <button type="button" class="menu-item composer-attach-menu-item" data-attach-pick="secret" role="menuitem"><span class="menu-item__title">Secret</span></button> </div> - <div - id="composer-secret-pick-menu" - class="menu composer-secret-pick-menu" - role="menu" - aria-label="Choose secret" - hidden - ></div> - <button - type="button" - class="composer-attach" - id="composer-attach-btn" - aria-label="Add attachment" - title="File or secret (also drag & drop and paste)" - aria-haspopup="menu" - aria-expanded="false" - aria-controls="composer-attach-menu" - > - <svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> - <path d="M21.44 11.05l-9.19 9.19a6 6 0 01-8.49-8.49l8.57-8.57A4 4 0 1117.95 8.83l-8.59 8.57a2 2 0 01-2.83-2.83l8.49-8.49"/> - </svg> - </button> + <div id="composer-secret-pick-menu" class="menu composer-secret-pick-menu" role="menu" aria-label="Choose secret" hidden></div> + + <!-- Textarea — full width, top --> <div class="composer-input-stack"> <textarea id="composer-input" name="content" - placeholder="Message…" + placeholder="Ask anything" rows="1" ></textarea> + <!-- Replaces the textarea when Full Power is selected but OpenClaw is off. + The toolbar (incl. the mode switcher) stays visible so the user can + switch mode right below this message. --> + <div class="composer-exec-msg" id="composer-exec-msg" aria-hidden="true"> + OpenClaw is off. Turn it on, or switch mode below. + </div> + <!-- Replaces the textarea when a Docker-required mode (Safe work) is + selected but Docker isn't running. The Install/Start button text and + action are set by syncDockerAvailability in iclaw.js. --> + <div class="composer-docker-msg" id="composer-docker-msg" aria-hidden="true"> + <span id="composer-docker-msg-text">Safe work needs Docker.</span> + <button type="button" class="composer-docker-btn" id="composer-docker-action">Install Docker</button> + <span class="composer-docker-size" id="composer-docker-size"></span> + <span class="composer-docker-or">or switch mode below.</span> + </div> + </div> + + <!-- Toolbar — bottom row --> + <div class="composer-toolbar"> + <div class="composer-toolbar__left"> + <!-- Attach --> + <button + type="button" + class="composer-toolbar-btn composer-attach" + id="composer-attach-btn" + aria-label="Add attachment" + title="File or secret (also drag & drop and paste)" + aria-haspopup="menu" + aria-expanded="false" + aria-controls="composer-attach-menu" + > + <svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> + <path d="M21.44 11.05l-9.19 9.19a6 6 0 01-8.49-8.49l8.57-8.57A4 4 0 1117.95 8.83l-8.59 8.57a2 2 0 01-2.83-2.83l8.49-8.49"/> + </svg> + </button> + + <!-- Mode selector --> + <% if (_modes.length > 1 && _defaultModeDef) { %> + <div class="composer-modes composer-toolbar-modes" id="composer-modes" data-default-mode="<%= _defaultMode %>" data-chat-mode="<%= _chatCurrentMode %>"> + <button + type="button" + class="composer-toolbar-btn composer-mode-btn" + id="composer-mode-btn" + data-mode="<%= _initialMode %>" + aria-haspopup="menu" + aria-expanded="false" + aria-controls="composer-mode-menu" + title="<%= _initialModeDef.description %>" + > + <span class="composer-mode-btn__label" id="composer-mode-label"><%= _initialModeDef.label %></span> + <svg xmlns="http://www.w3.org/2000/svg" width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> + <polyline points="6 9 12 15 18 9"/> + </svg> + </button> + <div id="composer-mode-menu" class="menu composer-mode-menu" role="menu" aria-label="Message mode" hidden> + <% _modes.forEach(function (m) { %> + <% if (m.ephemeral) { %><div class="composer-mode-menu-sep" role="separator" aria-hidden="true"></div><% } %> + <button type="button" class="menu-item composer-mode-menu-item" role="menuitemradio" data-mode="<%= m.id %>" data-requires-docker="<%= m.requiresDocker ? '1' : '0' %>" aria-checked="<%= m.id === _initialMode ? 'true' : 'false' %>"> + <span class="menu-item__title"><%= m.label %></span> + <span class="composer-mode-menu-item__desc"><%= m.description %></span> + </button> + <% }); %> + </div> + </div> + <% } %> + + <!-- Secure Mode network toggle --> + <button + type="button" + class="composer-toolbar-btn composer-network-toggle" + id="composer-network-toggle-btn" + aria-label="Toggle network access" + title="Network is off - click to enable" + data-network="off" + hidden + > + <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> + <circle cx="12" cy="12" r="10"/> + <line x1="2" y1="12" x2="22" y2="12"/> + <path d="M12 2a15.3 15.3 0 0 1 4 10 15.3 15.3 0 0 1-4 10 15.3 15.3 0 0 1-4-10 15.3 15.3 0 0 1 4-10z"/> + </svg> + </button> + + <!-- Work Mode folders --> + <button + type="button" + class="composer-toolbar-btn composer-work-folders" + id="composer-work-folders-btn" + aria-label="Work Mode folders" + title="Configure allowed folders for Work Mode" + hidden + > + <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> + <path d="M22 19a2 2 0 0 1-2 2H4a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2h5l2 3h9a2 2 0 0 1 2 2z"/> + </svg> + <span class="composer-work-folders__count" id="composer-work-folders-count"></span> + </button> + </div> + + <div class="composer-toolbar__right"> + <!-- Mic --> + <% if (typeof sttEnabled !== 'undefined' && sttEnabled) { %> + <button type="button" class="composer-mic" id="composer-mic-btn" data-state="idle" aria-label="Record voice" aria-pressed="false" title="Record voice"> + <svg class="composer-mic__idle" xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> + <path d="M12 2a3 3 0 0 0-3 3v7a3 3 0 0 0 6 0V5a3 3 0 0 0-3-3z"/> + <path d="M19 10v2a7 7 0 0 1-14 0v-2"/> + <line x1="12" y1="19" x2="12" y2="22"/> + </svg> + <svg class="composer-mic__busy" xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" aria-hidden="true"> + <path d="M21 12a9 9 0 1 1-6.219-8.56"/> + </svg> + </button> + <% } %> + + <!-- Send --> + <button type="submit" class="composer-send" id="composer-send-btn" aria-label="Send (hold to schedule)"> + <svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> + <path d="M12 19V5M5 12l7-7 7 7"/> + </svg> + </button> + </div> </div> - <button - type="submit" - class="composer-send" - id="composer-send-btn" - aria-label="Send (hold to schedule)" - > - <svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> - <path d="M12 19V5M5 12l7-7 7 7"/> - </svg> - </button> <% if (typeof sendHintShow !== 'undefined' && sendHintShow) { %> <!-- Discovery pill — surfaces the long-press menu (scheduled msg / create @@ -210,6 +314,24 @@ </div> </div> </div> + + <!-- Secure Mode workspace indicator — shares the bottom row with the secret UI --> + <div class="secure-workspace-bar" id="secure-workspace-bar" hidden> + <span class="secure-workspace-bar__label">Secure workspace</span> + <span class="secure-workspace-bar__size" id="secure-workspace-size" hidden></span> + <span class="secure-workspace-bar__sep" id="secure-workspace-sep">·</span> + <span class="secure-workspace-bar__ttl" id="secure-workspace-ttl"></span> + <button type="button" class="secure-workspace-bar__change" id="secure-workspace-change">change</button> + <button type="button" class="secure-workspace-bar__action" id="secure-workspace-export" title="Copy the sandbox out to a folder on your computer">Export</button> + <button type="button" class="secure-workspace-bar__action" id="secure-workspace-apply" title="Copy the sandbox's new/changed files back to your original folders">Apply changes</button> + <button type="button" class="secure-workspace-bar__destroy" id="secure-workspace-destroy" title="Delete the sandbox copy and everything in it">Destroy</button> + <!-- Anchored to the bar so it pops up in place instead of shifting layout --> + <div class="secure-ttl-menu menu" id="secure-ttl-menu" role="menu" hidden> + <button type="button" class="menu-item" data-ttl="1" role="menuitem">1 day</button> + <button type="button" class="menu-item" data-ttl="7" role="menuitem">7 days</button> + <button type="button" class="menu-item" data-ttl="30" role="menuitem">30 days</button> + </div> + </div> <div class="composer-secret-modal" id="composer-secret-modal" hidden> <div class="composer-secret-modal__backdrop" id="composer-secret-modal-backdrop"></div> <div class="composer-secret-modal__panel" role="dialog" aria-modal="true" aria-labelledby="composer-secret-modal-title"> @@ -289,3 +411,23 @@ </div> </div> </div> + +<!-- Work Mode folders modal --> +<div class="work-folders-modal" id="work-folders-modal" hidden> + <div class="work-folders-modal__backdrop" id="work-folders-backdrop"></div> + <div class="work-folders-modal__panel" role="dialog" aria-modal="true" aria-labelledby="work-folders-title"> + <header class="work-folders-modal__head"> + <div class="work-folders-modal__head-row"> + <div> + <h3 class="work-folders-modal__title" id="work-folders-title">Work Mode folders</h3> + <p class="work-folders-modal__hint muted">AI can only access these folders. New folders are read-only — click the access pill to allow writing.</p> + </div> + <button type="button" class="btn btn--primary btn--sm" id="work-folders-browse-btn"> + <svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.25" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M22 19a2 2 0 0 1-2 2H4a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2h5l2 3h9a2 2 0 0 1 2 2z"/></svg> + Browse + </button> + </div> + </header> + <ul class="work-folders-list" id="work-folders-list"></ul> + </div> +</div> diff --git a/views/partials/head.ejs b/views/partials/head.ejs index 0e8286c..f3e2cc6 100644 --- a/views/partials/head.ejs +++ b/views/partials/head.ejs @@ -26,6 +26,7 @@ window.__ICLAW_PROJECT_LOGO_EMOJIS__ = <%- JSON.stringify(projectLogoEmojis || []) %>; window.__ICLAW_PROJECTS__ = <%- JSON.stringify(typeof projectsMini !== 'undefined' ? projectsMini : []) %>; window.__ICLAW_SCHEDULED_CHAT_COUNTS__ = <%- JSON.stringify(typeof scheduledChatCounts !== 'undefined' ? scheduledChatCounts : {}) %>; + window.__ICLAW_DEV__ = <%- JSON.stringify(typeof devMode !== 'undefined' ? !!devMode : false) %>; </script> </head> <body> diff --git a/views/partials/sidebar.ejs b/views/partials/sidebar.ejs index 93560fb..917e101 100644 --- a/views/partials/sidebar.ejs +++ b/views/partials/sidebar.ejs @@ -11,7 +11,7 @@ <div class="sidebar-top"> <div class="sidebar-toolbar"> <a - href="/settings" + href="/settings/voice-ask" class="sidebar-settings-btn<%= typeof activeSettings !== 'undefined' && activeSettings ? ' active' : '' %>" aria-label="Settings" title="Settings" diff --git a/views/project.ejs b/views/project.ejs index 8556143..b84956f 100644 --- a/views/project.ejs +++ b/views/project.ejs @@ -119,6 +119,18 @@ data-tab-base="Memory" data-tab-count="<%= facts.length %>" >Memory</button> + <button + type="button" + class="project-tab" + role="tab" + id="project-tab-skills" + aria-controls="project-panel-skills" + aria-selected="false" + tabindex="-1" + data-project-tab="skills" + data-tab-base="Skills" + data-tab-count="<%= (skills || []).length %>" + >Skills</button> <button type="button" class="project-tab" @@ -231,6 +243,58 @@ </div> </div> + <div + id="project-panel-skills" + class="project-panel" + role="tabpanel" + aria-labelledby="project-tab-skills" + hidden + > + <div class="project-sheet project-sheet--flush project-sheet--stacked"> + <ul id="skills-list" class="project-chats"> + <% (skills || []).forEach(function (s) { %> + <li class="skill" data-skill-id="<%= s.id %>" data-skill-version="<%= s.version %>"> + <div class="project-row-head muted"> + <% if (s.project_id === null) { %> + <span class="skill-scope-badge" title="Available to every project">Global</span> + <% } %> + <% if (s.source_chat_id) { %> + <a href="/chats/<%= s.source_chat_id %>" class="project-chat-source"><%= (s.source_chat_title || '').trim() || 'Chat' %></a> + <% } else { %> + <span>—</span> + <% } %> + </div> + <input + class="skill-name" + aria-label="Skill name" + value="<%= s.name %>" + spellcheck="false" + /> + <textarea + class="skill-description" + aria-label="Skill summary" + rows="2" + ><%= s.description %></textarea> + <details class="skill-body-details"> + <summary>View / edit procedure</summary> + <textarea + class="skill-body" + aria-label="Skill procedure (SKILL.md)" + rows="10" + ><%= s.body %></textarea> + </details> + <div class="fact-meta"> + <button type="button" class="fact-delete skill-delete" aria-label="Remove skill">Remove</button> + </div> + </li> + <% }); %> + <% if (!(skills || []).length) { %> + <li class="project-chats-empty muted">No skills yet. Accept a skill suggestion in a chat for this project.</li> + <% } %> + </ul> + </div> + </div> + <div id="project-panel-links" class="project-panel" diff --git a/views/settings.ejs b/views/settings.ejs index 80934a5..eec4197 100644 --- a/views/settings.ejs +++ b/views/settings.ejs @@ -10,6 +10,76 @@ <h1 class="settings-title">Settings</h1> </header> + <nav class="settings-tabs" aria-label="Settings sections"> + <a class="settings-tab<%= settingsTab === 'voice-ask' ? ' is-active' : '' %>" href="/settings/voice-ask"<%- settingsTab === 'voice-ask' ? ' aria-current="page"' : '' %>>Voice & Ask</a> + <a class="settings-tab<%= settingsTab === 'remote-access' ? ' is-active' : '' %>" href="/settings/remote-access"<%- settingsTab === 'remote-access' ? ' aria-current="page"' : '' %>>Remote Access</a> + </nav> + + <% if (settingsTab === 'voice-ask') { %> + <!-- ─────────── Voice & Ask (OpenRouter) ─────────── --> + <section class="settings-section"> + <header class="settings-section-head"> + <div class="settings-section-head-text"> + <h2 class="settings-section-title"> + <svg class="settings-section-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"> + <path d="m12 3-1.9 5.8a2 2 0 0 1-1.3 1.3L3 12l5.8 1.9a2 2 0 0 1 1.3 1.3L12 21l1.9-5.8a2 2 0 0 1 1.3-1.3L21 12l-5.8-1.9a2 2 0 0 1-1.3-1.3z"/><path d="M5 3v4"/><path d="M3 5h4"/> + </svg> + Voice & Ask + </h2> + <p class="settings-section-desc"> + Paste your OpenRouter key to unlock: + </p> + <ul class="settings-section-feature-list"> + <li>🎙 <strong>Voice messages</strong> — speak instead of type</li> + <li>💬 <strong>Ask mode</strong> — quick answers without agent execution</li> + <li>🗂 <strong>Work mode</strong> — AI works inside your selected folders only, file writes need your approval</li> + <li>✏️ <strong>Auto-named chats</strong> — titles generated automatically</li> + </ul> + <p class="settings-section-desc settings-section-desc--sub">The key stays on this device.</p> + </div> + </header> + + <% if (openRouter.hasKey) { %> + <div class="or-card" id="or-card"> + <div class="or-status-row"> + <span class="or-dot" aria-hidden="true"></span> + <div class="or-status-text"> + <div class="or-status-title">Connected</div> + <code class="or-key"><%= openRouter.maskedKey %></code> + </div> + <button type="button" class="btn btn--ghost btn--sm btn--danger" id="or-disconnect">Disconnect</button> + </div> + <div class="or-usage" id="or-usage"> + <div class="or-usage-loading"> + <span class="ra-spinner" aria-hidden="true"></span><span>Loading usage…</span> + </div> + </div> + </div> + <% } else { %> + <div class="or-card or-connect" id="or-card"> + <form class="or-form" id="or-form"> + <input + type="password" + id="or-key-input" + class="or-input" + placeholder="Paste your key (sk-or-…)" + autocomplete="off" + spellcheck="false" + aria-label="OpenRouter API key" + /> + <button type="submit" class="btn btn--primary btn--sm" id="or-connect">Connect</button> + </form> + <p class="or-err" id="or-err" role="alert" hidden></p> + <a class="or-help" href="https://openrouter.ai/keys" target="_blank" rel="noopener noreferrer"> + Don’t have a key? Get one at openrouter.ai/keys + <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M7 17 17 7"/><path d="M7 7h10v10"/></svg> + </a> + </div> + <% } %> + </section> + <% } %> + + <% if (settingsTab === 'remote-access') { %> <!-- ─────────── Remote Access ─────────── --> <section class="settings-section"> <header class="settings-section-head"> @@ -126,10 +196,12 @@ <% } %> </div> </section> + <% } %> </div> </main> </div> +<% if (settingsTab === 'remote-access') { %> <!-- ─────────── New-tunnel modal ─────────── --> <dialog id="ra-modal" class="ra-modal" aria-labelledby="ra-modal-title" aria-describedby="ra-modal-desc"> <form method="dialog" class="ra-modal-form" id="ra-modal-form"> @@ -191,6 +263,7 @@ </footer> </form> </dialog> +<% } %> <style> /* ───────────── Page shell ───────────── */ @@ -203,7 +276,7 @@ overflow-y: auto; } .settings-inner { width: 100%; } -.settings-head { margin-bottom: var(--space-7); } +.settings-head { margin-bottom: var(--space-6); } .settings-title { margin: 0; font-size: 1.5rem; @@ -213,6 +286,37 @@ color: var(--text); } +/* ───────────── Section tabs (each its own page) ───────────── */ +.settings-tabs { + display: flex; + flex-wrap: wrap; + gap: var(--space-2); + margin: 0 0 var(--space-8); +} +.settings-tab { + display: inline-flex; + align-items: center; + padding: var(--space-3) var(--space-6); + border-radius: var(--radius-pill); + border: 1px solid color-mix(in srgb, var(--border) 80%, transparent); + background: var(--surface); + color: var(--muted); + font-size: var(--text-sm); + font-weight: 600; + letter-spacing: -0.01em; + text-decoration: none; + transition: color 0.15s ease, border-color 0.15s ease, background 0.15s ease; +} +.settings-tab:hover { + color: var(--text); + border-color: color-mix(in srgb, var(--text) 22%, var(--border)); +} +.settings-tab.is-active { + color: var(--btn-fg); + background: var(--btn-bg); + border-color: var(--btn-bg); +} + /* ───────────── Section ───────────── */ .settings-section { margin: 0 0 var(--space-9); } .settings-section-head { margin: 0 0 var(--space-5); } @@ -246,6 +350,23 @@ color: var(--muted); max-width: 26rem; } +.settings-section-desc--sub { + margin-top: var(--space-2); +} +.settings-section-feature-list { + margin: var(--space-2) 0 0; + padding-left: 0; + list-style: none; + display: flex; + flex-direction: column; + gap: var(--space-1); + font-size: var(--text-sm); + color: var(--muted); + max-width: 26rem; +} +.settings-section-feature-list li { + line-height: 1.45; +} .ra-new-btn { flex-shrink: 0; gap: var(--space-2); @@ -686,14 +807,121 @@ padding: 0 var(--space-5) var(--space-5); } +/* ───────────── Voice & Ask (OpenRouter) ───────────── */ +.or-card { + background: var(--surface); + border: 1px solid color-mix(in srgb, var(--border) 80%, transparent); + border-radius: var(--radius-lg); + padding: var(--space-5); +} + +/* Connected — status row */ +.or-status-row { display: flex; align-items: center; gap: var(--space-3); } +.or-dot { + flex-shrink: 0; + width: 8px; height: 8px; + border-radius: 50%; + background: var(--approve, #2da44e); + box-shadow: 0 0 0 3px color-mix(in srgb, var(--approve, #2da44e) 22%, transparent); +} +.or-status-text { flex: 1; min-width: 0; } +.or-status-title { + font-size: var(--text-sm); + font-weight: 600; + color: var(--text); + letter-spacing: -0.01em; +} +.or-key { + display: block; + margin-top: 2px; + font-family: ui-monospace, SFMono-Regular, Menlo, monospace; + font-size: var(--text-xs); + color: var(--muted); +} + +/* Connected — usage readout */ +.or-usage { + margin-top: var(--space-5); + padding-top: var(--space-5); + border-top: 1px solid color-mix(in srgb, var(--border) 55%, transparent); +} +.or-usage-loading { + display: flex; align-items: center; gap: var(--space-2); + font-size: var(--text-sm); color: var(--muted); font-style: italic; +} +.or-usage-grid { display: flex; gap: var(--space-10); } +.or-usage-value { + font-size: 1.5rem; + font-weight: 600; + letter-spacing: -0.02em; + color: var(--text); + font-variant-numeric: tabular-nums; + font-feature-settings: "tnum"; + line-height: 1.1; +} +.or-usage-label { margin-top: 3px; font-size: var(--text-xs); color: var(--muted); } +.or-bar { + margin-top: var(--space-5); + height: 5px; + border-radius: var(--radius-pill); + background: color-mix(in srgb, var(--text) 9%, transparent); + overflow: hidden; +} +.or-bar-fill { + height: 100%; + border-radius: var(--radius-pill); + background: var(--accent, var(--apple-blue)); + transition: width 0.4s ease; +} +.or-usage-muted { font-size: var(--text-sm); color: var(--muted); } + +/* Not connected — connect form */ +.or-form { display: flex; gap: var(--space-3); align-items: center; } +.or-input { + flex: 1; + min-width: 0; + margin: 0; + padding: var(--space-4) var(--space-5); + border: 1px solid var(--border); + border-radius: var(--radius-md); + background: color-mix(in srgb, var(--text) 3%, var(--surface)); + color: var(--text); + font-family: ui-monospace, SFMono-Regular, Menlo, monospace; + font-size: var(--text-sm); + transition: border-color 0.15s ease, box-shadow 0.15s ease; +} +.or-input::placeholder { color: color-mix(in srgb, var(--muted) 75%, transparent); } +.or-input:focus { + outline: none; + border-color: color-mix(in srgb, var(--text) 22%, var(--border)); + box-shadow: var(--focus-ring); +} +.or-err { margin: var(--space-3) 0 0; font-size: var(--text-sm); color: var(--danger); } +.or-err[hidden] { display: none; } +.or-help { + display: inline-flex; + align-items: center; + gap: var(--space-2); + margin-top: var(--space-5); + font-size: var(--text-sm); + color: var(--muted); + text-decoration: none; +} +.or-help > svg { width: 13px; height: 13px; flex-shrink: 0; } +.or-help:hover { color: var(--text); } +.or-help:hover { text-decoration: underline; text-underline-offset: 2px; } + @media (max-width: 520px) { .settings-page { padding: var(--space-7) var(--space-4) var(--space-8); } .settings-section-head-row { flex-direction: column; align-items: stretch; } .ra-new-btn { justify-content: center; } .ra-tunnel-head { align-items: flex-start; } + .or-form { flex-direction: column; align-items: stretch; } + .or-usage-grid { gap: var(--space-8); } } </style> +<% if (settingsTab === 'remote-access') { %> <script> (function () { const list = document.getElementById('ra-list'); @@ -1227,6 +1455,113 @@ tickRemaining(); })(); </script> +<% } %> + +<% if (settingsTab === 'voice-ask') { %> +<script> +(function () { + const card = document.getElementById('or-card'); + if (!card) return; + + function usd(n) { + if (!isFinite(n) || n <= 0) return '$0'; + if (n < 0.01) return '$' + n.toFixed(4); + if (n < 1) return '$' + n.toFixed(3); + return '$' + n.toFixed(2); + } + + /* ─── connect ─── */ + const form = document.getElementById('or-form'); + if (form) { + const input = document.getElementById('or-key-input'); + const btn = document.getElementById('or-connect'); + const err = document.getElementById('or-err'); + form.addEventListener('submit', async function (e) { + e.preventDefault(); + const key = ((input && input.value) || '').trim(); + if (err) err.hidden = true; + if (!key) { if (input) input.focus(); return; } + btn.disabled = true; + const orig = btn.textContent; + btn.textContent = 'Connecting…'; + try { + const res = await fetch('/api/openrouter/key', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ key }), + }); + const data = await res.json().catch(function () { return {}; }); + if (!res.ok) { + if (err) { err.textContent = data.error || ('Could not save key (HTTP ' + res.status + ')'); err.hidden = false; } + btn.disabled = false; + btn.textContent = orig; + return; + } + // Reload so the connected state + composer (mic, Ask) pick up the key. + location.reload(); + } catch (e2) { + if (err) { err.textContent = 'Could not save key: ' + e2.message; err.hidden = false; } + btn.disabled = false; + btn.textContent = orig; + } + }); + } + + /* ─── disconnect ─── */ + const disconnectBtn = document.getElementById('or-disconnect'); + if (disconnectBtn) { + disconnectBtn.addEventListener('click', async function () { + if (!confirm('Disconnect OpenRouter? Voice messages, Ask mode, and Work mode turn off until you add a key again.')) return; + disconnectBtn.disabled = true; + const orig = disconnectBtn.textContent; + disconnectBtn.textContent = 'Disconnecting…'; + try { + const res = await fetch('/api/openrouter/key', { method: 'DELETE' }); + if (!res.ok) { alert('Failed to disconnect'); disconnectBtn.disabled = false; disconnectBtn.textContent = orig; return; } + location.reload(); + } catch (e3) { + alert('Failed to disconnect: ' + e3.message); + disconnectBtn.disabled = false; + disconnectBtn.textContent = orig; + } + }); + } + + /* ─── usage ─── */ + const usageEl = document.getElementById('or-usage'); + if (usageEl) { + (async function () { + try { + const res = await fetch('/api/openrouter/usage'); + const data = await res.json().catch(function () { return {}; }); + if (!res.ok || !data.usage) { + usageEl.innerHTML = '<div class="or-usage-muted">Couldn’t load usage right now.</div>'; + return; + } + const u = data.usage; + const used = Number(u.usage || 0); + const remaining = u.remaining == null ? null : Number(u.remaining); + const limit = u.limit == null ? null : Number(u.limit); + + let html = '<div class="or-usage-grid">'; + html += '<div><div class="or-usage-value">' + usd(used) + '</div><div class="or-usage-label">spent</div></div>'; + if (remaining != null) { + html += '<div><div class="or-usage-value">' + usd(remaining) + '</div><div class="or-usage-label">remaining</div></div>'; + } + html += '</div>'; + if (limit != null && limit > 0) { + const pct = Math.max(0, Math.min(100, (used / limit) * 100)); + html += '<div class="or-bar"><div class="or-bar-fill" style="width:' + pct.toFixed(1) + '%"></div></div>'; + } + usageEl.innerHTML = html; + } catch (e4) { + usageEl.innerHTML = '<div class="or-usage-muted">Couldn’t load usage right now.</div>'; + } + })(); + } +})(); +</script> +<% } %> <script src="/js/iclaw.js?v=1"></script> diff --git a/views/welcome.ejs b/views/welcome.ejs new file mode 100644 index 0000000..32a3ed2 --- /dev/null +++ b/views/welcome.ejs @@ -0,0 +1,227 @@ +<%- include('partials/head', { title: 'Welcome to iClaw' }) %> + +<div class="onb"> + <div class="onb-card-shell" role="dialog" aria-labelledby="onb-title"> + + <!-- Step 1 — pick a power source --> + <section class="onb-step" id="onb-choose"> + <h1 class="onb-title" id="onb-title">Welcome to iClaw</h1> + <p class="onb-sub">iClaw works with the world’s best AI models - connect one to begin</p> + + <button type="button" class="onb-option" id="onb-pick-openrouter"> + <span class="onb-option-main"> + <span class="onb-option-name">Connect OpenRouter</span> + <span class="onb-option-desc"> + Pay as you go - about $5 lasts a long time, and we’ll guide you + </span> + </span> + <span class="onb-option-chevron" aria-hidden="true">›</span> + </button> + + <button type="button" class="onb-option onb-option--disabled" disabled> + <span class="onb-option-main"> + <span class="onb-option-name"> + Use iClaw credits + <span class="onb-badge">Coming soon</span> + </span> + <span class="onb-option-desc"> + No setup - we handle the AI for you + </span> + </span> + </button> + + <button type="button" class="onb-skip" id="onb-skip">Skip for now</button> + </section> + + <!-- Step 2 — paste the OpenRouter key --> + <section class="onb-step" id="onb-key" hidden> + <button type="button" class="onb-back" id="onb-back" aria-label="Back">‹ Back</button> + <h1 class="onb-title">Connect OpenRouter</h1> + + <ol class="onb-steps"> + <li> + Sign in at + <a href="https://openrouter.ai/keys" target="_blank" rel="noopener noreferrer">openrouter.ai/keys</a> + </li> + <li>Add about $5 - it goes a long way</li> + <li>Click <strong>Create Key</strong>, then copy and paste it below</li> + </ol> + + <form class="onb-key-form" id="onb-key-form" autocomplete="off"> + <input + type="password" + id="onb-key-input" + class="onb-key-input" + placeholder="sk-or-..." + aria-label="OpenRouter API key" + spellcheck="false" + autocapitalize="off" + /> + <button type="submit" class="btn btn--primary onb-key-submit" id="onb-key-submit"> + Connect + </button> + </form> + <p class="onb-key-error" id="onb-key-error" role="alert" hidden></p> + <p class="onb-key-note">Your key stays on this computer only</p> + </section> + + <!-- Honest background-prep line, shared by both steps --> + <div class="onb-prep" id="onb-prep" aria-live="polite" hidden> + <span class="onb-prep-dot" id="onb-prep-dot"></span> + <span class="onb-prep-text" id="onb-prep-text"></span> + </div> + </div> +</div> + +<script> +(function () { + var stepChoose = document.getElementById('onb-choose'); + var stepKey = document.getElementById('onb-key'); + var prepText = document.getElementById('onb-prep-text'); + var prepDot = document.getElementById('onb-prep-dot'); + + function show(step) { + stepChoose.hidden = step !== 'choose'; + stepKey.hidden = step !== 'key'; + if (step === 'key') { + var input = document.getElementById('onb-key-input'); + if (input) input.focus(); + } + } + + document.getElementById('onb-pick-openrouter').addEventListener('click', function () { + show('key'); + }); + document.getElementById('onb-back').addEventListener('click', function () { + show('choose'); + }); + + function complete() { + return fetch('/api/onboarding/complete', { method: 'POST' }); + } + + document.getElementById('onb-skip').addEventListener('click', function () { + complete().finally(function () { window.location.href = '/'; }); + }); + + var form = document.getElementById('onb-key-form'); + var errorEl = document.getElementById('onb-key-error'); + var submitBtn = document.getElementById('onb-key-submit'); + var keyInput = document.getElementById('onb-key-input'); + var noCreditOkFor = null; // key the user confirmed despite a $0 balance + + function resetBtn() { + submitBtn.disabled = false; + submitBtn.textContent = 'Connect'; + } + function showError(msg, warn) { + errorEl.textContent = msg; + errorEl.classList.toggle('is-warn', !!warn); + errorEl.hidden = false; + } + + var INVALID = { + empty: 'Paste your OpenRouter key first.', + unauthorized: 'That key was rejected by OpenRouter. Double-check you copied it in full.', + http: 'OpenRouter couldn’t verify that key. Try again in a moment.', + network: 'Couldn’t reach OpenRouter to verify the key — check your connection.', + }; + + // Save the (already-validated) key, mark onboarding done, enter the app. + function saveAndEnter(key) { + return fetch('/api/openrouter/key', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ key: key }), + }) + .then(function (res) { return res.json().then(function (b) { return { ok: res.ok, body: b }; }); }) + .then(function (r) { + if (!r.ok) { + showError((r.body && r.body.error) || 'Could not save the key.'); + resetBtn(); + return; + } + return complete().then(function () { window.location.href = '/'; }); + }); + } + + form.addEventListener('submit', function (e) { + e.preventDefault(); + var key = keyInput.value.trim(); + errorEl.hidden = true; + if (!key) { showError(INVALID.empty); return; } + + // Second click after a "no credit" warning → proceed without re-checking. + if (noCreditOkFor === key) { submitBtn.disabled = true; submitBtn.textContent = 'Connecting…'; saveAndEnter(key); return; } + + submitBtn.disabled = true; + submitBtn.textContent = 'Verifying…'; + fetch('/api/openrouter/validate', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ key: key }), + }) + .then(function (res) { return res.json().catch(function () { return {}; }); }) + .then(function (v) { + if (!v || !v.valid) { + showError(INVALID[(v && v.reason)] || INVALID.http); + resetBtn(); + return; + } + // Valid but no credit: warn once, let them continue anyway. + if (typeof v.remaining === 'number' && v.remaining <= 0) { + noCreditOkFor = key; + showError('Key works, but it has no credit yet — paid models won’t run until you top up. Click Connect again to continue anyway.', true); + resetBtn(); + submitBtn.textContent = 'Continue anyway'; + return; + } + submitBtn.textContent = 'Connecting…'; + return saveAndEnter(key); + }) + .catch(function () { showError(INVALID.network); resetBtn(); }); + }); + // Editing the key clears a prior "continue anyway" confirmation. + keyInput.addEventListener('input', function () { noCreditOkFor = null; }); + + // Background-prep poller: an honest line about Docker / OpenClaw readiness. + // Never blocks anything — purely informational. + // Only worth a line when something is actually happening or needs attention. + // When everything's fine (or we're still checking) we say nothing — a green + // "all good" badge is just noise for a non-technical user. + var prep = document.getElementById('onb-prep'); + function describe(env) { + // Keep onboarding jargon-free for non-technical users: only surface a + // neutral "setting up" line while the sandbox image downloads. Docker's + // state (stopped/missing) is NOT shown here — it's handled automatically + // (started on demand) and surfaces in-context later, with an Install + // button, only if the user actually picks Safe work. + if (env.docker === 'pulling') return 'Setting up your workspace…'; + return null; + } + function settled(env) { + return env.docker === 'ready' || env.docker === 'missing' || env.docker === 'stopped'; + } + function poll() { + fetch('/api/onboarding/status') + .then(function (r) { return r.json(); }) + .then(function (env) { + var msg = describe(env); + if (msg) { + prepText.textContent = msg; + if (prep) prep.hidden = false; + } else if (prep) { + prep.hidden = true; + } + // Green dot only means "all set". Attention states (missing/stopped) + // keep the amber pulsing dot; 'ready' is hidden anyway. + prepDot.classList.toggle('is-done', env.docker === 'ready'); + if (!settled(env)) setTimeout(poll, 1500); + }) + .catch(function () { setTimeout(poll, 3000); }); + } + poll(); +})(); +</script> + +<%- include('partials/foot') %>