From 96696bc6a8a97f2a11d0a50489501665b354fe6d Mon Sep 17 00:00:00 2001 From: Matheus Oliveira Date: Wed, 15 Jul 2026 19:43:27 -0300 Subject: [PATCH 1/2] docs: add Console admin UI page and navigation Document the previously-undocumented /console server-rendered admin UI (setup wizard, memory/skills/cognition/config/schedule pages) as a new operator-level page, with nav entries in the Roq template and the Antora nav. Signed-off-by: Matheus Oliveira --- docs/modules/ROOT/nav.adoc | 1 + docs/modules/ROOT/pages/console.adoc | 51 ++++++++++++++++++++++++++++ site/content/console.adoc | 51 ++++++++++++++++++++++++++++ site/templates/layouts/default.html | 1 + 4 files changed, 104 insertions(+) create mode 100644 docs/modules/ROOT/pages/console.adoc create mode 100644 site/content/console.adoc diff --git a/docs/modules/ROOT/nav.adoc b/docs/modules/ROOT/nav.adoc index a6c6e582..112e8955 100644 --- a/docs/modules/ROOT/nav.adoc +++ b/docs/modules/ROOT/nav.adoc @@ -8,6 +8,7 @@ * xref:architecture.adoc[Architecture Overview] * xref:composition.adoc[Composition] +* xref:console.adoc[Console] * xref:memory.adoc[Memory] * xref:storage.adoc[Storage] * xref:skills.adoc[Skills] diff --git a/docs/modules/ROOT/pages/console.adoc b/docs/modules/ROOT/pages/console.adoc new file mode 100644 index 00000000..23702fc4 --- /dev/null +++ b/docs/modules/ROOT/pages/console.adoc @@ -0,0 +1,51 @@ += Console +:page-title: Console +:page-description: The optional server-rendered admin UI - first-run setup plus memory, skills, cognition, config and schedule management. + +The console is an optional server-rendered admin UI at `/console`. It is a first-run setup wizard and a set of management pages over the agent's existing admin API - memory, skills, cognition, configuration, and the background schedule. It is built with Qute templates and https://htmx.org[HTMX], so it needs no Node build and no separate frontend. + +The console never re-implements admin logic. Its read views inject the same SPIs the agent uses (`FactStore`, `SkillStore`, the live Quarkus `Scheduler`, the config-metadata index); every mutating action is driven from the page by HTMX against the existing `/api/admin/*` endpoints. So the UI and the API can never drift, and anything the console does is equally scriptable with `curl`. + +== Enabling and reaching it + +The console is the composition capability `console`. It ships in the reference `app` (it is in the baked `agent.yml`), so a stock build already serves it; a slimmed distribution can leave it out (see xref:composition.adoc[Composition]). When present, browse to `/console` - `http://localhost:8080/console` in dev mode, or `http://localhost:8742/console` for the containerized run. + +Every console page is `@Authenticated` and carries no login of its own: it reuses whatever mechanism the app already enforces on `/api/admin/*`. In the bundled distribution that is HTTP Basic over an Argon2id credential (`qlawkus` / `dev` in dev mode; see xref:secrets.adoc[Secrets]). Add `quarkus-oidc` to the app and the console is behind SSO with no console-side change. + +== First-run setup wizard + +On first boot the agent cannot yet talk to an LLM, so the Overview shows a setup banner linking to the wizard at `/console/setup`. The wizard writes nothing itself: it orchestrates the same primitives documented elsewhere - the keystore writer (secrets, applied on restart) and the composition admin service (a staged manifest, applied on the next rebuild). + +It is split into two phases by the closed-world boundary, because some steps only make sense once a capability has actually been compiled in: + +[cols="1,3",options="header"] +|=== +|Phase |Steps + +|*A - before a rebuild* |The LLM base URL, chat model and API key (saved to the keystore), and the capability selection (staged as an `agent.yml` manifest). These are all you need to get the agent talking and to choose what to build next. +|*B - after a rebuild* |The interactive steps a capability unlocks once it is baked in: a bot token for each messaging adapter that is present, and Google authorization for the Google Workspace tools. Each Phase B step is shown only for a capability the running app already carries. +|=== + +So a typical bring-up is: run the wizard's Phase A, rebuild and redeploy with the staged manifest (see xref:composition.adoc#_the_restart_contract[the restart contract]), then return to the wizard for Phase B on the capabilities you turned on. + +== Management pages + +[cols="1,3",options="header"] +|=== +|Page |What it does + +|*Overview* (`/console`) |A live status panel - app version, the baked composition posture and capability set, and a server clock that ticks on every HTMX poll to show the refresh is live. Hosts the first-run setup banner. +|*Memory* (`/console/memory`) |The most recent facts, the fact sources in use, and the episodic journals. Review, curate and purge are HTMX calls to `/api/admin/memory/*`. See xref:memory.adoc[Memory]. +|*Skills* (`/console/skills`) |The current skill index. Pin, delete, curate and the lifecycle sweep call `/api/admin/skills/*`. See xref:skills.adoc[Skills]. +|*Cognition* (`/console/cognition`) |Reconcile and migrate between the markdown and pgvector representations, over `/api/admin/cognition/*`. Shown only when the `cognition.pgvector` capability is baked in (the console does not depend on that extension - it reads the manifest to decide). See xref:storage.adoc[Storage]. +|*Config* (`/console/config`) |The full configuration editor: one section per documented config root, generated from the same `quarkus-config-doc` metadata that builds xref:config-reference.adoc[the config reference], never a hand-maintained list. Each property shows its live effective value and source; secrets never appear. Saving routes by config phase - `RUN_TIME` properties to `/api/admin/runtime-toggles` (apply on restart), `BUILD_TIME` ones to `/api/admin/config-overrides` (apply on the next rebuild). See xref:composition.adoc#_the_configuration_editor[Composition > the configuration editor]. +|*Schedule* (`/console/schedule`) |The five background jobs (memory review, memory curation, episodic consolidation, skill curation, skill lifecycle), sourced from the live Quarkus `Scheduler` so cron and previous/next fire times always match what is actually registered. Editing a cron reuses the `RUN_TIME` runtime-toggle path; triggering a job now hits its manual `/api/admin/*` endpoint. +|=== + +A note the Config and Schedule pages surface: when a property's effective value already comes from a source that outranks the runtime-toggle override file (an environment variable, `.env`, or a `-D` flag), the page warns that saving there will not take effect until that higher-priority source is removed - so a silent no-op is caught in the UI rather than by trial and error. + +== What's next? + +* xref:composition.adoc[Composition] - the manifest the setup wizard stages, and the rebuild contract that applies it +* xref:secrets.adoc[Secrets] - the Argon2id credential the console authenticates against, and the keystore the wizard writes to +* xref:config-reference.adoc[Configuration Reference] - every property the config editor can render diff --git a/site/content/console.adoc b/site/content/console.adoc new file mode 100644 index 00000000..23702fc4 --- /dev/null +++ b/site/content/console.adoc @@ -0,0 +1,51 @@ += Console +:page-title: Console +:page-description: The optional server-rendered admin UI - first-run setup plus memory, skills, cognition, config and schedule management. + +The console is an optional server-rendered admin UI at `/console`. It is a first-run setup wizard and a set of management pages over the agent's existing admin API - memory, skills, cognition, configuration, and the background schedule. It is built with Qute templates and https://htmx.org[HTMX], so it needs no Node build and no separate frontend. + +The console never re-implements admin logic. Its read views inject the same SPIs the agent uses (`FactStore`, `SkillStore`, the live Quarkus `Scheduler`, the config-metadata index); every mutating action is driven from the page by HTMX against the existing `/api/admin/*` endpoints. So the UI and the API can never drift, and anything the console does is equally scriptable with `curl`. + +== Enabling and reaching it + +The console is the composition capability `console`. It ships in the reference `app` (it is in the baked `agent.yml`), so a stock build already serves it; a slimmed distribution can leave it out (see xref:composition.adoc[Composition]). When present, browse to `/console` - `http://localhost:8080/console` in dev mode, or `http://localhost:8742/console` for the containerized run. + +Every console page is `@Authenticated` and carries no login of its own: it reuses whatever mechanism the app already enforces on `/api/admin/*`. In the bundled distribution that is HTTP Basic over an Argon2id credential (`qlawkus` / `dev` in dev mode; see xref:secrets.adoc[Secrets]). Add `quarkus-oidc` to the app and the console is behind SSO with no console-side change. + +== First-run setup wizard + +On first boot the agent cannot yet talk to an LLM, so the Overview shows a setup banner linking to the wizard at `/console/setup`. The wizard writes nothing itself: it orchestrates the same primitives documented elsewhere - the keystore writer (secrets, applied on restart) and the composition admin service (a staged manifest, applied on the next rebuild). + +It is split into two phases by the closed-world boundary, because some steps only make sense once a capability has actually been compiled in: + +[cols="1,3",options="header"] +|=== +|Phase |Steps + +|*A - before a rebuild* |The LLM base URL, chat model and API key (saved to the keystore), and the capability selection (staged as an `agent.yml` manifest). These are all you need to get the agent talking and to choose what to build next. +|*B - after a rebuild* |The interactive steps a capability unlocks once it is baked in: a bot token for each messaging adapter that is present, and Google authorization for the Google Workspace tools. Each Phase B step is shown only for a capability the running app already carries. +|=== + +So a typical bring-up is: run the wizard's Phase A, rebuild and redeploy with the staged manifest (see xref:composition.adoc#_the_restart_contract[the restart contract]), then return to the wizard for Phase B on the capabilities you turned on. + +== Management pages + +[cols="1,3",options="header"] +|=== +|Page |What it does + +|*Overview* (`/console`) |A live status panel - app version, the baked composition posture and capability set, and a server clock that ticks on every HTMX poll to show the refresh is live. Hosts the first-run setup banner. +|*Memory* (`/console/memory`) |The most recent facts, the fact sources in use, and the episodic journals. Review, curate and purge are HTMX calls to `/api/admin/memory/*`. See xref:memory.adoc[Memory]. +|*Skills* (`/console/skills`) |The current skill index. Pin, delete, curate and the lifecycle sweep call `/api/admin/skills/*`. See xref:skills.adoc[Skills]. +|*Cognition* (`/console/cognition`) |Reconcile and migrate between the markdown and pgvector representations, over `/api/admin/cognition/*`. Shown only when the `cognition.pgvector` capability is baked in (the console does not depend on that extension - it reads the manifest to decide). See xref:storage.adoc[Storage]. +|*Config* (`/console/config`) |The full configuration editor: one section per documented config root, generated from the same `quarkus-config-doc` metadata that builds xref:config-reference.adoc[the config reference], never a hand-maintained list. Each property shows its live effective value and source; secrets never appear. Saving routes by config phase - `RUN_TIME` properties to `/api/admin/runtime-toggles` (apply on restart), `BUILD_TIME` ones to `/api/admin/config-overrides` (apply on the next rebuild). See xref:composition.adoc#_the_configuration_editor[Composition > the configuration editor]. +|*Schedule* (`/console/schedule`) |The five background jobs (memory review, memory curation, episodic consolidation, skill curation, skill lifecycle), sourced from the live Quarkus `Scheduler` so cron and previous/next fire times always match what is actually registered. Editing a cron reuses the `RUN_TIME` runtime-toggle path; triggering a job now hits its manual `/api/admin/*` endpoint. +|=== + +A note the Config and Schedule pages surface: when a property's effective value already comes from a source that outranks the runtime-toggle override file (an environment variable, `.env`, or a `-D` flag), the page warns that saving there will not take effect until that higher-priority source is removed - so a silent no-op is caught in the UI rather than by trial and error. + +== What's next? + +* xref:composition.adoc[Composition] - the manifest the setup wizard stages, and the rebuild contract that applies it +* xref:secrets.adoc[Secrets] - the Argon2id credential the console authenticates against, and the keystore the wizard writes to +* xref:config-reference.adoc[Configuration Reference] - every property the config editor can render diff --git a/site/templates/layouts/default.html b/site/templates/layouts/default.html index 7587e87f..32d59225 100644 --- a/site/templates/layouts/default.html +++ b/site/templates/layouts/default.html @@ -44,6 +44,7 @@ Platform Composition + Console Secrets Integrations From 4526166c445fbe0f4d80f0af237d930d55338a68 Mon Sep 17 00:00:00 2001 From: Matheus Oliveira Date: Wed, 15 Jul 2026 19:43:49 -0300 Subject: [PATCH 2/2] docs: verify site documentation against source and correct inaccuracies Checked every concrete claim (config properties, endpoints, tool names, defaults) against the code and fixed the errors found: pgvector reframed as durable embeddings rather than the default (markdown is the @DefaultBean; app ships hybrid); the missing searchMemories tool and POST /api/admin/memory/consolidate endpoint; the six cognition stores miscount; the Telegram webhook path (/api/webhook/telegram); the wrong context-TTL env var and non-existent TTS aliases; the Google token-vault framed as on by default (app disables it); and stale API_USER_PASSWORD references. Rebalanced depth (trimmed composition, cut the speculative skills roadmap) and added mermaid diagrams where they clarify (composition pipeline, skills lifecycle, storage reconcile-vs-migrate, Google OAuth flow). Signed-off-by: Matheus Oliveira --- docs/modules/ROOT/pages/composition.adoc | 21 ++--- docs/modules/ROOT/pages/google-workspace.adoc | 31 +++++-- docs/modules/ROOT/pages/index.adoc | 15 ++-- docs/modules/ROOT/pages/memory.adoc | 16 ++-- docs/modules/ROOT/pages/messaging.adoc | 88 +++++++++--------- docs/modules/ROOT/pages/secrets.adoc | 6 +- docs/modules/ROOT/pages/skills.adoc | 45 ++++++---- docs/modules/ROOT/pages/storage.adoc | 26 ++++-- docs/modules/ROOT/pages/voice.adoc | 89 +++++++++---------- site/content/composition.adoc | 21 ++--- site/content/google-workspace.adoc | 31 +++++-- site/content/index.adoc | 15 ++-- site/content/memory.adoc | 16 ++-- site/content/messaging.adoc | 88 +++++++++--------- site/content/secrets.adoc | 6 +- site/content/skills.adoc | 45 ++++++---- site/content/storage.adoc | 26 ++++-- site/content/voice.adoc | 89 +++++++++---------- 18 files changed, 378 insertions(+), 296 deletions(-) diff --git a/docs/modules/ROOT/pages/composition.adoc b/docs/modules/ROOT/pages/composition.adoc index 7b851585..bee55565 100644 --- a/docs/modules/ROOT/pages/composition.adoc +++ b/docs/modules/ROOT/pages/composition.adoc @@ -6,17 +6,20 @@ Qlawkus lets a single manifest, `agent.yml`, decide which capabilities are compo [NOTE] ==== -The manifest schema, the generator (`mvn qlawkus:generate`), the capability catalog, the `runtime` toggle tier, and dev-mode reload are all in place: the generator reconciles the pom against the capabilities the reactor's extensions declare, unknown names fail the build, the resolved set is reported, and the `runtime` block is applied at boot with an external override. Composing a capability in or out is a build-time concern by design (a rebuild, the same closed-world trade a native image makes); the `runtime` section is the part that changes without one. +Composing a capability in or out is a build-time concern by design: a rebuild, the same closed-world trade a native image makes. The `runtime` section is the only part that changes without one. ==== == Why a generator Quarkus resolves Maven dependencies *before* augmentation runs, and Maven does no tree-shaking, so a build step cannot add or drop dependencies for its own build. The only place a manifest can decide the dependencies is a step that runs *ahead* of the build. Qlawkus does this with a Maven plugin, the same idea as `quarkus extension add`: -[source,text] ----- -agent.yml -> mvn qlawkus:generate -> pom.xml -> mvn package (or quarkus build --native) ----- +++++ +
+graph LR + M["agent.yml
manifest"] -->|"mvn qlawkus:generate"| P["pom.xml"] + P -->|"mvn package / quarkus build --native"| A["agent artifact"] +
+++++ A deselected capability is left out of the pom entirely, so the result is lean by construction - not by relying on native-image dead-code elimination. @@ -140,7 +143,7 @@ All three require authentication. The staged manifest lives in the app's own sta == The configuration editor -The manifest above decides *which capabilities* exist. The same admin surface also lets you edit *any documented property value* - `qlawkus.*` in full, plus a small curated allowlist of `quarkus.*` properties - from `/console/config` (when the optional `console` capability is composed in) or directly over the API. The form is generated from the same `quarkus-config-doc` metadata that builds xref:config-reference.adoc[the config reference], so there is no separate list to keep in sync, and secret properties (API keys, password hashes, tokens) never appear here - they stay in xref:secrets.adoc[the keystore]. +The manifest above decides *which capabilities* exist. The same admin surface also lets you edit *any documented property value* - `qlawkus.*` in full, plus a small curated allowlist of `quarkus.*` properties - over the API, or from the xref:console.adoc[console] at `/console/config` when the optional `console` capability is composed in. The form is generated from the same `quarkus-config-doc` metadata that builds xref:config-reference.adoc[the config reference], so there is no separate list to keep in sync, and secret properties (API keys, password hashes, tokens) never appear here - they stay in xref:secrets.adoc[the keystore]. Each property is edited through one of two tiers, depending on its `ConfigPhase`: @@ -148,15 +151,13 @@ Each property is edited through one of two tiers, depending on its `ConfigPhase` |=== |Phase |Applies |Endpoint -|`RUN_TIME` |Live, on the next restart |`PUT`/`DELETE /api/admin/runtime-toggles` - writes straight into the runtime-toggle override file described above. +|`RUN_TIME` |On the next restart |`PUT`/`DELETE /api/admin/runtime-toggles` - writes straight into the runtime-toggle override file described above. |`BUILD_TIME` / `BUILD_AND_RUN_TIME_FIXED` |Next rebuild |`PUT`/`POST`/`DELETE /api/admin/config-overrides` - stages a `.properties` document, the same stage-then-promote pattern as the manifest. |=== A `BUILD_TIME` write is validated (documented, in-scope, not a secret) and staged only - never applied directly - exactly like the manifest. The staged document lives in its own file, `config-overrides.properties`, parallel to `agent.yml` rather than merged into it, so promoting it can overwrite the file wholesale without ever touching a hand-written line in `application.properties`. -== The schedule page - -`/console/schedule` lists the agent's five background jobs - memory review, memory curation, episodic consolidation, skill curation, and skill lifecycle - sourced from the live Quarkus `Scheduler` rather than a hand-maintained list, so cron and previous/next fire time always match what is actually registered. Editing a job's cron reuses the same `RUN_TIME` tier as the configuration editor above (`PUT`/`DELETE /api/admin/runtime-toggles`); triggering a job immediately hits its existing manual-trigger endpoint (`POST /api/admin/memory/{review,curate,consolidate}`, `POST /api/admin/skills/{curate,lifecycle}`). +The console surfaces the same two tiers as a form, and adds a xref:console.adoc#_management_pages[schedule page] that drives the background jobs. Both are just front-ends over these endpoints. == The restart contract diff --git a/docs/modules/ROOT/pages/google-workspace.adoc b/docs/modules/ROOT/pages/google-workspace.adoc index 6cef02d6..0557f876 100644 --- a/docs/modules/ROOT/pages/google-workspace.adoc +++ b/docs/modules/ROOT/pages/google-workspace.adoc @@ -77,7 +77,7 @@ export QLAWKUS_GOOGLE_STORAGE_ENABLED=true == 6. Credential Vault (Optional but Recommended) -By default, Qlawkus stores OAuth tokens in an encrypted PostgreSQL table using AES-256 + Argon2id. Enable with: +The credential vault persists the Google refresh token so authorization survives restarts. It is off by default in the bundled `app`; enable it with a passphrase: [source,bash] ---- @@ -85,15 +85,36 @@ export QLAWKUS_VAULT_ENABLED=true export QLAWKUS_VAULT_PASSPHRASE=your-strong-passphrase ---- -If disabled or no passphrase provided, tokens are kept in memory and lost on restart. +When enabled with a passphrase, the refresh token is encrypted with AES-256-GCM (the key is derived from the passphrase via Argon2id) and stored in the `qlawkus_google_auth` datasource. If the vault is disabled or no passphrase is set, tokens are kept in memory and lost on restart. Treat the passphrase as a secret: see xref:secrets.adoc[Secrets] to store it in the keystore under `qlawkus.google.vault.encryption-passphrase` instead of a plain environment variable. == 7. First Authorization Flow +Authorization uses the OAuth 2.0 authorization-code (loopback) flow. The agent never sees your Google password: it hands you a consent URL, and Google redirects a one-time code back to the callback, which the agent exchanges for a refresh token. + 1. Start the agent (dev mode or Docker) 2. Send a message that requires Google access (e.g., "List my emails") -3. The agent will return an authorization URL -4. Open the URL in a browser, sign in with your Google account, and grant permissions -5. You'll be redirected back to the agent, which stores the tokens +3. The agent calls its `startGoogleAuthorization` tool and returns an authorization URL +4. Open the URL in a browser, sign in with your Google account, and grant the requested scopes +5. Google redirects to `/api/google/oauth/callback`; the agent exchanges the code for a refresh token, stores it (see the vault above), and confirms via `checkGoogleAuthorization` +6. Ask again - the agent retries the original request automatically + +++++ +
+sequenceDiagram + participant U as You + participant A as Agent + participant G as Google + U->>A: "List my emails" + A-->>U: authorization URL (startGoogleAuthorization) + U->>G: open URL, sign in, grant scopes + G-->>A: redirect to /api/google/oauth/callback?code + A->>G: exchange code for refresh token + A->>A: store refresh token in the vault + A-->>U: authorized - retry your request +
+++++ + +The xref:console.adoc[Console] onboarding wizard can run this same authorization from a web page instead of chat. == Available tools diff --git a/docs/modules/ROOT/pages/index.adoc b/docs/modules/ROOT/pages/index.adoc index 0af7c7d4..582261d1 100644 --- a/docs/modules/ROOT/pages/index.adoc +++ b/docs/modules/ROOT/pages/index.adoc @@ -12,14 +12,11 @@ Inspired by projects like https://github.com/openclaw/openclaw[OpenClaw] and htt Most personal AI agents store memory as markdown files on disk. The model decides when to read them, which means it can forget to check. Qlawkus takes a different approach: **memory is injected into the prompt automatically, so it never depends on the model deciding to look - and the agent can still search it on demand** when it needs to dig deeper. -Every turn, the agent receives: +Every turn, the agent receives its own `Soul` (identity, mood, behavioral bias) and a curated `UserProfile` of the person it serves, plus the facts relevant to the current message - retrieved by a vector search that runs **before each reply**, not by a tool the model has to remember to call. -* A `Soul` - the agent's own identity, mood, and behavioral bias -* A `UserProfile` - a coherent, curated record of the human it serves +Facts are tagged by source (`semantic-extractor`, `remember-tool`, `episodic-consolidator`, `transcript`) and kept clean by background jobs: nightly deduplication merges near-duplicates, and a curation job distills scattered facts into the single `UserProfile`. Transcripts are embedded too, so the agent can semantically search what was actually said. -Below the surface, a https://github.com/pgvector/pgvector[pgvector]-backed fact store holds granular memories tagged by source (`semantic-extractor`, `remember-tool`, `episodic-consolidator`, `transcript`). A retrieval step runs **before each reply**, embedding the user's query and injecting the most relevant facts into the prompt. No tool call required. No model cooperation needed. - -Background jobs keep the store clean: nightly deduplication merges near-duplicates, and a curation job periodically distills scattered facts into a single, coherent `UserProfile`. Transcripts are embedded too, so the agent can semantically search what was actually said. +The store itself is pluggable, chosen at build time. Leave the Postgres extension out and the agent is genuinely **database-free** - markdown files plus an in-process embedding index (the default whenever no database backend is registered). Add https://github.com/pgvector/pgvector[pgvector] to keep the embeddings in a durable vector store instead; the bundled `app` ships the `hybrid` backend, keeping the markdown files as the source of truth and mirroring them into Postgres. The retrieval model is identical in every case. The result: an agent that always knows who it is, who its owner is, and what it has learned - regardless of channel, context length, or the model's willingness to call tools. @@ -29,7 +26,7 @@ Cognition:: A persistent `Soul` (identity, mood, focus) that shapes the agent's behavior every turn. Memory:: -pgvector-backed long-term memory with automatic retrieval, background curation, and transcript search. Always in context, never missed. +Long-term memory with automatic retrieval, background curation, and transcript search - always in context, never missed. File-based and database-free by default, with an optional pgvector backend for durable embeddings. Skills:: Procedural memory the agent writes, uses, and curates itself: reusable `SKILL.md` how-to procedures, injected as an index each turn and loaded on demand. @@ -40,6 +37,9 @@ Discord and Telegram adapters with voice in and out. Slack and WhatsApp adapters Tools:: Google Workspace (Gmail, Calendar, Drive, Sheets, Storage), a brag-document tool, and a sandboxed shell. +Console:: +An optional server-rendered admin UI at `/console`: a first-run setup wizard plus memory, skills, cognition, configuration, and schedule management. No Node build, no separate frontend. + Resilience:: A primary LLM with an Ollama fallback behind a circuit breaker. @@ -51,6 +51,7 @@ xref:quickstart.adoc[Quick Start] - Get a working agent running in minutes. * xref:architecture.adoc[Architecture] - How the modules fit together * xref:composition.adoc[Composition] - Let agent.yml decide which capabilities are built in +* xref:console.adoc[Console] - The optional server-rendered admin UI * xref:skills.adoc[Skills] - Procedural memory the agent writes and curates itself * xref:google-workspace.adoc[Google Workspace] - Gmail, Calendar, Drive, Sheets, Storage * xref:messaging.adoc[Messaging] - Discord and Telegram bots with voice diff --git a/docs/modules/ROOT/pages/memory.adoc b/docs/modules/ROOT/pages/memory.adoc index bfd7211f..c274b4de 100644 --- a/docs/modules/ROOT/pages/memory.adoc +++ b/docs/modules/ROOT/pages/memory.adoc @@ -10,7 +10,7 @@ Most agents handle long-term memory in one of two weak ways: they make recall *d * *Injected, not tool-dependent.* Relevant memory is retrieved and added to the prompt automatically before every reply. Recall never depends on the model choosing to search - reactive-only recall is unreliable, and the most important memory is the kind the model does not know it should look for. * *Retrieved, not wholesale.* Active memory uses Retrieval-Augmented Generation (RAG): each turn, only the facts relevant to the current message are selected (query-relevant top-K) and injected. The full store is never dumped into the prompt, so the per-turn token cost stays roughly constant as memory grows. -* *Searchable on top, not instead.* None of this removes deliberate search: the `searchTranscripts` tool lets the model query the full transcript archive on demand when it needs to reach past what was injected. Automatic injection guarantees recall; tool search is a complement, never the thing recall depends on. +* *Searchable on top, not instead.* None of this removes deliberate search. Two tools let the model reach past what was injected: `searchMemories` queries the curated long-term facts, and `searchTranscripts` queries the raw transcript archive (verbatim exchanges, finer-grained than distilled facts). Automatic injection guarantees recall; tool search is a complement, never the thing recall depends on. The result is recall that is both *reliable* (always runs) and *bounded* (only what is relevant), with on-demand search still available - instead of trading those off against each other. @@ -44,20 +44,24 @@ The agent maintains its own memory so it does not rot: |=== |Job |Schedule |What it does -|Episodic consolidation |nightly |Summarizes the day into a journal that active memory can surface. +|Episodic consolidation |nightly (`POST /api/admin/memory/consolidate`) |Summarizes the day into a journal that active memory can surface. |Memory review |nightly (`POST /api/admin/memory/review`) |Removes semantic near-duplicate facts. |Profile curation |nightly (`POST /api/admin/memory/curate`) |Folds facts into the owner profile, reconciling contradictions. Facts themselves are left untouched. |=== == Administration -* `GET /api/admin/memory` - summary of what is stored. -* `DELETE /api/admin/memory` - purge (all, or by source). -* `POST /api/admin/memory/review` and `POST /api/admin/memory/curate` - trigger the curation jobs manually. +All endpoints are `@Authenticated` and sit under `/api/admin/memory`: + +* `GET /api/admin/memory` - summary of what is stored; `GET /api/admin/memory/journals` lists the episodic journals. +* `DELETE /api/admin/memory` - purge, scoped by query parameter: `?all=true`, `?source=` (one of the source tags above), or `?includeJournals=true`. +* `POST /api/admin/memory/{review,curate,consolidate}` - trigger the three background jobs manually. + +The same actions are available from the xref:console.adoc[Console] admin UI. == Storage backends -Memory persistence is selected at build time with `qlawkus.cognition.backend` (`markdown`, `pgvector` or `hybrid`), the same switch that governs every cognition store. Whichever backend is active, fact retrieval is the same query-relevant top-K RAG path described above - the choice trades off persistence and scale, never the recall model. +Memory persistence is selected at build time with `qlawkus.cognition.backend` (`markdown`, `pgvector` or `hybrid`), the same switch that governs every cognition store. Whichever backend is active, fact retrieval is the same query-relevant top-K RAG path described above - the choice is about how embeddings are persisted (rebuilt in-process each boot versus stored durably in Postgres), never the recall model. The full treatment - the backend table, the optional `qlawkus-cognition-pgvector` extension and the database-free distribution, file locations, and how to reconcile or migrate data between markdown and pgvector - lives in xref:storage.adoc[Storage]. diff --git a/docs/modules/ROOT/pages/messaging.adoc b/docs/modules/ROOT/pages/messaging.adoc index 5a07694e..0d90d25a 100644 --- a/docs/modules/ROOT/pages/messaging.adoc +++ b/docs/modules/ROOT/pages/messaging.adoc @@ -1,29 +1,43 @@ = Messaging Setup (Discord & Telegram) +:page-title: Messaging Setup +:page-description: Configure the Discord and Telegram adapters, with voice, shared context, and message chunking -Qlawkus includes messaging adapters for Discord and Telegram with voice support. Slack and WhatsApp adapters exist but are not wired into the default `app/` module. +Qlawkus includes messaging adapters for Discord and Telegram with voice support. Slack and WhatsApp adapters exist but are not wired into the default `app/` module (only `qlawkus-messaging`, `qlawkus-messaging-discord`, and `qlawkus-messaging-telegram` are). -== How conversation context works +== Conversation context -When you chat with the agent via any channel (REST, Discord, Telegram), it maintains a working memory of the back-and-forth conversation. By default, all channels share the same conversation context (`QLAWKUS_AGENT_SHARED_CONTEXT_ENABLED=true`), meaning the agent is continuous across Discord, Telegram, and REST — the owner is one person regardless of channel. +When you chat with the agent via any channel (REST, Discord, Telegram), it keeps a working memory of the back-and-forth conversation. Two knobs govern that context, both on the `qlawkus.agent` root: -If the conversation goes idle for too long, the working memory is automatically cleared to prevent stale context from accumulating. The default idle timeout is 60 minutes. +*Shared context (default on).* All channels share the same conversation, so the agent stays continuous across Discord, Telegram, and REST (the owner is one person regardless of channel). Disable it to give each channel its own conversation: -You can also ask the agent to clear the context at any time. Say things like "clear the context", "reset", "start over", "new topic", or "forget our conversation" — the agent will wipe the short-term conversation and start fresh on the next message. This does **not** erase long-term saved facts about you (stored in pgvector); only the running back-and-forth is cleared. +[source,bash] +---- +export QLAWKUS_AGENT_SHARED_CONTEXT_ENABLED=false +---- + +*Idle reset (default 60 minutes).* If a conversation is idle for longer than the configured window, the working memory is cleared before the next message so stale context does not accumulate. Set the window to `0` to disable: + +[source,bash] +---- +export QLAWKUS_AGENT_CONTEXT_TTL_MINUTES=60 +---- + +You can also ask the agent to clear the context at any time. Say things like "clear the context", "reset", "start over", "new topic", or "forget our conversation", and the agent wipes the short-term conversation and starts fresh on the next message. This does *not* erase long-term saved facts about you; only the running back-and-forth is cleared. == Discord Bot === 1. Create a Discord Application 1. Go to https://discord.com/developers/applications -2. Click **New Application** > name it (e.g., "Qlawkus") -3. Go to **Bot** tab > **Add Bot** -4. **Public Bot:** Leave unchecked if only you will use it. When checked, anyone can install the bot. -5. Under **Privileged Gateway Intents**, enable **Message Content Intent** (required for the bot to read messages). Without this, the bot cannot see message text. -6. Go to **OAuth2** > **General** > copy **Application ID** +2. Click *New Application* > name it (e.g., "Qlawkus") +3. Go to *Bot* tab > *Add Bot* +4. *Public Bot:* Leave unchecked if only you will use it. When checked, anyone can install the bot. +5. Under *Privileged Gateway Intents*, enable *Message Content Intent* (required for the bot to read messages). Without this, the bot cannot see message text. +6. Go to *OAuth2* > *General* > copy *Application ID* === 2. Configure Permissions & Invite -1. Go to **OAuth2** > **URL Generator** +1. Go to *OAuth2* > *URL Generator* 2. Scopes: `bot` + `applications.commands` 3. Bot Permissions: * Send Messages @@ -45,7 +59,9 @@ export QLAWKUS_MESSAGING_DISCORD_RESPOND_TO_ALL_MESSAGES=true export QLAWKUS_MESSAGING_DISCORD_STARTUP_GREETING="👋 Qlawkus online and listening" ---- -Optional auth: +`QLAWKUS_MESSAGING_DISCORD_RESPOND_TO_ALL_MESSAGES` defaults to `true`. Set it to `false` to make the bot respond only to DMs and slash commands. + +Optional auth (restrict who may talk to the bot): [source,bash] ---- @@ -53,6 +69,8 @@ export QLAWKUS_MESSAGING_AUTH_ALLOWED_USERS_DISCORD=* # or comma-separated user IDs: 12345,67890 ---- +`*` (or leaving the list unset) allows everyone; a comma-separated list restricts to those user IDs. + === 4. Slash Commands The bot registers a `/qlawkus` command with subcommands: @@ -69,12 +87,12 @@ If you set `QLAWKUS_MESSAGING_DISCORD_GUILD_ID`, commands sync instantly to that 1. Open Telegram and message @BotFather 2. Send `/newbot` and follow the prompts -3. Copy the **Token** +3. Copy the *Token* === 2. Choose Mode: Polling vs Webhook -* **Polling** (default, simpler): Bot pulls updates. Works behind NAT, no public URL needed. -* **Webhook**: Telegram pushes updates to your HTTPS endpoint. Requires a public HTTPS URL. +* *Polling* (default, simpler): Bot pulls updates. Works behind NAT, no public URL needed. +* *Webhook*: Telegram pushes updates to your HTTPS endpoint. Requires a public HTTPS URL. === 3. Environment Variables @@ -94,48 +112,28 @@ Webhook: ---- export QLAWKUS_MESSAGING_TELEGRAM_BOT_TOKEN=your-bot-token export QLAWKUS_MESSAGING_TELEGRAM_MODE=webhook -export QLAWKUS_MESSAGING_TELEGRAM_WEBHOOK_URL=https://yourdomain.com/api/telegram/webhook +export QLAWKUS_MESSAGING_TELEGRAM_WEBHOOK_URL=https://yourdomain.com/api/webhook/telegram export QLAWKUS_MESSAGING_AUTH_ALLOWED_USERS_TELEGRAM=* ---- -The webhook path is `/api/telegram/webhook`. +The webhook path is `/api/webhook/telegram` (the Slack and WhatsApp adapters, when enabled, use `/api/webhook/slack` and `/api/webhook/whatsapp`). == Voice Support Voice messages are supported on both Discord and Telegram via file attachments (the bot does not join voice channels). When a user sends a voice message: -1. Audio attachment is downloaded -2. Audio is transcribed via STT (Whisper via Groq/OpenAI) -3. Agent processes the text -4. If TTS is configured, the agent can respond with voice (sent as an audio file attachment) - -Additionally, the orchestrator detects voice intent in text messages (e.g., "manda audio", "respond with voice") and automatically triggers TTS. +1. The audio attachment is downloaded +2. The audio is transcribed via STT (OpenAI-compatible Whisper, e.g. Groq or OpenAI) +3. The agent processes the text +4. If TTS is configured, the agent can reply with voice (sent as an audio file attachment) -For TTS setup, see xref:voice.adoc[Voice Setup]. +The agent decides to reply with voice by calling its `respondWithVoice` tool. Its system prompt tells it to do so whenever you ask for audio (phrases like "responde em audio", "me manda por voz", "reply by voice"). -== Shared Context Across Channels - -By default, the agent maintains a single conversation across REST, Discord, and Telegram. The owner is one person regardless of channel. - -To disable (separate conversations per channel): - -[source,bash] ----- -export QLAWKUS_AGENT_SHARED_CONTEXT_ENABLED=false ----- - -== Idle Conversation Reset - -If a conversation is idle for more than a configurable period, the working memory is cleared before the next message. This prevents stale context from accumulating. - -[source,bash] ----- -export QLAWKUS_MESSAGING_CONTEXT_TTL_MINUTES=60 ----- +For STT/TTS setup, see xref:voice.adoc[Voice Setup]. == Message Chunking -Long responses are automatically split to fit platform limits: +Long responses are automatically split to fit each platform's limit: [cols="1,1",options="header"] |=== @@ -155,7 +153,7 @@ Long responses are automatically split to fit platform limits: |Discord bot connects but doesn't respond |Message Content Intent not enabled in Developer Portal. |Slash commands not appearing |Wait up to 1 hour for global sync, or set `QLAWKUS_MESSAGING_DISCORD_GUILD_ID` for instant guild sync. -|Telegram webhook not receiving updates |Verify HTTPS URL is accessible, check logs for `setWebhook` errors. +|Telegram webhook not receiving updates |Verify the HTTPS URL is reachable and ends in `/api/webhook/telegram`; check logs for `setWebhook` errors. |Voice not working |Check STT/TTS config in xref:voice.adoc[Voice Setup]. |=== diff --git a/docs/modules/ROOT/pages/secrets.adoc b/docs/modules/ROOT/pages/secrets.adoc index a96cf382..9544b6ed 100644 --- a/docs/modules/ROOT/pages/secrets.adoc +++ b/docs/modules/ROOT/pages/secrets.adoc @@ -47,16 +47,16 @@ If you would rather not use `keytool`, the agent exposes a basic-auth gated admi [source,bash] ---- # Store a secret (alias = the config property it supplies) -curl -u "$API_USER:$API_USER_PASSWORD" -X PUT \ +curl -u "qlawkus:$ADMIN_PASSWORD" -X PUT \ http://localhost:8742/api/admin/secrets \ -H 'Content-Type: application/json' \ -d '{"alias":"quarkus.langchain4j.openai.\"primary\".api-key","value":""}' # List stored aliases (names only, never values) -curl -u "$API_USER:$API_USER_PASSWORD" http://localhost:8742/api/admin/secrets +curl -u "qlawkus:$ADMIN_PASSWORD" http://localhost:8742/api/admin/secrets # Remove a secret -curl -u "$API_USER:$API_USER_PASSWORD" -X DELETE \ +curl -u "qlawkus:$ADMIN_PASSWORD" -X DELETE \ "http://localhost:8742/api/admin/secrets?alias=quarkus.langchain4j.openai.%22primary%22.api-key" ---- diff --git a/docs/modules/ROOT/pages/skills.adoc b/docs/modules/ROOT/pages/skills.adoc index cfe6064a..8a8224f9 100644 --- a/docs/modules/ROOT/pages/skills.adoc +++ b/docs/modules/ROOT/pages/skills.adoc @@ -60,13 +60,26 @@ Skills grow and stay clean on their own, mirroring the fact-memory subsystem: == Lifecycle -Using a skill (`viewSkill`) marks it used. `SkillLifecycleJob` then ages unused skills by recency: - -`ACTIVE` -> `STALE` -> `ARCHIVED` - -* Archived skills leave the injected index but stay loadable by name, so nothing is lost. +Using a skill (`viewSkill`) marks it used. `SkillLifecycleJob` then ages unused skills by recency, and any use revives a skill straight back to `ACTIVE`: + +++++ +
+stateDiagram-v2 + [*] --> ACTIVE: created / distilled + ACTIVE --> STALE: unused 30 days + STALE --> ARCHIVED: unused 90 days + STALE --> ACTIVE: viewSkill + ARCHIVED --> ACTIVE: viewSkill + note right of ARCHIVED + Dropped from the injected + index, still loadable by name + end note +
+++++ + +* `STALE` skills are unused but still injected; `ARCHIVED` ones leave the injected index (to keep the prompt compact) but stay loadable by name, so nothing is lost. * Pinned skills never transition. Pin with `POST /api/admin/skills/{name}/pin`. -* Thresholds are config: `qlawkus.skills.lifecycle.stale-after-days`, `archive-after-days`, and `cron`. +* The day thresholds and the sweep cron are config: `qlawkus.skills.lifecycle.stale-after-days` (default 30), `archive-after-days` (default 90), and `cron`. Usage telemetry is stored where the backend lives: Postgres columns (`pgvector`/`hybrid`, migration `V7`) or a `.qlawkus-usage.json` sidecar next to the files (`markdown`). @@ -82,9 +95,9 @@ The cognition persistence backend is chosen at **build time** with `qlawkus.cogn |=== |Backend |Behavior -|`markdown` |`SKILL.md` files only, no database. The whole index is injected wholesale - the Hermes/OpenClaw model. -|`pgvector` |Skills in Postgres (the `skill` table, migration `V6`). The default; backward compatible. -|`hybrid` |`SKILL.md` files are the source of truth, mirrored into Postgres for ranking. +|`markdown` |`SKILL.md` files only, no database. This is the `@DefaultBean`, so it is what a build **without** the `cognition-pgvector` extension uses - a fully database-free distribution. +|`pgvector` |Skills in Postgres (the `skill` table, migration `V6`, lifecycle columns in `V7`). Durable server-side storage; requires the `cognition-pgvector` extension on the classpath. +|`hybrid` |`SKILL.md` files are the source of truth, mirrored into Postgres. This is what the reference `app` ships. |=== Because the choice is build-time fixed, a markdown-only distribution carries no datasource or Flyway dependency for skills at all. @@ -102,9 +115,11 @@ All endpoints are under `/api/admin/skills` (Basic auth, same as the rest of the |`DELETE /api/admin/skills/{name}` |Delete an owned skill. |`POST /api/admin/skills/curate` |Run curation now (dedup redundant skills). |`POST /api/admin/skills/lifecycle` |Run the lifecycle sweep now (age unused skills). -|`POST /api/admin/skills/{name}/pin` |Pin a skill so it never ages out. +|`POST /api/admin/skills/{name}/pin` |Pin a skill so it never ages out (`?pinned=false` to unpin). |=== +The optional xref:console.adoc[Console] admin UI drives all of these from a `/console/skills` page, so pinning, deleting, curating and running the lifecycle sweep are available without hand-rolling `curl`. + == Remote Skill Hub Everything above is *local* and offline. The optional `qlawkus-tools-skill-hub` extension adds the outward-facing registry client, so the agent can discover, install, and publish skills mid-conversation. It is a separate module: omit it and a locked-down distribution physically cannot reach a registry (stronger than a config flag). The default implementation is plain Java (`java.net.http`), native-friendly, with no external CLI; it is an overridable `@DefaultBean`. Qlawkus never hosts or manages a registry itself. @@ -124,15 +139,7 @@ Install and publish honor `qlawkus.skill-hub.approval-mode`: `yolo` acts immedia == Roadmap -Today a skill is procedural text. One extension is planned. - -=== Executable skills - -Because a skill is already a directory (`//SKILL.md`), it can carry sibling assets - bash or Python scripts the agent runs as part of the procedure. This is the Anthropic skill model (a `SKILL.md` plus assets). It is not wired up yet: the file and shell tools are confined to the *workspace* root, while skills live in their own isolated root, so neither writing nor running a script inside a skill folder works today. The plan is to author assets through the skill API (not the workspace file tool) and stage them into the workspace for execution, without loosening either confinement boundary. - -Note the overlap with in-process *dynamic tools* (e.g. runtime-compiled Groovy that registers a typed `@Tool`): both let the agent gain a self-authored capability. Bundled scripts win the common case - they run as a confined subprocess and work in native image - so they are the primary path for "executable skill". In-process dynamic tools remain a separate, niche axis for typed tools that need direct access to app internals (JVM-mode only, since GraalVM cannot load new bytecode at runtime). - -The remote Skill Hub above already ships search, install, and publish; a jskills-CLI backend (for hosts that already run the jskills tooling) remains a possible `@DefaultBean` override. +A skill is procedural *text* today. Because it already lives in its own directory (`//SKILL.md`), a planned extension is *executable* skills that carry sibling scripts the agent runs as part of the procedure (the Anthropic `SKILL.md`-plus-assets model). It is not wired up yet, since skills and the workspace file/shell tools live behind separate confinement boundaries. == What's next? diff --git a/docs/modules/ROOT/pages/storage.adoc b/docs/modules/ROOT/pages/storage.adoc index fe10b78a..113e4476 100644 --- a/docs/modules/ROOT/pages/storage.adoc +++ b/docs/modules/ROOT/pages/storage.adoc @@ -6,7 +6,7 @@ Qlawkus persists everything the agent knows behind a small set of store SPIs, wi == The stores -Six cognition stores (plus skills) sit behind SPIs in `qlawkus-client`, each with interchangeable backends: +Six stores sit behind SPIs in `qlawkus-client`, each with interchangeable backends: [cols="1,3",options="header"] |=== @@ -28,12 +28,12 @@ The backend is selected at build time with `qlawkus.cognition.backend`: |=== |Backend |Behavior -|`markdown` |Facts are `.md` files (editable, git-versionable) plus an in-process embedding index cached to JSON; journals are dated `.md` files; working memory is one append-only `.jsonl` per conversation; the persona is `soul.md` and the owner profile is `owner.md`. No database; single-node. This is the default whenever the pgvector extension is absent. -|`pgvector` |Facts are embeddings in Postgres; journals, skills, working memory, persona and owner profile are Postgres tables. Scales and is shared across instances. -|`hybrid` |`.md` files are the source of truth for the git-versionable knowledge (facts, dated journals, skills), mirrored into pgvector for retrieval and scale. The high-churn working memory and the live persona/profile singletons stay in Postgres. This is what the deployable `app` ships with. +|`markdown` |Facts are `.md` files (editable, git-versionable); their embeddings live in a langchain4j `InMemoryEmbeddingStore` that is rebuilt at boot, with a sibling `.embeddings.json` cache on disk so unchanged facts are not re-embedded. Journals are dated `.md` files; working memory is one append-only `.jsonl` per conversation; the persona is `soul.md` and the owner profile is `owner.md`. No database. This is the default whenever the pgvector extension is absent. +|`pgvector` |Facts are embeddings in Postgres; journals, skills, working memory, persona and owner profile are Postgres tables. The vectors are stored durably, so nothing is re-embedded at boot. +|`hybrid` |`.md` files are the source of truth for the git-versionable knowledge (facts, dated journals, skills), mirrored into pgvector so their embeddings persist durably. The high-churn working memory and the live persona/profile singletons stay in Postgres only. This is what the deployable `app` ships with. |=== -Whichever backend is active, fact retrieval is the same query-relevant top-K RAG path - the choice trades off persistence and scale, never the recall model. Journal text is always embedded through the fact store (`source=episodic-consolidator`), so journals stay searchable regardless of where the journal record itself lives. +Whichever backend is active, fact retrieval is the same query-relevant top-K RAG path - the choice is about how embeddings are persisted (rebuilt in-process each boot versus stored durably in Postgres), never the recall model. Journal text is always embedded through the fact store (`source=episodic-consolidator`), so journals stay searchable regardless of where the journal record itself lives. === Large facts are chunked @@ -68,11 +68,25 @@ In the Docker images these sit under the container `HOME` (`/home/default`), whi The backend is build-time fixed, so changing it means rebuilding. Because each cognition store keeps a parallel markdown representation alongside its Postgres form, Qlawkus can move data between the two - something a plain embedding store cannot do, since langchain4j's `EmbeddingStore`/`ChatMemoryStore` SPIs are isolated silos with no second representation to sync against. Both operations live in the `qlawkus-cognition-pgvector` extension, so they only run in a `pgvector` or `hybrid` build (a markdown-only build has no Postgres side in-process). +The two differ in direction and in whether they overwrite: + +++++ +
+graph LR + MD[("Markdown files")] + PG[("pgvector / Postgres")] + MD -->|reconcile: union, non-destructive| PG + PG -->|reconcile: union, non-destructive| MD + MD -.->|migrate direction=files-to-pg: overwrite| PG + PG -.->|migrate direction=pg-to-files: overwrite| MD +
+++++ + * **Reconcile** (`CognitionReconciler`) - a bidirectional, idempotent *union* of files and pgvector across all six stores. Collections (facts, journals, skills, working-memory conversations) are keyed by a stable identity (content hash, date, name, conversation id), so re-running is a no-op. The singletons (persona, owner profile) are filled only when one side is empty, never clobbering a customized side. In `hybrid` mode it runs once at boot (toggle with `qlawkus.cognition.reconcile-at-start`, default `true`), so a fresh switch into hybrid backfills the missing mirror automatically; it is also exposed as `POST /api/admin/cognition/reconcile`. * **Migrate** (`CognitionMigrator`, `POST /api/admin/cognition/migrate?direction=files-to-pg|pg-to-files`) - an explicit one-directional copy that *does* overwrite the destination singletons. Use it to move a deployment between backends: build with the pgvector extension present, point the file roots at the source/target `.md` tree, and run a direction. Because the persona is always seeded in Postgres, reconcile never imports a markdown `soul.md` into pg on its own (it cannot tell a customized persona from the seed); to carry a markdown persona into Postgres, run an explicit `migrate?direction=files-to-pg`. -A direct `markdown` <-> `pgvector` move therefore goes through a pgvector-enabled build: boot once (a `hybrid` build reconciles automatically, or call the migrate endpoint), then rebuild on the target backend. +A direct `markdown` <-> `pgvector` move therefore goes through a pgvector-enabled build: boot once (a `hybrid` build reconciles automatically, or call the migrate endpoint), then rebuild on the target backend. Both operations are also driven from the xref:console.adoc[Console] cognition page, which only appears when the pgvector extension is baked in. See xref:memory.adoc[Memory] for how facts are retrieved and injected, xref:skills.adoc[Skills] for the procedural side, and the xref:config-reference.adoc[configuration reference] for every storage knob. diff --git a/docs/modules/ROOT/pages/voice.adoc b/docs/modules/ROOT/pages/voice.adoc index cc334d5f..881b0817 100644 --- a/docs/modules/ROOT/pages/voice.adoc +++ b/docs/modules/ROOT/pages/voice.adoc @@ -1,60 +1,69 @@ = Voice Setup (STT / TTS) +:page-title: Voice Setup +:page-description: Enable Speech-to-Text transcription and Text-to-Speech replies on the messaging adapters -Qlawkus supports voice interactions on messaging platforms (Discord, Telegram): Speech-to-Text (STT) for transcribing voice messages, and Text-to-Speech (TTS) for spoken replies. Voice works through file attachments — the bot does not join voice channels. +Qlawkus supports voice interactions on messaging platforms (Discord, Telegram): Speech-to-Text (STT) for transcribing voice messages, and Text-to-Speech (TTS) for spoken replies. Voice works through file attachments (the bot does not join voice channels). + +The `WHISPER_*` and `TTS_*` names below are convenience environment aliases wired in the reference `app/` distribution (`app/src/main/resources/application.properties`); they map onto the `qlawkus.messaging.transcription.*` and `qlawkus.messaging.tts.*` config properties. See xref:config-reference.adoc[Configuration] for the full property list and their auto-derived `QLAWKUS_MESSAGING_*` env names. == Speech-to-Text (STT) -Uses OpenAI-compatible Whisper API (Groq or OpenAI). +Uses an OpenAI-compatible Whisper API (Groq or OpenAI). Transcription is disabled until an API key is set. === Environment Variables [source,bash] ---- export WHISPER_API_KEY=your-groq-or-openai-key -export WHISPER_BASE_URL=https://api.groq.com/openai # or https://api.openai.com -export WHISPER_MODEL=whisper-large-v3-turbo # or whisper-1 for OpenAI +export WHISPER_BASE_URL=https://api.groq.com/openai # default: https://api.openai.com +export WHISPER_MODEL=whisper-large-v3-turbo # default: whisper-1 ---- -**Providers:** -* **Groq** (recommended): Fast, free tier available. Base URL: `https://api.groq.com/openai` -* **OpenAI**: Standard Whisper API. Base URL: `https://api.openai.com` +*Providers:* + +* *Groq* (recommended): Fast, free tier available. Base URL: `https://api.groq.com/openai` +* *OpenAI*: Standard Whisper API. Base URL: `https://api.openai.com` === How it works 1. User sends a voice message on Discord/Telegram -2. Audio is downloaded and sent to Whisper API +2. Audio is downloaded and sent to the Whisper API 3. Transcribed text is processed by the agent as a normal message -4. Agent response follows normal flow (text or voice) +4. The agent response follows the normal flow (text or voice) == Text-to-Speech (TTS) -Supports multiple providers per language. The agent selects the provider based on the requested language. +Providers are keyed by the language the agent requests: `TTS_EN_*` is the provider used for English replies, `TTS_PT_*` for Portuguese, and so on. When the agent requests a language with no matching provider, `TTS_DEFAULT_LANGUAGE` is used. -=== Environment Variables (per language) +=== Global Settings -Configure providers with language-specific prefixes: `TTS_EN_*` for English, `TTS_PT_*` for Portuguese, etc. +[source,bash] +---- +export TTS_ENABLED=true # default: false (voice requests fall back to text) +export TTS_DEFAULT_LANGUAGE=en # default: en +---- + +=== Provider Variables (per language) [source,bash] ---- -# English provider -export TTS_EN_KIND=openai # openai or elevenlabs -export TTS_EN_BASE_URL=https://api.openai.com +# English provider (kind defaults to openai) +export TTS_EN_BASE_URL=https://api.groq.com/openai export TTS_EN_API_KEY=your-key -export TTS_EN_MODEL=tts-1 # or orpheus-v1-english for local -export TTS_EN_VOICE=alloy # alloy, echo, fable, onyx, nova, shimmer -export TTS_EN_RESPONSE_FORMAT=mp3 -export TTS_EN_FALLBACK=pt # fallback to Portuguese provider if this fails +export TTS_EN_MODEL=canopylabs/orpheus-v1-english +export TTS_EN_VOICE=troy -# Portuguese provider +# Portuguese provider (ElevenLabs) export TTS_PT_KIND=elevenlabs export TTS_PT_BASE_URL=https://api.elevenlabs.io export TTS_PT_API_KEY=your-elevenlabs-key export TTS_PT_MODEL=eleven_multilingual_v2 export TTS_PT_VOICE=your-voice-id -export TTS_PT_RESPONSE_FORMAT=mp3 ---- -=== Provider Types +Each provider also accepts `kind`, `response-format` (default `mp3`), and `fallback` (another provider key to try on failure). Set them through the config property `qlawkus.messaging.tts.providers..` (env `QLAWKUS_MESSAGING_TTS_PROVIDERS__`). + +=== Provider Kinds [cols="1,2,3",options="header"] |=== @@ -64,32 +73,18 @@ export TTS_PT_RESPONSE_FORMAT=mp3 | `elevenlabs` | ElevenLabs API | `https://api.elevenlabs.io` |=== -=== Global Settings - -[source,bash] ----- -export TTS_ENABLED=true # enable/disable TTS globally -export TTS_DEFAULT_LANGUAGE=en # fallback language ----- - === How it works -1. Agent decides to respond with voice (via `respondWithVoice` tool) -2. Agent specifies language (e.g., `pt` for Portuguese) -3. Corresponding provider (`TTS_PT_*`) generates audio -4. Audio is sent as a voice message on the platform - -== Voice on messaging platforms - -Voice messages on Discord and Telegram work through file attachments — the bot does **not** join voice channels. When a user sends a voice message, it is downloaded as an audio attachment, transcribed via STT, processed, and the agent can respond with voice (also as a file attachment). - -The orchestrator also detects voice intent in text messages (e.g., "send audio", "reply by voice", "manda audio") and automatically triggers TTS for the response. +1. The agent decides to reply with voice by calling its `respondWithVoice` tool (its system prompt tells it to do so when you ask for audio, e.g. "responde em audio", "reply by voice") +2. The tool call carries the ISO 639-1 language code of the reply (e.g. `pt` or `en`) +3. The provider for that language (`TTS_PT_*`, `TTS_EN_*`, ...) synthesizes the audio +4. The audio is sent as a voice message on the platform (falling back to text if TTS is disabled or synthesis fails) == Testing Voice -1. Configure STT and at least one TTS provider +1. Configure STT and at least one TTS provider, and set `TTS_ENABLED=true` 2. Send a voice message to the bot on Discord/Telegram -3. Bot should transcribe, process, and reply (voice if agent uses the tool) +3. The bot should transcribe, process, and reply (with voice when the agent uses the tool) == Troubleshooting @@ -97,11 +92,11 @@ The orchestrator also detects voice intent in text messages (e.g., "send audio", |=== | Issue | Solution -| "Transcription disabled" | Set `WHISPER_API_KEY` -| "TTS provider not configured" | Set `TTS__*` for the language -| Voice message not playing | Ensure `TTS_ENABLED=true` and the correct `TTS__*` vars are set -| Wrong language voice | Agent must specify language in `respondWithVoice` tool call -| Audio format errors | Try `mp3` for `TTS_*_RESPONSE_FORMAT` +| Transcription does nothing | Set `WHISPER_API_KEY` (a blank key disables STT) +| Voice request answered as text | Set `TTS_ENABLED=true` and configure the provider for that language +| TTS provider not configured | Set `TTS__*` for the requested language, or point it at `TTS_DEFAULT_LANGUAGE` +| Wrong language voice | The agent must pass the right language code to `respondWithVoice` +| Audio format errors | Set the provider's `response-format` to `mp3` |=== See xref:config-reference.adoc[Configuration] for all voice-related environment variables. diff --git a/site/content/composition.adoc b/site/content/composition.adoc index 7b851585..bee55565 100644 --- a/site/content/composition.adoc +++ b/site/content/composition.adoc @@ -6,17 +6,20 @@ Qlawkus lets a single manifest, `agent.yml`, decide which capabilities are compo [NOTE] ==== -The manifest schema, the generator (`mvn qlawkus:generate`), the capability catalog, the `runtime` toggle tier, and dev-mode reload are all in place: the generator reconciles the pom against the capabilities the reactor's extensions declare, unknown names fail the build, the resolved set is reported, and the `runtime` block is applied at boot with an external override. Composing a capability in or out is a build-time concern by design (a rebuild, the same closed-world trade a native image makes); the `runtime` section is the part that changes without one. +Composing a capability in or out is a build-time concern by design: a rebuild, the same closed-world trade a native image makes. The `runtime` section is the only part that changes without one. ==== == Why a generator Quarkus resolves Maven dependencies *before* augmentation runs, and Maven does no tree-shaking, so a build step cannot add or drop dependencies for its own build. The only place a manifest can decide the dependencies is a step that runs *ahead* of the build. Qlawkus does this with a Maven plugin, the same idea as `quarkus extension add`: -[source,text] ----- -agent.yml -> mvn qlawkus:generate -> pom.xml -> mvn package (or quarkus build --native) ----- +++++ +
+graph LR + M["agent.yml
manifest"] -->|"mvn qlawkus:generate"| P["pom.xml"] + P -->|"mvn package / quarkus build --native"| A["agent artifact"] +
+++++ A deselected capability is left out of the pom entirely, so the result is lean by construction - not by relying on native-image dead-code elimination. @@ -140,7 +143,7 @@ All three require authentication. The staged manifest lives in the app's own sta == The configuration editor -The manifest above decides *which capabilities* exist. The same admin surface also lets you edit *any documented property value* - `qlawkus.*` in full, plus a small curated allowlist of `quarkus.*` properties - from `/console/config` (when the optional `console` capability is composed in) or directly over the API. The form is generated from the same `quarkus-config-doc` metadata that builds xref:config-reference.adoc[the config reference], so there is no separate list to keep in sync, and secret properties (API keys, password hashes, tokens) never appear here - they stay in xref:secrets.adoc[the keystore]. +The manifest above decides *which capabilities* exist. The same admin surface also lets you edit *any documented property value* - `qlawkus.*` in full, plus a small curated allowlist of `quarkus.*` properties - over the API, or from the xref:console.adoc[console] at `/console/config` when the optional `console` capability is composed in. The form is generated from the same `quarkus-config-doc` metadata that builds xref:config-reference.adoc[the config reference], so there is no separate list to keep in sync, and secret properties (API keys, password hashes, tokens) never appear here - they stay in xref:secrets.adoc[the keystore]. Each property is edited through one of two tiers, depending on its `ConfigPhase`: @@ -148,15 +151,13 @@ Each property is edited through one of two tiers, depending on its `ConfigPhase` |=== |Phase |Applies |Endpoint -|`RUN_TIME` |Live, on the next restart |`PUT`/`DELETE /api/admin/runtime-toggles` - writes straight into the runtime-toggle override file described above. +|`RUN_TIME` |On the next restart |`PUT`/`DELETE /api/admin/runtime-toggles` - writes straight into the runtime-toggle override file described above. |`BUILD_TIME` / `BUILD_AND_RUN_TIME_FIXED` |Next rebuild |`PUT`/`POST`/`DELETE /api/admin/config-overrides` - stages a `.properties` document, the same stage-then-promote pattern as the manifest. |=== A `BUILD_TIME` write is validated (documented, in-scope, not a secret) and staged only - never applied directly - exactly like the manifest. The staged document lives in its own file, `config-overrides.properties`, parallel to `agent.yml` rather than merged into it, so promoting it can overwrite the file wholesale without ever touching a hand-written line in `application.properties`. -== The schedule page - -`/console/schedule` lists the agent's five background jobs - memory review, memory curation, episodic consolidation, skill curation, and skill lifecycle - sourced from the live Quarkus `Scheduler` rather than a hand-maintained list, so cron and previous/next fire time always match what is actually registered. Editing a job's cron reuses the same `RUN_TIME` tier as the configuration editor above (`PUT`/`DELETE /api/admin/runtime-toggles`); triggering a job immediately hits its existing manual-trigger endpoint (`POST /api/admin/memory/{review,curate,consolidate}`, `POST /api/admin/skills/{curate,lifecycle}`). +The console surfaces the same two tiers as a form, and adds a xref:console.adoc#_management_pages[schedule page] that drives the background jobs. Both are just front-ends over these endpoints. == The restart contract diff --git a/site/content/google-workspace.adoc b/site/content/google-workspace.adoc index 6cef02d6..0557f876 100644 --- a/site/content/google-workspace.adoc +++ b/site/content/google-workspace.adoc @@ -77,7 +77,7 @@ export QLAWKUS_GOOGLE_STORAGE_ENABLED=true == 6. Credential Vault (Optional but Recommended) -By default, Qlawkus stores OAuth tokens in an encrypted PostgreSQL table using AES-256 + Argon2id. Enable with: +The credential vault persists the Google refresh token so authorization survives restarts. It is off by default in the bundled `app`; enable it with a passphrase: [source,bash] ---- @@ -85,15 +85,36 @@ export QLAWKUS_VAULT_ENABLED=true export QLAWKUS_VAULT_PASSPHRASE=your-strong-passphrase ---- -If disabled or no passphrase provided, tokens are kept in memory and lost on restart. +When enabled with a passphrase, the refresh token is encrypted with AES-256-GCM (the key is derived from the passphrase via Argon2id) and stored in the `qlawkus_google_auth` datasource. If the vault is disabled or no passphrase is set, tokens are kept in memory and lost on restart. Treat the passphrase as a secret: see xref:secrets.adoc[Secrets] to store it in the keystore under `qlawkus.google.vault.encryption-passphrase` instead of a plain environment variable. == 7. First Authorization Flow +Authorization uses the OAuth 2.0 authorization-code (loopback) flow. The agent never sees your Google password: it hands you a consent URL, and Google redirects a one-time code back to the callback, which the agent exchanges for a refresh token. + 1. Start the agent (dev mode or Docker) 2. Send a message that requires Google access (e.g., "List my emails") -3. The agent will return an authorization URL -4. Open the URL in a browser, sign in with your Google account, and grant permissions -5. You'll be redirected back to the agent, which stores the tokens +3. The agent calls its `startGoogleAuthorization` tool and returns an authorization URL +4. Open the URL in a browser, sign in with your Google account, and grant the requested scopes +5. Google redirects to `/api/google/oauth/callback`; the agent exchanges the code for a refresh token, stores it (see the vault above), and confirms via `checkGoogleAuthorization` +6. Ask again - the agent retries the original request automatically + +++++ +
+sequenceDiagram + participant U as You + participant A as Agent + participant G as Google + U->>A: "List my emails" + A-->>U: authorization URL (startGoogleAuthorization) + U->>G: open URL, sign in, grant scopes + G-->>A: redirect to /api/google/oauth/callback?code + A->>G: exchange code for refresh token + A->>A: store refresh token in the vault + A-->>U: authorized - retry your request +
+++++ + +The xref:console.adoc[Console] onboarding wizard can run this same authorization from a web page instead of chat. == Available tools diff --git a/site/content/index.adoc b/site/content/index.adoc index 0af7c7d4..582261d1 100644 --- a/site/content/index.adoc +++ b/site/content/index.adoc @@ -12,14 +12,11 @@ Inspired by projects like https://github.com/openclaw/openclaw[OpenClaw] and htt Most personal AI agents store memory as markdown files on disk. The model decides when to read them, which means it can forget to check. Qlawkus takes a different approach: **memory is injected into the prompt automatically, so it never depends on the model deciding to look - and the agent can still search it on demand** when it needs to dig deeper. -Every turn, the agent receives: +Every turn, the agent receives its own `Soul` (identity, mood, behavioral bias) and a curated `UserProfile` of the person it serves, plus the facts relevant to the current message - retrieved by a vector search that runs **before each reply**, not by a tool the model has to remember to call. -* A `Soul` - the agent's own identity, mood, and behavioral bias -* A `UserProfile` - a coherent, curated record of the human it serves +Facts are tagged by source (`semantic-extractor`, `remember-tool`, `episodic-consolidator`, `transcript`) and kept clean by background jobs: nightly deduplication merges near-duplicates, and a curation job distills scattered facts into the single `UserProfile`. Transcripts are embedded too, so the agent can semantically search what was actually said. -Below the surface, a https://github.com/pgvector/pgvector[pgvector]-backed fact store holds granular memories tagged by source (`semantic-extractor`, `remember-tool`, `episodic-consolidator`, `transcript`). A retrieval step runs **before each reply**, embedding the user's query and injecting the most relevant facts into the prompt. No tool call required. No model cooperation needed. - -Background jobs keep the store clean: nightly deduplication merges near-duplicates, and a curation job periodically distills scattered facts into a single, coherent `UserProfile`. Transcripts are embedded too, so the agent can semantically search what was actually said. +The store itself is pluggable, chosen at build time. Leave the Postgres extension out and the agent is genuinely **database-free** - markdown files plus an in-process embedding index (the default whenever no database backend is registered). Add https://github.com/pgvector/pgvector[pgvector] to keep the embeddings in a durable vector store instead; the bundled `app` ships the `hybrid` backend, keeping the markdown files as the source of truth and mirroring them into Postgres. The retrieval model is identical in every case. The result: an agent that always knows who it is, who its owner is, and what it has learned - regardless of channel, context length, or the model's willingness to call tools. @@ -29,7 +26,7 @@ Cognition:: A persistent `Soul` (identity, mood, focus) that shapes the agent's behavior every turn. Memory:: -pgvector-backed long-term memory with automatic retrieval, background curation, and transcript search. Always in context, never missed. +Long-term memory with automatic retrieval, background curation, and transcript search - always in context, never missed. File-based and database-free by default, with an optional pgvector backend for durable embeddings. Skills:: Procedural memory the agent writes, uses, and curates itself: reusable `SKILL.md` how-to procedures, injected as an index each turn and loaded on demand. @@ -40,6 +37,9 @@ Discord and Telegram adapters with voice in and out. Slack and WhatsApp adapters Tools:: Google Workspace (Gmail, Calendar, Drive, Sheets, Storage), a brag-document tool, and a sandboxed shell. +Console:: +An optional server-rendered admin UI at `/console`: a first-run setup wizard plus memory, skills, cognition, configuration, and schedule management. No Node build, no separate frontend. + Resilience:: A primary LLM with an Ollama fallback behind a circuit breaker. @@ -51,6 +51,7 @@ xref:quickstart.adoc[Quick Start] - Get a working agent running in minutes. * xref:architecture.adoc[Architecture] - How the modules fit together * xref:composition.adoc[Composition] - Let agent.yml decide which capabilities are built in +* xref:console.adoc[Console] - The optional server-rendered admin UI * xref:skills.adoc[Skills] - Procedural memory the agent writes and curates itself * xref:google-workspace.adoc[Google Workspace] - Gmail, Calendar, Drive, Sheets, Storage * xref:messaging.adoc[Messaging] - Discord and Telegram bots with voice diff --git a/site/content/memory.adoc b/site/content/memory.adoc index bfd7211f..c274b4de 100644 --- a/site/content/memory.adoc +++ b/site/content/memory.adoc @@ -10,7 +10,7 @@ Most agents handle long-term memory in one of two weak ways: they make recall *d * *Injected, not tool-dependent.* Relevant memory is retrieved and added to the prompt automatically before every reply. Recall never depends on the model choosing to search - reactive-only recall is unreliable, and the most important memory is the kind the model does not know it should look for. * *Retrieved, not wholesale.* Active memory uses Retrieval-Augmented Generation (RAG): each turn, only the facts relevant to the current message are selected (query-relevant top-K) and injected. The full store is never dumped into the prompt, so the per-turn token cost stays roughly constant as memory grows. -* *Searchable on top, not instead.* None of this removes deliberate search: the `searchTranscripts` tool lets the model query the full transcript archive on demand when it needs to reach past what was injected. Automatic injection guarantees recall; tool search is a complement, never the thing recall depends on. +* *Searchable on top, not instead.* None of this removes deliberate search. Two tools let the model reach past what was injected: `searchMemories` queries the curated long-term facts, and `searchTranscripts` queries the raw transcript archive (verbatim exchanges, finer-grained than distilled facts). Automatic injection guarantees recall; tool search is a complement, never the thing recall depends on. The result is recall that is both *reliable* (always runs) and *bounded* (only what is relevant), with on-demand search still available - instead of trading those off against each other. @@ -44,20 +44,24 @@ The agent maintains its own memory so it does not rot: |=== |Job |Schedule |What it does -|Episodic consolidation |nightly |Summarizes the day into a journal that active memory can surface. +|Episodic consolidation |nightly (`POST /api/admin/memory/consolidate`) |Summarizes the day into a journal that active memory can surface. |Memory review |nightly (`POST /api/admin/memory/review`) |Removes semantic near-duplicate facts. |Profile curation |nightly (`POST /api/admin/memory/curate`) |Folds facts into the owner profile, reconciling contradictions. Facts themselves are left untouched. |=== == Administration -* `GET /api/admin/memory` - summary of what is stored. -* `DELETE /api/admin/memory` - purge (all, or by source). -* `POST /api/admin/memory/review` and `POST /api/admin/memory/curate` - trigger the curation jobs manually. +All endpoints are `@Authenticated` and sit under `/api/admin/memory`: + +* `GET /api/admin/memory` - summary of what is stored; `GET /api/admin/memory/journals` lists the episodic journals. +* `DELETE /api/admin/memory` - purge, scoped by query parameter: `?all=true`, `?source=` (one of the source tags above), or `?includeJournals=true`. +* `POST /api/admin/memory/{review,curate,consolidate}` - trigger the three background jobs manually. + +The same actions are available from the xref:console.adoc[Console] admin UI. == Storage backends -Memory persistence is selected at build time with `qlawkus.cognition.backend` (`markdown`, `pgvector` or `hybrid`), the same switch that governs every cognition store. Whichever backend is active, fact retrieval is the same query-relevant top-K RAG path described above - the choice trades off persistence and scale, never the recall model. +Memory persistence is selected at build time with `qlawkus.cognition.backend` (`markdown`, `pgvector` or `hybrid`), the same switch that governs every cognition store. Whichever backend is active, fact retrieval is the same query-relevant top-K RAG path described above - the choice is about how embeddings are persisted (rebuilt in-process each boot versus stored durably in Postgres), never the recall model. The full treatment - the backend table, the optional `qlawkus-cognition-pgvector` extension and the database-free distribution, file locations, and how to reconcile or migrate data between markdown and pgvector - lives in xref:storage.adoc[Storage]. diff --git a/site/content/messaging.adoc b/site/content/messaging.adoc index 5a07694e..0d90d25a 100644 --- a/site/content/messaging.adoc +++ b/site/content/messaging.adoc @@ -1,29 +1,43 @@ = Messaging Setup (Discord & Telegram) +:page-title: Messaging Setup +:page-description: Configure the Discord and Telegram adapters, with voice, shared context, and message chunking -Qlawkus includes messaging adapters for Discord and Telegram with voice support. Slack and WhatsApp adapters exist but are not wired into the default `app/` module. +Qlawkus includes messaging adapters for Discord and Telegram with voice support. Slack and WhatsApp adapters exist but are not wired into the default `app/` module (only `qlawkus-messaging`, `qlawkus-messaging-discord`, and `qlawkus-messaging-telegram` are). -== How conversation context works +== Conversation context -When you chat with the agent via any channel (REST, Discord, Telegram), it maintains a working memory of the back-and-forth conversation. By default, all channels share the same conversation context (`QLAWKUS_AGENT_SHARED_CONTEXT_ENABLED=true`), meaning the agent is continuous across Discord, Telegram, and REST — the owner is one person regardless of channel. +When you chat with the agent via any channel (REST, Discord, Telegram), it keeps a working memory of the back-and-forth conversation. Two knobs govern that context, both on the `qlawkus.agent` root: -If the conversation goes idle for too long, the working memory is automatically cleared to prevent stale context from accumulating. The default idle timeout is 60 minutes. +*Shared context (default on).* All channels share the same conversation, so the agent stays continuous across Discord, Telegram, and REST (the owner is one person regardless of channel). Disable it to give each channel its own conversation: -You can also ask the agent to clear the context at any time. Say things like "clear the context", "reset", "start over", "new topic", or "forget our conversation" — the agent will wipe the short-term conversation and start fresh on the next message. This does **not** erase long-term saved facts about you (stored in pgvector); only the running back-and-forth is cleared. +[source,bash] +---- +export QLAWKUS_AGENT_SHARED_CONTEXT_ENABLED=false +---- + +*Idle reset (default 60 minutes).* If a conversation is idle for longer than the configured window, the working memory is cleared before the next message so stale context does not accumulate. Set the window to `0` to disable: + +[source,bash] +---- +export QLAWKUS_AGENT_CONTEXT_TTL_MINUTES=60 +---- + +You can also ask the agent to clear the context at any time. Say things like "clear the context", "reset", "start over", "new topic", or "forget our conversation", and the agent wipes the short-term conversation and starts fresh on the next message. This does *not* erase long-term saved facts about you; only the running back-and-forth is cleared. == Discord Bot === 1. Create a Discord Application 1. Go to https://discord.com/developers/applications -2. Click **New Application** > name it (e.g., "Qlawkus") -3. Go to **Bot** tab > **Add Bot** -4. **Public Bot:** Leave unchecked if only you will use it. When checked, anyone can install the bot. -5. Under **Privileged Gateway Intents**, enable **Message Content Intent** (required for the bot to read messages). Without this, the bot cannot see message text. -6. Go to **OAuth2** > **General** > copy **Application ID** +2. Click *New Application* > name it (e.g., "Qlawkus") +3. Go to *Bot* tab > *Add Bot* +4. *Public Bot:* Leave unchecked if only you will use it. When checked, anyone can install the bot. +5. Under *Privileged Gateway Intents*, enable *Message Content Intent* (required for the bot to read messages). Without this, the bot cannot see message text. +6. Go to *OAuth2* > *General* > copy *Application ID* === 2. Configure Permissions & Invite -1. Go to **OAuth2** > **URL Generator** +1. Go to *OAuth2* > *URL Generator* 2. Scopes: `bot` + `applications.commands` 3. Bot Permissions: * Send Messages @@ -45,7 +59,9 @@ export QLAWKUS_MESSAGING_DISCORD_RESPOND_TO_ALL_MESSAGES=true export QLAWKUS_MESSAGING_DISCORD_STARTUP_GREETING="👋 Qlawkus online and listening" ---- -Optional auth: +`QLAWKUS_MESSAGING_DISCORD_RESPOND_TO_ALL_MESSAGES` defaults to `true`. Set it to `false` to make the bot respond only to DMs and slash commands. + +Optional auth (restrict who may talk to the bot): [source,bash] ---- @@ -53,6 +69,8 @@ export QLAWKUS_MESSAGING_AUTH_ALLOWED_USERS_DISCORD=* # or comma-separated user IDs: 12345,67890 ---- +`*` (or leaving the list unset) allows everyone; a comma-separated list restricts to those user IDs. + === 4. Slash Commands The bot registers a `/qlawkus` command with subcommands: @@ -69,12 +87,12 @@ If you set `QLAWKUS_MESSAGING_DISCORD_GUILD_ID`, commands sync instantly to that 1. Open Telegram and message @BotFather 2. Send `/newbot` and follow the prompts -3. Copy the **Token** +3. Copy the *Token* === 2. Choose Mode: Polling vs Webhook -* **Polling** (default, simpler): Bot pulls updates. Works behind NAT, no public URL needed. -* **Webhook**: Telegram pushes updates to your HTTPS endpoint. Requires a public HTTPS URL. +* *Polling* (default, simpler): Bot pulls updates. Works behind NAT, no public URL needed. +* *Webhook*: Telegram pushes updates to your HTTPS endpoint. Requires a public HTTPS URL. === 3. Environment Variables @@ -94,48 +112,28 @@ Webhook: ---- export QLAWKUS_MESSAGING_TELEGRAM_BOT_TOKEN=your-bot-token export QLAWKUS_MESSAGING_TELEGRAM_MODE=webhook -export QLAWKUS_MESSAGING_TELEGRAM_WEBHOOK_URL=https://yourdomain.com/api/telegram/webhook +export QLAWKUS_MESSAGING_TELEGRAM_WEBHOOK_URL=https://yourdomain.com/api/webhook/telegram export QLAWKUS_MESSAGING_AUTH_ALLOWED_USERS_TELEGRAM=* ---- -The webhook path is `/api/telegram/webhook`. +The webhook path is `/api/webhook/telegram` (the Slack and WhatsApp adapters, when enabled, use `/api/webhook/slack` and `/api/webhook/whatsapp`). == Voice Support Voice messages are supported on both Discord and Telegram via file attachments (the bot does not join voice channels). When a user sends a voice message: -1. Audio attachment is downloaded -2. Audio is transcribed via STT (Whisper via Groq/OpenAI) -3. Agent processes the text -4. If TTS is configured, the agent can respond with voice (sent as an audio file attachment) - -Additionally, the orchestrator detects voice intent in text messages (e.g., "manda audio", "respond with voice") and automatically triggers TTS. +1. The audio attachment is downloaded +2. The audio is transcribed via STT (OpenAI-compatible Whisper, e.g. Groq or OpenAI) +3. The agent processes the text +4. If TTS is configured, the agent can reply with voice (sent as an audio file attachment) -For TTS setup, see xref:voice.adoc[Voice Setup]. +The agent decides to reply with voice by calling its `respondWithVoice` tool. Its system prompt tells it to do so whenever you ask for audio (phrases like "responde em audio", "me manda por voz", "reply by voice"). -== Shared Context Across Channels - -By default, the agent maintains a single conversation across REST, Discord, and Telegram. The owner is one person regardless of channel. - -To disable (separate conversations per channel): - -[source,bash] ----- -export QLAWKUS_AGENT_SHARED_CONTEXT_ENABLED=false ----- - -== Idle Conversation Reset - -If a conversation is idle for more than a configurable period, the working memory is cleared before the next message. This prevents stale context from accumulating. - -[source,bash] ----- -export QLAWKUS_MESSAGING_CONTEXT_TTL_MINUTES=60 ----- +For STT/TTS setup, see xref:voice.adoc[Voice Setup]. == Message Chunking -Long responses are automatically split to fit platform limits: +Long responses are automatically split to fit each platform's limit: [cols="1,1",options="header"] |=== @@ -155,7 +153,7 @@ Long responses are automatically split to fit platform limits: |Discord bot connects but doesn't respond |Message Content Intent not enabled in Developer Portal. |Slash commands not appearing |Wait up to 1 hour for global sync, or set `QLAWKUS_MESSAGING_DISCORD_GUILD_ID` for instant guild sync. -|Telegram webhook not receiving updates |Verify HTTPS URL is accessible, check logs for `setWebhook` errors. +|Telegram webhook not receiving updates |Verify the HTTPS URL is reachable and ends in `/api/webhook/telegram`; check logs for `setWebhook` errors. |Voice not working |Check STT/TTS config in xref:voice.adoc[Voice Setup]. |=== diff --git a/site/content/secrets.adoc b/site/content/secrets.adoc index a96cf382..9544b6ed 100644 --- a/site/content/secrets.adoc +++ b/site/content/secrets.adoc @@ -47,16 +47,16 @@ If you would rather not use `keytool`, the agent exposes a basic-auth gated admi [source,bash] ---- # Store a secret (alias = the config property it supplies) -curl -u "$API_USER:$API_USER_PASSWORD" -X PUT \ +curl -u "qlawkus:$ADMIN_PASSWORD" -X PUT \ http://localhost:8742/api/admin/secrets \ -H 'Content-Type: application/json' \ -d '{"alias":"quarkus.langchain4j.openai.\"primary\".api-key","value":""}' # List stored aliases (names only, never values) -curl -u "$API_USER:$API_USER_PASSWORD" http://localhost:8742/api/admin/secrets +curl -u "qlawkus:$ADMIN_PASSWORD" http://localhost:8742/api/admin/secrets # Remove a secret -curl -u "$API_USER:$API_USER_PASSWORD" -X DELETE \ +curl -u "qlawkus:$ADMIN_PASSWORD" -X DELETE \ "http://localhost:8742/api/admin/secrets?alias=quarkus.langchain4j.openai.%22primary%22.api-key" ---- diff --git a/site/content/skills.adoc b/site/content/skills.adoc index cfe6064a..8a8224f9 100644 --- a/site/content/skills.adoc +++ b/site/content/skills.adoc @@ -60,13 +60,26 @@ Skills grow and stay clean on their own, mirroring the fact-memory subsystem: == Lifecycle -Using a skill (`viewSkill`) marks it used. `SkillLifecycleJob` then ages unused skills by recency: - -`ACTIVE` -> `STALE` -> `ARCHIVED` - -* Archived skills leave the injected index but stay loadable by name, so nothing is lost. +Using a skill (`viewSkill`) marks it used. `SkillLifecycleJob` then ages unused skills by recency, and any use revives a skill straight back to `ACTIVE`: + +++++ +
+stateDiagram-v2 + [*] --> ACTIVE: created / distilled + ACTIVE --> STALE: unused 30 days + STALE --> ARCHIVED: unused 90 days + STALE --> ACTIVE: viewSkill + ARCHIVED --> ACTIVE: viewSkill + note right of ARCHIVED + Dropped from the injected + index, still loadable by name + end note +
+++++ + +* `STALE` skills are unused but still injected; `ARCHIVED` ones leave the injected index (to keep the prompt compact) but stay loadable by name, so nothing is lost. * Pinned skills never transition. Pin with `POST /api/admin/skills/{name}/pin`. -* Thresholds are config: `qlawkus.skills.lifecycle.stale-after-days`, `archive-after-days`, and `cron`. +* The day thresholds and the sweep cron are config: `qlawkus.skills.lifecycle.stale-after-days` (default 30), `archive-after-days` (default 90), and `cron`. Usage telemetry is stored where the backend lives: Postgres columns (`pgvector`/`hybrid`, migration `V7`) or a `.qlawkus-usage.json` sidecar next to the files (`markdown`). @@ -82,9 +95,9 @@ The cognition persistence backend is chosen at **build time** with `qlawkus.cogn |=== |Backend |Behavior -|`markdown` |`SKILL.md` files only, no database. The whole index is injected wholesale - the Hermes/OpenClaw model. -|`pgvector` |Skills in Postgres (the `skill` table, migration `V6`). The default; backward compatible. -|`hybrid` |`SKILL.md` files are the source of truth, mirrored into Postgres for ranking. +|`markdown` |`SKILL.md` files only, no database. This is the `@DefaultBean`, so it is what a build **without** the `cognition-pgvector` extension uses - a fully database-free distribution. +|`pgvector` |Skills in Postgres (the `skill` table, migration `V6`, lifecycle columns in `V7`). Durable server-side storage; requires the `cognition-pgvector` extension on the classpath. +|`hybrid` |`SKILL.md` files are the source of truth, mirrored into Postgres. This is what the reference `app` ships. |=== Because the choice is build-time fixed, a markdown-only distribution carries no datasource or Flyway dependency for skills at all. @@ -102,9 +115,11 @@ All endpoints are under `/api/admin/skills` (Basic auth, same as the rest of the |`DELETE /api/admin/skills/{name}` |Delete an owned skill. |`POST /api/admin/skills/curate` |Run curation now (dedup redundant skills). |`POST /api/admin/skills/lifecycle` |Run the lifecycle sweep now (age unused skills). -|`POST /api/admin/skills/{name}/pin` |Pin a skill so it never ages out. +|`POST /api/admin/skills/{name}/pin` |Pin a skill so it never ages out (`?pinned=false` to unpin). |=== +The optional xref:console.adoc[Console] admin UI drives all of these from a `/console/skills` page, so pinning, deleting, curating and running the lifecycle sweep are available without hand-rolling `curl`. + == Remote Skill Hub Everything above is *local* and offline. The optional `qlawkus-tools-skill-hub` extension adds the outward-facing registry client, so the agent can discover, install, and publish skills mid-conversation. It is a separate module: omit it and a locked-down distribution physically cannot reach a registry (stronger than a config flag). The default implementation is plain Java (`java.net.http`), native-friendly, with no external CLI; it is an overridable `@DefaultBean`. Qlawkus never hosts or manages a registry itself. @@ -124,15 +139,7 @@ Install and publish honor `qlawkus.skill-hub.approval-mode`: `yolo` acts immedia == Roadmap -Today a skill is procedural text. One extension is planned. - -=== Executable skills - -Because a skill is already a directory (`//SKILL.md`), it can carry sibling assets - bash or Python scripts the agent runs as part of the procedure. This is the Anthropic skill model (a `SKILL.md` plus assets). It is not wired up yet: the file and shell tools are confined to the *workspace* root, while skills live in their own isolated root, so neither writing nor running a script inside a skill folder works today. The plan is to author assets through the skill API (not the workspace file tool) and stage them into the workspace for execution, without loosening either confinement boundary. - -Note the overlap with in-process *dynamic tools* (e.g. runtime-compiled Groovy that registers a typed `@Tool`): both let the agent gain a self-authored capability. Bundled scripts win the common case - they run as a confined subprocess and work in native image - so they are the primary path for "executable skill". In-process dynamic tools remain a separate, niche axis for typed tools that need direct access to app internals (JVM-mode only, since GraalVM cannot load new bytecode at runtime). - -The remote Skill Hub above already ships search, install, and publish; a jskills-CLI backend (for hosts that already run the jskills tooling) remains a possible `@DefaultBean` override. +A skill is procedural *text* today. Because it already lives in its own directory (`//SKILL.md`), a planned extension is *executable* skills that carry sibling scripts the agent runs as part of the procedure (the Anthropic `SKILL.md`-plus-assets model). It is not wired up yet, since skills and the workspace file/shell tools live behind separate confinement boundaries. == What's next? diff --git a/site/content/storage.adoc b/site/content/storage.adoc index fe10b78a..113e4476 100644 --- a/site/content/storage.adoc +++ b/site/content/storage.adoc @@ -6,7 +6,7 @@ Qlawkus persists everything the agent knows behind a small set of store SPIs, wi == The stores -Six cognition stores (plus skills) sit behind SPIs in `qlawkus-client`, each with interchangeable backends: +Six stores sit behind SPIs in `qlawkus-client`, each with interchangeable backends: [cols="1,3",options="header"] |=== @@ -28,12 +28,12 @@ The backend is selected at build time with `qlawkus.cognition.backend`: |=== |Backend |Behavior -|`markdown` |Facts are `.md` files (editable, git-versionable) plus an in-process embedding index cached to JSON; journals are dated `.md` files; working memory is one append-only `.jsonl` per conversation; the persona is `soul.md` and the owner profile is `owner.md`. No database; single-node. This is the default whenever the pgvector extension is absent. -|`pgvector` |Facts are embeddings in Postgres; journals, skills, working memory, persona and owner profile are Postgres tables. Scales and is shared across instances. -|`hybrid` |`.md` files are the source of truth for the git-versionable knowledge (facts, dated journals, skills), mirrored into pgvector for retrieval and scale. The high-churn working memory and the live persona/profile singletons stay in Postgres. This is what the deployable `app` ships with. +|`markdown` |Facts are `.md` files (editable, git-versionable); their embeddings live in a langchain4j `InMemoryEmbeddingStore` that is rebuilt at boot, with a sibling `.embeddings.json` cache on disk so unchanged facts are not re-embedded. Journals are dated `.md` files; working memory is one append-only `.jsonl` per conversation; the persona is `soul.md` and the owner profile is `owner.md`. No database. This is the default whenever the pgvector extension is absent. +|`pgvector` |Facts are embeddings in Postgres; journals, skills, working memory, persona and owner profile are Postgres tables. The vectors are stored durably, so nothing is re-embedded at boot. +|`hybrid` |`.md` files are the source of truth for the git-versionable knowledge (facts, dated journals, skills), mirrored into pgvector so their embeddings persist durably. The high-churn working memory and the live persona/profile singletons stay in Postgres only. This is what the deployable `app` ships with. |=== -Whichever backend is active, fact retrieval is the same query-relevant top-K RAG path - the choice trades off persistence and scale, never the recall model. Journal text is always embedded through the fact store (`source=episodic-consolidator`), so journals stay searchable regardless of where the journal record itself lives. +Whichever backend is active, fact retrieval is the same query-relevant top-K RAG path - the choice is about how embeddings are persisted (rebuilt in-process each boot versus stored durably in Postgres), never the recall model. Journal text is always embedded through the fact store (`source=episodic-consolidator`), so journals stay searchable regardless of where the journal record itself lives. === Large facts are chunked @@ -68,11 +68,25 @@ In the Docker images these sit under the container `HOME` (`/home/default`), whi The backend is build-time fixed, so changing it means rebuilding. Because each cognition store keeps a parallel markdown representation alongside its Postgres form, Qlawkus can move data between the two - something a plain embedding store cannot do, since langchain4j's `EmbeddingStore`/`ChatMemoryStore` SPIs are isolated silos with no second representation to sync against. Both operations live in the `qlawkus-cognition-pgvector` extension, so they only run in a `pgvector` or `hybrid` build (a markdown-only build has no Postgres side in-process). +The two differ in direction and in whether they overwrite: + +++++ +
+graph LR + MD[("Markdown files")] + PG[("pgvector / Postgres")] + MD -->|reconcile: union, non-destructive| PG + PG -->|reconcile: union, non-destructive| MD + MD -.->|migrate direction=files-to-pg: overwrite| PG + PG -.->|migrate direction=pg-to-files: overwrite| MD +
+++++ + * **Reconcile** (`CognitionReconciler`) - a bidirectional, idempotent *union* of files and pgvector across all six stores. Collections (facts, journals, skills, working-memory conversations) are keyed by a stable identity (content hash, date, name, conversation id), so re-running is a no-op. The singletons (persona, owner profile) are filled only when one side is empty, never clobbering a customized side. In `hybrid` mode it runs once at boot (toggle with `qlawkus.cognition.reconcile-at-start`, default `true`), so a fresh switch into hybrid backfills the missing mirror automatically; it is also exposed as `POST /api/admin/cognition/reconcile`. * **Migrate** (`CognitionMigrator`, `POST /api/admin/cognition/migrate?direction=files-to-pg|pg-to-files`) - an explicit one-directional copy that *does* overwrite the destination singletons. Use it to move a deployment between backends: build with the pgvector extension present, point the file roots at the source/target `.md` tree, and run a direction. Because the persona is always seeded in Postgres, reconcile never imports a markdown `soul.md` into pg on its own (it cannot tell a customized persona from the seed); to carry a markdown persona into Postgres, run an explicit `migrate?direction=files-to-pg`. -A direct `markdown` <-> `pgvector` move therefore goes through a pgvector-enabled build: boot once (a `hybrid` build reconciles automatically, or call the migrate endpoint), then rebuild on the target backend. +A direct `markdown` <-> `pgvector` move therefore goes through a pgvector-enabled build: boot once (a `hybrid` build reconciles automatically, or call the migrate endpoint), then rebuild on the target backend. Both operations are also driven from the xref:console.adoc[Console] cognition page, which only appears when the pgvector extension is baked in. See xref:memory.adoc[Memory] for how facts are retrieved and injected, xref:skills.adoc[Skills] for the procedural side, and the xref:config-reference.adoc[configuration reference] for every storage knob. diff --git a/site/content/voice.adoc b/site/content/voice.adoc index cc334d5f..881b0817 100644 --- a/site/content/voice.adoc +++ b/site/content/voice.adoc @@ -1,60 +1,69 @@ = Voice Setup (STT / TTS) +:page-title: Voice Setup +:page-description: Enable Speech-to-Text transcription and Text-to-Speech replies on the messaging adapters -Qlawkus supports voice interactions on messaging platforms (Discord, Telegram): Speech-to-Text (STT) for transcribing voice messages, and Text-to-Speech (TTS) for spoken replies. Voice works through file attachments — the bot does not join voice channels. +Qlawkus supports voice interactions on messaging platforms (Discord, Telegram): Speech-to-Text (STT) for transcribing voice messages, and Text-to-Speech (TTS) for spoken replies. Voice works through file attachments (the bot does not join voice channels). + +The `WHISPER_*` and `TTS_*` names below are convenience environment aliases wired in the reference `app/` distribution (`app/src/main/resources/application.properties`); they map onto the `qlawkus.messaging.transcription.*` and `qlawkus.messaging.tts.*` config properties. See xref:config-reference.adoc[Configuration] for the full property list and their auto-derived `QLAWKUS_MESSAGING_*` env names. == Speech-to-Text (STT) -Uses OpenAI-compatible Whisper API (Groq or OpenAI). +Uses an OpenAI-compatible Whisper API (Groq or OpenAI). Transcription is disabled until an API key is set. === Environment Variables [source,bash] ---- export WHISPER_API_KEY=your-groq-or-openai-key -export WHISPER_BASE_URL=https://api.groq.com/openai # or https://api.openai.com -export WHISPER_MODEL=whisper-large-v3-turbo # or whisper-1 for OpenAI +export WHISPER_BASE_URL=https://api.groq.com/openai # default: https://api.openai.com +export WHISPER_MODEL=whisper-large-v3-turbo # default: whisper-1 ---- -**Providers:** -* **Groq** (recommended): Fast, free tier available. Base URL: `https://api.groq.com/openai` -* **OpenAI**: Standard Whisper API. Base URL: `https://api.openai.com` +*Providers:* + +* *Groq* (recommended): Fast, free tier available. Base URL: `https://api.groq.com/openai` +* *OpenAI*: Standard Whisper API. Base URL: `https://api.openai.com` === How it works 1. User sends a voice message on Discord/Telegram -2. Audio is downloaded and sent to Whisper API +2. Audio is downloaded and sent to the Whisper API 3. Transcribed text is processed by the agent as a normal message -4. Agent response follows normal flow (text or voice) +4. The agent response follows the normal flow (text or voice) == Text-to-Speech (TTS) -Supports multiple providers per language. The agent selects the provider based on the requested language. +Providers are keyed by the language the agent requests: `TTS_EN_*` is the provider used for English replies, `TTS_PT_*` for Portuguese, and so on. When the agent requests a language with no matching provider, `TTS_DEFAULT_LANGUAGE` is used. -=== Environment Variables (per language) +=== Global Settings -Configure providers with language-specific prefixes: `TTS_EN_*` for English, `TTS_PT_*` for Portuguese, etc. +[source,bash] +---- +export TTS_ENABLED=true # default: false (voice requests fall back to text) +export TTS_DEFAULT_LANGUAGE=en # default: en +---- + +=== Provider Variables (per language) [source,bash] ---- -# English provider -export TTS_EN_KIND=openai # openai or elevenlabs -export TTS_EN_BASE_URL=https://api.openai.com +# English provider (kind defaults to openai) +export TTS_EN_BASE_URL=https://api.groq.com/openai export TTS_EN_API_KEY=your-key -export TTS_EN_MODEL=tts-1 # or orpheus-v1-english for local -export TTS_EN_VOICE=alloy # alloy, echo, fable, onyx, nova, shimmer -export TTS_EN_RESPONSE_FORMAT=mp3 -export TTS_EN_FALLBACK=pt # fallback to Portuguese provider if this fails +export TTS_EN_MODEL=canopylabs/orpheus-v1-english +export TTS_EN_VOICE=troy -# Portuguese provider +# Portuguese provider (ElevenLabs) export TTS_PT_KIND=elevenlabs export TTS_PT_BASE_URL=https://api.elevenlabs.io export TTS_PT_API_KEY=your-elevenlabs-key export TTS_PT_MODEL=eleven_multilingual_v2 export TTS_PT_VOICE=your-voice-id -export TTS_PT_RESPONSE_FORMAT=mp3 ---- -=== Provider Types +Each provider also accepts `kind`, `response-format` (default `mp3`), and `fallback` (another provider key to try on failure). Set them through the config property `qlawkus.messaging.tts.providers..` (env `QLAWKUS_MESSAGING_TTS_PROVIDERS__`). + +=== Provider Kinds [cols="1,2,3",options="header"] |=== @@ -64,32 +73,18 @@ export TTS_PT_RESPONSE_FORMAT=mp3 | `elevenlabs` | ElevenLabs API | `https://api.elevenlabs.io` |=== -=== Global Settings - -[source,bash] ----- -export TTS_ENABLED=true # enable/disable TTS globally -export TTS_DEFAULT_LANGUAGE=en # fallback language ----- - === How it works -1. Agent decides to respond with voice (via `respondWithVoice` tool) -2. Agent specifies language (e.g., `pt` for Portuguese) -3. Corresponding provider (`TTS_PT_*`) generates audio -4. Audio is sent as a voice message on the platform - -== Voice on messaging platforms - -Voice messages on Discord and Telegram work through file attachments — the bot does **not** join voice channels. When a user sends a voice message, it is downloaded as an audio attachment, transcribed via STT, processed, and the agent can respond with voice (also as a file attachment). - -The orchestrator also detects voice intent in text messages (e.g., "send audio", "reply by voice", "manda audio") and automatically triggers TTS for the response. +1. The agent decides to reply with voice by calling its `respondWithVoice` tool (its system prompt tells it to do so when you ask for audio, e.g. "responde em audio", "reply by voice") +2. The tool call carries the ISO 639-1 language code of the reply (e.g. `pt` or `en`) +3. The provider for that language (`TTS_PT_*`, `TTS_EN_*`, ...) synthesizes the audio +4. The audio is sent as a voice message on the platform (falling back to text if TTS is disabled or synthesis fails) == Testing Voice -1. Configure STT and at least one TTS provider +1. Configure STT and at least one TTS provider, and set `TTS_ENABLED=true` 2. Send a voice message to the bot on Discord/Telegram -3. Bot should transcribe, process, and reply (voice if agent uses the tool) +3. The bot should transcribe, process, and reply (with voice when the agent uses the tool) == Troubleshooting @@ -97,11 +92,11 @@ The orchestrator also detects voice intent in text messages (e.g., "send audio", |=== | Issue | Solution -| "Transcription disabled" | Set `WHISPER_API_KEY` -| "TTS provider not configured" | Set `TTS__*` for the language -| Voice message not playing | Ensure `TTS_ENABLED=true` and the correct `TTS__*` vars are set -| Wrong language voice | Agent must specify language in `respondWithVoice` tool call -| Audio format errors | Try `mp3` for `TTS_*_RESPONSE_FORMAT` +| Transcription does nothing | Set `WHISPER_API_KEY` (a blank key disables STT) +| Voice request answered as text | Set `TTS_ENABLED=true` and configure the provider for that language +| TTS provider not configured | Set `TTS__*` for the requested language, or point it at `TTS_DEFAULT_LANGUAGE` +| Wrong language voice | The agent must pass the right language code to `respondWithVoice` +| Audio format errors | Set the provider's `response-format` to `mp3` |=== See xref:config-reference.adoc[Configuration] for all voice-related environment variables.