Skip to content

feat(agent): MCP-driven agent session path (parallel to chat) - #155

Open
esnunes wants to merge 4 commits into
mainfrom
2e1ba0db-c167-4c19-9176-566c2a27916f
Open

feat(agent): MCP-driven agent session path (parallel to chat)#155
esnunes wants to merge 4 commits into
mainfrom
2e1ba0db-c167-4c19-9176-566c2a27916f

Conversation

@esnunes

@esnunes esnunes commented May 20, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a new MCP-driven agent session path that runs alongside the existing chat path. Destila now operates as an MCP server reachable over HTTP+SSE on /mcp, authenticated by a global bearer token. A Go bridge in cmd/destila-mcp/ translates Claude Code's stdio MCP frames to Destila's HTTP shape, insulating Elixir from upstream MCP wire-protocol drift.

The chat path (WorkflowRunnerLive, Destila.AI.*, the 27 existing feature files, lib/destila/workflows/*) is fully untouched — this is a strict parallel rollout per the plan.

Implements docs/plans/2026-05-20-001-feat-mcp-driven-agent-sessions-plan.md end-to-end (U1–U10) plus follow-up fixes from a multi-agent code review.

What's new

  • HTTP+SSE MCP transport under /mcp with bearer auth (Plug.Crypto.secure_compare), case-insensitive Bearer prefix, RFC 7235 compliant
  • Go bridge at cmd/destila-mcp/ (greenfield Go subdir, separate release artifact)
  • New schemas + migration — agent_sessions, agent_session_events, plus a nullable agent_session_id FK on workflow_session_metadata so exports are shared. SQLite table-rebuild with explicit up/down. Application-level validate_exactly_one_session invariant + unique index on (agent_session_id, phase_name, key) for upsert
  • Orchestrator — SessionServer GenServer (per session), EventRouter, Registry + DynamicSupervisor wired into the application supervision tree
  • Four MCP tool handlers — session (phase_complete / suggest_phase_complete / export), ask_user_question (non-blocking, returns immediately with question_id), service (typed-error stub until ServiceManager is refactored), exports_read (cross-phase context recovery)
  • YAML workflow loader — priv/workflows/*.yaml cached in :persistent_term at boot, with a bundled example.yaml
  • Embedded host mode — launches `claude` in Destila.Terminal, writes per-session .mcp.json (chmod 0600) and system-prompt file
  • External host mode — surfaces connection info (bridge path, MCP URL, token, session id) and paste-target events
  • AgentSessionLive at /agent-sessions/:id — export-first layout, no chat textarea, exports panel + collapsible tool-call event log + embedded terminal or external-host panel + question panel + handoff modal
  • AgentSessionCreateLive at /agent-sessions/new — workflow + host-mode picker, optional project
  • Crafting board — additive "New agent-driven session" card next to the existing chat-path entry
  • features/mcp_driven_session.feature — 23 Gherkin scenarios + MockMCPClient test harness
  • scripts/mcp_smoke.sh + docs/mcp_smoke_test.md — manual transport smoke test

Test plan

  • mix test passes — 921 tests (37 new tests for schemas, sessions, server, router, tool handlers, auth plug, controller, LiveView render, create flow)
  • mix compile --warnings-as-errors clean
  • mix ecto.rollback + mix ecto.migrate round-trip cleanly (rebuild preserves chat-path rows on down)
  • Go bridge builds cleanly (go build ./cmd/destila-mcp/)
  • Walkthrough video captured at `docs/videos/mcp-driven-agent-session.mp4` showing crafting board → create form → external-host session detail page
  • Manual smoke test (scripts/mcp_smoke.sh) once DESTILA_MCP_TOKEN is set

Code review pass

Multi-agent review (correctness, security, data-migrations, reliability, project-standards, adversarial) surfaced and auto-fixed:

  • Security: constant-time bearer compare, 0600 file modes on .mcp.json + system-prompt
  • Correctness: strict {:ok, _} = matches replaced with case branches in every tool handler and SSE connect/close handler so Ecto errors surface as JSON-RPC envelopes instead of crashing the per-session GenServer
  • Reliability: SSE chunk-error handling, PubSub unsubscribe on every exit path, safe payload encoding, JSON-RPC §4.1 notifications no longer get responses, 60s tool-call timeout with try/catch, WorkflowLoader.load_all wrapped in try/rescue, LiveView falls back gracefully when SessionServer is idled
  • Data integrity: explicit migration up/down, unique index + on_conflict: {:replace, ...} on agent exports
  • Standards: Workflow.Phase extracted to its own file (no nested modules), String.to_atom removed from agent-input path

Residual follow-up (not in this PR)

  • EmbeddedHost.start_phase is not yet called by SessionServer — the embedded mode still needs the boot-phase wiring, and the resolved agent_command needs to be passed into Terminal.Server (which currently hard-codes `tmux attach`)
  • ask_user_question answers don't reach the agent yet — answer delivery via embedded stdin or external paste needs SessionServer.handle_call({:answer_question, ...}) to route by host_mode
  • ServiceTool returns a typed error stub — ServiceManager.execute/3 needs an agent-session entry point before this can be wired

These are design decisions intentionally scoped out of the auto-fix pass.

🤖 Generated with Claude Code

esnunes and others added 4 commits May 20, 2026 07:04
@esnunes @claude
docs(plans): add plan for MCP-driven agent sessions (parallel path)
6467386 
Captures the deep plan to pivot Destila to an MCP-server model where
the agent communicates only via tool calls, running alongside the
existing chat-based path. Front-loads the HTTP+SSE transport smoke
test as U1 to derisk Claude Code compatibility before investing in
schema, UI, and host-mode work.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@esnunes @claude
docs(plans): deepen MCP-driven agent sessions plan
c090b39 
Targeted deepening pass against the priority areas where the first pass
was thin:

- U1 transport: concrete bridge<->Destila HTTP shape (JSON-RPC methods,
  response envelopes, headers, SSE event format). Clarifies the bridge
  isolates upstream MCP wire-protocol drift from Destila.
- U2 schemas: confirmed SQLite via ecto_sqlite3 ~> 0.17. Replaced the
  table-level CHECK with changeset-level validation (CHECK on existing
  tables is awkward in SQLite ALTER); added three indexes for the new
  query shapes.
- U6 handoff race: concrete sub-state machine with 30s spawn-to-active
  gate, 5s soft-stop / 10s hard-kill escalation, FIFO buffered stdin
  flushed with 50ms inter-write delay, crash-mid-handoff handling.
- U10 mock harness: promoted MockMCPClient from sketch to a 10-function
  public API. Specified Mimic-stub registration in test_helper.exs.
- Scenario coverage audit: added the two scenarios the first pass left
  unlinked ("Session log records only tool-call events" in U3 tests,
  "User types directly into the embedded terminal" in U8 tests).

No implementation-unit boundaries changed; no U-IDs were renumbered.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@esnunes @claude
feat(agent): add MCP-driven agent session path (parallel to chat)
468550c 
Implements the full plan at docs/plans/2026-05-20-001-feat-mcp-driven-agent-sessions-plan.md.

The new agent path runs alongside the existing chat path (untouched).
Destila now operates as an MCP server reachable over HTTP+SSE on /mcp,
authenticated by a global bearer token from DESTILA_MCP_TOKEN. A new Go
bridge in cmd/destila-mcp/ translates Claude Code's stdio MCP frames to
Destila's HTTP shape, insulating the Elixir code from upstream MCP wire
protocol drift.

The new path:
- adds agent_sessions and agent_session_events tables, plus a nullable
  agent_session_id FK on workflow_session_metadata so exports are shared
- introduces Destila.Agent.* modules: schemas, context, session GenServer,
  event router, four tool handlers (session/ask_user_question/service/
  exports_read), embedded + external host modes, YAML workflow loader
- mounts /mcp routes (POST rpc + GET events SSE) under a new pipeline
- adds AgentSessionLive (export-first UI, no chat textarea) and
  AgentSessionCreateLive at /agent-sessions and /agent-sessions/new
- adds a "New agent-driven session" card to the crafting board next to
  the existing chat-path entry
- adds features/mcp_driven_session.feature, the MockMCPClient harness,
  scripts/mcp_smoke.sh for manual smoke tests, and docs/mcp_smoke_test.md
- 37 new tests cover schemas, sessions, server, router, tool handlers,
  auth plug, controller, LiveView render, and create flow

The chat path (WorkflowRunnerLive, Destila.AI.*, the 27 existing feature
files, lib/destila/workflows/*) is fully untouched; existing 921-test
suite still passes.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@esnunes @claude
fix(agent): address P1/P2 findings from multi-agent code review
4c5bfd8 
Auto-applied fixes from the ce-code-review pass over commit 468550c.

Security / auth:
- MCP bearer token compared with Plug.Crypto.secure_compare to close
  the timing-side-channel on a global, long-lived credential. Bearer
  prefix is now case-insensitive (RFC 7235) and missing-token config
  is rejected as 401 instead of raising 500.
- Per-session tmpdir under /tmp now chmod 0o700; .mcp.json and the
  system-prompt file written 0o600 so the token isn't readable by
  other local users on shared hosts.
- ExternalHost.connection_info and McpConfigWriter no longer crash
  the LiveView render or terminal spawn when :mcp_token is missing.

Correctness / reliability:
- Replace strict {:ok, _} = ... matches in session, ask_user_question,
  and exports_read tool handlers with case branches so Ecto errors
  surface as JSON-RPC error envelopes instead of crashing the
  per-session GenServer.
- SessionServer.mark_connected/disconnected no longer crashes the
  GenServer when update_session returns {:error, _}.
- handle_tool_call wraps GenServer.call in try/catch with an explicit
  60s timeout so a slow tool returns -32603 instead of a 500.
- AgentSessionLive answers a question against an idled-out SessionServer
  with a flash instead of a MatchError crash.
- SSE controller: handle MatchError on first chunk write, unsubscribe
  on every exit path, safely encode arbitrary payloads via Jason.encode/1,
  drop unused after timeout from 60min to 5min.
- EventRouter no longer returns a JSON-RPC error envelope for a
  notification request (per JSON-RPC 2.0 §4.1).
- WorkflowLoader.load_all/0 now boots inside try/rescue so a malformed
  YAML file is logged loudly instead of silently degrading the app.

Data / migration:
- Migration converted to explicit up/down. Down preserves chat-path
  rows and drops the agent-only ones explicitly (with NOT NULL
  workflow_session_id documented as a destructive caveat).
- Add unique_index on (agent_session_id, phase_name, key) so
  Sessions.record_export/2 can use on_conflict {:replace, ...} to
  upsert instead of duplicating rows on retry.

Project standards:
- Extract Destila.Agent.Workflow.Phase to its own file (CLAUDE.md:
  never nest modules).
- Replace String.to_atom(action) in service_tool.ex with a stub that
  returns a typed error (CLAUDE.md: never String.to_atom on user input;
  ServiceManager.execute signature is incompatible — wiring deferred).
- Fix mcp_driven_session.feature scenario tag to match an existing
  scenario name.

Test isolation:
- auth_plug_test.exs setup now restores the previous :mcp_token value
  instead of deleting it, so later LiveView tests that read the env
  via ExternalHost don't break.

Remaining residual work (not in this commit): EmbeddedHost wiring is
still incomplete — SessionServer never calls start_phase, the terminal
command isn't passed to Terminal.Server, and ask_user_question answers
aren't routed back to the agent. These need design decisions beyond
auto-fix scope.

All 921 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@esnunes