Data Analysis Agent v0.1 — Phase 1: CSV Upload + Question → Answer + Chart#90
Closed
madhyamakist wants to merge 55 commits into
Closed
Data Analysis Agent v0.1 — Phase 1: CSV Upload + Question → Answer + Chart#90madhyamakist wants to merge 55 commits into
madhyamakist wants to merge 55 commits into
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace google-genai dep with openai SDK - Add OpenRouterLLMProvider using openai.OpenAI(base_url=openrouter) - Rename DATAANALYSIS_GEMINI_API_KEY → DATAANALYSIS_OPENROUTER_API_KEY - Default model: google/gemini-2.5-flash (via OpenRouter) - All 16 tests pass Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add LLMResult dataclass (text + input/output/total tokens + cost) - OpenRouterLLMProvider extracts usage from response; estimates cost via a per-model pricing table (labelled "approx." in the UI) - StubLLMProvider returns zeros (shown as "$0.000000 (stub)") - Add 5 new columns to query_records: input/output/total_tokens, estimated_cost_usd, api_request_count + Alembic migration - Answer page shows a Usage table; history page shows token/cost inline Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New design system in base.html: CSS variables, stat pills, upload zone, breadcrumbs, action bars, proper button variants - Home: hero layout with feature highlights and styled upload zone - Dataset: header with column chips and history shortcut - Answer: answer body block, stat pills for token/cost usage - History: timeline cards with inline usage badges, empty state - Error: centered friendly error page - Fix stub banner copy (Gemini → OpenRouter) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The agent now generates SQL queries against the full CSV data, executes them in an in-memory SQLite DB, and repeats until the LLM signals a final answer — instead of guessing from a 5-row sample. - load_data: loads full CSV into in-memory SQLite (keyed by run_id) - plan_query: LLM generates next SQL or emits "FINAL ANSWER: ..." - execute_query: validates SELECT, runs SQL, appends result to history - Iterates until FINAL ANSWER or max_agent_iterations (default 10) - Accumulates token usage across all LLM calls per query - Adds iteration_count to QueryRecord (new migration) - Spec updated: 07-agent-graph.md + capabilities/02-nl-query-iterative.md - All 16 tests pass Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Lesson learned from phase-3 redesign: the original pipeline passed a 5-row CSV sample to the LLM for a single-shot answer — correct in structure but wrong in practice for any dataset larger than the sample. Changes: - ai-agents.md: add Rule #9 (data-access agents must use ReAct loop, never sample-and-guess) and Section 10 (full design pattern: loop shape, termination protocol, max iterations guard, self-correction on action errors, in-memory cache pattern, pre-coding checklist) - phases.md: add Phase 2 gate item #8 — stub must simulate ≥2 iterations (action + FINAL ANSWER) to prove the loop works, not just the nodes - 01-vision.md: update to reflect actual built state (OpenRouter, ReAct loop, inline history, session management) - 07-agent-graph.md: add mandatory six-question pre-coding checklist at the top of the file, answered for this project Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… entity Core concept: each data source registers Tools with ToolCapabilities in SQLite. The agent loop loads the tool registry at runtime and dispatches actions generically — SQL query on CSV is the first tool type; API calls, GraphQL, and shell execution slot in without changing the loop. Changes: - 01-vision: update to reflect data sources + sessions as primary entities; tool registry pattern explained - 02-architecture: add Tool Registry layer to component map; update data flow - 04-data-model: add DataSource, Tool, ToolCapability entities; rename Dataset→DataSource; add Session entity (between DataSource and QueryRecord); document bootstrap sequence (upload → 3 records created atomically) - 05-api: update endpoints to /datasources/* and /sessions/*; add session create/delete endpoints - 06-ui: redesign screens — home is now Data Sources list; add DataSource detail (sessions list); rename dataset page to session chat page - 07-agent-graph: update state to include tools list; rename plan_query/ execute_query to plan_action/execute_action; add tool prompt format section; update termination to handle JSON tool calls not bare SQL - capabilities/: update index; rename/rewrite 01 as connect-csv; update 02 to tool-agnostic ReAct loop; add 03-sessions capability Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pass)
Redesign from monolithic dataset/query pipeline to a modular tool registry
architecture where data sources register executable tools with typed capabilities.
Data model:
- Replace DatasetRow with DataSourceRow, ToolRow, ToolCapabilityRow, SessionRow
- QueryRecordRow now links to Session (not Dataset)
- CSV upload atomically creates DataSource + Tool (csv_query) + ToolCapability (run_query)
- Migration e2c8b5a1d9f3 handles schema transition; batch mode added to alembic env
Agent graph:
- plan_query/execute_query → plan_action/execute_action
- load_data now fetches Tool registry from DB and passes it via state["tools"]
- plan_action builds prompt with tool descriptions; LLM emits JSON tool calls
- execute_action dispatches by tool type (csv_query dispatches to in-memory SQLite)
- Self-correction and max-iterations guard unchanged
Stub provider:
- Returns JSON tool call on first call: {"capability": "run_query", "parameters": {...}}
- Returns FINAL ANSWER on second call (validates 2-iteration loop)
API routes:
- /datasources/upload, /datasources/{id}, /sessions/{id}, /sessions/{id}/query
- Old /upload and /datasets/* routes removed
- New routes.py replaces datasets.py
Templates:
- home.html: Data Sources list + upload form
- datasource.html (new): sessions list per data source
- session.html (new): inline Q&A chat page with loading overlay + reasoning trace
Tests (21 passed):
- Golden path: upload → create session → query → answer inline
- Pipeline: full ReAct loop with tool registry fixture
- Unit: new domain models (DataSource, Tool, ToolCapability, Session)
- Unit: new DB models with FK chain
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…/22 tests)
DataModel:
- SessionRow loses data_source_id column
- New SessionDataSourceRow join table (session_id, data_source_id composite PK)
- Session can now span multiple data sources; each brings its own tools
- Migration c7d4e3f2a1b8 creates join table, migrates existing FK, drops old column
Agent graph:
- AgentState drops data_source_id and csv_path fields
- load_data fetches ALL data sources for the session via join table, loads each
CSV into its own table in a shared in-memory SQLite (table name = sanitized filename)
- run_pipeline signature simplified: only session_id + question needed
API:
- POST /datasources/{id}/sessions removed; replaced by POST /sessions
- POST /sessions accepts list of data_source_ids (checkbox multi-select)
- DELETE /sessions/{id} redirects to / (home) instead of datasource detail
UI (home.html):
- Two-column layout: Data Sources (left) + Sessions (right)
- Both are first-level — no hierarchy between them
- New Session form expands inline with checkbox list of all data sources
- Data Source cards have inline delete confirmation
- Session cards show source chips, query count, Open + Delete actions
- session.html header shows chips for all attached data sources
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix _build_plan_prompt hardcoding "Table: data" — now shows actual
table names derived from filenames (e.g. global_tech_stocks)
- Fix _load_tool_registry to include data_source_id in each tool dict
so load_data patches tool configs correctly for multi-source sessions
- Fix stub provider hardcoding FROM data — now parses real table name
from prompt
- Add 60s timeout to OpenRouter API calls to prevent infinite hangs
- Run pipeline in background thread so POST /query returns 303
immediately instead of blocking for the full LLM execution time
- Add GET /sessions/{id}/query/{qr_id}/status polling endpoint
- Session page auto-shows loading overlay and polls every 2s; uses
window.location.reload() on completion to avoid same-URL no-op bug
- Add Cache-Control: no-store on session page to prevent browser
caching stale pending state
- Add structured JSON file logging via structlog with RotatingFileHandler;
configurable via DATAANALYSIS_LOG_FILE and DATAANALYSIS_LOG_LEVEL
- Update .gitignore to exclude *.db, uploads/, logs/, and test caches
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… + UI form Deployment pipeline: - Dockerfile: uv-based two-phase build (deps layer cached separately from src) - docker-entrypoint.sh: runs alembic migrations then starts uvicorn - .dockerignore: excludes tests, .env, runtime data, .git from image - .github/workflows/ci.yml: runs pytest on every branch/PR with stub LLM - .github/workflows/deploy.yml: test → build → push to GHCR → trigger Render hook - render.yaml: Render Blueprint for web service with 1 GB persistent disk Agent + UI fixes (from previous session): - nodes.py: self-correcting ReAct loop; STDDEV/VARIANCE custom SQLite aggregates; schema grouped by real table names (fixes "no such table: data" infinite loop) - session.html: form uses onsubmit instead of onclick to avoid browser submit cancellation; Cache-Control: no-store added to prevent reload loops Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s (Section 11) Reconciles learnings from the data-analysis-agent ReAct implementation: - Section 10 (ReAct loop): expanded self-correction rules — unknown capability, malformed JSON, and non-SELECT SQL are now classified as recoverable (feed back and loop) rather than fatal; only infra failures are fatal. Pre-coding checklist gains three new mandatory questions about tool inventory, invocation format, and registration triggers. - Section 11 (new): Tool Registry and Capability Dispatch — defines the Tool abstraction (anatomy, categories, registry pattern), the LLM invocation format, capability dispatch by tool type with recoverable-error handling, how to prompt the LLM with available tools including multi-source sessions, and non-negotiable security boundaries per tool category. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces db/database.py — a self-contained Database class whose only public method is execute(sql, params) -> list[dict]. All engine creation, connection pooling, and session management are hidden inside the class. session.py is now a thin bridge: get_session() and create_db_session() delegate to Database internally so all existing callers stay unchanged. Switch to PostgreSQL by setting: DATAANALYSIS_DATABASE_URL=postgresql+psycopg2://user:pass@host:5432/db uv pip install data-analysis-agent[postgres] Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On upload, FileIngester converts the raw file to Parquet (Snappy-compressed)
and extracts per-column dtype + nullability metadata. The master DB stores:
- file_path — original upload preserved as-is
- parquet_path — uploads/parquet/<uuid>.parquet
- schema_json — [{name, dtype, nullable}] per column
- column_names_json — backward-compat list of names
The agent's load_data node reads from Parquet (dtype-preserving, faster)
with a CSV fallback for pre-migration uploads. Delete cleans up both files.
Supports .csv, .xlsx, .xls, .json uploads. pyarrow added as core dep.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…disk FileIngester gains ingest_stream() which reads from the upload stream into memory and writes only the Parquet output — the raw CSV/XLSX/JSON never touches the filesystem. upload_csv() calls ingest_stream() instead of saving the raw file first. Delete route no longer needs to clean up a raw file. shutil and pandas imports removed from routes.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After Parquet conversion, _generate_tool_descriptions() prompts the LLM
with the filename, column schema (name + dtype + nullable), row count, and
a 5-row sample. The model returns {"tool_description", "capability_description"}
which are stored in ToolRow and ToolCapabilityRow respectively.
Strips markdown fences before JSON parsing (Gemini wraps despite instruction).
Falls back silently to hardcoded templates on any error so uploads never fail.
Stub updated to return valid JSON for the <node:describe_tool> prompt tag.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
POST /datasources/{id}/sync reads the existing Parquet file and schema
from the master DB, re-prompts the LLM, and updates ToolRow.description
and ToolCapabilityRow.description in place. Falls back to hardcoded
templates if the LLM call fails (same path as upload).
Home page data source cards gain a Sync button above Delete. On click
the button label changes to "Syncing…" and is disabled until the page
reloads to prevent double-submits.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…<=25 lines Every method now has a pydoc docstring and is <=25 lines (135 methods, 0 over). Dead code removed: - api/datasets.py (stale, referenced non-existent DatasetRow, never wired in) - tools/csv_parser.py (unused since the Parquet ingestion refactor) graph/nodes.py (460-line monolith with 5 responsibilities) split into: - sql_aggregates.py — STDDEV/VARIANCE SQLite aggregates - data_cache.py — per-run in-memory connection cache - tool_registry.py — load tools/sources from the DB - planning.py — build_plan_prompt, decomposed into section builders - execution.py — tool-call parse/dispatch/SQL exec, recoverable vs fatal - loading.py — Parquet/CSV -> in-memory SQLite - persistence.py — mark_completed / mark_failed - nodes.py — now just 5 thin node functions wiring the above api/routes.py (god-router, 9 routes, 3 entities) split per spec "one router per domain entity": - home.py, datasources.py, sessions.py, queries.py - _repository.py — shared 404 lookups and session read helpers Shared logic extracted: - tools/table_naming.py — single source for SQL-safe table names - tools/descriptions.py — LLM description service (moved out of the route layer) Behavior is unchanged: all 22 tests pass and a live PostgreSQL smoke test (upload->Parquet, sync, session, query) returns the correct aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Rewrite product spec (vision, architecture, data-model, api, agent-graph,
capabilities 00/01/02) and tech-stack around MCP: each data source is wrapped
by an in-process FastMCP server over DuckDB/Parquet; the agent is an MCP
client; tools are flat; LLM tool-call format becomes {"tool","arguments"};
Tool/ToolCapability tables removed, descriptions move onto data_sources.
- Add mcp==1.28.0 (pinned; v2 removes the in-memory transport) and duckdb>=1.1,<2.
- Phase-0 spike validated in-memory transport, auto inputSchema, DuckDB native
aggregates, and FastMCP raised-exception -> isError=True mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_sources - db: remove ToolRow/ToolCapabilityRow; add tool_description + capability_description columns to DataSourceRow. domain: remove unused Tool/ToolCapability models. - alembic b8e1f0a2c3d4: add columns, back-fill from the old tables, drop them; reversible downgrade recreates + restores. Round-trip verified on a temp DB. - api/datasources: upload/sync write the description columns; delete stops touching tool tables; dead helpers removed. - graph/tool_registry: Phase-1 shim synthesises the legacy nested tool shape from the description columns so the sync pipeline still runs until the Phase-3 cutover. - tests updated; full suite green (20 passed) incl. golden path + direct run_pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ated) - graph/mcp/csv_server.py: build_server() wraps ONE Parquet in a FastMCP server with a read-only run_query tool backed by a DuckDB view; SELECT-only + DuckDB errors surface as isError=True (recoverable), missing Parquet raises (fatal). - graph/mcp_pool.py: the only importer of mcp.shared.memory; holds per-source servers + DuckDB conns across nodes and opens TRANSIENT ClientSessions per op (namespaced tool keys, routing, idempotent close, version guard). No graph wiring yet. - settings: add mcp_max_result_rows (200). - tests/unit/graph: 10 isolated tests via the real in-memory MCP client. Spec correction: a LangGraph spike proved each node runs in its own asyncio task, so an MCP ClientSession cannot span nodes (AnyIO cancel-scope error). Updated architecture + agent-graph specs: pool holds plain objects across nodes; sessions are transient per node. Full suite: 30 passed (graph still on the Phase-1 shim). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e-load path
- nodes: 5 nodes -> async def; load_data opens the MCP pool + list_tools; execute_action
routes through pool.call_tool; finalize/handle_error await close_pool.
- runner: stays sync but drives asyncio.run(ainvoke) with a try/finally close_pool backstop.
- execution: parse MCP-native {"tool","arguments"}; SQLite executor removed.
- planning: DuckDB dialect, namespaced tool names, {"tool","arguments"} format.
- tool_registry: shim replaced by load_sources_for_session (sources only).
- stub: emit {"tool","arguments"} with the namespaced tool name; "[1] tool:" sentinel.
- templates: trace reads step.tool/step.arguments; drop dead {% if tool %} block.
- delete graph/loading.py, graph/data_cache.py, graph/sql_aggregates.py.
- test_pipeline fixture writes a real Parquet (CSV fallback gone). README -> MCP/DuckDB.
Gate: uv run pytest = 30 passed (stub mode, real MCP+DuckDB E2E); live server smoke
(3 repeated queries) completed with no loop/task errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ol + memory) Rewrite the product/engineering spec for the next build: - MCP pool is one PER SESSION (lazy build, reused, idle/LRU evicted, closed on delete/shutdown), not per query; queries serialized by a per-session lock. - Per-query loop shrinks to plan_action -> execute_action -> finalize/handle_error (no load_data); tools/schema read from the SessionPoolManager by session_id. - Durable per-session memory via LangGraph SqliteSaver (thread_id = session_id); state splits into durable `conversation` (reducer) + per-query scratch (reset via input). - All MCP/tool code moves under tools/ (tools/mcp/server.py, tools/mcp/pool.py). - Checkpoint store is a separate SQLite file, not Alembic-managed. Files: 07-agent-graph, 02-architecture, 04-data-model, capabilities/02 + 03, tech-stack. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- git mv graph/mcp/csv_server.py -> tools/mcp/server.py - git mv graph/mcp_pool.py -> tools/mcp/pool.py - move tests -> tests/unit/tools/; update all import paths - (tools/mcp is safe alongside the installed `mcp` SDK: absolute imports resolve to the SDK) Per-query pool behavior is unchanged this phase; the session-scoped rewrite is phase C. Full suite: 30 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tools/mcp/pool.py: SessionPoolManager — one MCP pool per session, built lazily on first query and reused; per-session threading.Lock serializes queries (DuckDB conn not concurrency-safe); idle/LRU eviction that skips in-use (locked) sessions; close on demand. - graph: drop load_data; entry = plan_action; nodes read tools/schema from the manager by session_id; execute_action routes via manager.call_tool; finalize/handle_error no longer close the pool. agent.py exposes build_graph(); runner holds the session lock, acquires the pool (lazy build), then ainvoke. Deleted graph/tool_registry.py (folded into the pool). - api: warm pool on session create, close on session delete, close affected pools on datasource delete, close_all on app shutdown. settings: max_session_pools, idle seconds. - state: tools/column_names/row_count no longer in state. - tests: rewrite pool tests for the manager (build/route/reuse/LRU/close); conftest resets it. Gate: 33 passed; live smoke — 3 queries in one session, exactly one pool build, no errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Memory keyed by thread_id = session_id. runner compiles the graph per query with an AsyncSqliteSaver (file-backed → durable across the ephemeral per-query savers and across restarts); _fresh_input resets all per-query scratch while the checkpointer restores the conversation. planning renders a "Conversation so far" block. - state.conversation is a plain last-value channel; finalize writes the FULL updated list. (A reducer double-appends on checkpoint resume — overwrite is idempotent under replay.) - settings.checkpoint_db; lifespan ensures its dir; .env.example/Dockerfile/render put it on the persistent disk. dep: langgraph-checkpoint-sqlite. agent.py drops the unused compiled graph. - tests: multi-turn memory test (conversation accumulates across two queries); fixtures point the checkpoint DB at tmp. README/stack/structure updated. Gate: 34 passed; live durability — 2 turns, restart, 3rd query in same session shows all 3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…evel addressing
Reframe the spec so a "tool" is a NAMED dataset addressed by a URI: internal parquet:///{name}
(a directory of Parquet files, one CSV -> one table) or external postgresql:// (BETA, flag-gated).
Tool canonical name = dataset name; one capability per table. LLM call becomes two-level:
{"tool":"<dataset>","capability":"<table>","arguments":{"query":"SELECT ..."}}.
- 04-data-model: DataSourceRow reused as the Dataset row (+uri/last_synced_at/connection_error;
legacy per-source columns deprecated) + new DatasetTableRow child (one row per table).
- 05-api: upload (dataset_name+type), add-csv, sync-over-all-tables, delete-dir, connection-check
before commit, external 501 when flag off.
- 07-agent-graph: two-level format + grouped prompt + action_history {tool,capability,arguments,...}.
- 02-architecture/06-ui/01-vision/capabilities: dataset terminology, per-dataset MCP server,
connector seam (tools/connectors/), per-table capabilities, within-dataset JOINs.
- tech-stack: DuckDB postgres ext + psycopg2 (BETA) + datasets_dir + enable flag; connector seam.
- secret-hygiene: DB URIs are secret-bearing; render via DatasetURI.display() only.
- project-layout: tools/connectors/ + datasets_dir.
Rewrites done via a parallel workflow against one canonical contract; consistency-checked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cols + migration
- db/models.py: DataSourceRow reused as the Dataset row (+ uri, last_synced_at, connection_error;
dataset_uri property default-derives parquet:///{name}; type default 'parquet'; legacy per-source
columns kept nullable). New DatasetTableRow child (one row per table; UNIQUE(dataset_id,table_name);
JSON column/schema accessors).
- settings: datasets_dir (uploads/datasets) + enable_external_datasets (default off).
- alembic c3d4e5f6a7b8 (down_revision b8e1f0a2c3d4): add columns, create dataset_tables, Python
backfill of one child per legacy row preserving parquet_path + setting uri/type; reversible.
- tests: DatasetTableRow + dataset_uri + unique-constraint; migration backfill + downgrade.
Old query pipeline still runs on the deprecated columns (dataset cutover is phase 3). 39 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…server
- tools/connectors/uri.py: DatasetURI — parse + a display() that strips credentials (every log/
template/error/stored field must use it); internal parquet:///{name}, external {type}://...
- tools/connectors/base.py: DatasetConnector Protocol + DatasetConnectionError + get_connector
factory (lazy-imports the external connector so its driver stays optional; flag-gated).
- tools/connectors/parquet.py: ParquetConnector — connection_check (each parquet readable),
discover_tables, build_server (one DuckDB conn, a view per file).
- tools/mcp/server.py: shared build_dataset_server (one query_{table} tool per table, within-
dataset JOINs), register_parquet_view, new_connection. Old build_server kept until phase 3.
- tests: DatasetURI (creds hidden), ParquetConnector check + multi-table build + JOIN.
No pool/graph change yet. 44 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lity)
- tools/mcp/pool.py: one MCP server PER DATASET via get_connector; SessionPool holds datasets ->
per-table capabilities; grouped snapshot(); two-level call_tool(session, dataset, capability,
args); lock-safe close() (waits for in-flight query); legacy-synth fallback builds one table
from a child-less DataSourceRow so pre-cutover uploads keep working.
- graph: planning groups tools by dataset (capability per table, JOIN-sibling note) + two-level
response format; execution/nodes parse + route {tool,capability,arguments}; state + stub +
session.html updated to the two-level shape.
- tools/mcp/server.py: removed dead build_server/_open_view (datasets use build_dataset_server).
- connectors/base.py: accept legacy 'csv' type as an internal parquet dataset.
- tests: rewrote test_mcp_pool for the dataset/two-level API (JOIN, namespacing, eviction); removed
superseded test_csv_server.
Gate: 38 passed (multi-table JOIN, two-level routing, dataset-namespaced tables, durable memory).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…connection-check
- api/datasources.py: upload requires dataset_name + dataset_type (parquet default); writes to
datasets_dir/{id}/{table}.parquet, creates the Dataset + first DatasetTableRow, regenerates
descriptions, runs connection_check BEFORE commit (400, credential-free on failure), rejects
duplicate names. New POST /add-csv appends a table (auto-suffix on collision) + closes attached
pools. /sync regenerates descriptions over ALL tables and records a sanitized connection_error.
/delete rmtrees the dataset dir + child rows + session links + pools. External URI -> 501 when
DATAANALYSIS_ENABLE_EXTERNAL_DATASETS is off (introspection path wired for phase 6).
- tools/descriptions.py: generate_dataset_descriptions(dataset, tables) -> tool desc + per-table
capability map; stub describe branch emits the new shape.
- tests: 6 dataset-API tests; golden path uploads now send dataset_name. 44 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, add-csv - home.py: per-dataset view-model (name, type, credential-free uri_display via DatasetURI.display(), table_count, total_rows, columns sample, last_synced_at, connection_error); pass enable_external. - home.html: "Datasets" terminology; add-a-dataset form requires a name + a type chooser (CSV default; "Connect a database" -> URI field, shown only when external is enabled); dataset cards show type badge, table count, credential-free URI, connection-error badge, last sync, and an "Add CSV" reveal posting to /add-csv; session-create lists datasets (keeps data_source_ids). - golden-path home assertion -> "Datasets". Gate: 44 passed; live smoke — create named dataset, add a 2nd table, session + 2 queries (memory), one pool build, no errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tools/connectors/postgres.py: psycopg2 connection-check + information_schema introspection (each wrapped in a hard wall-clock timeout); DuckDB ATTACH ... (TYPE postgres, READ_ONLY) + one view per table for the query path (shares _run_select/JOINs with parquet); all errors sanitized through DatasetURI.display() so credentials never leak. get_connector dispatches to it only when DATAANALYSIS_ENABLE_EXTERNAL_DATASETS is on (psycopg2 imported lazily). - tests: mocked-psycopg2 connection-check/introspection + credential-sanitization + flag gating; one live test behind the external_db marker (skipped unless DATAANALYSIS_TEST_PG_URI is set). - pyproject: register the external_db marker. Gate: 49 passed, 1 skipped (live PG path is BETA, not part of the offline gate). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sistent disk - README: features/structure/stack reflect datasets (named, multi-table, add-csv, external Postgres BETA), the tools/connectors/ seam, and two-level addressing. - .env.example / Dockerfile / render.yaml: add DATAANALYSIS_DATASETS_DIR (on /data so internal parquet datasets survive redeploys) + DATAANALYSIS_ENABLE_EXTERNAL_DATASETS (default off). - Close the session report. Final: 49 passed, 1 skipped; alembic upgrade head -> c3d4e5f6a7b8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ts children) MCP server is now the primary entity, backed 1:1 by a dataset; children are tools/resources/prompts generated by a 5-stage sync pipeline (versioned, soft-delete-only) and served over a JSON-RPC endpoint. Agent uses single-level addressing over a generic SQL tool (Phase A); hybrid wiring + add/update cascades specced for Phase B. Clean rebuild — fresh schema, no backfill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…le-level agent
Clean rebuild (no aliases/backfill): McpServerRow (mcp_servers, unique name+uri,
version, dataset_schema/physical_tables metadata) with mcp_tools/resources/prompts
children (version + soft-delete envelope, partial-unique over active rows). Fresh
single initial migration 835cdb8ae996.
- tools/sync: 5-stage LLM pipeline (title->schema->entities->tools->prompts),
transactional versioned soft-delete apply, generated-SQL validation; stub +5 branches.
- tools/mcp/dispatch: DB-backed MCP JSON-RPC (tools/resources/prompts list+read/get+call,
keyset cursor pagination, -32601/-32602/isError) at POST /mcpserver/{id}.
- api/mcpserver: upload (CSV->parquet, returns URI) -> create (1:1, connection-check, sync)
-> sync/add-csv/delete + detail; sessions use mcp_server_ids.
- agent flattened to single-level {tool,arguments} over a generic read-only query tool
(hardened SELECT guard); conversation kept a plain channel.
- UI: home (upload->create two-step), session (server chips/trace), mcpserver detail;
vestigial templates removed. Tests rewritten: 54 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nv var Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…essing Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ent tools
POST /mcpserver/{id} gains tools|prompts|resources add/update (JSON-RPC). Each
applies a client-supplied capability + an ADDITIVE downstream cascade
(resource->tools->prompts, tool->prompts) — insert/update only, never soft-deletes
a sibling (vs the full /sync which prunes). One transaction, one version bump;
add rejects active key / update requires one (-32602); tools/add|update guard the
sql_template at write time (SELECT-only + compile-check + $param<->schema); failed
mutation persists nothing; success refreshes pools post-commit. Unknown method -32601.
Hybrid agent: pool serves active generated tools; single-level addressing gains an
optional 'capability' (absent => free SQL, golden path unchanged; present => run the
tool's SQL with bound params via the shared read-only guard). bind_params factored
into server.py. +24 tests; 78 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ect schema resource Adversarial review (verified findings): - SECURITY (high): the read-only SELECT guard blocked keywords but not DuckDB file-reading table functions (read_text/read_csv/read_parquet/read_blob/glob/...), allowing arbitrary host-file read via agent free-SQL, generated tools, and a persisted tools/add template. _guard_select now denies these (one chokepoint = write-time validation + every runtime path). Verified exploit refused; legit view queries unaffected. - MEDIUM: granular update now uses PATCH semantics (keep unspecified fields) and refuses to edit/add the sync-managed 'schema' resource (was: replace semantics clobbered it). - LOW: refreshed the dispatch.py module docstring (write surface). +4 tests; spec guard wording synced. 82 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… UI issues Single-page shell (index.html over base.html, shared _partials/fragments macros) replacing the per-page templates; mcpserver→database rename across api/tests; sqlite/mongodb/snowflake connectors; offset-window "Load more" pagination (_pagination/_view/fragments) + stats widget. Fixes from re-test: - Stale list after delete (clicking a just-deleted row 404'd): render() now sets Cache-Control: no-store on every shell render, so the post-delete 303→GET / always re-renders fresh instead of serving a cached list. - Opening a session/database showed a blank gap: nav links raise the full-page loading overlay (showPageLoading) until the server-rendered page lands. - Connection-URI hint was one combined string: each DATABASE_TYPES entry now carries a uri_hint; the field placeholder shows only the selected type's shape. - Pagination "invisible": default ui_page_size 20→5, so lists past 5 (e.g. a database's resources) now show the Load-more control. Gate: 92 passed, 1 skipped (70 unit + 22 integration); browser-driven Playwright verification of all four fixes, zero console errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… remove api/fragments.py
Consolidate the list surfaces and make pagination uniform across the app:
- Delete api/fragments.py. Its list endpoints move to the routers they belong to —
GET /sessions (sessions.py), GET /databases (database.py), GET /sessions/{id}/queries
(queries.py) — each a rows-only HTML fragment with the next keyset cursor in an
X-Next-Cursor header.
- Drop the bespoke GET /database/{id}/{kind} capability endpoint: the UI now reads
tools/list / resources/list / prompts/list from the existing MCP JSON-RPC surface
(POST /database/{id}). Those list items carry the management UI's edit payload in an
MCP-reserved `_meta` (tool execution, resource kind/size/content, prompt template) —
no signature change, agents/external clients ignore it.
- Replace offset windows with keyset (cursor) pagination everywhere (api/_pagination.py
keyset_page); the JSON-RPC surface was already cursor-based. Default page size 5
(ui_page_size + mcp_list_page_size).
- UI: every list (sessions, databases, chat thread, 3 capability lists) is loaded by its
own AJAX call AFTER the shell renders, and paged with Previous/Next buttons over a
client-side cursor stack. The shell carries only header counts + active-entity metadata.
Capability rows are client-rendered from the JSON-RPC response; edit modals read _meta.
Gate: 101 passed, 1 skipped (adds unit keyset_page + integration list-endpoint/_meta tests).
Browser-driven Playwright verification: empty shell → AJAX populate, Previous/Next on
sessions/resources/thread (cursor stack, ends disabled correctly), edit modals from _meta,
zero console errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s/Next
The conversation thread now opens on the newest turns pinned to the bottom and pulls
the next OLDER page when the user scrolls to the top, prepending it with stable scroll
until there are no older turns — same keyset-cursor surface (GET /sessions/{id}/queries,
X-Next-Cursor), no API change. Sessions / databases / capability lists keep Previous/Next.
- index.html: replace the queries Pager with setupChatThread() (scroll-to-top → loadOlder
→ insertAdjacentHTML afterbegin + re-anchor); drop the #queries-pager markup and the
now-unused _scrollThreadBottom.
- spec 05-api/06-ui: document the thread as infinite-scroll over the same cursor.
Verified (Playwright): opens at newest (pinned bottom), scroll-up prepends older pages in
chronological order, stops at the oldest, no pager, zero console errors. 101 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DuckDB requires pytz to convert TIMESTAMP WITH TIME ZONE columns into Python datetime objects. Without it, any query touching a timestamptz column (e.g. the PostgreSQL pg_meta connector) failed at fetch time with "Required module 'pytz' failed to import", surfaced to the agent as "Action execution failed". pytz was a transitive expectation, never declared — now pinned explicitly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
McpToolRow: execution_json + output_schema_json + annotations_json → meta_json
{"execution":{}, "input_schema":{}, "output_schema":{}, "annotations":{}}
McpPromptRow: arguments_json + template_json → meta_json
{"arguments":[], "template":{}}
McpResourceRow: kind column removed; kind is now stored inside content_json
{"kind":"primary_entity", ...existing fields...}
Backward-compat property setters/getters keep pipeline.py and other callers
working without changes. McpResourceRow.kind is a hybrid_property so
SQLAlchemy filter expressions (kind == "schema" / kind != "schema") still
work via type_coerce+JSON subscript (json_extract on SQLite, ->> on PG).
Postgres DB dropped and recreated fresh from new models. One test assertion
updated: r.content now always includes "kind" as per the new schema.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The two header tab buttons were redundant: each database row already has an
Inspect button that navigates to /database/{id}, which renders the database
pane server-side (active_tab="database"). Removed the tab nav, the now-dead
switchTab() JS, and the unused .hdr-tab / .site-header__tabs CSS. EER still
renders on load via the existing DOMContentLoaded → renderEER() hook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- LangGraph 6-node pipeline: load_dataset → plan_analysis → execute_code → reason_answer → handle_error → finalize - POST /api/files/upload: CSV upload, pandas schema extraction, SQLite registry - POST /api/analysis/run: real Gemini API (flash for code-gen, pro for reasoning) - execute_code: sandboxed exec() with AST forbidden-import check + 30s timeout - Frontend: Next.js chat UI with sidebar, drag-drop upload, inline Plotly.js charts - 8 integration tests (real Gemini API) + 26 unit tests + 3 Playwright E2E tests - LangSmith tracing wired; structlog observability on every node - Phase 2 stubs: PostgreSQL connect (501), session persistence (clearly labelled) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both side cards now slide OVER the main layout instead of occupying flow space, so the main area always uses full width: - Sessions sidebar: absolute within .app-shell, translateX off-screen by default, toggled by a new hamburger button in the header (top-left). - MCP capabilities (database view): absolute overlay sliding in from the right, toggled by a "MCP capabilities" button in the db-head (and a × to close). The EER diagram now spans the full body width. Anchoring both panels to .app-shell (an ancestor of the scrolling .main) keeps them pinned and lets .app-shell's overflow:hidden clip them when closed. Escape closes either panel. No API changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace type_coerce JSON subscript (broken on PostgreSQL) with a custom _KindFromJson FunctionElement that compiles to json_extract on SQLite and CAST(col AS jsonb) ->> 'kind' on PostgreSQL. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… harness/ restructure Merges 9 remote commits (harness updates) into 51 local commits (full agent implementation). Resolved conflicts in favor of local application code (spec/, src/, alembic/, tests/, pyproject.toml). Accepted upstream harness changes: - New zero-shot-build/fix/sync commands and skills - spec/engineering/ → harness/patterns/ + harness/rules/ reorganization - Updated .claude/agents/ (agent-builder, qa-auditor, spec-writer) - Replaced .github/agents/ and AGENTS.md with harness-native structure - Rejected remote's simple Next.js frontend (we use SSR templates) - Rejected remote's phase-1 alembic migrations (we have MCP server schema) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a .section-loading spinner overlay inside the tool, prompt, resource, and schema modals so the user gets immediate feedback when Save is clicked. The overlay covers the modal for the duration of the async RPC call; it clears on error (so the user can retry) and the page replaces it on success. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Personal on-demand data analysis agent: upload a CSV, ask a plain-English question, receive a text answer + interactive Plotly.js chart rendered inline in the chat UI. Raw data never leaves the machine — the agent generates Pandas code, runs it locally in a sandboxed exec() environment, and sends only an aggregated result sample (≤500 rows) to Gemini for final reasoning.
What Phase 1 added (this commit)
Phase 2 stubs (clearly labelled in UI)
How to verify Phase 1
Stack
Python 3.11 + FastAPI + LangGraph + Google Gemini (gemini-2.5-flash / gemini-2.5-pro) + SQLite (app metadata) + Next.js 15 + Tailwind CSS v4 + Plotly.js
Phases
PR
#90
🤖 Generated with Claude Code