fix(query): respect workspace llm_provider/llm_model when no request override given#242
Open
RomySaputraSihananda wants to merge 58 commits into
Open
Conversation
Spec covers async pre-pipeline enrichment using EdgeParser (text mode) + local OpenAI-compatible VLM to generate summary, topic, language, and keywords from the first 5 pages before full graph processing runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dd default test - Add missing "metadata_enrich" arm to TaskType FromStr in postgres.rs to prevent deserialization failure for MetadataEnrich tasks fetched from DB - Replace path-qualified uuid::Uuid with plain Uuid in MetadataEnrichData fields to match the existing use uuid::Uuid import in data.rs - Extend test_metadata_enrich_data_serialization to assert serde default applies max_pages=5 when the field is absent from JSON Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- MetadataEnrichProcessor: EdgeParser + local VLM, writes summary/topic/language/keywords - enrich_queue added to AppState (dedicated ChannelTaskQueue capacity 200) - PDF upload pre-generates document_id shared between MetadataEnrich + PdfProcessing tasks - Dedicated enrichment WorkerPool (4 workers, 5min timeout) started in main.rs - enrichment_status written to KV at upload time so clients can poll immediately Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…sePdfConverter import path
- main pool: initial_retry_delay 5000ms → 2000ms (faster recovery) - main pool: max_tasks_per_tenant default → 0 (no cap for single-tenant) - enrichment pool: concurrent default → (num_cpus/2).max(4) (scales with machine) - task/enrich queues: capacity 100/200 → 500 each (handles burst uploads) - .env.example: document WORKER_THREADS default, add MAX_TASKS_PER_TENANT=0 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Includes all enrichment pipeline env vars (ENRICHMENT_*), worker config (MAX_TASKS_PER_TENANT, TASK_PROCESSING_TIMEOUT_SECS), and WSL2-compatible host.docker.internal extra_hosts for Ollama access. Also adds .env.docker.example as a copy-paste starting point. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch postgres service to ghcr.io prebuilt image (no local build needed), remove unused POSTGRES_PORT mapping, rename volume to edgequake-pg-data to match upstream quickstart convention. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When enable_topic_scope=true is set on a query: 1. Scans KV for enrichment_topic across all workspace documents 2. Uses LLM to classify which topics are relevant to the query 3. Restricts vector/graph retrieval to matching document IDs Falls back gracefully to full search when: - Fewer than 2 distinct topics exist - LLM classification returns no matches - Classification call fails (non-fatal warning) Works for both /query and /query/stream endpoints. Ignored when document_filter is already set. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migration 026 hardcoded allowed task types without metadata_enrich, causing INSERT to fail with a constraint violation → HTTP 500 on every PDF upload since the enrichment task could not be persisted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DocumentSummary now includes enrichment_status, enrichment_topic, enrichment_summary, enrichment_language, enrichment_keywords, and enrichment_completed_at. These are extracted from the same KV metadata key that the enrichment pipeline writes to, so the UI can use enrichment_topic for the grouped accordion view. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds POST /api/v1/documents/{id}/enrich endpoint that re-queues a
MetadataEnrich task for any PDF document regardless of its current
enrichment_status. Resets enrichment fields in KV before queueing
so the VLM runs fresh.
Frontend: "Extract Topic & Summary" menu item (Sparkles icon) in the
document three-dot menu, visible only for PDF documents.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
## Enrichment - Add `tags` field (3-6 subtopic chips) to enrichment prompt and KV storage - `EnrichmentResponse` now parses `tags` from VLM JSON response - `DocumentSummary` and `DocumentDetailResponse` expose `enrichment_tags` - list.rs, track_status.rs, detail.rs all extract `enrichment_tags` from KV ## Query filters (BE + MCP + SDK) - `QueryRequest` / `StreamQueryRequest`: add optional `topic` and `tags` params - `topic`: case-insensitive exact match against `enrichment_topic` - `tags`: match docs containing ANY of the given tags (case-insensitive) - Priority chain: `document_filter` → `topic` → `tags` → `enable_topic_scope` - `topic_resolver`: add `resolve_topic_filter` and `resolve_tags_filter` functions - MCP `query` tool: expose `topic` and `tags` parameters - TypeScript SDK types updated accordingly ## Frontend - Document table row: show `enrichment_tags` as chip pills below title - Document group accordion: show union of tags in group header (max 5) - Preview panel: add Topic & Tags section with Sparkles icon - Preview panel: fix Content Preview — falls back to `enrichment_summary` for PDF docs - Preview panel bug fix: topic/tags now shown (was missing from `DocumentDetailResponse`) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Lower MAX_CONCURRENT_EXTRACTIONS to 2 (from 16) and raise CHUNK_TIMEOUT_SECS to 300s to prevent all-chunk timeouts when running against Ollama/LMStudio on consumer hardware. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Prompt now instructs LLM to pick exactly one from the allowed list - Backend validates LLM output and falls back to Uncategorized if topic is outside the list (prevents hallucinated topics like "Cross-Cultural Communication" slipping through) - MCP query tool uses z.enum() for strict type-safe topic parameter - SDK exports EnrichmentTopic union type for consumer type safety Allowed topics: Politics, Indonesia, Psychology, Business, Communication, Technology, Science, SocialMedia, Religion, Media, AI, Culinary, Finance, Literature, Law, Sports, Education, History, Uncategorized Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tion Implements MCP JSON-RPC 2.0 over HTTP at POST /mcp, enabling clients like Claude Desktop or MCP-compatible tools to connect directly to the EdgeQuake API without the npm package. Supported methods: - initialize (MCP handshake) - ping - tools/list (returns query + list_documents tools) - tools/call (dispatches to query or list_documents) - notifications/* (silently accepted) Tools exposed: - query: RAG query with topic filter support - list_documents: lists documents from KV store Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ackage Adds 14 new tools matching @romysaputrasihanandaa/edgequake-mcp: - document_upload, document_list, document_get, document_delete, document_status - graph_search_entities, graph_get_entity, graph_entity_neighborhood, graph_search_relationships - health - workspace_list, workspace_create, workspace_get, workspace_delete, workspace_stats Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The global sota_engine uses the binary-default provider (anthropic/claude-3.5-sonnet) which is unavailable on the OpenRouter account. Fix by resolving the default workspace's LLM config (openrouter/gpt-oss-120b) and passing it as override to query_with_llm_provider, matching the behaviour of the regular query API endpoints. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ables Querying _ag_label_vertex/_ag_label_edge triggers PostgreSQL's table inheritance scan which includes all child tables without using their specific indexes (idx_edge_start_id, idx_node_props_gin, etc.). Switching to the concrete "Node" and "EDGE" label tables avoids the Parallel Append inheritance overhead and lets the planner use existing indexes — fixing the 30s statement timeout on hybrid graph queries. Also raises the default EDGEQUAKE_GRAPH_QUERY_TIMEOUT_SECS from 30s to 120s in docker-compose as a safety net during large graph traversals. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the ad-hoc topic list (Politics, Indonesia, AI, etc.) with the standard Amazon Books category hierarchy. Covers fiction, nonfiction, and specialty categories for better cross-document classification consistency and recognizability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…endpoint Re-queues MetadataEnrich tasks for documents whose enrichment_topic is from the legacy taxonomy. Pass ?force=true to re-enrich all documents regardless of current topic. Useful after updating the ALLOWED_TOPICS taxonomy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…delete/status/health/workspace - Add topic filter parameter to document_list tool (Amazon Books taxonomy) - Update query tool topic enum to new taxonomy - Sync ALLOWED_TOPICS constant to new taxonomy - Hide tools from MCP listing: document_upload, document_delete, document_status, health, workspace_list, workspace_create, workspace_get, workspace_delete, workspace_stats Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…is visible error_details was set on the in-memory stats_with_lineage struct but never written to KV storage, making it impossible to diagnose partial_failure documents via the API or DB. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- detail.rs / list.rs: fall back to error_details when error_message is absent so
partial_failure cause is visible in both list and detail endpoints
- text_insert.rs: populate error_details for the 0-entity extraction case so the
root cause ("Extracted 0 entities") is stored in KV and returned by the API
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…atible base URLs Custom servers (LMDeploy, vLLM) expose model names with "/" paths (e.g. "/models/Qwen2-VL-72B"). When EDGEQUAKE_CHAT_BASE_URL / OPENAI_BASE_URL points to such a server the model name is not an Ollama-style conflict, so the has_mismatch flag should not fire for that pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Large documents (4000+ entities with long LLM-extracted descriptions) cause the inlined UNWIND literal to exceed the 30s statement_timeout. Reducing to 100 per batch makes each query ~5x smaller and stays well within the limit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…skips LLM Previously clear_pipeline_checkpoint was called unconditionally, so a partial_failure (e.g. graph storage timeout) wiped the checkpoint. The next reprocess could not find it and re-ran expensive LLM extraction from scratch. Now the checkpoint is only cleared when final_status == "completed". On partial_failure the checkpoint survives, and reprocessing loads it and skips directly to the storage retry stage. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… be updated") When multiple workers process documents containing shared entities (e.g. "Google", "United States") simultaneously, AGE returns "Entity failed to be updated: 3" on the second writer's MERGE. Add a 3-attempt backoff retry (200 ms × attempt) in both upsert_nodes_batch and upsert_edges_batch so transient conflicts resolve without failing the entire document. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…phan recovery overwrites Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…NALITY WHERE key = ANY($1) does not guarantee result order matches the input slice. recover_orphaned_documents zips metadata_keys with the returned values, so mismatched order caused completed docs to be paired with non-terminal values and incorrectly reset to pending on every API restart. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Queries always fell through to the system default model (e.g. gpt-5-mini)
even when the workspace had llm_provider and llm_model configured in the
database. llm_override was only created from the request body, skipping
the workspace-level config entirely.
Add workspace LLM as second priority in the override chain: