feat: complete GenIE web app with 3-agent pipeline, encrypted key vault and real-time monitor - #2
Open
RMSantista wants to merge 3 commits into
Open
feat: complete GenIE web app with 3-agent pipeline, encrypted key vault and real-time monitor#2RMSantista wants to merge 3 commits into
RMSantista wants to merge 3 commits into
Conversation
…lt and real-time monitor Implements the design handoff (handoff.html) on the existing Python/FastAPI backend, making GenIE fully operational end to end: Frontend (spec/web): faithful static SPA port of the prototype — model picker with key status, input/output connection panels (URL, local folder, database, REST API, upload/download), extraction prompt, output format, agent monitor with live SSE log and tabular/JSON result preview with signed download links. Agents (spec/extraction/agents): Conector (all I/O, no LLM), Localizador (LLM extraction with chunking and retry/backoff), Organizador (LLM formatting with passthrough when no format is given) and the Orchestrator driving the pipeline with real-time events. API (spec/api/v1): /models, /keys (AES-256-GCM encrypted vault, masked previews only), /uploads (sanitized, size/extension limits), /runs (create, inspect, cancel, SSE stream with Last-Event-ID replay) and /downloads (HMAC-signed, 15-minute links). Security fixes: - API keys encrypted at rest (AES-256-GCM); no endpoint returns plaintext - CORS restricted to explicit allowlist; credentials disabled - Filesystem connector confined to allowed roots (anti path traversal) - Upload filename sanitization + extension allowlist + size limits - LLM factory cache no longer embeds key material (SHA-256 digest) - Transient credentials kept in memory only, never logged or streamed - data/ fully gitignored (master key, vault, uploads, outputs) Also: current model catalog (Gemini 2.5, GPT-4o, Claude Sonnet 4.6/Haiku 4.5), content parsers for CSV/XLSX/HTML/JSON, 35 new tests (76 total, all passing), updated README and .env.example. https://claude.ai/code/session_01CKjevqGgfWLggV1DG1Tmpq
…manual Review fixes and improvements before the PR: Fixes: - JobManager.stream no longer yields duplicate events when emissions race with history replay (tracks last yielded seq); covered by test - Job eviction prefers finished jobs over running ones - Frontend stops SSE reconnection after repeated failures (server restart) instead of retrying forever - pyproject ruff config migrated to [tool.ruff.lint] (deprecation) - health endpoint cleanup (timezone-aware timestamp, version 1.0.0) Reliability improvements: - Native JSON mode on providers: response_format json_object (OpenAI) and response_mime_type application/json (Gemini) - Larger output budgets: Gemini 16384 (thinking tokens), OpenAI/Claude 8192 - Updated provider default models (gemini-2.5-flash, claude-sonnet-4-6) - CSV artifacts written as UTF-8 with BOM so Excel pt-BR renders accents - Stale upload/output batches cleaned up on startup (24h TTL) New: inline 'text' input type — integrating apps (TabEx) send pre-extracted OCR text in a single POST /runs call, no multipart upload needed. Docs: docs/MANUAL.md — full usage manual covering standalone mode (web UI walkthrough), plugin mode (REST contract with curl/Python/JS examples) and the TabEx integration: replaces TabEx's layout-specific regex extraction (extrairResultados) with GenIE via Apps Script polling, enabling layout-independent extraction and automatic new-exam columns. 79 tests passing. https://claude.ai/code/session_01CKjevqGgfWLggV1DG1Tmpq
Closes the remaining Phase 1 items from docs/guides/GENIE-TODO.md
(Quality Gate 1.4) after reviewing the original implementation plan:
- Config CRUD endpoints (POST/GET/PUT/DELETE /api/v1/configs) backed by a
new file-based ConfigStore (data/configs/{id}.json, safe id validation)
- Search Library endpoints (GET /api/v1/library/patterns, /patterns/{id},
/stats with usage aggregates)
- ExtractionEngine now loads the stored ExtractionConfig: uses its
extraction_instructions, output schema and provider/model, and honors
behavior.use_search_library / auto_create_patterns flags (decision nº 1:
generic, not general — previously config_id was accepted but ignored)
- scripts/test_llm_connection.py (planned Stage 1.2.2 utility)
Test coverage raised from 65% to 81% (plan target: >= 80%), 127 tests:
- tests/unit/test_llm_providers.py (mocked clients; planned Stage 1.2.2)
- tests/integration/test_end_to_end.py (config -> LLM -> pattern saved ->
library hit with zero LLM cost -> force_llm bypass; planned Stage 1.4.4)
- tests/unit/test_search_library.py (matcher + success-rate moving average)
- connector sqlite input, inline text input, HTTP delivery, XLSX parsing
- providers endpoint tests
Restructuring (alignment with the plan):
- Removed empty spec/api/v1/routes/ and spec/output/formatters/ (not in plan)
- Moved setup.sh and test-genie.py into scripts/
- Moved root status docs (IMPLEMENTATION-SUMMARY, HOMOLOGATION-CHECKLIST,
FINAL-RELEASE-NOTES, TESTE-RAPIDO, PHASE-1-STATUS) to docs/history/
- Updated living docs: GENIE-TODO.md checkboxes (Phase 1 complete, dated)
+ addendum for the web app delivery; PROJECT-AUDIT.md status appendix
- Updated stale default model ids in models/config.py
https://claude.ai/code/session_01CKjevqGgfWLggV1DG1Tmpq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the design handoff (handoff.html) on the existing Python/FastAPI
backend, making GenIE fully operational end to end:
Frontend (spec/web): faithful static SPA port of the prototype — model picker
with key status, input/output connection panels (URL, local folder, database,
REST API, upload/download), extraction prompt, output format, agent monitor
with live SSE log and tabular/JSON result preview with signed download links.
Agents (spec/extraction/agents): Conector (all I/O, no LLM), Localizador
(LLM extraction with chunking and retry/backoff), Organizador (LLM formatting
with passthrough when no format is given) and the Orchestrator driving the
pipeline with real-time events.
API (spec/api/v1): /models, /keys (AES-256-GCM encrypted vault, masked
previews only), /uploads (sanitized, size/extension limits), /runs (create,
inspect, cancel, SSE stream with Last-Event-ID replay) and /downloads
(HMAC-signed, 15-minute links).
Security fixes:
Also: current model catalog (Gemini 2.5, GPT-4o, Claude Sonnet 4.6/Haiku 4.5),
content parsers for CSV/XLSX/HTML/JSON, 35 new tests (76 total, all passing),
updated README and .env.example.
https://claude.ai/code/session_01CKjevqGgfWLggV1DG1Tmpq