Feat/moondream#2
Open
ralfferreira wants to merge 45 commits into
Open
Conversation
…ream2 batch processor with delta encoding - Add vision_ollama module imports (OllamaBatchProcessor, ScreenshotDescription) to backend/cmd/brain/main.py - Replace Ollama chat test with OllamaBatchProcessor initialization using moondream model in BrainWorker.load_models() - Add process_snapshots_batch method (134 lines) to process multiple snapshots using Moondream2 batch processing with delta encoding - Add _format_moondream_description
…d brain worker with Phi-3.5 hybrid clustering
…and Phi-3.5 episodic memory system
…lidate into main guides
… endpoints with offset parameter - Add offset parameter parsing (default 0) to handleMemoryTimeline and handleMemoryByType in handlers_memory.go - Update GetMemoryTimeline and GetMemoryByType database methods to accept offset parameter and add OFFSET clause to SQL queries - Redesign chat.html UI with Tailwind dark theme (zinc palette), Phosphor icons, and improved message styling
…dashboard UI with Tailwind zinc theme - Add search query parameter parsing (q) to handleEpisodesTimeline in handlers_episodes.go - Update SQL query builder to support dynamic WHERE clauses with ILIKE search on title and summary fields - Replace hardcoded query parameter binding with dynamic args slice and argCount tracking - Add search input field with magnifying glass icon to dashboard controls - Redesign dashboard.html with Tailwind dark
… add audio processing debug utility - Change brain worker interval from 5 to 10 minutes in daemon/main.go - Add separate episodic clustering ticker at 30-minute intervals (previously ran sequentially after brain) - Remove episodic worker from startup sequence to prevent immediate execution - Add independent timer handling with separate select cases for brain and episodic workers - Add debug_audio.py utility (141 lines) to diagnose audio segment
…s to prevent XSS and improve performance - Add currentEpisodes global array to store episodes data in dashboard.html - Update loadTimeline to populate currentEpisodes array after fetching episodes - Change openModal parameter from JSON object to numeric index - Update episode card onclick handler to pass array index instead of serialized JSON object - Add null check in openModal to validate episode exists before rendering modal
- Delete debug_audio.py (141 lines) that was used for diagnosing audio segment processing status - Script provided database inspection for audio segments, transcription status, and episodization readiness
- Add Answer, Sources, Mode, and Error fields to ChatResponse - Parse JSON output from RAG script with fallback to raw text - Return structured error responses in JSON format - Fix chat request body to use 'message' field instead of 'question' - Update run-all-multimodal.bat to use absolute paths and remove PostgreSQL dependency - Remove obsolete setup-venv.bat script
- Change from executable path to working directory for backend path resolution - Add verification that RAG script exists before execution - Capture and log stderr separately from stdout for better debugging - Include stderr in error messages when RAG script fails - Log stderr as debug info for successful runs to capture status messages
- Create audiodetect Go module using Windows Core Audio API - Detect actual audio-emitting process via IAudioSessionManager2 - Return active audio session process name instead of active window - Add fallback to active window detection when helper unavailable - Use audio source detection for loopback, active window for microphone
…ty validation - Create example_enhanced_memory.py demonstrating complete workflow from activity data to knowledge graph queries - Add entity existence validation before creating relationships to prevent FOREIGN KEY constraint errors - Validate entity existence before creating embeddings and episode links - Extract relations between entities in episodization pipeline - Add domain field to MemoryItem with 'work' default - Improve robust
…ation - Add comprehensive grounding instructions to entity extraction prompts - Force extraction of all apps from activity log as Tool entities with auto-fallback - Add source_snippet field to entities and relations for verification - Implement confidence threshold (0.75) with rejection logging - Fix markdown code block parsing to handle ```json with/without newlines - Add regex patterns to clean malformed JSON fields and broken
…nd structured output - Add format_graph_context_for_llm() to transform raw graph data into organized markdown - Deduplicate entities by type (Project/Tool/Person) and name - Calculate relationship confidence from weight and occurrence count - Prioritize typed relations (used/worked_on/discussed) over co_occurred_with - Format context with clear entity groups, relationship chains, and evidence snippets - Update LLM prompt to reference
…r domain detection
…m 3 to 1 - Add generate_episode_metadata_unified() to extract title, summary, entities, and relations in single LLM call - Reduce episode processing time from ~30s to ~10-12s per episode - Generate consistent entity IDs based on type+name for cross-episode deduplication - Include confidence thresholds (0.7) and fallback to simple extraction on failure - Enrich entities and relations with full metadata including episode_id and source
…c entity extraction - Reduce LLM calls to only title+summary generation (entities/relations via heuristics) - Add intelligent item sampling (beginning+middle+end) to reduce context size from full episode to 15 items - Extract entities directly from app names, window titles, and content keywords without LLM - Generate relations from co-occurrence patterns (uses/works_on/used_for) with confidence scores - Add enforce_max_duration_post_split
… with content inference - Remove LLM boundary refinement stage to reduce processing time (trust high-confidence heuristics >=0.7) - Add enforce_max_duration_post_split() to guarantee 45-minute episode cap after clustering - Replace generate_episode_summary_with_phi() with generate_episode_metadata_unified() for single-call metadata extraction - Add intelligent primary app detection with 3-tier fallback: app_name → window_title extraction
…th schema migration - Add AppConfigFilter for category-based filtering (WORK/PERSONAL/BLOCKED) with capture mode control - Fix entity ID generation to normalize names (remove .exe, handle special chars, deduplicate) - Add entity schema migration to remove UNIQUE(type, name) constraint causing conflicts - Change entity upsert to use ON CONFLICT(id) instead of (type, name) - Add graceful handling of UNIQUE constraint violations with fall
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.