feat(rag): Native RAG & Hybrid Vector Search Engine with Lucene & OpenSearch Parity

[ruisantana]

Motivation & Context

This PR started out of curiosity to see whether it would be viable to embed a native RAG (Retrieval-Augmented Generation) engine directly into IPED processing. As the architecture evolved and proved to be genuinely viable for production use, I decided to share this implementation so maintainers can test it and evaluate whether this concept adds value to IPED.

The solution is designed with a highly configurable architecture:

• Flexible choice of embedding models (bge-m3) and LLM providers (local offline models like qwen3:4b via Ollama as well as cloud APIs like Gemini/Claude).
• Configurable whitelists/blacklists for evidence types to embed.
• Targeted scope reduction: Ability to restrict RAG queries to filtered or checked table item selections using IPED's native filters.
• Dedicated Analysis UI Panel: Integrated Swing docking panel (RAGAssistantPanel.java) featuring multi-session management, incremental token-by-token response rendering, interactive evidence source tables, and double-click navigation.
• Complementary hybrid retrieval combining BM25 exact lexical search with KNN dense vector search using Reciprocal Rank Fusion (RRF).
• Full parity between standalone local Lucene KNN indexing and corporate OpenSearch REST clusters.

This work can certainly evolve further with community collaboration. By providing pre-computed case embeddings and a hybrid retrieval API, it can naturally act as a high-performance backend for AI assistant interfaces and MCP agents built on top of IPED.

---

Summary

This PR introduces a native Retrieval-Augmented Generation (RAG) & Hybrid Vector Search Engine to IPED. It empowers forensic examiners to perform case-wide semantic search and vector-assisted retrieval across large evidence repositories.

The architecture provides full feature parity between Local Lucene KNN and Corporate OpenSearch REST Clusters, combining dense vector retrieval with classic BM25 lexical search via Reciprocal Rank Fusion (RRF).

It serves as a core retrieval backend for case search and AI assistant extensions.

---

Key Features

1. Integrated Analysis UI Panel (RAGAssistantPanel.java):
   • Built-in desktop docking assistant tab in IPED-SearchApp with conversation history persistence, incremental response rendering (token by token as the LLM generates), selection scope restriction (chkScopeOnly), and evidence source tables with double-click item navigation.

*Figure 1: RAG AI Assistant UI integrated into IPED-SearchApp.*

2. Targeted Scope Reduction & Item Selection Filtering (allowedItemIds):

   • Enables examiners to restrict vector/hybrid queries to specific items checked in the IPED table or filtered using IPED's native filter controls (e.g., date ranges, file types, specific senders).
   • Drastically eliminates evidence noise in multi-terabyte cases and optimizes retrieval speed.

3. Dual Vector Store Backends (Lucene & OpenSearch Parity):

   • Local Lucene KNN: Indexes text fragment embeddings using HNSW vector fields directly in the local case index.
   • OpenSearch Cluster REST API: Fully mapped knn_vector indices with bulk REST ingestion and REST KNN search capabilities for enterprise multi-node setups.
   • Automatic fallback to local Lucene if the OpenSearch cluster is unreachable at startup.

4. Complementary Hybrid Search (BM25 + Dense KNN via RRF):

   • Combines BM25 exact lexical matching (names, numbers, CPFs) with dense semantic vector search.
   • Merges candidate results using Reciprocal Rank Fusion (RRF) with similarity threshold guards to eliminate hallucinations.

5. Pluggable RAG Engine Backend:

   • Functions directly within IPED forensic search and acts as a high-performance retrieval engine for AI Assistants and Chatbot interfaces.

6. Multi-Provider LLM & Remote Integration (Ollama qwen3:4b, OpenAI gpt-4o/gpt-4o-mini, Gemini, Claude):

   • Seamless integration with local/remote offline LLMs (Ollama with models like qwen3:4b), OpenAI endpoints (gpt-4o, gpt-4o-mini, vLLM, LMStudio), and cloud APIs (Google Gemini, Anthropic Claude).
   • Confirms full compatibility for both local air-gapped workstations and distributed remote GPU inference servers in forensic lab networks (validated extensively using a dedicated remote GPU workstation with an NVIDIA RTX A4000 16GB VRAM).
   • Calibrated similarity thresholds (chunkSimilarityThreshold) and retrieval limits (maxRetrievedChunks = 5) to fit LLM context windows efficiently.

---

System Overview

flowchart TD
    subgraph Phase1["1. Ingestion Phase"]
        A[Extract & Fragment Text] --> B[Generate Embeddings - bge-m3]
        B --> C[Store Vectors in Lucene HNSW / OpenSearch KNN]
    end

    subgraph Phase2["2. Query Phase"]
        D[User Prompt & Scope Selection] --> E[Scope Filter - Selection / IPED Filters]
        E --> F1[BM25 Lexical Search]
        E --> F2[KNN Vector Search]
        F2 --> G[Apply Vector Similarity Threshold >= 0.62]
        F1 --> H[RRF Fusion Ranking]
        G --> H
        H --> I[Display in RAGAssistantPanel & Source Table]
        I --> J[LLM Response Generation - qwen3:4b / Gemini / Claude]
    end

    C --> D

Loading