Install required libraries

pip install -r requirements.txt

🤖 Autonomous Machine Learning Specialist Agent (MachineLearningRAG)

An industrial-grade, end-to-end Retrieval-Augmented Generation (RAG) platform designed to act as an autonomous Machine Learning Specialist Agent. Built natively to optimize performance under strict cost-free, open-source constraints on a native WSL2 Linux kernel, this platform eliminates heavy black-box third-party frameworks. It provides a highly tailored, deterministic knowledge base engine that ingests scientific ML literature, compiles high-dimensional text embeddings, and executes low-level hybrid searches to resolve complex machine learning and algorithmic troubleshooting queries.

---

🏗️ System Architecture & Low-Level Data Flow

The platform utilizes a decoupled, high-throughput pipeline designed to map complex mathematical text structures into high-performance retrieval topologies.

---

⚡ Core Technical Features & Systems Design

1. Ingestion Pipeline with Overlap Safeguards

• The Engineering Problem: Machine learning literature is heavily saturated with complex mathematical notation, LaTeX symbols, and code blocks. Naive character-based text splitters blindly sever equations in half, destroying semantic integrity.
• The Solution: Implemented a robust sliding-window token chunking matrix (chunk_size=1000 tokens, overlap=200 tokens). This ensures that multi-line algorithmic steps, tensor shape definitions, and mathematical structures are kept unified within an exact semantic boundary.

2. Dual-Engine Hybrid Fusion Querying

• The Retrieval Problem: Dense vector search handles abstract concepts gracefully but fails catastrophically on exact keyword metrics (e.g., searching for specific hyperparameter keys like scale_pos_weight or a paper's exact publication year).
• The Solution: Configured an explicit, native Elasticsearch 8.x schema pairing a tokenized text index (BM25 Keyword Engine) with a Hierarchical Navigable Small World (HNSW) graph topology (Dense kNN Vector Engine) executing Cosine similarity metrics. The orchestrator dispatches a joint, single-round REST query to merge exact syntax validation with deep conceptual understanding.

3. Native Linux OS & Kernel Synchronization

• The Architecture Advantage: Completely bypassed Docker or abstract cloud layers to maximize hardware capability. The database daemon runs natively on the laptop's Linux subsystem, hard-capping the JVM heap at a safe -Xms1g -Xmx1g boundary. This leaves the remaining host RAM entirely unallocated, allowing the Linux Kernel Page Cache to keep the index segments and HNSW matrix graphs entirely hot in volatile memory for sub-millisecond retrieval speeds.

---

🛠️ Repository Blueprint & Directory Layout

MachineLearningRAG/
│
├── app/
│   ├── app.py                       # Interactive Streamlit Web Chatbot UI Cockpit
│   └── search_engine.py             # Compound Hybrid BM25 + kNN Vector Search Engine
│
├── data_pipeline/
│   └── parse_and_chunk.py           # Robust sliding-window PDF parser & token compiler
│
├── indexing/
│   └── index_documents.py           # Database version-aligned vector schema ingestion pipeline
│
├── knowledge_base/                  # Local directory housing target textbooks and research papers
│   └── CatBoost.pdf                 # Reference benchmark document: arXiv:1706.09516
│
├── .gitignore                       # Explicitly masks environment runtimes and large binary data maps
├── requirements.txt                 # Clean, verified production-grade open-source package stack
└── LICENSE                          # MIT open-source certification

---

🚀 Native Linux Installation & Deployment

1. Clone & Set Up the Python Virtual Environment

Navigate to your local Linux directory, clone the project code, and spin up the isolated execution environment:

git clone https://github.com
cd MachineLearningRAG

# Build and activate the isolated virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Upgrade pipeline dependencies
pip install -r requirements.txt

2. Populating the Knowledge Base

To run a local query benchmark, place your target textbooks or research papers inside the local directory (knowledge_base/CatBoost.pdf) and initialize the vectorizer:

# Execute the high-performance indexing script
python indexing/index_documents.py

The script will parse the target file, load the BGE transformer model weights, establish connection pipelines to port 9200, and populate your Lucene storage arrays natively.

3. Launch the AI Agent Cockpit

# Launch the interactive Streamlit chatbot application
streamlit run app/app.py

Open http://localhost:8501 in your browser. Type an advanced machine learning prompt (e.g., "How does CatBoost partition categorical features?") and view the agent's real-time, explainable context retrieval layers.

---

⚖License

Distributed under the MIT License. See LICENSE for more information.