An AI-powered news research tool that lets you load news articles from URLs and ask questions about their content using Retrieval-Augmented Generation (RAG). The tool uses advanced NLP techniques to understand and analyze news content, providing insightful answers with proper citations.
- Load and analyze multiple news articles simultaneously
- Automatic HTML parsing and text extraction
- Smart chunking for optimal context preservation
- Natural language question answering
- Context-aware responses using RAG
- Support for multiple Groq LLM models (Mixtral, Llama 3, Gemma)
- Semantic search using FAISS vector database
- Relevant context extraction from documents
- Configurable similarity search parameters
- Source tracking for every answer
- Expandable document previews
- Confidence scoring for retrieved information
- Fast inference with Groq's LPU technology
- Persistent vector storage for repeated use
- Efficient batch processing of multiple articles
- Clean Streamlit-based UI
- Real-time progress indicators
- Interactive chat interface
- One-click sample questions
| Component | Technology | Purpose |
|---|---|---|
| Frontend | Streamlit | Web interface |
| Backend | Python 3.9+ | Core logic |
| AI Framework | LangChain | LLM orchestration |
| Vector Database | FAISS | Similarity search |
| Embeddings | HuggingFace (all-MiniLM-L6-v2) | Text vectorization |
| LLM Provider | Groq API | Fast inference |
| Document Loading | AsyncHtmlLoader, Html2TextTransformer | Web scraping |
| Text Processing | RecursiveCharacterTextSplitter | Document chunking |
Before you begin, ensure you have:
- Python 3.9 or higher
python --version
- Groq API Key (free at console.groq.com)
- Git (for version control)
-
Clone the Repository
git clone https://github.com/yourusername/news-research-tool.git cd news-research-tool -
Create Virtual Environment
python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate
-
Install Dependencies
pip install -r requirements.txt
-
Set Up Environment Variables Create a .env file in the project root:
GROQ_API_KEY=your_groq_api_key_here
streamlit run src/main.py- Document Processing Pipeline
URLs → HTML Download → Text Extraction → Chunking → Embeddings → FAISS Index
- Query Processing
User Question → Embedding → Similarity Search → Context Retrieval → LLM Answer Generation
- RAG Architecture
The tool implements Retrieval-Augmented Generation (RAG):
- Retrieval: Finds relevant document chunks using FAISS vector similarity
- Augmentation: Combines retrieved chunks with the user's question
- Generation: Uses Groq LLM to generate accurate, context-aware answers
- Support for PDF and DOCX files
- Batch processing for large article collections
- Multi-language support
- Advanced analytics and visualization
- Topic modeling and trend analysis
- User authentication and saved sessions
- API endpoint for programmatic access
- Integration with news APIs (NewsAPI, GDELT)
MIT License