AI Research Paper Intelligence System

A full-stack AI research paper intelligence platform for semantic search, analysis, and recommendations using machine learning. Built with React, FastAPI, and Jupyter notebooks.

Features

• Semantic Search: Find relevant papers using Sentence Transformers and FAISS vector similarity
• Keyword Extraction: Automatically extract important keywords from papers
• Technical Entity Extraction: Identify models, datasets, metrics, and frameworks
• PDF Analyzer: Upload and analyze research PDFs
• Paper Q&A: Ask questions about paper content
• Recommendations: Find similar papers based on vector embeddings
• Demo Mode: Works out-of-box with sample data (full dataset optional)

Prerequisites

• Python 3.8+ (tested with Python 3.14)
• Node.js 16+ and npm/yarn
• Git for version control
• Optional: Google Colab for running notebooks with GPU

Project Architecture

Design Principle

ML development lives in Jupyter notebooks (.ipynb files only), not in production code. The backend connects the UI with processed outputs from notebooks.

Generated Data Files (from Notebooks)

After running notebooks, copy these files to backend/data/:

• cleaned_arxiv_papers.csv — Processed paper metadata (title, abstract, etc.)
• arxiv_embeddings.npy — Vector embeddings from Sentence Transformers
• faiss_index.index — FAISS index for fast similarity search

Note: Without these files, the app runs in demo mode using 6 sample papers.

Dataset Used

Dataset: CShorten/ML-ArXiv-Papers from Hugging Face

Amount used in the final notebooks: 50,000 research papers

Main fields used:

• title
• abstract
• paper_text = title + abstract

Folder Structure

AI_Research_Paper_Intelligence_System/
├── backend/                          # FastAPI backend
│   ├── main.py                       # App entry point
│   ├── schemas.py                    # Request/response models
│   ├── requirements.txt              # Python dependencies
│   ├── routes/                       # API route handlers
│   │   ├── search.py
│   │   ├── keywords.py
│   │   ├── entities.py
│   │   ├── recommend.py
│   │   ├── summarize.py
│   │   ├── pdf_analyzer.py
│   │   └── qa.py
│   ├── services/                     # Business logic
│   │   ├── search_service.py
│   │   ├── keyword_service.py
│   │   ├── entities_service.py
│   │   ├── recommendation_service.py
│   │   ├── summarizer_service.py
│   │   ├── pdf_service.py
│   │   └── qa_service.py
│   ├── utils/                        # Utilities
│   │   ├── config.py
│   │   └── __init__.py
│   ├── data/                         # Generated dataset files (gitignored)
│   └── README.md
│
├── frontend/                         # React + Vite frontend
│   ├── src/
│   │   ├── App.jsx                   # Main app component
│   │   ├── main.jsx                  # Entry point
│   │   ├── components/               # Reusable components
│   │   │   ├── Navbar.jsx
│   │   │   └── ResultCard.jsx
│   │   ├── pages/                    # Page components
│   │   │   ├── Home.jsx
│   │   │   ├── SearchPapers.jsx
│   │   │   ├── PdfAnalyzer.jsx
│   │   │   ├── PaperQA.jsx
│   │   │   └── Dashboard.jsx
│   │   ├── services/                 # API client
│   │   │   └── api.js
│   │   └── styles/
│   │       └── global.css
│   ├── index.html
│   ├── package.json
│   └── README.md
│
├── notebooks/                        # Jupyter notebooks (ML development)
│   ├── 01_eda_and_embeddings.ipynb   # Data processing & embeddings
│   ├── 02_search_engine.ipynb        # FAISS index creation
│   ├── 03_pdf_paper_analyzer.ipynb   # PDF parsing
│   └── 04_recommendation_system.ipynb # Recommendation logic
│
├── docs/                             # Documentation
│   └── PROJECT_NOTES.md
│
├── verify_project.py                 # Project validation script
├── .gitignore                        # Git ignore rules
└── README.md                         # This file

Quick Start

Option 1: Demo Mode (No Setup Required)

Start immediately with 6 sample papers:

# Terminal 1: Start backend
cd backend
pip install -r requirements.txt
uvicorn main:app --reload --app-dir . --port 8000

# Terminal 2: Start frontend
cd frontend
npm install
npm run dev

Open http://localhost:5173 in your browser.

Option 2: Full Dataset (Complete Setup)

Step 1: Generate Dataset with Notebooks

Run notebooks in sequence in Google Colab or local Jupyter:

# Local setup
pip install jupyter pandas numpy torch transformers sentence-transformers faiss-cpu scikit-learn
jupyter notebook notebooks/

Run notebooks in order:

1. 01_eda_and_embeddings.ipynb

   • Loads CShorten/ML-ArXiv-Papers dataset
   • Outputs: cleaned_arxiv_papers.csv, arxiv_embeddings.npy

2. 02_search_engine.ipynb

   • Creates FAISS index for fast search
   • Outputs: faiss_index.index

3. 03_pdf_paper_analyzer.ipynb

   • PDF extraction and processing

4. 04_recommendation_system.ipynb

   • Similarity-based recommendations

Step 2: Copy Generated Files

# Copy outputs from notebooks to backend
cp cleaned_arxiv_papers.csv backend/data/
cp arxiv_embeddings.npy backend/data/
cp faiss_index.index backend/data/

Step 3: Start Application

# Terminal 1: Backend
cd backend
pip install -r requirements.txt
uvicorn main:app --reload --app-dir . --port 8000

# Terminal 2: Frontend
cd frontend
npm install
npm run dev

Endpoints:

• Frontend: http://localhost:5173 (or http://localhost:5174, etc.)
• Backend API: http://127.0.0.1:8000
• API Docs: http://127.0.0.1:8000/docs

API Endpoints

Core

• GET / — Project info
• GET /health — Health check

Search & Discovery

• POST /api/search — Semantic search by query

  • Body: {query: string, top_k: int, include_summary: bool}
  • Returns: Array of matching papers with scores

• POST /api/recommend — Find similar papers

  • Body: {paper_index: int, top_k: int}
  • Returns: Recommended papers with similarity scores

Analysis

• POST /api/keywords — Extract keywords

  • Body: {text: string}
  • Returns: {keywords: []}

• POST /api/entities — Extract technical entities

  • Body: {text: string}
  • Returns: {models: [], datasets: [], metrics: [], frameworks: []}

• POST /api/summarize — Summarize text

  • Body: {text: string}
  • Returns: {summary: string}

• POST /api/ask-paper — Q&A on paper text

  • Body: {question: string, paper_text: string, top_k: int}
  • Returns: {answers: [string]}

File Processing

• POST /api/upload-pdf — Upload and analyze PDF
  • Body: multipart form with file (PDF)
  • Returns: Extracted text, sections, keywords, entities

Interactive API docs: http://127.0.0.1:8000/docs

Dataset

Source: CShorten/ML-ArXiv-Papers (Hugging Face Hub)

Size: 50,000 research papers (optional for full deployment)

Fields Used:

• title — Paper title
• abstract — Paper abstract
• paper_text — Combined title + abstract for embedding

Troubleshooting

Backend won't start

Issue: Could not import module "main"

Solution: Ensure you're in the project root, then:

uvicorn main:app --reload --app-dir backend --port 8000

Frontend shows blank page

Issue: "React is not defined" error in console

Solution: React import is missing in components. Ensure all components import React:

import React from 'react';

Port already in use

Solution: Change port:

uvicorn main:app --port 8001          # Backend
npm run dev -- --port 5175            # Frontend

Missing dependencies

Backend:

cd backend
pip install -r requirements.txt

Frontend:

cd frontend
npm install
# or clear cache: npm cache clean --force

Large file warnings

If notebooks output files >100MB, add them to .gitignore (already done for:

• backend/data/*.csv
• backend/data/*.npy
• backend/data/*.index )

Development Notes

Demo Fallback Mode

The backend includes 6 sample papers and runs without external dataset files:

SAMPLE_PAPERS = [
    {"title": "Attention Based Neural Machine Translation", ...},
    {"title": "Convolutional Neural Networks for Medical Image Analysis", ...},
    # ... 4 more
]

Use this for quick testing without running notebooks.

V1 Scope

This is a demo/educational project. Future enhancements:

• User authentication & multi-user DB
• Real-time arXiv crawling
• Model fine-tuning
• Production deployment (Docker, load balancing)
• Citation graph analysis
• Advanced filtering & faceted search

the backend uses a small built-in demo dataset and TF-IDF fallback search. This is only for testing the app flow. For the actual ML system, run the notebooks and copy the generated files into backend/data/.