Multisource Search Engine for Developers

A domain-specific search engine that indexes and searches developer-related content (e.g. GitHub repositories, Dev.to articles) and serves ranked results through a REST API and a React frontend. When a query returns weak or no results, it logs the gap and triggers background crawls to fill it, so the index improves over time.

Status: work in progress. The backend search, caching, background crawling, analytics, and a basic UI are functional. The project is not deployed; it runs locally via Docker Compose. See Testing & Quality for the automated test suite and CI.

Features

• Full-text search over indexed documents using Elasticsearch (BM25), with phrase boosting and a fuzzy-match fallback for typo tolerance.
• Filters and facets by source, tags, and language.
• Autocomplete suggestions using Elasticsearch search_as_you_type.
• Redis caching (cache-aside) on search results, with cache invalidation on reindex.
• Background crawling with BullMQ (queue, worker, scheduler) to ingest content from external sources.
• Demand-driven recrawling: low-result and low-quality queries are tracked and can trigger gated background crawls.
• Search analytics: logged searches, result counts, click tracking, and analytics endpoints.
• Input validation with Zod and centralized error handling.
• Rate limiting on API routes (returns HTTP 429 when exceeded).
• API documentation via Swagger UI at /api/docs.

Tech Stack

• Frontend: React, TypeScript, Vite
• Backend: Node.js, Express (v5), TypeScript
• Database: PostgreSQL with Prisma ORM
• Search: Elasticsearch
• Cache: Redis
• Background jobs: BullMQ
• Testing: Vitest, Supertest
• CI: GitHub Actions

Architecture

React client
   │  HTTP
   ▼
Express API ──► Redis cache ──(miss)──► Elasticsearch
   │                                        ▲
   │                                        │ index
   ▼                                        │
PostgreSQL ◄── Crawler / ingestion ◄── BullMQ worker ◄── External APIs

The backend follows a layered structure: route → validator (Zod) → controller → service → repository.

Project Structure

.
├── client/                 # React + TypeScript + Vite frontend
└── server/                 # Node + Express + TypeScript backend
    ├── prisma/             # Prisma schema and migrations
    ├── src/
    │   ├── config/         # env, logger, prisma, redis, queue, elasticsearch
    │   ├── controllers/    # request handlers
    │   ├── crawler/        # ingestion + per-source fetchers/normalizers
    │   ├── jobs/           # BullMQ job definitions and scheduler
    │   ├── middleware/     # validation, error handling, rate limiting, logging
    │   ├── repositories/   # data access (Prisma)
    │   ├── routes/         # route definitions
    │   ├── search/         # Elasticsearch search service and indexing
    │   ├── services/       # business logic (cache, analytics, demand, documents)
    │   └── validators/     # Zod schemas
    |   └── worker/         # worker processes
    |   └── types/
    |   └── utills/  
    └── test/               # Vitest + Supertest tests

Getting Started

Prerequisites

• Node.js 20+
• Docker and Docker Compose (for PostgreSQL, Elasticsearch, and Redis)

Setup

1. Clone and install

   git clone https://github.com/sarbik99/op-search-engine.git
   cd op-search-engine/server
   npm install

2. Configure environment

   Copy the example file and fill in the values:

   cp .env.example .env

   Key variables (see src/config/env.ts for the full list and defaults):

   Variable	Description	Default
   DATABASE_URL	PostgreSQL connection string	(required)
   ELASTICSEARCH_URL	Elasticsearch URL	http://localhost:9200
   REDIS_URL	Redis URL	redis://localhost:6379
   PORT	API port	5000
   CACHE_TTL_SECONDS	Search cache TTL	300
   SEARCH_PROVIDER	elasticsearch or postgres	elasticsearch
   LOW_QUALITY_MIN_RESULTS	Threshold for low-result detection	5
   LOW_QUALITY_MIN_TOP_SCORE	Soft threshold for low top score	10
   GITHUB_TOKEN	GitHub token for crawling (optional)	(optional)

3. Start backing services (PostgreSQL, Elasticsearch, Redis)

   docker compose up -d

4. Run database migrations and generate the Prisma client

   npx prisma generate
   npx prisma migrate deploy

5. Ingest some data (so search returns results)

   npm run ingest:devto
   npm run ingest:github   # requires GITHUB_TOKEN
   npm run reindex         # index documents into Elasticsearch

6. Run the API and worker

   npm run dev          # API server
   npm run worker       # background worker (separate terminal)

7. Run the frontend

   cd ../client
   npm install
   npm run dev

API

Interactive documentation is available at /api/docs (Swagger UI) when the server is running.

Main endpoints:

Method	Path	Description
GET	/api/health	Health check
GET	/api/search	Search documents (filters, paging)
GET	/api/search/autocomplete	Autocomplete suggestions
GET	/api/documents	Document CRUD
GET	/api/analytics/*	Search analytics
POST	/api/crawl	Trigger a background crawl
POST	/api/demand/process	Process demand-driven recrawls

Example:

curl "http://localhost:5000/api/search?q=redis%20pubsub&source=DEVTO&limit=10"

Testing & Quality

The backend has an automated test suite using Vitest (unit) and Supertest (HTTP integration). Tests mock Elasticsearch, Redis, and PostgreSQL, so they run quickly and require no external services.

Current coverage of behavior:

• Validators (unit): query and autocomplete Zod schemas.
• Search route (integration): 200 for a valid query and 400 for a missing query, exercising routing, validation, and the controller.
• Rate limiter: allows requests up to the limit and returns 429 beyond it.

Run the tests:

cd server
npm test              # run once
npm run test:watch    # watch mode
npm run test:coverage # with coverage report

Continuous Integration

A GitHub Actions workflow (.github/workflows/ci.yml) runs the full test suite on every push and pull request, on a clean Ubuntu runner. It installs dependencies, generates the Prisma client, and runs the tests. A failing test fails the pipeline.

Possible Future Work

• Deployment to managed services (PostgreSQL, Elasticsearch, Redis, worker).
• Additional content sources (StackOverflow, MDN, npm).
• Semantic/vector search and hybrid ranking (BM25 + embeddings).
• User features: search history, bookmarks, saved collections.