Skip to content

niconiahi/docudocu

BranchesTags

Repository files navigation

docudocu

A hybrid search engine for developer documentation, designed with domain-driven principles.

Repository Status: This repository is frozen and serves as documentation and learning material only. See Teaching Skill for what kind of help is available.

Table of Contents

Documentation Index

Architecture

Document Description
docs/DESCRIPTION.md Start here. Complete architectural description with DDD bounded contexts, component interactions, and design decisions
docs/docudocu-class-diagram.png Visual class diagram showing all components and relationships
docs/architecture.puml PlantUML source for the diagram

API Reference

Document Description
CHANGELOG.md Complete API reference organized by feature step. All types, functions, methods, constants

Issue Specifications

Detailed specifications for each feature, organized by development step:

Step Directory Description Issues
0 linear-issues/00-licensing/ License verification, activate/license commands DOCU-001 to DOCU-004
1 linear-issues/01-update/ Data directory, sources, fetcher, snapshots DOCU-005 to DOCU-009
2 linear-issues/02-index/ Normalization, parsing, chunking, SQLite, Ollama, indexes DOCU-010 to DOCU-017
3 linear-issues/03-query/ Hybrid search, deduplication, formatter DOCU-018 to DOCU-021
4 linear-issues/04-status/ Status command DOCU-022
5 linear-issues/05-mcp/ MCP protocol, tools, serve command DOCU-023 to DOCU-025
6 linear-issues/06-backend/ License generation, payment setup DOCU-026 to DOCU-027
7 linear-issues/07-sync/ Package.json parser, lockfile parser, sync command DOCU-028 to DOCU-030
8 linear-issues/08-search-quality/ Search quality improvements (90% → 100%) DOCU-031

Summary: linear-issues/README.md — Overview of all 31 issues with estimates and dependency graph

Skills (Claude Code Instructions)

Skill Purpose
.claude/skills/teaching/SKILL.md Teaching mode: Explanations and Go code snippets only
.claude/skills/go/SKILL.md Go naming conventions (snake_case, SCREAMING_SNAKE_CASE)
.claude/skills/testing/SKILL.md Testing strategy and patterns
.claude/skills/documentation/SKILL.md Documentation and changelog conventions
.claude/skills/command_runner/SKILL.md Mage commands reference
.claude/skills/version_control/SKILL.md Git conventions (read-only)
.claude/skills/issue_tracking/SKILL.md Linear issue status conventions
.claude/skills/search_quality/SKILL.md Search quality principles (generic, not library-specific)

Package Reference

internal/indexer

Purpose: Text normalization, markdown parsing, and document chunking pipeline

File Description
normalize.go Converts raw markdown to normalized content (CRLF→LF, removes frontmatter, collapses blank lines, generates SHA256 hash)
parse.go Parses markdown into structured blocks (HeadingBlock, CodeBlock, ParagraphBlock) with heading hierarchy and anchors
chunk.go Splits parsed documents into searchable chunks (50-400 tokens, merges tiny fragments, generates unique IDs)

internal/db

Purpose: SQLite database layer for persisting chunks and metadata

File Description
db.go Database connection management, migrations (WAL mode), CRUD operations for chunks, targets, and metadata

internal/embedding

Purpose: Integration with Ollama for generating vector embeddings

File Description
ollama.go HTTP client for Ollama embedding service, supports custom model/URL/timeout, ping/health checks, batch embedding

internal/search

Purpose: Hybrid search combining BM25 keyword matching and semantic vector similarity

File Description
bm25.go BM25 probabilistic ranking index with term frequency analysis, fuzzy matching (Levenshtein), linguistic word expansion
vector.go Vector index using cosine similarity (1536-dimensional embeddings)
hybrid.go Combines BM25 and vector search with configurable alpha weighting (0-1), scoring, deduplication, topic detection
preprocess.go Query preprocessing with typo correction via Hunspell, vocabulary context awareness
formatter.go Formats search results into LLM-ready markdown with token limits, citations, confidence scores
section.go Section deduplication logic
wordforms/expander.go Linguistic word expansion library

internal/commands

Purpose: CLI command implementations (dispatcher layer)

File Description
activate.go License activation from key
license.go Display license status
update.go Download documentation from GitHub releases
index.go Build BM25 and vector indexes from downloaded docs
query.go Hybrid search with formatting (supports --raw, --top-k, --alpha flags)
serve.go MCP (Model Context Protocol) server for Claude Code integration
status.go Show documentation index status (fetched, indexed, updated)
sync.go Automatic multi-package indexing from package.json
help.go CLI help text with all commands and examples

internal/mcp

Purpose: Model Context Protocol server for Claude Code integration

File Description
server.go JSON-RPC 2.0 server, request/response handling, error codes
tools.go MCP tool definitions (search, index, etc.)

internal/license

Purpose: Ed25519 license verification and management

File Description
license.go Parses and validates Ed25519-signed licenses (base64-encoded), extracts email/timestamp
guard.go Enforces license requirement (RequireLicense checks)
storage.go Persists license to ~/.docudocu/license.json
pubkey.go Public key for license verification

internal/targets

Purpose: Target/package registry and version parsing

File Description
targets.go Registry of supported targets (svelte, svelte-kit, etc.) with GitHub metadata, version parsing

internal/source

Purpose: GitHub documentation fetching and extraction

File Description
github.go Downloads GitHub release tarballs, extracts documentation, validates checksums (SHA256)

internal/storage

Purpose: Data directory and path management

File Description
paths.go Standardized paths for targets, versions, indexes, databases (respects DOCUDOCU_DATA_DIR env var)

internal/version

Purpose: Version metadata tracking and management

File Description
version.go Stores/retrieves version metadata (fetched_at, content_hash, release_url), manages symlinks to current version

internal/lockfile

Purpose: package.json and lockfile parsing for auto-sync

File Description
package.go Parses package.json to extract dependencies and devDependencies
pnpm.go pnpm-lock.yaml parsing for version resolution

internal/hunspell

Purpose: Spell checking via C bindings to Hunspell

File Description
hunspell.go CGO bindings to Hunspell C library for typo detection/correction

Test Suites

Unit Tests

Every package has *_test.go files. Run with mage test.

Integration Tests (integration/)

Task-oriented tests for complete user workflows:

File Purpose Pipeline
index_documentation_from_files_test.go Full indexing pipeline markdown → normalize → parse → chunk → embed → index
search_documentation_test.go Hybrid search and result ranking query → preprocess → BM25+vector → hybrid score → format
threshold_gap_test.go Score gap detection validation Edge case handling
threshold_min_score_test.go Minimum score threshold validation Confidence filtering

Stress Tests (stress/)

17 scenario-based tests validating search quality across different query types:

# Test Category Query Example
01 Typo Tolerance Fuzzy matching "rective variables"
02 Conceptual Query Semantic understanding "share state between components"
03 Multi-Term Balance Multi-term handling "$state $derived runes"
04 Ambiguous Term API/Reference lookup "store"
05 Troubleshooting Intent recognition "load function running twice"
06 Comparison Intent recognition "difference page.js page.server.js"
07 Cross-Cutting Multi-term handling "authentication session cookies"
08 Non-Existent Edge case "graphql integration"
09 Acronym Expansion Fuzzy matching "SSR CSR hydration"
10 Code Pattern API/Reference lookup "$effect(() => { })"
11 Niche Feature API/Reference lookup "snapshot restore form state"
12 API Parameters API/Reference lookup "cookies.set options path domain"
13 Migration Syntax Edge case "on:click onclick svelte 5"
14 Framework Crossover Semantic understanding "useEffect equivalent"
15 Partial Term Fuzzy matching "preload data before navigation"
16 Error Handling Intent recognition "500 error handle server"
17 Constraint Query Intent recognition "render page without javascript disabled"

Status: All 17 tests passing (100%)

Each test directory contains:

  • README.md — Test description and expected results
  • *_test.go — Test implementation
  • reports/ — Version history of test results

Build & Commands

Mage Commands (magefile.go)

Command Description
mage build Compile binary to ./docudocu
mage test Run all unit tests
mage test:verbose Run tests with verbose output
mage test:integration Run integration tests
mage test:integration TestSearch Run specific integration test
mage clean Remove binary and ~/.docudocu directory
mage ollama:status Check Ollama server status
mage ollama:pull Pull embedding model
mage ollama:setup Full Ollama setup
mage ollama:run Start Ollama server

CLI Commands

docudocu help                    # Show help
docudocu activate <key>          # Activate license
docudocu license                 # Show license status

docudocu update svelte           # Fetch latest docs
docudocu index svelte            # Index documentation
docudocu query svelte "query"    # Search documentation
docudocu status svelte           # Show status

docudocu serve                   # Start MCP server
docudocu sync                    # Sync from package.json

Dependencies (go.mod)

Dependency Purpose
magefile/mage Build automation
modernc.org/sqlite Pure Go SQLite (no CGO for db)
Hunspell (system) CGO bindings for spell checking

Entry Point (cmd/docudocu/main.go)

Simple switch statement routing commands to internal/commands package functions.

Quick Reference

Bounded Contexts

┌─────────────────────────────────────────────────────────────────────┐
│  INDEXING CONTEXT                                                   │
│  "How do we transform raw docs into searchable units?"              │
│                                                                     │
│  Components: NormalizedContent, Document, Block*, Chunk             │
│  Package: internal/indexer                                          │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              │  []Chunk (shared kernel)
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│  PERSISTENCE CONTEXT                                                │
│  "How do we durably store and retrieve data?"                       │
│                                                                     │
│  Components: Database, StoredChunk, Target                          │
│  Package: internal/db                                               │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              │  Database, loaded indexes
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│  SEARCH CONTEXT                                                     │
│  "How do we find the most relevant content?"                        │
│                                                                     │
│  Components: HybridSearcher, BM25Index, VectorIndex, OllamaClient   │
│  Package: internal/search, internal/embedding                       │
└─────────────────────────────────────────────────────────────────────┘

Shared Kernel: Chunk

The central abstraction everything operates on:

type Chunk struct {
    ID          string      // Unique identifier
    TargetID    string      // Documentation source
    Version     string      // Documentation version
    FilePath    string      // Original file
    HeadingPath []string    // Breadcrumb navigation
    Anchor      string      // URL anchor
    Content     string      // The actual text
    ContentHash string      // SHA256 for incremental indexing
    HasCode     bool        // Contains code blocks?
    TokenCount  int         // For relevance scoring
}

Data Flow

GitHub Release
    → internal/source (fetch tarball)
    → internal/version (metadata)
    → internal/indexer (normalize → parse → chunk)
    → internal/db (store chunks)
    → internal/embedding (vector generation)
    → internal/search (BM25 + Vector indexes)
    → formatter (LLM context)

Data Directory

~/.docudocu/
├── license.key                  # License file
├── targets/
│   └── svelte/
│       ├── current -> versions/5.48.5/
│       └── versions/
│           └── 5.48.5/
│               ├── meta.json
│               └── documentation/
├── indexes/
│   └── svelte/
│       ├── vectors.gob          # Vector index
│       └── bm25.gob             # BM25 index
└── db/
    └── docudocu.sqlite          # Chunks, targets, state

Other Directories

Directory Purpose
playground/ Sample Svelte documentation for testing
stress/tool/ Utilities for running stress tests

File Lookup

If you need... Read this
Overall architecture docs/DESCRIPTION.md
Visual diagram docs/docudocu-class-diagram.png
API reference CHANGELOG.md
How indexing works linear-issues/02-index/
How search works linear-issues/03-query/
How MCP works linear-issues/05-mcp/
Search quality techniques linear-issues/08-search-quality/
Go naming conventions .claude/skills/go/SKILL.md
Testing patterns .claude/skills/testing/SKILL.md
Mage commands .claude/skills/command_runner/SKILL.md
All issues overview linear-issues/README.md
Stress test scenarios stress/README.md

Learning with This Repository

This repository is configured for teaching mode. You can:

  1. Ask for explanations — From simple to detailed
  2. Request Go code snippets — Isolated examples for specific APIs
  3. Understand the architecture — Read DESCRIPTION.md and ask questions

See Teaching Skill for full details.

About

Generate documentation from actual .md files from source and consult it locally through an MCP tool

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages