Skip to content

ambicuity/KINDX

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

164 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 ██╗  ██╗██╗███╗   ██╗██████╗ ██╗  ██╗
 ██║ ██╔╝██║████╗  ██║██╔══██╗╚██╗██╔╝
 █████╔╝ ██║██╔██╗ ██║██║  ██║ ╚███╔╝
 ██╔═██╗ ██║██║╚██╗██║██║  ██║ ██╔██╗
 ██║  ██╗██║██║ ╚████║██████╔╝██╔╝ ██╗
 ╚═╝  ╚═╝╚═╝╚═╝  ╚═══╝╚═════╝ ╚═╝  ╚═╝

KINDX

MCP-Compatible Local-First Node.js TypeScript License: MIT OpenSSF Scorecard

KINDX is a local-first hybrid CLI/MCP search engine for personal knowledge bases and agentic workflows, combining BM25, vector retrieval, and LLM reranking locally via node-llama-cpp/GGUF. It provides a command-line interface, a Model Context Protocol (MCP) server, and HTTP endpoints for indexing files, retrieving documents, and storing scoped agent memories.

The core package is a Node.js/TypeScript project. It can run local GGUF models through node-llama-cpp, or use an OpenAI-compatible remote backend such as Ollama or LM Studio for embedding, expansion, and reranking.

Features

  • Index local collections configured in YAML under the KINDX config directory.
  • Search with BM25 full-text queries, vector similarity, or hybrid structured queries.
  • Retrieve documents by path, virtual kindx:// path, glob, or docid.
  • Serve MCP tools over stdio or Streamable HTTP.
  • Expose HTTP endpoints for health, metrics, structured query, streaming query, and MCP.
  • Store scoped agent memories with text and semantic search.
  • Watch collections and incrementally update the index.
  • Run diagnostics, cleanup, backup, restore, and shard scheduler status commands.
  • Optionally extract text from PDF and DOCX files when local system tools are available.
  • Provide TypeScript schema/client packages and a Python LangChain retriever wrapper.

Tech Stack

  • Node.js >=20, TypeScript ESM, npm workspaces
  • SQLite through better-sqlite3; optional SQLCipher-capable runtime through better-sqlite3-multiple-ciphers
  • SQLite FTS5 and sqlite-vec
  • node-llama-cpp for local GGUF inference
  • @modelcontextprotocol/sdk for MCP
  • chokidar, fast-glob, yaml, and zod
  • Vitest for TypeScript tests
  • Python >=3.10 for python/kindx-langchain and training/
  • Docker runtime based on node:22-bookworm-slim

Project Structure

.
├── bin/kindx                  # npm executable wrapper for the compiled CLI
├── engine/                    # CLI, MCP/HTTP server, data layer, inference, memory, diagnostics
├── packages/kindx-schemas/    # shared Zod schemas and TypeScript types
├── packages/kindx-client/     # typed TypeScript client for KINDX HTTP/MCP APIs
├── python/kindx-langchain/    # Python retriever wrapper
├── specs/                     # Vitest test suite and fixtures
├── tooling/                   # release, benchmark, QA, and development utilities
├── training/                  # query-expansion training and evaluation tools
├── capabilities/kindx/        # packaged KINDX skill files
├── demo/                      # sample data, recipes, demos, and comparison fixtures
├── openclaw-integration/      # separate integration subtree
├── Dockerfile                 # multi-stage production image
├── package.json               # root package, workspace, scripts, and CLI metadata
└── sample-catalog.yml         # example collection configuration

Architecture

KINDX is organized around a TypeScript CLI, an MCP/HTTP integration layer, a SQLite-backed repository layer, and local or OpenAI-compatible inference.

Component Overview

graph TB
    subgraph CLI["CLI Layer (engine/kindx.ts)"]
        CMD["Command Parser"]
    end

    subgraph Engine["Core Engine"]
        REPO["Repository (engine/repository.ts)"]
        INF["Inference (engine/inference.ts)"]
        CAT["Catalogs (engine/catalogs.ts)"]
        REN["Renderer (engine/renderer.ts)"]
        RT["Runtime (engine/runtime.ts)"]
    end

    subgraph Protocol["Integration Layer"]
        MCP["MCP Server (engine/protocol.ts)"]
    end

    subgraph Storage["Data Layer"]
        SQLite["SQLite + FTS5"]
        VEC["sqlite-vec Vectors"]
        CACHE["LLM Response Cache"]
    end

    subgraph Models["Inference Backends"]
        EMBED["Embedding model"]
        RERANK["Rerank model"]
        EXPAND["Expansion model"]
    end

    CMD --> REPO
    CMD --> CAT
    CMD --> REN
    MCP --> REPO
    REPO --> RT
    REPO --> INF
    RT --> SQLite
    RT --> VEC
    INF --> EMBED
    INF --> RERANK
    INF --> EXPAND
    REPO --> CACHE
    CAT --> SQLite
Loading

Hybrid Retrieval Pipeline

flowchart TD
    Q["User Query"] --> SPLIT

    subgraph SPLIT["Query Processing"]
        ORIG["Original Query"]
        EXP["Optional Expansion"]
    end

    SPLIT --> P1 & P2 & P3

    subgraph P1["Original Query"]
        BM25_1["BM25 / FTS5"]
        VEC_1["Vector Search"]
    end

    subgraph P2["Expanded Query"]
        BM25_2["BM25 / FTS5"]
        VEC_2["Vector Search"]
    end

    subgraph P3["HyDE Query"]
        BM25_3["BM25 / FTS5"]
        VEC_3["Vector Search"]
    end

    P1 & P2 & P3 --> RRF

    subgraph RRF["Result Fusion"]
        FUSE["Reciprocal Rank Fusion"]
        BONUS["Strong retrieval signal preservation"]
        TOPK["Candidate selection"]
    end

    RRF --> RERANK_STEP

    subgraph RERANK_STEP["Optional Reranking"]
        LLM["Cross-encoder reranker"]
    end

    RERANK_STEP --> BLEND

    subgraph BLEND["Final Scoring"]
        B1["Retrieval score"]
        B2["Vector score"]
        B3["Rerank score"]
    end

    BLEND --> RESULTS["Ranked Results"]
Loading

Prerequisites

  • Node.js 20 or newer.
  • npm.
  • Git, if you use collection update commands or contribute to the repository.
  • Python 3.10 or newer for Python integration tests and training tools.
  • Optional local tools:
    • pdftotext for better PDF extraction.
    • unzip for DOCX extraction.
    • A C/C++ build toolchain for native Node modules when prebuilt binaries are unavailable.

Getting Started

Install the published CLI:

npm install -g @ambicuity/kindx
kindx --help

Configure MCP clients with the global binary:

{
  "mcpServers": {
    "kindx": {
      "command": "kindx",
      "args": ["mcp"]
    }
  }
}

For local development from source:

git clone https://github.com/ambicuity/KINDX.git
cd KINDX
npm install
npm run build

For local CLI development, run the TypeScript entry point through the package script:

npm run kindx -- --help

To link the built CLI as kindx on your machine:

npm link
kindx --help

Indexing A Collection

The quickest path is init, which registers a collection, indexes metadata, and generates embeddings:

npm run kindx -- init . --name notes

You can also run the steps separately:

npm run kindx -- collection add . --name docs
npm run kindx -- update
npm run kindx -- embed

Search the index:

npm run kindx -- search "authentication"
npm run kindx -- vsearch "how does authentication work"
npm run kindx -- query "authentication configuration"

Retrieve documents:

npm run kindx -- get "docs/architecture.md"
npm run kindx -- get "#abc123"
npm run kindx -- multi-get "docs/**/*.md"

Application Entry Points

Entry point Purpose
npm run kindx -- <args> Runs engine/kindx.ts with tsx for local development.
bin/kindx npm package executable wrapper. Requires npm run build first in a clone.
node dist/kindx.js <args> Runs the compiled CLI directly.
kindx mcp Starts the MCP server over stdio.
kindx mcp --http [--daemon] Starts the MCP HTTP server.
Docker CMD Runs node dist/kindx.js mcp --http --port 8181.

Available Scripts

Command Description
npm run prepare Installs repository hooks when .git exists.
npm run build Compiles engine/**/*.ts to dist/ and makes dist/kindx.js executable.
npm run build:packages Builds @ambicuity/kindx-schemas and @ambicuity/kindx-client.
npm run build:all Builds the CLI and workspace packages.
npm test Builds the CLI, then runs Vitest tests in specs/.
npm run test:packages Runs package tests for schemas and client.
npm run test:python Runs Python unittest discovery for python/kindx-langchain/tests.
npm run test:openclaw-integration Runs the configured OpenClaw integration Vitest target through pnpm.
npm run test:all Builds all TypeScript targets, then runs root, package, and Python tests.
npm run kindx -- <args> Runs the CLI TypeScript source with tsx.
npm run index Runs kindx index.
npm run vector Runs kindx vector.
npm run search Runs kindx search.
npm run vsearch Runs kindx vsearch.
npm run rerank Runs kindx rerank.
npm run perf:warm-daemon Runs the warm daemon benchmark helper.
npm run perf:llm-pool-contention Runs the LLM pool contention benchmark helper.
npm run bench:quality Runs enforced BM25 quality benchmarks.
npm run bench:latency Runs latency benchmarks.
npm run bench:regressions Runs enforced insert regression benchmarks.
npm run bench:daemon Runs daemon load benchmarks.
npm run bench:all Runs all enforced benchmark tracks.
npm run bench:full Runs all benchmark tracks including optional tracks.
npm run bench:section6 Runs the Section 6 benchmark script.
npm run qa:customer-pov Runs the P0 customer point-of-view launch gate.
npm run qa:customer-pov:p0 Writes P0 customer point-of-view results to tooling/artifacts/customer-pov-p0.json.
npm run qa:customer-pov:p1 Writes P1 customer point-of-view results to tooling/artifacts/customer-pov-p1.json.
npm run qa:customer-pov:p2 Writes P2 customer point-of-view results to tooling/artifacts/customer-pov-p2.json.
npm run qa:customer-pov:p3 Writes P3 customer point-of-view results to tooling/artifacts/customer-pov-p3.json.
npm run qa:customer-pov:all Writes all-phase customer point-of-view results to tooling/artifacts/customer-pov-all.json.
npm run arch:status Runs kindx arch status.
npm run arch:refresh Runs kindx arch refresh.
npm run inspector Starts the MCP inspector against tsx engine/kindx.ts mcp.
npm run release Runs tooling/release.sh.

CLI Commands

Run kindx --help or npm run kindx -- --help for the full command reference.

Common commands:

Command Description
kindx query <query> Hybrid search with expansion and reranking.
kindx query $'lex: ...\nvec: ...' Structured query document with typed subqueries.
kindx search <query> BM25/FTS full-text search.
kindx vsearch <query> Vector similarity search.
kindx get <file>[:line] Retrieve one document by path or docid.
kindx multi-get <pattern> Retrieve multiple documents by glob or comma-separated list.
kindx collection add <path> [--name <name>] [--mask <glob>] Register a collection.
kindx collection list List configured collections.
kindx context add/list/rm Manage human-written context annotations.
kindx ls [collection[/path]] Inspect indexed files.
kindx update [--pull] Re-index configured collections.
kindx embed [-f] [--resume] Generate or refresh vector embeddings.
kindx watch [collections...] Run the incremental indexing daemon.
kindx status Show index and collection health.
kindx pull [--refresh] Download or check default local GGUF models.
kindx cleanup Clear caches and vacuum the database.
kindx doctor Run deterministic health diagnostics.
kindx repair --check-only Dry-run integrity and repair checks.
kindx backup <create|verify|restore> [path] Manage SQLite backups.
kindx scheduler status Show shard sync checkpoint and queue status.
kindx verify-wipe Scan for residual local index artifacts.
kindx memory <subcommand> Manage scoped agent memories.
kindx arch <status|build|import|refresh> Run optional Arch sidecar integration commands.
kindx migrate chroma <path> Migrate a ChromaDB SQLite file.
kindx migrate openclaw <path> Migrate an OpenClaw repository to use KINDX.

Build

npm run build
npm run build:packages
npm run build:all

The root build compiles engine/**/*.ts into dist/. The package build compiles the workspace packages under packages/.

Testing

npm test
npm run test:packages
npm run test:python
npm run test:all

npm test runs npm run build first through the pretest script.

Configuration

KINDX reads runtime configuration from environment variables. The repository includes .env.example, but the application does not load dotenv files automatically. Export variables in your shell, set them in your process manager, or pass them to Docker with -e.

Collection configuration is stored as YAML. By default, the config file is ~/.config/kindx/index.yml; the example shape is shown in sample-catalog.yml.

Index data is stored in SQLite. By default, the database path is under ~/.cache/kindx/, unless overridden.

Environment Variables

Variable Required Description
INDEX_PATH No Overrides the SQLite database path.
KINDX_CONFIG_DIR No Overrides the KINDX config directory.
XDG_CONFIG_HOME No Base directory used for config when KINDX_CONFIG_DIR is unset.
XDG_CACHE_HOME No Base directory used for cache, index, PID, model, and Arch artifact paths.
HOME No Used to resolve default config/cache paths and ~/ collection paths.
KINDX_MCP_TOKEN No Bearer token for HTTP MCP/query authentication in single-tenant mode.
KINDX_HTTP_CONCURRENCY No Maximum concurrent HTTP requests. Default: 150.
KINDX_MAX_CONCURRENCY_PER_TENANT No Maximum concurrent HTTP requests per tenant. Default: 10.
KINDX_RATE_LIMIT_BURST No Per-tenant rate-limit burst size. Default: 100.
KINDX_RATE_LIMIT_MS No Rate-limit window in milliseconds. Default: 1000.
KINDX_SQLITE_DRIVER No Forces a SQLite driver module. By default KINDX tries better-sqlite3-multiple-ciphers, then better-sqlite3.
KINDX_ENCRYPTION_KEY No Enables keyed SQLCipher runtime support when using a compatible SQLite driver.
KINDX_LLM_BACKEND No Set to remote to use the OpenAI-compatible backend; otherwise local node-llama-cpp is used.
KINDX_CPU_ONLY No Set to 1 to force local model execution on CPU.
KINDX_EMBED_MODEL No Local embedding model URI or model override used by inference.
KINDX_RERANK_MODEL No Local reranking model URI.
KINDX_GENERATE_MODEL No Local generation/query-expansion model URI.
KINDX_OPENAI_BASE_URL No Base URL for the OpenAI-compatible remote backend. Default: http://localhost:11434/v1.
KINDX_OPENAI_API_KEY No API key for the remote backend. If set, KINDX sends it as a bearer token.
KINDX_OPENAI_EMBED_MODEL No Remote embedding model name.
KINDX_OPENAI_GENERATE_MODEL No Remote generation model name.
KINDX_OPENAI_RERANK_MODEL No Remote reranking model name.
KINDX_EXPAND_CONTEXT_SIZE No Context size for expansion sessions.
KINDX_RERANK_CONTEXT_SIZE No Context size for reranking sessions.
KINDX_VRAM_RESERVE_MB No Reserve budget used by local model/device planning.
KINDX_LLM_POOL_SIZE No Size override for the default LLM pool.
KINDX_QUERY_TIMEOUT_MS No Query timeout guard. 0 disables it.
KINDX_INFLIGHT_DEDUPE No In-flight query deduplication mode. Supported value in code: join; other values disable joining.
KINDX_QUERY_REPLAY_DIR No Directory for query replay artifacts.
KINDX_MAX_RERANK_CANDIDATES No Maximum rerank candidate ceiling.
KINDX_RERANK_TIMEOUT_MS No Timeout budget for reranking before fallback.
KINDX_RERANK_QUEUE_LIMIT No Queue length cap for reranking.
KINDX_RERANK_CONCURRENCY No Rerank worker concurrency.
KINDX_RERANK_DROP_POLICY No Rerank backpressure policy: timeout_fallback or wait.
KINDX_VECTOR_FANOUT_WORKERS No Parallel worker cap for vector fanout.
KINDX_ANN_ENABLE No Enables ANN routing for sharded collections. Default: 1.
KINDX_ANN_CENTROIDS No Requested centroid count for ANN shard routing.
KINDX_ANN_PROBE_COUNT No Number of ANN centroid probes per shard. Default: 4.
KINDX_ANN_SHORTLIST No ANN shortlist size per shard.
KINDX_EXTRACTOR_PDF No Enables PDF extraction. Default: enabled.
KINDX_EXTRACTOR_DOCX No Enables DOCX extraction. Default: enabled.
KINDX_EXTRACTOR_FALLBACK_POLICY No PDF fallback behavior: fallback or strict.
CHOKIDAR_USEPOLLING No Set to 1 or true to make the watch daemon use polling.
CHOKIDAR_INTERVAL No Polling interval for chokidar. Default: 100.
KINDX_ARCH_ENABLED No Enables Arch sidecar build/import/refresh commands.
KINDX_ARCH_AUGMENT_ENABLED No Enables Arch hints in query results.
KINDX_ARCH_AUTO_REFRESH_ON_UPDATE No Rebuilds/imports Arch artifacts after kindx update.
KINDX_ARCH_REPO_PATH No Path to the external Arch repository. Default: ./tmp/arch.
KINDX_ARCH_PYTHON_BIN No Python executable for Arch commands. Default: python3.
KINDX_ARCH_COLLECTION No Collection name for imported Arch artifacts. Default: __arch.
KINDX_ARCH_ARTIFACT_DIR No Distilled Arch artifact directory.
KINDX_ARCH_MIN_CONFIDENCE No Minimum Arch confidence: EXTRACTED, INFERRED, or AMBIGUOUS.
KINDX_ARCH_MAX_HINTS No Maximum Arch hints per query. Default: 3.
KINDX_LOG_LEVEL No Logging verbosity: DEBUG, INFO, WARN, or ERROR.
KINDX_LOG_JSON No Set to 1 or true for JSON logs.
NO_COLOR No Disables colored CLI output.
KINDX_ENABLE_MAINTENANCE_TOOLS No Exposes maintenance MCP tools such as status, arch_status, memory_stats, memory_history, memory_mark_accessed, memory_delete, and memory_bulk.
KINDX_TRUST_UPDATE_CMDS No Trusts configured collection update commands when set to 1 or true.
KINDX_MCP_SERVERS_JSON No Inline MCP control-plane server configuration.
KINDX_TRUST_PROJECT No Marks a project-scoped MCP control-plane config as trusted.

HTTP And MCP Server

Start the stdio MCP server:

kindx mcp

Start the HTTP server:

kindx mcp --http
kindx mcp --http --port 8080
kindx mcp --http --daemon
kindx mcp stop

HTTP mode uses bearer-token authentication for protected endpoints when KINDX_MCP_TOKEN is configured or when multi-tenant RBAC is enabled. /health and /metrics are intentionally unauthenticated.

HTTP Endpoints

Endpoint Description
GET /health Liveness response with status and uptime.
GET /metrics Prometheus text metrics.
POST /query Structured retrieval endpoint.
POST /search Alias for POST /query.
POST /query/stream Streaming structured query endpoint.
POST /mcp MCP Streamable HTTP endpoint.

POST /query accepts this request shape:

{
  "searches": [
    { "type": "lex", "query": "authentication" },
    { "type": "vec", "query": "how authentication is configured" },
    { "type": "hyde", "query": "Authentication is configured with tokens, secrets, and middleware." }
  ],
  "limit": 10,
  "minScore": 0,
  "candidateLimit": 40,
  "collections": ["docs"]
}

type must be one of lex, vec, or hyde.

MCP Tools

The server registers these core tools:

Tool Description
query Structured search across one or more lex, vec, or hyde subqueries.
get Retrieve one document by path or docid.
multi_get Retrieve multiple documents by glob or list.
memory_put Store or update a scoped memory.
memory_search Search scoped memories by semantic or text mode.
arch_query Retrieve optional Arch integration hints.

Additional maintenance tools are registered only when KINDX_ENABLE_MAINTENANCE_TOOLS is set: status, arch_status, memory_history, memory_stats, memory_mark_accessed, memory_delete, and memory_bulk.

The TypeScript client exposes helpers for /query, get, multi_get, status, memory_put, memory_search, memory_history, and memory_mark_accessed.

Docker

Build the image:

docker build -t kindx .

Run the HTTP MCP server on port 8181:

docker run --rm \
  -p 8181:8181 \
  -v kindx-data:/data \
  -e KINDX_MCP_TOKEN=<token> \
  kindx

The Docker image sets HOME=/data, KINDX_LOG_JSON=1, and KINDX_SQLITE_DRIVER=better-sqlite3-multiple-ciphers.

Workspace Packages

@ambicuity/kindx-schemas

Shared Zod schemas and TypeScript types for KINDX request and response payloads.

npm run build -w @ambicuity/kindx-schemas
npm run test -w @ambicuity/kindx-schemas

@ambicuity/kindx-client

Typed TypeScript client for the KINDX HTTP and MCP APIs.

npm run build -w @ambicuity/kindx-client
npm run test -w @ambicuity/kindx-client

Python Integration

python/kindx-langchain contains an installable Python package named kindx-langchain.

python3 -m unittest discover -s python/kindx-langchain/tests -v

The package requires Python >=3.10. Its optional langchain extra installs langchain-core>=0.3.0.

Training Tools

training/ contains Python tooling for query-expansion fine-tuning and evaluation. It is a separate Python project named qmd-finetune with Python >=3.10 and dependencies declared in training/pyproject.toml.

See training/README.md for training-specific commands.

Database And Storage

KINDX stores indexed content in SQLite. The schema includes:

  • content for content-addressed document bodies.
  • documents for collection/path metadata and active document mapping.
  • documents_fts for FTS5 full-text search.
  • content_vectors and a sqlite-vec virtual table for vector search.
  • llm_cache for LLM expansion/rerank cache entries.
  • memory, audit, and AI usage tables initialized by the corresponding engine modules.

Collection definitions live in YAML config, not in the SQLite collections table.

How It Works

Indexing Flow

flowchart LR
    COL["Collection Config"] --> GLOB["Glob Pattern Scan"]
    GLOB --> FILES["Supported Files"]
    FILES --> INGEST["Ingest / Extract Text"]
    INGEST --> PARSE["Title + Content Hash"]
    PARSE --> SQL["Store in SQLite"]
    SQL --> FTS["FTS5 Full-Text Index"]
Loading

Embedding Flow

Documents are split into chunks before embedding. The chunker prefers natural document boundaries such as headings, paragraphs, and code fences where possible.

flowchart LR
    DOC["Document"] --> CHUNK["Smart Chunking"]
    CHUNK --> FMT["Embedding Text"]
    FMT --> LLM["Embedding Backend"]
    LLM --> STORE["sqlite-vec Vectors"]
    CHUNK --> META["Chunk Metadata"]
    META --> STORE
Loading

Local inference defaults are defined in engine/inference.ts and can be overridden with KINDX_EMBED_MODEL, KINDX_RERANK_MODEL, and KINDX_GENERATE_MODEL. Set KINDX_LLM_BACKEND=remote to use the OpenAI-compatible backend instead of local GGUF execution.

Contributing

See CONTRIBUTING.md for contributor setup and workflow details. At minimum, verify the root build and tests before opening a pull request:

npm run build
npm test

Security issues should be reported according to SECURITY.md.

License

MIT. See LICENSE.

About

Local-first hybrid cli search engine (BM25 + Vector) for personal knowledge bases and agentic workflows. Runs entirely on-device via node-llama-cpp.

Topics

Resources

License

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Sponsor this project

  •  

Packages

 
 
 

Contributors