Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
192 lines (146 loc) · 7.1 KB

DEVELOPMENT.md

File metadata and controls

192 lines (146 loc) · 7.1 KB
Raw

ChatCrystal Development Guide

English | 简体中文

This guide covers repository structure, architecture, development commands, testing, and release workflows.

Project Overview

ChatCrystal is a local-first AI conversation crystallization tool. It imports conversations from AI coding tools, generates structured notes with LLMs, builds embeddings for semantic search, and exposes both UI and MCP workflows.

Monorepo Layout

ChatCrystal/
├── shared/                  # Shared TypeScript types
├── server/                  # Fastify backend, CLI, MCP server
├── client/                  # React SPA
├── electron/                # Electron main and preload processes
├── skills/                  # Publishable ChatCrystal agent skills
├── docs/                    # Maintainer and user documentation
├── scripts/                 # Release and utility scripts
└── site/                    # Project website

Tech Stack

Layer Technology
Backend Node.js, Fastify v5, TypeScript
Frontend Vite v8, React 19, Tailwind CSS v4, TanStack React Query v5
Desktop Electron, electron-builder
Database sql.js WASM SQLite
LLM Vercel AI SDK v6
Embeddings vectra local vector index
Queue p-queue
File watching chokidar

Development Commands

npm run dev                   # Server 3721 + client 13721
npm run build                 # Build server and client
npm start                     # Production server
npm run lint                  # Biome + client ESLint
npm run lint:fix              # Apply safe lint fixes
npm run test -w server        # Server tests
npm run dev:electron          # Electron dev mode
npm run build:electron        # Build Windows installer
npm run pack:electron         # Build unpacked Electron app
npm run eval:experience -w server

npm run eval:experience -w server runs the offline calibration suite for the experience quality gate.

Runtime Data

Runtime data is stored in config.json and chatcrystal.db under the active data directory.

Default data directory:

  • CLI, MCP, npm package, repository checkout, and Electron: ~/.chatcrystal/data
  • Explicit override: DATA_DIR

Electron sets ELECTRON=true, DATA_DIR, and ELECTRON_PACKAGED when applicable.

Data Flow

AI tool conversation files
  -> SourceAdapter scan/parse
  -> Import service deduplication
  -> SQLite conversations/messages
  -> Summarization queue
  -> LLM structured note generation
  -> Embedding generation
  -> vectra semantic index
  -> REST API, UI, CLI, MCP

Summarization Pipeline

ChatCrystal uses turn-based transcript preparation before summarization:

  1. Split messages into user-assistant turns.
  2. Keep the user instruction plus the first and last substantial assistant replies in each turn.
  3. Score turns by instruction length and assistant engagement.
  4. Always include the first turn and final turns.
  5. Fill the remaining budget with high-value middle turns.
  6. Compress skipped turns into one-line previews.

Structured output uses Vercel AI SDK generateObject() with Zod schemas. This avoids fragile JSON extraction and lets schema validation retry invalid model output.

Source Adapters

Add a new source by implementing SourceAdapter:

interface SourceAdapter {
  name: string;
  displayName: string;
  detect(): Promise<SourceInfo | null>;
  scan(): Promise<ConversationMeta[]>;
  parse(meta: ConversationMeta): Promise<ParsedConversation>;
}

Built-in adapters:

Adapter Data Source Format
claude-code ~/.claude/projects/**/*.jsonl JSONL conversation log
codex ~/.codex/sessions/**/rollout-*.jsonl JSONL event stream
cursor Cursor workspaceStorage/state.vscdb SQLite KV store
trae Trae workspaceStorage/state.vscdb SQLite KV store
copilot VS Code workspaceStorage/chatSessions/*.jsonl JSONL snapshots

Create the adapter under server/src/parser/adapters/ and register it in server/src/parser/index.ts.

API Surface

Key REST endpoints:

Method Path Description
GET /api/status Server status and statistics
GET /api/config Current config with secrets redacted
POST /api/config Update provider config
POST /api/import/scan Trigger import
GET /api/conversations List conversations
GET /api/conversations/:id Conversation detail
POST /api/conversations/:id/summarize Summarize one conversation
POST /api/summarize/batch Batch summarization
GET /api/notes List notes
GET /api/notes/:id Note detail
GET /api/search?q=...&expand=true Semantic search
GET /api/graph/projection Bounded graph projection for the UI
GET /api/relations/graph Legacy note relation graph data
GET /api/queue/status Queue status

Knowledge Graph

The default graph UI uses /api/graph/projection?level=tag. It renders tags as knowledge-point nodes and connects tags that co-occur in the same note. Tag edge strength is normalized as cooccurrence_count / sqrt(tagA_note_count * tagB_note_count), then filtered and capped before it reaches the client.

The note relation graph remains available through /api/relations/graph and /api/graph/projection?level=note for compatibility and relation-aware workflows.

The relation system supports these relation types:

Relation Meaning
CAUSED_BY Causation
LEADS_TO Leads to
RESOLVED_BY Resolved by
SIMILAR_TO Similar topic
CONTRADICTS Contradiction
DEPENDS_ON Dependency
EXTENDS Extension
REFERENCES Reference

Relations can be discovered by LLM, added manually, or followed during semantic search expansion.

Testing

Primary verification:

npm run test -w server
npm run build
npm run lint
npm run eval:experience -w server

Use focused server tests while iterating, then run the full commands before committing.

Release

npm run release                    # Full release: npm + Electron, tag v*
npm run release -- minor
npm run release -- major
npm run release -- 1.0.0
npm run release:electron -- 1.0.1  # Electron-only release, tag electron-v*
npm run release:npm -- 1.0.1       # npm-only release, tag npm-v*

Use scripts/release.mjs for releases. Avoid manually bumping versions, committing, tagging, and pushing unless you are doing an explicit recovery flow.

Release tag behavior:

  • v* tags trigger both npm publishing and Electron GitHub Release builds. Use this only when both root package.json and server/package.json should move together.
  • electron-v* tags trigger Electron-only GitHub Release builds. Use this for desktop-only changes, including Electron main/preload/tray/packaging changes and Electron-gated client UI.
  • npm-v* tags publish only the npm package. Use this for CLI, server, MCP, or package-content changes.
  • The npm package version comes from server/package.json, not the root package. A v* release will fail with npm E403 if server/package.json still points to an already-published version.