Production hardening: parallel indexing + closed-loop session memory#42
Merged
Conversation
…atform Dashboard (CSRF + path traversal): - Reject mutating requests with cross-origin Sec-Fetch-Site - Validate target_path stays inside project_dir via PathOutsideProjectError - Reject absolute / traversal paths and unknown files in delete_file - URL-encode project name in export Content-Disposition header Chunker (truth-in-advertising): - Register tree-sitter-typescript so .ts/.tsx use real AST chunking instead of silently falling through to line-range fallback - Use the TSX grammar for .tsx files - README: drop PHP from AST table (no tree-sitter-php dep) and surface it under fallback chunking Hot-path performance: - graph_store.neighbors_for_files: single SQL query joining nodes+edges replaces N+1 (per-result get_nodes_by_file + 2x get_neighbors) - mcp_server._estimate_full_file_tokens: use stat().st_size instead of reading every result file fully into memory on every search Cross-platform (Linux + macOS): - services.py: start ollama/dashboard with start_new_session=True so they survive parent shell signals - services.py: ollama install hint covers both ollama.com and brew - utils.resolve_cce_binary: include /opt/homebrew/bin (Apple Silicon) and /opt/local/bin (MacPorts) in candidate paths Tests: 264 passed (0 failed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compression cache (chunk_id, level) → table in vector_store - Same chunk no longer re-summarised on every retrieval; LocalBackend exposes get_cached_compression / put_cached_compression; Compressor checks cache first, persists after computing. RemoteBackend is duck-type-skipped so remote-mode behaviour is unchanged. - Cache rows are dropped on delete_by_file and clear() so a re-index can't surface stale summaries. Compressor.is_available cached with 30s TTL - Was probing localhost on every compress() call; the loop runs many times per session and the probe is small but not free. Retriever: drop RRF saturation, intent-aware FTS boost - Old `min(rrf*60, 1.0)` clamped almost every candidate to ~1.0 so the RRF term carried no gradient and confidence_score dominated. Replaced with rank-normalisation against the best score in the candidate set. - When the parsed intent is CODE_LOOKUP, FTS gets a 1.5x weight so exact identifier matches don't get drowned by fuzzy semantic hits. Query parser: keep code-prefix verbs for code lookups - Words like get/set/find/save/validate were unconditional stop-words, so "show me get functions" extracted no keywords. Now stripped only when the intent isn't CODE_LOOKUP. Session recall: vector search instead of substring grep - Decision recorded as "Use JWT with RS256" was invisible to a recall on "auth"; now the topic is embedded against each candidate entry and the matches are ranked by cosine similarity (>= 0.35). Falls back to the prior substring path on embedding failure. - Window expanded from 20 to 50 sessions. Token accounting: split retrieval and compression savings - mcp_server now exposes _split_savings; cli savings prints both percentages (retrieval vs full files / compression vs raw chunks) separately instead of a single conflated headline. JSON output keeps the old fields for back-compat and adds the two new ones. - README headline rewritten — the "70% saved" framing compared against a strawman (wholesale-paste of full files); replaced with a three-tier comparison (paste / targeted Read / CCE) and an explanation of the split metrics. Tests: 264 passed (0 failed). test_cli_savings updated for the new display format. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… auth, docs
Closed the cross-session memory loop end-to-end so the value prop actually
holds: the system used to depend on Claude knowing to call record_decision /
session_recall, but the generated CLAUDE.md never told it to. Now it does,
and bootstrap surfaces real history instead of a topic-grep that usually
missed.
P0 — closed-loop memory and crash safety
- CLAUDE.md template (cli._CCE_CLAUDE_MD_BLOCK) instructs Claude when to
call record_decision / record_code_area / session_recall, with explicit
triggers and format guidance. Without this, recording was opt-in by Claude
and never happened.
- Bootstrap pulls "most recent decisions" via SessionCapture.get_recent_decisions
instead of _search_sessions("decision") — the old path matched the literal
word "decision" and almost always returned nothing.
- _atomic_write_text helper (tempfile + os.replace) used for stats.json,
state.json, and per-session JSON. Plain write_text truncated the target
before writing; a crash mid-write left zero-byte files that next-load
silently treated as empty, losing all stats.
- SessionCapture: threading.RLock around every _active read/write; new
get_session_snapshot() so cross-module reads from mcp_server go through
the lock instead of reaching into the dict directly. Eliminates a real
race between concurrent record_* + _persist_current_session calls.
P1 — operability
- docs/operations.md: systemd unit (Linux) + launchd plist (macOS),
healthcheck endpoints, auth checklist, resource sizing, what's still not
supervised. First production-readiness doc the project ships.
- Optional dashboard bearer token: CCE_DASHBOARD_TOKEN env var. When set,
mutating endpoints require Authorization: Bearer; constant-time compare.
Dashboard JS picks up ?token=... from the URL and attaches the header
via a fetch monkey-patch — works without per-call code changes. CLI
prints the token-bearing URL when configured.
- vector_store: replaced bare `except Exception: return []` with logged
warnings on count, file_chunk_counts, search, clear. Schema corruption
no longer looks identical to "no results".
P2 — auto-capture and pruning
- SessionCapture.touch_files / get_top_touched_files: every file appearing
in a context_search result or opened via expand_chunk gets its touch
counter bumped automatically. Bootstrap's working_state then shows
"Recently touched files (prior session): a.py (12), b.py (7), ..." so
Claude sees where the last session was working without needing
record_code_area calls.
- SessionCapture.prune_old_sessions: when more than 100 session files
accumulate, the oldest are consolidated into decisions_log.json
(decisions only — the durable signal) and the source files removed.
Runs automatically at MCP server start and on demand via
`cce sessions prune`. get_recent_decisions reads from the consolidated
archive too, so old decisions remain reachable post-prune.
Tests: 264 passed (0 failed). All changes go through existing test paths
without modifications.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drives a real ContextEngineMCP (full object graph — LocalBackend, Embedder,
HybridRetriever, Compressor, BootstrapBuilder, SessionCapture) the same way
the stdio transport would, without needing a Claude or Anthropic API key.
Three tests:
1. test_decision_recorded_then_recalled_across_restart
The load-bearing one. Session 1 records a decision, server tears down,
session 2 starts fresh against the same on-disk storage. Asserts:
- The decision is on disk after record_decision (atomic flush actually
landed)
- get_recent_decisions() surfaces it in session 2 (the bootstrap fix
from fdd66bc — was previously a topic-grep that almost always missed)
- _search_sessions("naming convention") finds the "snake_case" decision
via paraphrase, proving recall is embedding-based
2. test_touched_files_auto_capture_survives_restart
Indexes a real file, runs context_search in session 1, asserts auth.py
ended up in touched_files automatically. Restarts; asserts the next
session's load_recent_sessions sees it. Closes the loop on the auto-
capture fix from the same commit — Claude doesn't need to call
record_code_area for "where you left off" to work.
3. test_prune_consolidates_old_sessions_into_decisions_log
110 session files, threshold=100, keep=50. Asserts 60 are pruned, 60
decisions land in decisions_log.json, 50 source files remain, and
archived decisions are still surfaced via get_recent_decisions so
long-lived projects don't silently drop old context.
Tests: 267 passed (264 existing + 3 new).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Indexing parallelism (the user-visible win) - pipeline._run_indexing_locked: chunking now runs through asyncio.to_thread + asyncio.gather across each batch instead of serially per file. tree-sitter is a C extension that releases the GIL, so this gives real concurrency on CPU-bound parsing. - pipeline: per-file `await backend.delete_by_file()` removed from the inner loop. Files-to-replace are accumulated and a single backend.delete_by_files fires once per re-index batch, collapsing N×3 small SQLite roundtrips (vector + FTS + graph per file) into 3 one-shot DELETE-IN queries. - Same treatment for the deleted-files cleanup pass at end of pipeline. - vector_store / fts_store / graph_store / local_backend: each now exposes delete_by_files(paths). Old delete_by_file delegates to the plural form. - fts_store._ingest_sync: switched per-row INSERT loop → executemany. ~30-50% faster bulk ingest on 1000+ chunks per the prior perf review. Net: a fresh `cce index` on a typical Python project runs noticeably faster, and the difference grows linearly with file count. CLAUDE.md upgrade path - Added a versioned tag (`<!-- cce-block-version: 2 -->`) and end marker (`<!-- /cce-block -->`) bracketing the block. _ensure_claude_md detects three states: missing → write; current version → no-op; stale (older versioned block OR pre-versioned legacy) → in-place replace preserving surrounding user content. Without this, projects installed before the record/recall instructions landed kept the old block forever — the cross-session memory loop never closed for existing users. Reliability - utils.atomic_write_text: shared helper. Replaces two duplicated _atomic_write_text copies in mcp_server.py + session_capture.py. Now always mkdir's parent so a deleted-mid-run sessions dir doesn't fail the write. - Compressor: asyncio.Lock around the 30-second Ollama probe cache so N concurrent compress() calls during a stale TTL window don't all fire the probe in parallel (single-flight via lock + recheck). - _cosine_sim: explicit length check + debug log. Mismatched embedding lengths now return 0 (no match) instead of silently truncating via zip — guards against a model swap mid-process or corrupted cached vector returning a misleading similarity. - SessionCapture.prune_old_sessions: fcntl.flock advisory lock on .prune.lock in the sessions dir. Two `cce serve` processes (or a serve + a `cce sessions prune`) starting at the same time no longer race the read-append-write on decisions_log.json — last-write-wins would have clobbered one process's appended decisions. Falls through unlocked on platforms without fcntl (Windows isn't a deploy target). Tests: 267 passed (no regressions), 4:19s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skip re-embedding chunks whose content is unchanged across index runs. SQLite store keyed by SHA-256 of chunk content; on re-index, only cache misses go through the embedding model. On a typical re-index after editing a few files, 95%+ of embeddings come from cache. Ported from #41 (rajkumarsakthivel) with the following improvements: 1. **Binary float32 storage instead of JSON.** Vectors go through struct.pack/unpack — same encoding the sqlite-vec store uses elsewhere. ~4× smaller on disk: a 384-dim cache for a 10k-chunk project drops from ~60MB to ~15MB. 2. **Thread-safe SQLite connection.** `check_same_thread=False` matches the rest of the storage layer. The cache is now called from asyncio.to_thread workers (post-parallelism rewrite) — the original would have raised in that path. 3. **try/finally for cache.close().** Pipeline now closes the cache even if recording cache_hits/cache_misses raises after embed(). Guarantees no leaked SQLite handle. 4. **prune_orphans(known_hashes).** New method drops cached entries whose content hash is no longer present in the live index. Pipeline calls it automatically on `cce index --full` so the cache doesn't grow monotonically forever (concrete on a high-churn project: prior to this, every chunk content variant ever seen accumulated). Refuses to operate on an empty known-set as a safety guard. 5. **Tests use pytest.approx for float comparison.** Round-tripping through float32 loses sub-bit precision; exact equality would fail for non-trivial vectors. Added tests for prune_orphans (with the empty-set safety check) — the upstream PR didn't cover pruning. Other surfaces: - `cce status` shows cache size and disk footprint when the cache exists for the current project - `cce index` summary line includes "X% cache hit" when at least one hit was served, so users see the speedup directly - README adds section 7 ("Content-Hash Embedding Cache") and the roadmap entry; section numbers shifted (Cross-Session Memory is now section 8) Tests: 279 passed (267 prior + 12 new), 4:21s. Co-Authored-By: Raj <rajkumar.sakti@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the full suite from 4:21 → 2:50 (35% faster, 1:31 saved per run). Configuration: - Capped at -n 4 to match Embedder._PARALLEL and keep peak memory under ~1.5 GB (each worker process loads its own ~60 MB ONNX model). Override with `pytest -n auto` (use all cores) or `-n 1` (serial). - --dist=loadgroup so xdist_group marks pin tests to a single worker. Ollama-touching tests pinned to a single worker via `pytestmark = pytest.mark.xdist_group(name="ollama")` in test_compressor.py + test_ollama_client.py. Without this, four workers race on phi3:mini (which serves requests serially) and the 30s per-call timeout flakes — observed once before this fix. Audit notes: - All other tests use tmp_path → already worker-isolated. - monkeypatch.chdir / os.chdir paths are per-test and per-process; xdist runs each worker in its own process so cwd doesn't leak. - Compression tests share localhost:11434 (real Ollama) but are now group-pinned so concurrency is bounded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bumping to 0.3.0. Branch shipped substantial work since 0.2.11 (parallel indexing, embedding cache, closed-loop session memory, dashboard auth, honest token accounting, +279 tests parallel via pytest-xdist) — semver calls for a minor bump. CI: fix fastembed download race introduced by pytest-xdist - Four xdist workers were racing on /tmp/fastembed_cache during concurrent first-time downloads, one would win and others would read a partial model_optimized.onnx and fail with NO_SUCHFILE. - New ci.yml step pre-warms the cache in a single process before pytest starts; workers then hit the populated cache. Local runs weren't affected because the model was already cached. Version sync - src/context_engine/__init__.py was stuck at 0.1.0 (dead since the initial commit). Bumped to 0.3.0 to match pyproject.toml. README cleanups - Status banner showed v0.2.9 → v0.3.0 - "Claude can call context_search and eight other MCP tools" was off-by-one — there are nine tools total (eight companions plus context_search itself). Reworded to remove the ambiguity. Indexer: file-size cap to prevent OOM - pipeline._safe_read now returns None (skip) for any single file larger than 2 MB. Without this, an accidentally-committed log dump or vendored bundle would be loaded fully into memory. 2 MB easily covers normal source (the largest CPython stdlib module is ~250 KB) while ruling out files you'd never want in a semantic index anyway. Tests: 279 passed parallel (2:54s). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The package name foregrounded a single AI vendor, but the tool is vendor-neutral — it works as an MCP server for any client. Renamed for the marketplace listing without breaking existing installs. What changed - Display name "Claude Context Engine" → "Code Context Engine" everywhere (README, docs, banner, generated CLAUDE.md text, error messages). - PyPI package name `claude-context-engine` → `code-context-engine`. README install commands and PyPI badge updated. - MCP server identifier in Server() arg → "code-context-engine". - README install hint for the [http] extra updated. - Generated .gitignore block label updated. What stayed the same (back-compat) - Storage path `~/.claude-context-engine/` — every existing user has data there; renaming would orphan it. The path is internal and users don't type it. - Console script aliases: `cce` (canonical) and `code-context-engine` (new long alias) plus the legacy `claude-context-engine` so users on the prior install still have a working command. - Git hook marker (`# claude-context-engine hook`) — embedded in users' installed hooks; changing it would orphan our own hooks. - `.cce` and `.claude-context-engine` directory names in the watcher ignore list (matches both the new `.cce` ignore and the legacy storage-dir ignore). - pkg_version() lookups in cli.py and bootstrap.py now try "code-context-engine" first and fall back to the legacy name so unupgraded installs still report a version. Tests: 279 passed parallel (2:52s). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Net 165 lines removed. All flagged via vulture (≥80% confidence) and manually verified to have zero call sites. Removed - src/context_engine/daemon.py (entire file): the Daemon class wrapped the same backend / embedder / retriever / compressor / MCP wiring that cli._run_serve already does directly. Nothing imported it. The parallel code path was a leftover from before cli.py grew its own serve plumbing. - SessionCapture.record_question, get_top_touched_files: declared in earlier rounds with intent to wire later, never wired. Bootstrap reads touched_files directly from get_session_snapshot, the recall path never used record_question. - ContextEngineMCP._split_savings: added during the honest-accounting work, but cli._print_project does the math inline against stats and never called it. - Embedder._model_name: assigned, never read after the fastembed switch (resolution happens via the local `resolved` var instead). - Embedder's `import numpy as np`: leftover from the SentenceTransformer era; fastembed returns numpy arrays we just .tolist(). - savings_shortcut's `import sys as _sys`: unused alias. - Config.indexer_languages + its YAML mapping entries: declared and loadable but no consumer. Deduplicated - cli._configure_mcp and Manifest.save both rolled their own tempfile + os.replace pattern. Both now call utils.atomic_write_text (the helper extracted in fdd66bc). 30 lines of identical boilerplate collapsed into two one-liners. Quantization (no code change) - Embedding model is *already* int8-quantized: fastembed serves the qdrant `bge-small-en-v1.5-onnx-q` mirror by default (the `-q` is the quantized int8 build — that's why the cache is 60 MB instead of the ~133 MB FP32 weights). Vector storage in sqlite-vec is still float32; switching to int8 storage is feasible (~4× smaller index, ~1-2% recall@10 loss) but not worth shipping yet — typical project index lands at 5-60 MB total and isn't the bottleneck. Tests: 279 passed parallel (2:53s), no regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR hardens Code Context Engine (CCE) for production use by improving indexing throughput, making on-disk state/session persistence more crash-safe, strengthening dashboard security, and closing the loop on cross-session memory (record → persist → recall) with integration tests.
Changes:
- Add parallelized indexing + batched deletes/ingest, plus a content-hash SQLite embedding cache to avoid recomputing unchanged embeddings.
- Implement crash-safer persistence (atomic writes) and “closed-loop” session memory (auto-captured touched files, decision surfacing, pruning/consolidation).
- Harden the dashboard (CSRF guard + optional bearer token) and refresh docs/tests to match the new behaviors + package rename.
Reviewed changes
Copilot reviewed 39 out of 39 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_cli_savings.py | Updates savings assertions to match split retrieval vs compression reporting. |
| tests/integration/test_memory_loop.py | Adds end-to-end cross-session memory integration tests (record/recall, touched files, pruning). |
| tests/indexer/test_embedding_cache.py | Adds unit tests for new SQLite embedding cache and embedder integration. |
| tests/compression/test_ollama_client.py | Pins Ollama tests to an xdist group to reduce parallel flakiness. |
| tests/compression/test_compressor.py | Pins compressor Ollama tests to the same xdist group. |
| src/context_engine/utils.py | Introduces shared atomic_write_text helper; expands binary resolution paths. |
| src/context_engine/storage/vector_store.py | Adds chunk compression cache table + batched delete APIs; improves error logging. |
| src/context_engine/storage/local_backend.py | Adds batched delete API and compression cache passthrough methods. |
| src/context_engine/storage/graph_store.py | Adds multi-file neighbor query + batched delete API. |
| src/context_engine/storage/fts_store.py | Speeds ingest via executemany and adds batched delete API. |
| src/context_engine/services.py | Detaches child processes via start_new_session; improves Ollama missing message. |
| src/context_engine/serve_http.py | Fixes install hint for new package name; wires compressor cache. |
| src/context_engine/retrieval/retriever.py | Updates RRF blending/normalization and boosts FTS for code-lookup intent. |
| src/context_engine/retrieval/query_parser.py | Adds intent-aware stop-word handling to preserve code-prefix verbs. |
| src/context_engine/project_commands.py | Updates gitignore block label to new package name. |
| src/context_engine/integration/session_capture.py | Thread-safe session capture, touched-files tracking, pruning/consolidation, recent decision surfacing. |
| src/context_engine/integration/mcp_server.py | Adds session prune on start, atomic persistence, touched-file auto-capture, vector-based session recall. |
| src/context_engine/integration/bootstrap.py | Resolves version from new package name with fallback to legacy. |
| src/context_engine/indexer/pipeline.py | Adds path traversal guard, parallel chunking, batched deletes, embedding cache + pruning, file size cap. |
| src/context_engine/indexer/manifest.py | Switches manifest saves to shared atomic_write_text helper. |
| src/context_engine/indexer/embedding_cache.py | Adds SQLite embedding cache keyed by content hash (pack/unpack float32). |
| src/context_engine/indexer/embedder.py | Adds optional embedding cache support; uses batched cache lookups. |
| src/context_engine/indexer/chunker.py | Adds tree-sitter TypeScript/TSX language support. |
| src/context_engine/dashboard/server.py | Adds CSRF/auth middleware, path validation, safer export filename handling. |
| src/context_engine/dashboard/_page.py | Attaches optional bearer token to fetch() requests via URL param. |
| src/context_engine/daemon.py | Removes legacy daemon orchestration module. |
| src/context_engine/config.py | Removes indexer_languages config surface (no longer used). |
| src/context_engine/compression/compressor.py | Adds cached compression storage, Ollama probe TTL + single-flight lock. |
| src/context_engine/cli.py | Renames branding, versioned CLAUDE.md block upgrade path, sessions prune CLI, split savings output, cache stats display. |
| src/context_engine/init.py | Updates package description and version to 0.3.0. |
| pyproject.toml | Renames package, adds script aliases, enables pytest-xdist by default, warms model cache in CI. |
| docs/wiki/Home.md | Updates project name/branding. |
| docs/wiki/CLI-Reference.md | Updates name in examples (but still contains older version text). |
| docs/review-2026-04-17.md | Updates review doc branding header. |
| docs/operations.md | Adds ops guidance for systemd/launchd, auth checklist, healthchecks. |
| docs/index.html | Updates site branding strings. |
| README.md | Updates name, install instructions, honest savings framing, embedding cache docs. |
| CONTRIBUTING.md | Updates branding header. |
| .github/workflows/ci.yml | Pre-warms fastembed cache to avoid xdist worker download races. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- dashboard/server.py: read versioned manifest schema correctly so
/api/status, /api/files, /api/delete, /api/clear stop treating
__schema_version / files / last_git_sha as file paths. Clear writes
the current schema; delete preserves last_git_sha.
- indexer/pipeline.py: defer replacement deletes until after embed
succeeds, so a transient model-download or sqlite-vec failure can no
longer wipe a working index.
- storage/{fts,graph}_store.py: serialise the shared sqlite3 connection
with an RLock, mirroring VectorStore. Concurrent dashboard / MCP /
reindex calls running through asyncio.to_thread no longer race.
- cli.py serve: reload .context-engine.yaml from --project-dir AFTER
chdir so the project's config actually wins.
- cli.py uninstall: use _extract_existing_cce_block() so the current
<!-- cce-block-version: 2 --> CLAUDE.md block is removed (legacy
CCE:BEGIN/END markers still recognised).
- uv.lock: refresh stale lock (package rename + pytest-xdist).
- .gitignore: ignore local review/ notes.
Adds 10 regression tests; full suite 279 -> 289 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- indexer/pipeline.py: map ChunkType -> NodeType properly. Markdown, yaml, json, comment, and module-fallback chunks no longer all land as NodeType.CLASS, which had been polluting graph expansion. - storage/vector_store.py: when embedding dimension changes, also clear the `chunks` and `chunk_compressions` tables alongside `chunks_vec`. Previously only `chunks_vec` was dropped, leaving orphan rows that count_chunks() / file_chunk_counts() reported but search could never return. Adds 2 regression tests; full suite 289 -> 291 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 45 out of 47 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rajkumarsakthivel
temporarily deployed
to
github-pages
April 27, 2026 14:07 — with 
GitHub Pages
Inactive
fazleelahhee
added a commit
that referenced
this pull request
Apr 27, 2026
- utils.py atomic_write_text: pin encoding=utf-8, flush + fsync the tempfile, fsync the parent dir so the rename is durable across power loss. The "fully on disk" docstring claim now actually holds. - dashboard/server.py: route /api/clear, /api/delete, /api/compression manifest/stats/state writes through atomic_write_text. - cli.py _ensure_claude_md + uninstall: atomic_write_text so an interrupted upgrade can't truncate a user's CLAUDE.md. - vector/fts/graph stores delete_by_files: chunk file_paths / node_ids under SQLITE_PARAM_BATCH (500) so a project-wide prune touching thousands of removed files no longer trips `OperationalError: too many SQL variables`. - session_capture.get_recent_decisions: snapshot the nested decisions list under the lock too — `dict(s)` was a shallow copy that left session["decisions"] aliasing the live list. - mcp_server._search_sessions: include consolidated decisions from decisions_log.json. After prune_old_sessions deletes per-session files, archived rows now stay surfaceable via session_recall, in line with the prune CLI's docstring. - Promote SessionCapture.load_consolidated_decisions to public. - retriever.py: drop unused ParsedQuery import. - pyproject.toml: align [project.optional-dependencies].dev with [dependency-groups].dev so the install method doesn't decide which pytest version you get. - docs CLI-Reference.md + landing index.html: drop hard-coded v0.2.9 banner so the example doesn't drift on every release. Adds 4 regression tests; full suite 291 -> 295 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
Author
|
Addressed all 13 Copilot review comments in 947aa87.
Added regression tests:
Full suite 291 → 295 passing. |
fazleelahhee
added a commit
that referenced
this pull request
Apr 27, 2026
- embedding_cache.py: add RLock around the shared sqlite connection, matching VectorStore / FTSStore / GraphStore — `check_same_thread= False` only disables the ownership check, not concurrent access. - embedding_cache.py: namespace the cache by model name. Schema v2 PRIMARY KEY is (content_hash, model). Pre-v2 (model-less) tables are dropped on open since cached vectors had no model attribution and a model swap could otherwise return wrong-meaning embeddings — or with a different dim, break sqlite-vec ingest. Vectors are recomputable; cache is purely a speed-up so dropping is safe. - pipeline.py / cli.py: pass `model_name=config.embedding_model` when constructing EmbeddingCache. - Use chunked() helper instead of inline 500-batch loops for consistency with the other stores. Adds 2 regression tests (model-namespacing, v1→v2 drop). Full suite 295 -> 297 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
Author
|
Addressed the remaining 3 Copilot threads in 1d6ba1d:
Also unified the IN(...) batching across all four sqlite-backed stores (vector, fts, graph, embedding cache) on the shared Full suite 291 → 297 passing. |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five commits hardening CCE from "Alpha — solo use only" to fit-for-production.
Five themes, listed in commit order:
9492a3e) — dashboard CSRF + pathtraversal fixes; TypeScript actually wired into the chunker (was silently
falling through to line-range fallback); PHP removed from the AST table
since
tree-sitter-phpisn't even a dep; batched graph queries; macOSstart_new_session.faef864) — RRF de-saturated(was clamping every candidate to ~1.0, drowning exact identifier matches
under fuzzy semantic hits); intent-aware stop-words so
get/set/findsurvive code-lookup queries; per-
(chunk_id, level)compression cache sothe same chunk isn't re-summarised on every retrieval; vector-similarity
session recall (paraphrases now match — recording "Use JWT with RS256"
is now findable via
session_recall("auth")); split retrieval-vs-compression token-savings reporting (the old "70%" headline conflated
them and benchmarked against a strawman of pasting whole files).
fdd66bc) — the load-bearing change. The generated
CLAUDE.mdnow explicitly tells Claudewhen to call
record_decision/record_code_area/session_recall(without this, recording was opt-in by Claude and never happened).
Bootstrap pulls most-recent decisions instead of grepping for the
literal word "decision".
tempfile + os.replaceatomic writes forstats.json/state.json/ per-session JSON.threading.RLockaround
SessionCapture._active. Auto-capture of files surfaced viacontext_search/expand_chunkso "where you left off" workswithout explicit
record_code_areacalls. Old-session pruningconsolidates >100 sessions into
decisions_log.jsonso long-livedprojects don't silently drop old context. Optional dashboard bearer
token via
CCE_DASHBOARD_TOKEN.vector_storelogs warnings insteadof silently swallowing exceptions. New
docs/operations.mdwithsystemd unit + launchd plist + healthcheck guidance.
f88fbea) — drives areal
ContextEngineMCP(full object graph, no mocks, no__new__bypass) the same way the stdio transport would, no Anthropic API
key required. Three tests: record-then-recall-across-restart with
paraphrase recall; touched-files survive restart; prune consolidates
correctly.
f4d5025) — chunkingnow
asyncio.to_thread+asyncio.gatherper batch (tree-sitterreleases the GIL); per-file
await delete_by_fileremoved from theinner loop, single batched
delete_by_filesper batch collapsesN×3 SQLite roundtrips into 3
DELETE INs; FTS ingest switched toexecutemany. CLAUDE.md upgrade path via versioned tag + end markerso old installs actually get upgraded. Shared
atomic_write_textin
utils.py. Compressorasyncio.Locksingle-flights the Ollamaprobe.
_cosine_simlength check.prune_old_sessionscross-process advisory
fcntl.flock.Stats
Test plan
cce initin a fresh project writes the new versionedCLAUDE.mdblock (look for
<!-- cce-block-version: 2 -->)cce initreplaces it in place (preserves surrounding user content)
cce dashboardworks withoutCCE_DASHBOARD_TOKENset; withthe env var set, write-actions 401 if the token is wrong
cce index --fullon a 500+ file project shows a meaningfulspeedup vs. the prior version (chunking parallelism + batched
deletes + executemany compound)
record_decision("Use snake_case", "PEP 8")then restart →session_recall("naming convention")returns the decision (paraphrase match via vector similarity)
tests/integration/test_memory_loop.pypasses locally as asmoke test of the above
Notes for review
chunk_compressionsis a new table — created on first
_ensure_tables()call soexisting indexes pick it up automatically on next start.
against a strawman (wholesale-paste of full files). Replaced with
a three-tier comparison (paste / targeted Read / CCE) and an
explanation of the split metrics. Honest is better than impressive.
every backward-compat surface preserved.
🤖 Generated with Claude Code