Skip to content

Prevent embedded ChromaDB 1 MiB FFI buffer overflow during high-limit --deep queries #21

Description

@nmdra

Describe the Bug

When executing --deep chunk-by-chunk hidden connection discovery with higher limits (notebrain hidden "<note>" --deep --limit=8) on notes containing numerous indexed chunks and rich metadata, the query fails with:

Error: hidden connections deep: semantic search for query "§ Concepts > Platform Abstractions": semantic search: error loading query projections: failed to decode get records response: unexpected end of JSON input: upstream ChromaDB embedded 1 MiB FFI limit exceeded on large records payload; try lowering --top-k or --limit

Root Cause Analysis

  1. Embedded ChromaDB (CGO_ENABLED=1) FFI Ceiling:
    In embedded persistent mode, chroma-go (pkg/api/v2) communicates with the underlying C/SQLite engine via CGO/FFI (libchroma). Query responses (including chunk IDs, distances, document text, and metadata maps like heading_path, title, and tags) are serialized across the CGO boundary as a JSON string. This FFI bridge enforces a hard buffer ceiling of 1 MiB (1,048,576 bytes) per query response. If the JSON payload exceeds this ceiling, it truncates (unexpected end of JSON input), triggering NoteBrain's wrapChromaErr() check (internal/store/query.go:L1287).

  2. Multiplier Chaining in --deep Mode:
    When --limit=8 is passed to HiddenConnectionsDeep, candidate multipliers expand across layers:

    • HiddenConnectionsDeep calls MultiSemanticSearch with fetchLimit = max(limit * 2, 15) = 16.
    • MultiSemanticSearch expands candidate pool: subLimit = max(fetchLimit * 2, 20) = 32, subTopK = max(topKPerNote * 2, 6) = 6.
    • semanticSearch calculates total records to request from ChromaDB: fetchCount = max(subLimit * 3, subLimit * subTopK) = max(32 * 3, 32 * 6) = 192.

    Requesting 192 chunk records (WithNResults(192)) along with IncludeMetadatas, IncludeDistances on a single section query ("§ Concepts > Platform Abstractions") pushes the returned JSON string above 1 MiB when notes have long frontmatter properties or deep heading paths.

Proposed Solution

To permanently prevent multiplier chaining (fetchCount = 192+) from causing FFI buffer overflows:

  1. Safety Ceiling on NResults in semanticSearch (internal/store/query.go):
    Cap fetchCount at a safe upper bound (<= 100) regardless of input multiplier chaining:

    fetchCount := max(limit*3, limit*topKPerNote)
    if fetchCount > 100 {
        fetchCount = 100 // Safe ceiling well below 1 MiB FFI string limit
    }
  2. Memory Pagination for High Limits (limit > 30):
    If a user explicitly requests a very large limit (> 30), execute paginated query slices (Offset: 0..100, Offset: 100..200) and merge results in Go memory instead of requesting > 100 records across CGO in one payload.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinghelp wantedExtra attention is needed

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions