Describe the Bug
When executing --deep chunk-by-chunk hidden connection discovery with higher limits (notebrain hidden "<note>" --deep --limit=8) on notes containing numerous indexed chunks and rich metadata, the query fails with:
Error: hidden connections deep: semantic search for query "§ Concepts > Platform Abstractions": semantic search: error loading query projections: failed to decode get records response: unexpected end of JSON input: upstream ChromaDB embedded 1 MiB FFI limit exceeded on large records payload; try lowering --top-k or --limit
Root Cause Analysis
-
Embedded ChromaDB (CGO_ENABLED=1) FFI Ceiling:
In embedded persistent mode, chroma-go (pkg/api/v2) communicates with the underlying C/SQLite engine via CGO/FFI (libchroma). Query responses (including chunk IDs, distances, document text, and metadata maps like heading_path, title, and tags) are serialized across the CGO boundary as a JSON string. This FFI bridge enforces a hard buffer ceiling of 1 MiB (1,048,576 bytes) per query response. If the JSON payload exceeds this ceiling, it truncates (unexpected end of JSON input), triggering NoteBrain's wrapChromaErr() check (internal/store/query.go:L1287).
-
Multiplier Chaining in --deep Mode:
When --limit=8 is passed to HiddenConnectionsDeep, candidate multipliers expand across layers:
HiddenConnectionsDeep calls MultiSemanticSearch with fetchLimit = max(limit * 2, 15) = 16.
MultiSemanticSearch expands candidate pool: subLimit = max(fetchLimit * 2, 20) = 32, subTopK = max(topKPerNote * 2, 6) = 6.
semanticSearch calculates total records to request from ChromaDB: fetchCount = max(subLimit * 3, subLimit * subTopK) = max(32 * 3, 32 * 6) = 192.
Requesting 192 chunk records (WithNResults(192)) along with IncludeMetadatas, IncludeDistances on a single section query ("§ Concepts > Platform Abstractions") pushes the returned JSON string above 1 MiB when notes have long frontmatter properties or deep heading paths.
Proposed Solution
To permanently prevent multiplier chaining (fetchCount = 192+) from causing FFI buffer overflows:
-
Safety Ceiling on NResults in semanticSearch (internal/store/query.go):
Cap fetchCount at a safe upper bound (<= 100) regardless of input multiplier chaining:
fetchCount := max(limit*3, limit*topKPerNote)
if fetchCount > 100 {
fetchCount = 100 // Safe ceiling well below 1 MiB FFI string limit
}
-
Memory Pagination for High Limits (limit > 30):
If a user explicitly requests a very large limit (> 30), execute paginated query slices (Offset: 0..100, Offset: 100..200) and merge results in Go memory instead of requesting > 100 records across CGO in one payload.
Describe the Bug
When executing
--deepchunk-by-chunk hidden connection discovery with higher limits (notebrain hidden "<note>" --deep --limit=8) on notes containing numerous indexed chunks and rich metadata, the query fails with:Root Cause Analysis
Embedded ChromaDB (
CGO_ENABLED=1) FFI Ceiling:In embedded persistent mode,
chroma-go(pkg/api/v2) communicates with the underlying C/SQLite engine via CGO/FFI (libchroma). Query responses (including chunk IDs, distances, document text, and metadata maps likeheading_path,title, andtags) are serialized across the CGO boundary as a JSON string. This FFI bridge enforces a hard buffer ceiling of 1 MiB (1,048,576bytes) per query response. If the JSON payload exceeds this ceiling, it truncates (unexpected end of JSON input), triggering NoteBrain'swrapChromaErr()check (internal/store/query.go:L1287).Multiplier Chaining in
--deepMode:When
--limit=8is passed toHiddenConnectionsDeep, candidate multipliers expand across layers:HiddenConnectionsDeepcallsMultiSemanticSearchwithfetchLimit = max(limit * 2, 15) = 16.MultiSemanticSearchexpands candidate pool:subLimit = max(fetchLimit * 2, 20) = 32,subTopK = max(topKPerNote * 2, 6) = 6.semanticSearchcalculates total records to request from ChromaDB:fetchCount = max(subLimit * 3, subLimit * subTopK) = max(32 * 3, 32 * 6) = 192.Requesting
192chunk records (WithNResults(192)) along withIncludeMetadatas, IncludeDistanceson a single section query ("§ Concepts > Platform Abstractions") pushes the returned JSON string above 1 MiB when notes have long frontmatter properties or deep heading paths.Proposed Solution
To permanently prevent multiplier chaining (
fetchCount = 192+) from causing FFI buffer overflows:Safety Ceiling on
NResultsinsemanticSearch(internal/store/query.go):Cap
fetchCountat a safe upper bound (<= 100) regardless of input multiplier chaining:Memory Pagination for High Limits (
limit > 30):If a user explicitly requests a very large
limit(> 30), execute paginated query slices (Offset: 0..100,Offset: 100..200) and merge results in Go memory instead of requesting> 100records across CGO in one payload.