feat(index): configurable chunk size/overlap + embedder-window clamp (bo-jsxi)

[scbrown]

What

Makes line-based chunking configurable via [index].chunk_size / chunk_overlap and clamps every line/markdown-fallback chunk to the embedding model's token window so dense chunks aren't silently truncated at embed time.

Why (bo-jsxi)

Chunking was hardcoded (chunk_size=50, overlap=10 in parser.rs). A 50-line dense-code chunk can exceed the embedder's max_seq — 256 for the default all-MiniLM-L6-v2, 512 for bge-small/gte-small — and the embedder truncates the overflow silently, degrading retrieval for the dropped tail.

How

• Config: [index].chunk_size (50) + chunk_overlap (10).
• Parser: with_chunking(size, overlap, max_chunk_tokens) — sanitizes overlap < size and size >= 1. chunk_by_lines uses them.
• Clamp: when max_chunk_tokens > 0, each chunk is shrunk (by line) until its estimated tokens (~chars/4) fit the window; advance logic guards forward progress when a clamped chunk is shorter than the overlap. 0 = window unknown (API backend) = no clamp.
• Wiring: new Embedder::max_seq() accessor; index/watch/webhook pass the live window into the parser.

Tests

• estimate_tokens heuristic; with_chunking overlap/size sanitization
• configurable size+overlap chunk boundaries
• clamp splits a dense 50-line window into multiple chunks, every chunk <= cap, full coverage start→end
• config parse + defaults

cargo build/clippy clean (0 errors); config + parser + embedder suites pass.

Note: scoped to the line-based + markdown-fallback chunkers named in the bead. Tree-sitter semantic chunks (whole functions) are intentionally not split (splitting breaks semantics); they remain subject to the embedder's existing truncation as before.

🤖 Generated with Claude Code