Bench/gyst memorybench adapter

[chaydavs]

What changed

Why

Testing

• bun test passes
• bun run lint passes
• Eval MRR@5 >= 0.90
• No secrets in source code

Summary by CodeRabbit

• Bug Fixes

  • Improved search matching for natural-language queries by reducing noisy question words and pronouns in query expansion.
  • Added a fallback search pass that can still return relevant results when the initial match is too strict.

• Documentation

  • Added benchmark documentation covering methodology, analysis, improvements, and a results overview.
  • Included updated benchmark reports and summaries for multiple evaluation runs.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a909f29e-8fee-44ce-a57d-7292ab33d446

📥 Commits

Reviewing files that changed from the base of the PR and between 1ab6367 and d44819f.

📒 Files selected for processing (15)

• benchmarks/ANALYSIS.md
• benchmarks/IMPROVEMENTS.md
• benchmarks/METHODOLOGY.md
• benchmarks/README.md
• benchmarks/reports/gyst-base100.summary.json
• benchmarks/reports/gyst-baseline.summary.json
• benchmarks/reports/gyst-h1-haiku-100.summary.json
• benchmarks/reports/gyst-h2-100.summary.json
• benchmarks/reports/gyst-h2h3-haiku-100.summary.json
• benchmarks/reports/gyst-heldout-haiku.summary.json
• src/store/query-expansion.ts
• src/store/search.ts
• tests/store/query-expansion-stopwords.test.ts
• tests/store/query-expansion.test.ts
• tests/store/search-or-fallback.test.ts

---

📝 Walkthrough

Walkthrough

Adds two FTS5 retrieval improvements: an OR-mode fallback in searchByBM25 that retries with term1 OR term2 ... when implicit-AND returns zero rows, and an expanded FTS5_PROBLEM_WORDS set stripping question words, pronouns, and auxiliaries before matching. Six benchmark JSON reports and four markdown documents recording methodology, analysis, and results are also added.

Changes

Retrieval Improvements and Benchmark Documentation

Layer / File(s)	Summary
H3: Expanded stopword list src/store/query-expansion.ts, tests/store/query-expansion-stopwords.test.ts, tests/store/query-expansion.test.ts	FTS5_PROBLEM_WORDS extended with question words, pronouns, possessives, and auxiliaries/modals; existing tests updated and a new stopword-specific suite added.
H2: BM25 OR-mode fallback src/store/search.ts, tests/store/search-or-fallback.test.ts	toOrModeExpression helper rewrites flat token lists to term1 OR term2 ...; searchByBM25 retries with OR expression when AND returns zero rows; tests validate AND preservation, OR fallback, and false-positive guard.
Benchmark report JSONs benchmarks/reports/gyst-*.summary.json	Six JSON run summaries added covering baseline, base100, h2-100, h1-haiku-100, h2h3-haiku-100, and heldout-haiku runs with accuracy, token, and latency metrics.
Benchmark docs benchmarks/README.md, benchmarks/METHODOLOGY.md, benchmarks/ANALYSIS.md, benchmarks/IMPROVEMENTS.md	Documents added covering LongMemEval evaluation setup, hypothesis ranking, retrieval-miss analysis, Phase 5 changelog, and reproduction steps.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Poem

🐇 Hop, hop—the OR gate swings wide,
No longer does AND let good results hide.
"Why," "how," and "should" get tossed in the bin,
So only the real content words filter in.
The benchmarks now sing from 13% to more—
A rabbit rejoices at the new retrieval score! 🎉

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch bench/gyst-memorybench-adapter

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}