Feat/benchmarking

[KIvanow]

Summary

Adds outcome-aware self-tuning to the Monitor's threshold recommendation engine and benchmarks it on four public datasets (STSb, SICK, SemBenchmarkLmArena, PAWS-Wiki). The autotuner loosens when too strict, tightens when too loose, and does nothing when the threshold is already optimal — with zero degradation across every configuration tested.

Changes

• Add signal quality guards to the recommendation engine — require the uncertain-hit fraction of all operations (not just hits) to exceed 15% before tightening; require hit rate > 80% before treating distant-hits as a tighten
  signal
  • Add recall-cost guard — block tighten if estimated hit loss exceeds 15% of current hits
  • Add outcome tracking — after each adjustment, compare current signal metrics against the snapshot from the previous adjustment; stop if the triggering signal did not improve by 20%
  • Add velocity dampening — progressive step-size reduction for consecutive same-direction adjustments with oscillation detection
  • Write tuning history with metrics snapshots in the apply dispatcher to enable outcome tracking
  • Add STSb dataset loader (8,628 pairs, 3 genre categories, continuous scores)
  • Add SICK dataset loader (9,927 pairs, score-derived categories)
  • Add --match-threshold CLI option for continuous-score datasets
  • Add autotune-full mode combining rerank + LLM judge + Monitor-driven autotuning
  • Improve judge prompt — semantic equivalence framing instead of Q&A, strip "Answer:" prefix
  • Add timeout retry with exponential backoff on Monitor API calls

Benchmark results (bare → autotune):

• STSb θ=0.20: +2.8% F1 (loosened 0.20 → 0.22)
• STSb θ=0.40: +8.6% precision (tightened 0.40 → 0.30)
• SICK θ=0.10: +2.1% F1 (loosened 0.10 → 0.145)
• SemBenchmarkLmArena θ=0.40: +2.9% F1 (tightened 0.40 → 0.30)
• PAWS-Wiki: 0% change (wall confirmed, autotuner correctly does nothing)

Checklist

• Unit / integration tests added
• Docs added / updated
• Roborev review passed — run roborev review --branch or /roborev-review-branch in Claude Code (internal)
• Competitive analysis done / discussed (internal)
• Blog post about it discussed (internal)

---

Note

Medium Risk
Changes proprietary threshold recommendation and apply paths that affect production cache behavior; benchmark and docs changes are lower risk but autotune now depends on Monitor API credentials and external calls.

Overview
Strengthens Monitor-driven semantic cache threshold tuning with richer signals (uncertain hits, distant weak hits, near-misses, low hit rate), a recall-cost guard, outcome tracking against prior adjustments, and velocity/oscillation dampening. Approved changes now append tuning history with metric snapshots in Valkey, and live config reads prefer the threshold field in __config.

Expands the cache-benchmark harness: STSb and SICK loaders, --match-threshold, autotune-full (rerank + judge + Monitor autotune), and BetterDB autotune wired to the Monitor recommend → propose → approve API (replacing in-process tuning), plus judge prompt tweaks and Monitor API retries. Adds Python semantic cache package documentation and minor benchmark/README/lockfile updates.

^{Reviewed by Cursor Bugbot for commit 5af5e5f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[KIvanow]

THis is fine and expected

[KIvanow]

THis is fine and expected

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6436ea9. Configure here.}