Skip to content

Phase 2 oracle: 20-query recall@10 with real SPECTER2 + BGE-M3 #3

Description

@deOliveira-R

Manual acceptance test required before tagging v1.0.

Per plan §6 Phase 2: the 20-query recall@10 oracle and <500ms cold-start oracle need the real library + real models loaded. `tools/phase2_search_oracle.py` already exists; it needs a curated queries file.

Steps

  1. Full Zotero migration first: `grimoire migrate zotero` (~8 min for 15k items).
  2. `grimoire index` to populate SPECTER2 item embeddings (~8 min) and BGE-M3 chunks (overnight, CPU-only).
  3. Draft 20 queries + gold answers (item IDs or DOIs) from your research notes — the ones you'd actually ask.
  4. Wire the queries file path into `tools/phase2_search_oracle.py` if not already supported.
  5. Run the oracle against the full corpus; compare keyword / semantic / hybrid.

Acceptance

  • 20 curated queries + gold-answer item IDs captured (commit `tools/phase2_queries.jsonl` or similar).
  • Hybrid recall@10 ≥ 0.90 per plan target.
  • Cold-start semantic query latency < 500ms on the TrueNAS host.
  • Regression baseline stored in `project_oracle_results.md` memory or repo docs for future regressions.

Notes

  • Small queries file (20 lines) is fine; format as `{query, gold_item_ids: [..], notes: "..."}`.
  • BGE-M3 indexing is the slow part — schedule overnight. SPECTER2 alone is ~8 min.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions