docs: README rewrite + test fixes + CI workflow#22
Merged
Conversation
added 7 commits
June 7, 2026 18:41
- HybridReranker: 4-signal weighted fusion (RRF, embedding, intent boost, content quality) - MultiQueryExpander: LLM-based query variant generation for improved recall - ResultMerger: multi-variant result merging (sum_score/max_score/rrf_again) - RerankerConfig: YAML-configurable weights and intent boost tables - Pipeline integration with backward-compatible API - RetrievalTrace extended with hybrid reranking metadata - 47 new unit tests (all passing) - English README with screenshots, badges, contributing guide - OpenSpec change proposal + design + tasks + spec
1. Evidence list capped at _MAX_EVIDENCE=15, sorted by score. Prevents unbounded context growth across reformulate iterations. 2. _llm_guided_search now MERGES with existing evidence instead of replacing it. Same dedup-by-chunk_id pattern as _reformulate_search. Previously, LLM-guided search would discard all prior good evidence. 3. Both paths now log total evidence count after modification.
hybrid_reranker: - N+1 query → batch WHERE id IN (...) for _load_doc_embeddings - bare except:pass → logger.warning for embedding load failures - keyword density: CJK substring + Latin word boundary matching - _normalize_minmax: uniform values return 0.5 (neutral) not 1.0 - _is_real_embedding_provider: check embed/encode methods first query_expander: - log message no longer leaks query content (PII risk) pipeline: - remove redundant outer try/except around query expansion - add logger.warning for YAML config load failures reranker_config: - ContentQualityConfig.__post_init__ validates length/density constraints - from_yaml logs warning when file not found result_merger: - unknown strategy logs warning before fallback - multi_query marker deduplication - hardcoded k=60 → DEFAULT_RRF_K constant - representative selection by rrf_score, not sources length draft_agent: - initial evidence capped to _MAX_EVIDENCE (sorted by score)
hybrid_reranker (+10): - TestIsRealEmbeddingProvider: 6 cases (real/fake/none/encode/unknown) - TestKeywordDensityEdgeCases: 4 cases (Latin boundary, CJK, mixed) - TestHybridReranker: top_k truncation + overflow query_expander (+5): - default intent, num_variants limit, whitespace variant, non-string parse result_merger (+5): - unknown strategy fallback, highest-rrf representative, multi_query dedup, rrf_again precise score verification reranker_config (+7): - all signals missing, ContentQualityConfig validation (3 cases), malformed YAML, empty YAML validation
GitHub Actions workflow that runs on every PR: - Installs Open Code Review CLI - Configures MiMo-V2.5 as LLM backend - Reviews PR diff and posts results as PR comment - Requires MIMO_API_KEY and MIMO_BASE_URL repo secrets
- Chinese-first hook with quantified metrics (60% auto-send, 40% human review) - 3-column screenshot grid at top - 3-step quick start (down from 7) - Chinese mermaid architecture diagram - Bilingual feature comparison table - Reduced from 254 to 183 lines (-28%)
🤖 AI Code Review (Open Code Review + MiMo)Review ResultsAutomated review by Open Code Review + Xiaomi MiMo-V2.5 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5f8e15ae2e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }, | ||
| method="POST", | ||
| ) | ||
| with urllib.request.urlopen(req, timeout=self._timeout) as resp: |
There was a problem hiding this comment.
When enable_query_expansion=True and TICKETPILOT_LLM_API_KEY is set, this path sends the retrieval query to /chat/completions, making retrieval nondeterministic and dependent on an external LLM. That violates the repository-level AGENTS.md non-negotiable boundary that classification, risk, retrieval, and drafting must be deterministic with no LLM calls in the pipeline, so this should be replaced with a deterministic expander or kept outside the retrieval pipeline entirely.
Useful? React with 👍 / 👎.
Comment on lines
+218
to
+221
| signals.append(RerankSignal( | ||
| name="embedding_similarity", weight=w, | ||
| raw_value=sim, normalized_value=sim, # already in [0,1] | ||
| contribution=w * sim, |
There was a problem hiding this comment.
For real embedding providers, _cosine_similarity() can return values in [-1, 1], but the reranker records that raw value as a normalized [0, 1] signal and multiplies it by the configured weight. If any candidate has a negative cosine similarity, its final score gets a negative contribution even though the config assumes non-negative normalized signals, which can distort ranking relative to the other signals; clamp or map cosine to [0, 1] before contributing it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR introduces an upgraded retrieval stack in TicketPilot by adding optional multi-query expansion, cross-variant result merging/dedup, and a new hybrid (multi-signal) reranker, while extending RetrievalTrace for better explainability and updating docs/tests/CI accordingly.
Changes:
- Add hybrid reranking components:
RerankerConfig(YAML-loadable),HybridReranker(4-signal fusion), andResultMerger(multi-query merge strategies). - Integrate optional query expansion + hybrid reranking into
hybrid_retrieval()/retrieve_evidence(), and extendRetrievalTracewith new metadata fields. - Add unit tests for the new modules, refresh README/docs/OpenSpec artifacts, and add an AI code review GitHub Actions workflow.
Reviewed changes
Copilot reviewed 21 out of 24 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
tests/unit/test_result_merger.py |
Adds unit coverage for merge strategies and dedup behavior. |
tests/unit/test_reranker_config.py |
Adds validation + YAML loading tests for reranker configuration. |
tests/unit/test_query_expander.py |
Adds tests for LLM output parsing, filtering, and fallback behavior. |
tests/unit/test_hybrid_reranker.py |
Adds tests for signal math, ordering, and embedding-provider downgrade. |
src/ticketpilot/retrieval/traces.py |
Extends RetrievalTrace schema with query-expansion/rerank metadata fields. |
src/ticketpilot/retrieval/retrieve_evidence.py |
Threads new retrieval params into hybrid_retrieval() calls. |
src/ticketpilot/retrieval/result_merger.py |
New module to merge/dedup results across query variants. |
src/ticketpilot/retrieval/reranker_config.py |
New YAML-configurable reranker weights/boosts + validation/helpers. |
src/ticketpilot/retrieval/query_expander.py |
New LLM-backed query variant generator with robust parsing + fallback. |
src/ticketpilot/retrieval/pipeline.py |
Integrates query expansion, per-variant retrieval, merge, and hybrid reranking. |
src/ticketpilot/retrieval/hybrid_reranker.py |
New multi-signal reranker with trace-friendly signal breakdown. |
src/ticketpilot/drafting/draft_agent.py |
Caps evidence list and merges evidence instead of replacing to prevent context growth. |
README.md |
Rewrites README (Chinese-first) and updates retrieval architecture messaging. |
openspec/changes/add-hybrid-retrieval-reranking/tasks.md |
Adds implementation task plan for the new retrieval/rerank stack. |
openspec/changes/add-hybrid-retrieval-reranking/specs/hybrid-reranking/spec.md |
Adds requirements/spec scenarios for the new system. |
openspec/changes/add-hybrid-retrieval-reranking/proposal.md |
Adds proposal context and goals for hybrid reranking + query expansion. |
openspec/changes/add-hybrid-retrieval-reranking/design.md |
Adds design/architecture details and file manifest for the change. |
LICENSE |
Adds MIT license file. |
docs/technical/retrieval_architecture.md |
Documents hybrid reranking and multi-query expansion in the technical docs. |
CONTRIBUTING.md |
Adds contributor setup + quality gate instructions and architecture notes. |
config/reranker.yaml |
Adds default reranker weights/boost tables and content quality params. |
.github/workflows/code-review.yml |
Adds an AI code review workflow using Open Code Review + MiMo. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+18
to
22
| logger = logging.getLogger(__name__) | ||
| from ticketpilot.retrieval.rrf import DEFAULT_RRF_K, rrf_fusion | ||
| from ticketpilot.retrieval.schema.knowledge import DocType | ||
| from ticketpilot.retrieval.traces import RetrievalTrace | ||
| from ticketpilot.retrieval.traces import FusedResult, RetrievalTrace | ||
| from ticketpilot.retrieval.vector_search import get_hnsw_params, vector_search |
Comment on lines
+123
to
+125
| if enable_query_expansion: | ||
| expander = MultiQueryExpander() | ||
| query_variants = expander.expand(query, intent or "") |
Comment on lines
+212
to
+222
| # Signal 2: Embedding similarity | ||
| w = weights.get("embedding_similarity", 0.0) | ||
| if is_real_embedding and cand.chunk_id in doc_embeddings: | ||
| sim = _cosine_similarity(query_embedding, doc_embeddings[cand.chunk_id]) | ||
| else: | ||
| sim = 0.0 | ||
| signals.append(RerankSignal( | ||
| name="embedding_similarity", weight=w, | ||
| raw_value=sim, normalized_value=sim, # already in [0,1] | ||
| contribution=w * sim, | ||
| )) |
Comment on lines
+296
to
+299
| for row in cur.fetchall(): | ||
| cid = UUID(row[0]) | ||
| emb_str = row[1] | ||
| if emb_str: |
| for result_set in result_sets: | ||
| for r in result_set: | ||
| score_sums[r.chunk_id] += r.rrf_score | ||
| # Keep the version with most info (prefer one with both keyword+vector) |
| # 🎫 TicketPilot | ||
|
|
||
| AI Customer Service Copilot for cross-border e-commerce — **deterministic, no-LLM-in-pipeline, full-chain traceability**. | ||
| **中文客服工单 AI 分拣系统 — 确定性管线，零 LLM 调用，全链路可追溯** |
| ```bash | ||
| git clone https://github.com/lennney/ticketpilot.git | ||
| cd ticketpilot | ||
| - **管线内零 LLM 调用** — 分类、风险、检索、评分全部确定性执行，结果可复现 |
Comment on lines
+63
to
+66
| The pipeline is **deterministic by design** — no LLM calls in the core pipeline | ||
| (classification, risk, retrieval, confidence scoring). LLM is only used in | ||
| `DraftAgent` for reply generation. This is intentional: it means the pipeline | ||
| is fully testable without mocking LLM responses. |
Comment on lines
+127
to
+129
| #### Scenario: Existing hybrid_retrieval call without new params | ||
| - **WHEN** hybrid_retrieval() is called without enable_query_expansion and reranker_config | ||
| - **THEN** uses defaults (expansion enabled, default reranker config) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
.github/workflows/code-review.ymlfor automated AI review