From c4fe2f385f11e7d9aa00395575c0a84acd7cd42d Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Sun, 7 Jun 2026 18:41:03 +0000 Subject: [PATCH 1/7] docs: add MIT LICENSE and fix README clone URL - Add LICENSE file with MIT license text - Fix README Quick Start clone URL from yourusername to lennney Closes #5 Closes #6 --- LICENSE | 21 +++++++++++++++++++++ README.md | 2 +- 2 files changed, 22 insertions(+), 1 deletion(-) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..5c92e9f --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 lennney + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 7ae2a58..2afc257 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,7 @@ bash scripts/demo.sh ### Manual Setup ```bash -git clone https://github.com/yourusername/ticketpilot.git +git clone https://github.com/lennney/ticketpilot.git cd ticketpilot pip install uv From 1f102a328215c06ac50c68444cb2fb1f5c23901b Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 8 Jun 2026 03:01:47 +0000 Subject: [PATCH 2/7] feat: hybrid retrieval reranking + open source README - HybridReranker: 4-signal weighted fusion (RRF, embedding, intent boost, content quality) - MultiQueryExpander: LLM-based query variant generation for improved recall - ResultMerger: multi-variant result merging (sum_score/max_score/rrf_again) - RerankerConfig: YAML-configurable weights and intent boost tables - Pipeline integration with backward-compatible API - RetrievalTrace extended with hybrid reranking metadata - 47 new unit tests (all passing) - English README with screenshots, badges, contributing guide - OpenSpec change proposal + design + tasks + spec --- CONTRIBUTING.md | 70 ++++ README.md | 184 +++++++---- config/reranker.yaml | 40 +++ docs/assets/dashboard-charts.png | Bin 0 -> 18268 bytes docs/assets/dashboard-heatmap.png | Bin 0 -> 30386 bytes docs/assets/dashboard-overview.png | Bin 0 -> 19060 bytes docs/technical/retrieval_architecture.md | 68 +++- .../add-hybrid-retrieval-reranking/design.md | 306 ++++++++++++++++++ .../proposal.md | 118 +++++++ .../specs/hybrid-reranking/spec.md | 159 +++++++++ .../add-hybrid-retrieval-reranking/tasks.md | 111 +++++++ src/ticketpilot/retrieval/hybrid_reranker.py | 300 +++++++++++++++++ src/ticketpilot/retrieval/pipeline.py | 278 +++++++++++----- src/ticketpilot/retrieval/query_expander.py | 136 ++++++++ src/ticketpilot/retrieval/reranker_config.py | 150 +++++++++ src/ticketpilot/retrieval/result_merger.py | 146 +++++++++ .../retrieval/retrieve_evidence.py | 17 +- src/ticketpilot/retrieval/traces.py | 30 +- tests/unit/test_hybrid_reranker.py | 169 ++++++++++ tests/unit/test_query_expander.py | 67 ++++ tests/unit/test_reranker_config.py | 101 ++++++ tests/unit/test_result_merger.py | 81 +++++ 22 files changed, 2378 insertions(+), 153 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 config/reranker.yaml create mode 100644 docs/assets/dashboard-charts.png create mode 100644 docs/assets/dashboard-heatmap.png create mode 100644 docs/assets/dashboard-overview.png create mode 100644 openspec/changes/add-hybrid-retrieval-reranking/design.md create mode 100644 openspec/changes/add-hybrid-retrieval-reranking/proposal.md create mode 100644 openspec/changes/add-hybrid-retrieval-reranking/specs/hybrid-reranking/spec.md create mode 100644 openspec/changes/add-hybrid-retrieval-reranking/tasks.md create mode 100644 src/ticketpilot/retrieval/hybrid_reranker.py create mode 100644 src/ticketpilot/retrieval/query_expander.py create mode 100644 src/ticketpilot/retrieval/reranker_config.py create mode 100644 src/ticketpilot/retrieval/result_merger.py create mode 100644 tests/unit/test_hybrid_reranker.py create mode 100644 tests/unit/test_query_expander.py create mode 100644 tests/unit/test_reranker_config.py create mode 100644 tests/unit/test_result_merger.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..e32f551 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,70 @@ +# Contributing to TicketPilot + +Thanks for your interest in contributing! Here's how to get started. + +## Development Setup + +```bash +git clone https://github.com/lennney/ticketpilot.git +cd ticketpilot +pip install uv +uv sync --group dev +docker compose up -d db +``` + +## Running Tests + +```bash +# Unit tests (no DB required, fast) +TICKETPILOT_SKIP_DB_TESTS=1 uv run pytest tests/ --ignore=tests/integration -q + +# Full tests (requires DB) +uv run pytest tests/ -v + +# Quality gate (must pass before PR) +bash scripts/run_quality_gate.sh +``` + +## Code Style + +- **Linter**: ruff (all rules enabled, no isort) +- **Type hints**: Required for all public functions +- **Docstrings**: Required for all public modules and classes +- **Tests**: Every new feature needs tests; coverage must stay ≥ 70% + +## How to Contribute + +### Reporting Bugs + +Open an issue with: +- Steps to reproduce +- Expected vs actual behavior +- Python version and OS + +### Submitting Changes + +1. Fork the repo +2. Create a branch: `git checkout -b feature/your-feature` +3. Make your changes with tests +4. Run the quality gate: `bash scripts/run_quality_gate.sh` +5. Submit a PR with a clear description + +### Good First Issues + +Look for issues labeled `good-first-issue`: + +- 📝 Documentation improvements +- 🧪 Test coverage for edge cases +- 🔧 Small bug fixes +- 🌐 Internationalization + +## Architecture Notes + +The pipeline is **deterministic by design** — no LLM calls in the core pipeline +(classification, risk, retrieval, confidence scoring). LLM is only used in +`DraftAgent` for reply generation. This is intentional: it means the pipeline +is fully testable without mocking LLM responses. + +## Questions? + +Open a discussion or comment on an existing issue. diff --git a/README.md b/README.md index 2afc257..2add7d3 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,42 @@ # TicketPilot -AI Customer Service Copilot for cross-border e-commerce — **deterministic, no-LLM-in-pipeline, full-chain traceability**. +[![Tests](https://img.shields.io/badge/tests-1%2C662-brightgreen)]() +[![Coverage](https://img.shields.io/badge/coverage-87%25-brightgreen)]() +[![Python](https://img.shields.io/badge/python-3.11%2B-blue)]() +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) +[![Docker](https://img.shields.io/badge/docker-compose-blue?logo=docker)]() -> TicketPilot chains intent classification, risk assessment, evidence retrieval, draft generation, and human review into a single pipeline. Human agents only need to judge the ~20% of tickets that actually require judgment. +**AI Customer Service Copilot for cross-border e-commerce — deterministic pipeline, full-chain traceability, hybrid retrieval.** -## What Makes TicketPilot Different +TicketPilot triages customer tickets through intent classification, risk assessment, hybrid evidence retrieval, and draft generation — then routes only the ~20% that need human judgment to a review console. The pipeline is fully deterministic (zero LLM calls), with every decision traceable from answer → citation → chunk → document. -| Feature | Typical Approach | TicketPilot | -|---------|-----------------|-------------| -| Confidence scoring | Binary (confident / not) | 4-dimensional weighted: retrieval + classification + citation + evidence density | -| Response routing | All-auto or all-human | 4-tier degradation: AUTO_SEND → CAUTIOUS → HUMAN_REVIEW → ESCALATION | -| Hallucination guard | None or simple keyword filter | 8-category forbidden promise detection (refund amounts, legal threats, etc.) | -| Retrieval | Simple vector search | Keyword FTS + Vector HNSW → RRF fusion with per-ranker contribution tracing | -| Traceability | None | Full chain: answer → citation → chunk → document (ClaimProvenance + RetrievalTrace) | -| Agent architecture | Single agent | Multi-agent orchestrator with intent-based routing to 5 specialized agents | -| Pipeline determinism | LLM-dependent | Rule-driven, zero LLM calls in pipeline | -| Calibration | Static thresholds | Feedback loop with isotonic regression calibration + reliability diagrams | -| Experimentation | Manual A/B | Built-in A/B experiment framework with comparison reports | -| Self-reflection | None | Skill seed learning from successful draft patterns | +> This is a portfolio project demonstrating production-grade RAG architecture patterns. All data is synthetic. + +--- + +## Why I Built This + +Most AI customer service demos hide the hard parts: how do you know the LLM isn't hallucinating? How do you decide which tickets need a human? How do you trace a wrong answer back to its source? + +TicketPilot answers these questions with engineering, not prompts: + +- **No black boxes** — every retrieval, classification, and confidence score is explainable +- **No hallucination risk in the pipeline** — LLM is only used for draft generation, with 8-category forbidden-promise detection +- **Hybrid retrieval that works** — keyword FTS + vector HNSW → RRF fusion → multi-signal reranking +- **Confidence you can calibrate** — 4-dimensional scoring with isotonic regression, not arbitrary thresholds + +--- + +## Screenshots + +### Confidence Monitoring Dashboard +![Dashboard Overview](docs/assets/dashboard-overview.png) + +### Tier & Agent Routing Distribution +![Dashboard Charts](docs/assets/dashboard-charts.png) + +### Intent × Risk Label Heatmap +![Dashboard Heatmap](docs/assets/dashboard-heatmap.png) ## Architecture @@ -25,7 +44,7 @@ AI Customer Service Copilot for cross-border e-commerce — **deterministic, no- graph TD A[Ticket Input] --> B[Intent Classifier] B --> C[Risk Assessor] - C --> D[Hybrid Retrieval
FTS + pgvector → RRF] + C --> D[Hybrid Retrieval
FTS + pgvector → RRF → Hybrid Rerank] D --> E[Multi-Agent Router] E --> F[Refund Agent] E --> G[Complaint Agent] @@ -55,60 +74,67 @@ graph TD V --> B ``` +## What Makes It Different + +| Feature | Typical RAG | TicketPilot | +|---------|------------|-------------| +| Retrieval | Single vector search | Keyword FTS + Vector HNSW → RRF → **4-signal hybrid reranking** | +| Confidence | Binary (confident / not) | 4-dimensional weighted: retrieval + classification + citation + evidence density | +| Routing | All-auto or all-human | 4-tier degradation: AUTO → CAUTIOUS → HUMAN_REVIEW → ESCALATION | +| Hallucination guard | None or keyword filter | 8-category forbidden promise detection (refund amounts, legal threats, etc.) | +| Traceability | None | Full chain: answer → citation → chunk → document | +| Agent architecture | Single agent | Multi-agent orchestrator with intent-based routing to 5 specialists | +| Pipeline determinism | LLM-dependent | Rule-driven, zero LLM calls in pipeline | +| Calibration | Static thresholds | Feedback loop with isotonic regression + reliability diagrams | +| Self-reflection | None | Skill seed learning from successful draft patterns | + +## Hybrid Retrieval Pipeline + +``` +Query → LLM Query Expansion (2 variants) + → Parallel Retrieval (keyword FTS + vector HNSW per variant) + → RRF Fusion (k=60) + → Multi-variant Merge (sum_score dedup) + → Hybrid Reranker (4-signal weighted fusion): + ├── RRF score (weight: 0.40) + ├── Embedding similarity (weight: 0.25) + ├── Intent metadata boost (weight: 0.20) + └── Content quality (weight: 0.15) + → Top-K Evidence + Full RetrievalTrace +``` + +The reranker is fully configurable via `config/reranker.yaml` — weights, intent boost tables, and content quality parameters are all externalized for A/B experimentation. + ## Key Modules ### Confidence & Routing - **ConfidenceScorer** — 4-dimensional scoring (retrieval 35%, classification 25%, citation 25%, evidence density 15%) - **DegradationRouter** — 4-tier routing based on confidence level - **Claim Guard** — Forbidden promise detection, citation coverage, risk acknowledgment -- **Citation Validator** — Luhn bank card check, unsupported claim detection ### Multi-Agent System - **Orchestrator** — Intent-based routing to specialized agents - **5 Specialists** — RefundAgent, ComplaintAgent, LogisticsAgent, TechnicalAgent, DefaultAgent -- **Per-agent prompt templates** — Each specialist uses domain-specific prompts -- **Self-Reflection Skills** — Agents learn from successful draft patterns via skill seed +- **Self-Reflection Skills** — Agents learn from successful draft patterns ### Retrieval - **Hybrid search** — PostgreSQL FTS + pgvector HNSW → RRF fusion -- **RetrievalTrace** — Full explainability: keyword rank, vector rank, RRF contribution per result -- **Context truncation** — Token-budget-aware truncation for retrieval results +- **Hybrid Reranker** — Multi-signal weighted fusion (RRF + embedding + intent + content quality) +- **Query Expansion** — LLM-generated query variants for improved recall +- **RetrievalTrace** — Full explainability with per-signal breakdown ### Feedback & Calibration - **FeedbackCollector** — Records (confidence, action, was_correct) from human reviews -- **CalibrationCurve** — 5-bucket reliability analysis with ECE - **IsotonicCalibrator** — Pure Python PAV algorithm for confidence calibration -- **ThresholdAdvisor** — Suggests optimal thresholds based on calibration data - **ReliabilityDiagram** — ASCII art visualization for terminal -### Evaluation & Experimentation +### Evaluation - **NLI Scorer** — Sentence decomposition, synonym expansion, negation detection - **Retrieval Metrics** — Precision@K, Recall@K, MRR, NDCG - **A/B Experiment Framework** — Same tickets, two configs, comparison report -- **Human Review Accuracy** — Precision/recall/F1 for review trigger correctness - -### Dashboard & Visualization -- **Confidence Dashboard** — Streamlit visualization of confidence distribution and tier routing -- **Retrieval Visualization** — Streamlit table + contribution chart for retrieval traces -- **Human Review Console** — Review interface with approve/edit/escalate/reject actions -- **Chat UI** — Multi-turn conversation interface with evidence panel and risk escalation - -### Observability -- **AgentTrace** — Append-only event stream per run -- **ClaimProvenance** — Answer → citation → chunk → document traceability -- **Provider Latency Measurement** — Benchmark script for LLM provider comparison ## Quick Start -### One-Click Demo - -```bash -# Check Docker, start DB, seed data, run demo, optionally launch dashboard -bash scripts/demo.sh -``` - -### Manual Setup - ```bash git clone https://github.com/lennney/ticketpilot.git cd ticketpilot @@ -126,38 +152,32 @@ uv run python scripts/ingest_knowledge.py uv run uvicorn ticketpilot.api:app --host 0.0.0.0 --port 8000 ``` +### One-Click Demo + +```bash +bash scripts/demo.sh +``` + ### Run Tests ```bash # Unit tests (no database required) TICKETPILOT_SKIP_DB_TESTS=1 uv run pytest tests/ --ignore=tests/integration -q -# Full quality gate +# Full quality gate (lint + tests + integration + openspec + secret scan) bash scripts/run_quality_gate.sh ``` -### Review Console +### Review Console & Dashboard ```bash +# Human review interface uv run streamlit run src/ticketpilot/review/console.py --server.port 8501 -``` - -### Dashboard -```bash +# Metrics dashboard uv run python scripts/run_dashboard.py ``` -### Calibration & Feedback - -```bash -# Run calibration with reflection data -uv run python scripts/calibrate_with_reflection.py - -# Run A/B threshold experiment -uv run python scripts/run_threshold_ab.py -``` - ## API Endpoints | Endpoint | Method | Description | @@ -175,20 +195,22 @@ uv run python scripts/run_threshold_ab.py src/ticketpilot/ ├── api/ # FastAPI endpoints + SSE streaming ├── classification/ # Intent classifier (deterministic, 8 classes) -├── config/ # Central confidence thresholds ├── confidence/ # 4-dimensional confidence scorer ├── degradation/ # 4-tier response router -├── drafting/ # DraftAgent, prompt builder, claim guard, citation validator -├── evaluation/ # RAGAS-style metrics, NLI scorer, retrieval metrics, A/B experiments -├── experiment/ # A/B experiment framework (Config + Runner + Reporter) +├── drafting/ # DraftAgent, claim guard, citation validator +├── evaluation/ # NLI scorer, retrieval metrics, A/B experiments +├── experiment/ # A/B experiment framework ├── feedback/ # Feedback collector, calibrator, threshold advisor ├── guardrails/ # PII detection, security scanning ├── intake/ # Ticket normalization, entity extraction ├── multi_agent/ # Orchestrator + 5 specialized agents -├── prompts/ # Per-agent prompt templates -├── retrieval/ # Hybrid retrieval (FTS + HNSW → RRF) -├── review/ # Streamlit review console, retrieval visualization -├── risk/ # Risk assessor + rules (8 flag types, 3 severity) +├── retrieval/ # Hybrid retrieval (FTS + HNSW → RRF → hybrid rerank) +│ ├── hybrid_reranker.py # Multi-signal weighted reranking +│ ├── query_expander.py # LLM query expansion +│ ├── result_merger.py # Multi-variant result merging +│ └── reranker_config.py # YAML-configurable weights +├── review/ # Streamlit review console +├── risk/ # Risk assessor (8 flag types, 3 severity) ├── schema/ # Pydantic data models ├── tracing/ # Provenance tracking └── triggers/ # CLI + webhook entry points @@ -203,9 +225,29 @@ src/ticketpilot/ └── Coverage: 87% (>= 70% enforced) ``` -## Portfolio +## Contributing + +Contributions welcome! Good first issues: + +- 📝 **Documentation** — Improve Chinese/English docs, add usage examples +- 🧪 **Test coverage** — Add edge case tests for retrieval or classification +- 🔧 **Bug fixes** — Check [Issues](https://github.com/lennney/ticketpilot/issues) for open bugs +- 🌐 **Internationalization** — Add multi-language support for the review console + +```bash +# Setup dev environment +uv sync --group dev +uv run pytest tests/ -v + +# Run quality gate before submitting PR +bash scripts/run_quality_gate.sh +``` + +## Technical Docs -See [docs/portfolio/index.md](docs/portfolio/index.md) for the project elevator pitch, architecture diagram, and key metrics. +- [Retrieval Architecture](docs/technical/retrieval_architecture.md) — Hybrid retrieval pipeline deep dive +- [Validation Policy](docs/technical/validation_policy.md) — Testing and quality gate rules +- [Portfolio](docs/portfolio/index.md) — Project elevator pitch and key metrics ## License diff --git a/config/reranker.yaml b/config/reranker.yaml new file mode 100644 index 0000000..43f5e4e --- /dev/null +++ b/config/reranker.yaml @@ -0,0 +1,40 @@ +# Hybrid Reranker Configuration +# Weights must sum to 1.0 + +weights: + rrf_score: 0.40 + embedding_similarity: 0.25 + intent_metadata_boost: 0.20 + content_quality: 0.15 + +# Intent -> doc_type boost values +# Higher value = stronger preference for that doc_type given the intent +intent_boost: + refund: + policy: 0.15 + faq: 0.10 + return_exchange: + policy: 0.15 + faq: 0.10 + account_issue: + policy: 0.10 + faq: 0.10 + technical_issue: + faq: 0.10 + case: 0.10 + product_consulting: + faq: 0.15 + logistics: + faq: 0.10 + case: 0.10 + complaint: + case: 0.15 + policy: 0.10 + other: {} + +content_quality: + optimal_length_min: 200 + optimal_length_max: 800 + keyword_density_weight: 0.5 + +num_query_variants: 2 diff --git a/docs/assets/dashboard-charts.png b/docs/assets/dashboard-charts.png new file mode 100644 index 0000000000000000000000000000000000000000..1ff3e561a9dace915a1a979e4ae3c6dc6236eaa9 GIT binary patch literal 18268 zcmeJFXE>Z)`!I@M2r1fqN3^K9QwUL#=%OSd5`rX%sG~%5BYGPpg&?8@i5djaqjy7e zqD3!b^wH}KW=z@3{XD-9?{WM;{g3^=`@`N}jJeje);ibO+nUfPT562vE}nxRi1E?G z`_CYV7W_+n_Y5`oh`$<%fuO&jNB8gQdL?h5nEgg@h!u)9Ib}zH>QTnErYYn6{weQk$J;fpMin+aym#mJUYLA>iJ4rNu6LRbSn9K95O?k9ix0 z4W3wm-KtrjSJ^gkZeHmp%~{`5Dmx8O&tG>TUc1iDAE~6VZ z*AP95p8H`Y=JO+C&$kc`8k#!Tci9Ayq_zB_6w?WI9G4tZb1h~DS$)_~&Pcz+Nr*V% zlg?YAzvl1Ts7rl+R>x;rQRYI97}F}c!R6b_xSu28;Gb87$aj9-hUH|LVm1;jNL2K- zu7`c&KR=ebI-}?~+8XlQaaw6TF(v6x&lq84o3?q=;|v?1CTdK#%uz#gx3w$o zy{cx`)hvAxpPemPfnvRxs^dk0vF(`zp;QAGmmao+_zP0Q8%}57a5QJK-MD_5P^$4o zlxVV?=X}`dQMgFajFGJCQ&?dD6OH?s_^Sh8af9#X+wE3|Gf2mcdhgh96$|3;)vT%K ze*1$w73m5e(`F`0-b(w7Zn@`AdG25HRIGJSMJ;;Y?x^f=@IB>uBPvzK-q2#5)HAwi ze~wXp40SkL)csXozd|YVWj9WTi`UeL(ziD;)UASEy>n`v;&*WPAVP?V^x$79@YdAq z5;^qW$P>11+)s=|SRIciZ5_u(KDO(hn7>Y+FfCiH!)EOzT* z%X0CJqv^HaTxrzdh;Yw6^u;eKKW_y6tF`|7@&B0e`d?Heo+>bXc_)6G*O=6C3B^{b zt7U3?^wfOi^s46;y_>$))=G`%CofD0HPBbrnjn?m=^a~ls@F6OjFHi9_9kSj1ln8o zYd2?+evNZzBrjz4fd#y!CWn}fHsX>dYkmlt4jT-595d}q2@0WTdm#6zl}F1?T{Qdf zVeRlZz-uuF&Ac1M>n|8({^?GG5aHkr^9+kieGpSq)FPjFU^@RR9G9(P>E$Tz3_8V~ zPLbs%h~X(U2meA_vH#`Ocolqt$;Hk)7dZmcH6WUom2QNJxU>i(l1Fzg2W#iU4lWrU z$m}6dh@1tP{*g`r60JkxQ-QbjeQ78?7)fzRB^;vb)V*1Z{wpz3k*m{8&A;>+4UKsD z8M8nJcV}F}f-y4PlFNC9p3|6Wwb}273-JkE*-e^IMVdFMI`ac z5Hq(=K^*5!i7U6u5`J5T@EV=}5Zv_sG!5tWEYd&A7E=pBoZ4V={4!|@cATSh#?3~S z@R)x;YY0cFuJ%S(LeN)DFdj8ZBQ{1s{Ti6$UWszD|6MmMHJkn|E0?^jH0U#%1y*?s z-U`D;zjbH!FW=m;4~kukXQvH*vr_DYddn9{zIYny)CFj|U7o#0o(rfN-9B6Y)W=K3 z?1N1q+T@TK$zu473c3{oPJiY!1y{M8P)VmXHFVt znM8jwx{E>xZ^Cz77iM6%+r@~so5ZJqQ@(}hI8K4ZD~g$x>MNFz7daA5jqnoy#{f$Z zHHTSK)|9F5%HbbP#-RPtX&bJcCvjRlm7E-b43`3_=nVHxqwQvp3Ihs-w0a4&!P(1* zsTifBdkaDwj6cgZ_sj7ag;IY|5mKn!GO@4d+fKj&B&v}NBD za8=x`qN4FYJK!(GIlw4)z}B-81+dAz@RT54TgCboso6_~OYHp*TxY@JYGCoGm3+!# zfCwYj*uJ@1gGR+*?{vTjaSymebfMr>e--8)wDqsLePeg&J|A>RB*h7K29*dl=Kz~4 z+bB`}IKV4!moHQP5pS(%tUMeDmWRf0bM@4 zaRw5f1ssf&-w}g3A335s&*JqesjT19`jR0K5+{W&Ud zY??^lbVjt)D$zsYfnd{sRhi?pVj4s~f@|k$P{F`a3)0>jR|Dp%g1PM0FkAvXwZT6q z4`&?TX^2CRn&TUc8*zY6L5yfm{?Z6?h@RZf%$%Y~LQR3d?IZD{JZyT$s7aYiJFc>q z)#ysw(kna-gsGaEz$jw7b5zX`=6iEM+vOfvW-72uZM_*0QspNGDN}$JLlhR`25RA_ z2qUzs%51hT0$?_pw9tw**ff%7W)D@Va!f)4z)tihdmU1F%L4p3)0;veP_Ic(c zX&O?&(#k%-+}Fhbl4>p91_~r!0i(|(+Uzt&=?F$zoqR@|o&Pw7!0i28y#G6@rJ>Wp zqCrEGDG_BA!_^q4B}oUZNP{i=ucZ0$BNmm-l*_++o-WlO*r@Cwkvz~h;35V#{;gRA zSn|2qw{o6|SbYz3{yGUKq6c9m9fC+I0Cf>9O5_e3e^w4+KL<;W^F~>N$)R8Q&ep1&q4#tfE;`>wfjm&FjuR|pwa0)KmPS{$x)sOaxoP| z2!iNVXu!O`IGi2TVC9T5!m{V+1O5LAr1HsM8ZpUbPi0^TuGi?HHpt0(rM9o31eP2F zOTw>Gj*DsDUNL#e*I9B7dszl_AC6|8$<(P|E6LXP}vMos;W43OL;4cT1KOyf*U`91YB_$ zU^0XWj<^za>479|>F1u0R0AOZct~eJci3OO+wU z#sz(hc9Q)ZFp(8>!+kO5xe+uVeKL)kysJca>ycC2igLW11FHmtxDU~|8SRG~Om+PHf;qM6rngVu3c`SaVoO$_WIkW#%>DXLaEx$N~P(GQYgIfqr5 z;c{lPpR5sul#79!ZrR!&srKae)85k7YWo2en?W`dmVm)+0NNI?Uz5tOE_w(?H5AlC z-wpvxhukIIp8dzj#hu4Z1N>ZVy-p5D&y( z-?$PzNc`ner*G*wdB5h@$V$5SOw0-J!oOn$0e#M^_kiFT=->zt)Bsqac5bDQ!tnAZ zn?6mSof@q;)NnbFs!!h@MeoR13~cbR)T{;|v!O`whJ$>$5^%P`QTF3}b~}PTCir%( zC?DAQ`0GG{slaeygmLrQ)SO}pL}h+oyu_(*0JBZ_TU5N~6S-2E@U%fUmWKWcoB^r; zlarOZ6t||ltZ`yn=wIf>>w9D;MFPtxTe!U9sBm}Ca=#ty&xoDLmPXLSoda3EjU?ZcK1HT6T z<*9ESU~)~x=%!ZAMH&?~Ed%ot+{6s#yo3U`05LJ6#O}VPs6X+C>ioW02MR#WkN=%f zUn?yBU4oDN)v?K-U}*$H1G%L^bTZOZcTQEX>6);0J`<@>Vul6~)Eu+ZhNifYgMnC` z_%koBr^P_n$Ntc)aUv|2i9xauD5m20T6StM|EzeEY&ox&EFe>GV*KU)ak}IRL!Ck# zFs*K}VDXT6M-hYX|C0;QPormeR7krZ$_UV8h%uuYa0GjCZ%A?1x`?_+&K4|j-++9Y zc+CP`oljv0Mhu^@bUstONwq@70v_3TZ?-iJ4FCCg+4|*EelE$n3*rz;0FXGnu-W{V z%5$yG;!;K#`3hdI>yWav0X3sDQz5n2LoSw1bw@_z8E8Ngc%deC_K&0`PX!m87#NKT z9)A%?*%NSWQ910?m-K?H*Eztlzj`B|!LrHm9va|@SEg}Og{*_(Tk2dMd}4UXqw$Su zMHJ`()&#>YI@wDB$$$`@=WU?cp7S;U03L-4oz(2*w53{8bAY8$z=5cZak67zKsee0 zMt@R!ofG;d4?=(ybvbt#acl)QqFHKoV_}FTa0{@kiLD0bO{@3GQIzE#>WM7iUkqrT zfY5e+r6BrtyVR_MB{hSxCt#_f??#{KO7FS$><4YcIN6^2FRZy0SEE=bKW+31*NI1I zP8tIK2m$9(pVS{WxUyXuPCfCx_vEmV;NdfSp(x2PlI<(jZEdOnlLLq**<`ZhkwM#k zw~+?hl=YN6zLsCkgSZ)`_lF9qZ5i``OuBpNm0xw^6cr&9kj~oAmjf@UEO`Z-3v@a; zLRS|K{istH(8(<}Ko+W96n5x*@$a`TI%R#AJ$jm`Xg?q70TZAegk?kjrs{ks8ybDG z_8-N-qN8(Mx1JYiYeThs2Q1Y4-90Mvg1FIg@bexp)8Ia|@1_Vz%8KnO>SDe`50&P@~p&vj%7j3z$pv|AOvo8GD68FeLM8`Oz%Rs9{2s6k6K6g zLAAh$B)`?5QlWDRy#@IIU7M^zrcI#ZZ&}2*p88i{Wfv0~u0x#d0@ok~vwemp*+rPl zC5RJ*z5_HTQsbAx$SH_)^Mn3H9VRMZE{P_}y#6+=GV?eMgEN!2WqWY_edv~&=|7&j zz(+ViFy>Xs)t8^&gahyUKVn|+CitTT={BTb$qbarQUW*?KR8n=I1}c%bF}cKyL4S; z!c>90|2}x)tqR?Ga2IcMBKiv8m3aMUaXK$91TsX$a(oR)SNRF8oITNF=NJ5CAx{3+ z6Jo%={i_H4dKo$Yx#eP3YI>*4lmS2Few%lsf_#8% zV_QGu@||f4@Xhj%sxWJxzUP0G8s;D163HxbO|2_)| zMlMLx4D<&hTaMTR6@du)*c^SXxzrJ$ny$1Y5DeQT0}rP)!3nrBM!L_Bvh%42;K7;% zz7t`C2p<4IHFNN)a~BDNXV$@*I$$46aKq-9>$!010cIDqICiaMpr%e7fz+Ps9E$VZ zv{WmnFQaicr#7fojL0TbD{;}=1e?yTv&w1R*Eyv25|qwE-%d=iX&0A?tixV1gw37P z!A!s)FESOGXpfB1o#Er`RL<~M00_isF;i72A+gzVyp&bMIf%n zH1yRkZ*3*dt`xBf<`7t6!%;S09A(z#j3aI2<`2JFNs2L3RK_9Ev{US4DPlO&fb|1Y z+*y^`^vj2<>>V3CxHS_b_74Ry74$@>bk5nR?aOjYS8H=+J?ZrRgGJ1yTug%cRgVe@ zB;uvfH;U>4IvQEe4=JCdYZ2LqGeqIEa|~xiLPa80t3KIk-LGQN916xb%6?X|XpJ7s zo%4-cRvH>6^wAq0@(Ia!z4UUN?;O8x23GTLJ;fwW@m`#CNt|-I z7^&@%G==0k7cAKzSv9+=(sF>|@%NmFeslSKKd%tSO(99Sfo&Hxy^miG<^WoO#>HkG zT-q6trn_ZWOG+XxS26XF@Xc@9S3L4|RxdlkMV&MkE9PhW%t{kONdDAWUJfN4H1d{HDPwe?TzhiJ!&3U&v3Tkrm)wRYa>qsRc4)!wOg ziG(pzeeQ_2Q$i0axSf6rbB)}(HF)Qe5(U#M0_Sm}aYownQ5uH;kNYRYYUR*Ch`imd z(@Z2}G2JZjBRS$KkV+~uYKDQs2U?UDrd`!gatM)L;Nus$|3+{merY(inDQFxYgt*R zNDw{)D?tIEAqA7YG%cpFkXx7aIf&-u*a?GU=)YZv)@*&E0-o4nV7Ho-cjGvf1G5o0kxtW(KuLHg)6z$&I<~Yo)|Fg?3;{SWDA8bHL zrNNffx6g8&Vg~4RV~KVWj7FW>CdCl3VS1!QFh!F#I`q8+Ftk>+haw&3EfdC z2b6toq;Uy+ZpD0^X;DpS5mV73c0)eH|9JWfGZBWPZ))1tmQzLw|44p@ooNed9*M6) zUf2ET@aSFTS#};TSG&cs+~kIe%ZVCm#n+v}d5gFwj0a*oWy&KuH6uGU`0gZnetW$b zTNr{KAqygGkHmV#svUs!paKpb-M!u4Nqb`c8fJejRmQO{66+NQIerDT|E&b?mLE^_ z+_7=GJN0vW`v!7~w($M2PECc{-nsL#{f*CLw%&|d!-R|u_G9=Yj@ZlK zi)xy5C%`Fx`Gj!cED_`=KQ3_Ic_8xi-7{)tYE>O>83 zi{2=tkgACDhaHk5yqy8Q26;8tKXT2;xxC1`b%n|B9LoN3HpRn&SVsdfoCobFRA_F@F5yw3qUSeZBBE;Yv!;M$uS$jRTQc}Qp2m+yWx zpr+0yK!ZbpP@%0UBXLQo_V(e|1W{tg7{u4nr{{V$$;i@A*?yn&9W+)|wqg|Te)i=ZqvW^n_ueED}66_`Naguf=but-KY^T|g8xame*GuM>_t~cy!?g4OF{xSOWo$s@<-5Mh6!`mOnpSfb zQes&pfYfX2X$k8=wRj=dU*7M%Iyuq&A2X-YT!8i%mr{LgE*|0T?7*+P5#5 z6bg6Rw4ir`23-?C*7Bb{8t7}@5h8#Cku<$i2z?dEL(?nnP-+&%Ae9ei#g!j~UmWND z!XRNWNqL5Z7-Yd|hg2Ew?4i=mMBRMwn6d&&3VARnm>Yyr!GTYa7B z{~D@Zg)96Y`^A91wpvn44Z)m_^!q2(st{>?gwBhV;|%tXOn`}sm#HNUj`8AE)KrGJ zWNBME{8p0Q$G)P4Iv)Uh`j!vh(Lq`~1^G&B-pH{uC!~*e80S4L_E_X%Q zQ?Os%_M;)|1f3TW%PohjT1O)gbQ8FzrO)k2T#0bq|H%bxjgfbm6A$6c&ev#J;o)o2 z#V?JaKrbL?28D%L<|t{9!ek|mrN%BTvy;|x!qLaagAt(E29Q$<1G%gjHr@kTUI>qm(0!+STKy@AO(CMDbx{yrC%aDS_(fZDoS7 zkmz8vH=;~jjQhy7pROeKBl$eUUj)!sUmp2}LRt?Az@{ksEiQiaJN)&OvcRn3Zw>`= z0G@w6O#GwGwfAxLR##(SRtbb##hd4*Z8hiDkrw>3b5X5z%AKSg@< zd=O^8&jNDufO=o-?F-}qGhS9A`SaI(1SPUC^r2eK<@Qt1EZ{3;o0+Q3CZ&Kzkj|$I zRI0Q0=R80gATN)}hovB&8T30|N8GO!8_9;mv?tH8pMt(= z0Ke!qfSXI2edtgxlS1Q;b1Y@6zC5KN)c>Q#tjaiDy-KC8(anz! zjn)?xB#dZdKHXr1n!Ev;l8tI}=RkE(lDxBR$Xw6Oai>PrYH--I6^oX_(FXDYLMK7Z_XeOneq0+Ea|vdR1c=Sdh0m;XgO@;$hcaAp zlh7QULs{zVCrg%pr*4SnkVYI0Gc%;!k_Ji>f(Rff`3?SECnQp_?4VN{iN+G!qnzm7 zbjYf)CxO$0`UPp%a4{xGSqXrTEvam;1vJF806J>)-VSo@;?d(}^i>0Iop>i;J&*$X zh6HKc`N0&*gf7WzXp1j+bh+sf5U3EOW8Y2{BZ8%%9lz2Zv&VgahX4AT+hTvS1JxWa@ngufo{kx@8-+s>8DGf8adZM z1a7fW{zfB*?q9nI-BPYCT#Cay6MJxz7`rkxTFd?`DmFpzNBz7PwsGiD9~2b?sKP0L z`t0DHRf~^Xi-)=S*Oh@oCl#JHWup1V`o(M0*WNQdiFIsrn>1Nm1v00?1!B^ z(GW;EK@FDXF`?{rmE1Vy#p&Dc&>x#I{8ZCRm{F9$!a7x4l=0h+yY z*33+K!{{(hd%CXBVl*P0?RVRf+ocf@MzS*v2? zLpyZ4q%f~Z)oW{uW(8OZY$V(mT{UN~G5YRnSy8S~aiUC3X$Kz=os&9P!;`wXt*Hbh zJXQDclX^^*|CpC6&BiOrt}M^Z(%fUtKU16+(hr}WLuKt zcPDFfd~}GwK7pJe_CW^XlFO|NdcC!~@v?C#hmsrBzCE>T3Y*LxSh&kni+x_C0@A-H zQ@E;r_i9*Ctv8u5{rG&tS1M`UV%eER_?^f3Ucc(Sb@XSrdh_j{D^;5}m)BTNyh`@- z!R6sbaJjdUN41U}ybs@z?6;8CCNd%S+gM$f`*`hh#6dFYe3Q7d-5A=vrO^J9$4S&{sS()#s;=HcRBcVZz++?V01bU#b!e42CUME$ zUe4)SwurI#=I5*3Ryyo6;<%Demv&kXZy;aI_ulR+50)LVMyir_U4Ua01O*2@tFtF% z6ArjS@2hPNI~XG~&D*gI+=Q%)eR6>cq#DiKB3}D(q&HrrPOH%shAXPgj!9uI-1rjC zW>Vq2^_hw+Vl%$m6tk2Plu~UmS2*@bp)i_d0VC3AyO{&}JE*d)8YiLBc%;)(%>;>kYAtsY9vNX3_XKC8plibD(uf0 zy=PetFyAF`ZKGls+_k6Odt#jX*|ltE?@xTc&28NPCxuxGrvjh0+S%^44#`qO5nX8yPQSbZ&d1gqQH z`-uGKck0cJa7AmRt|8bu$@TPg~oN>M(&!oF2sFG5_eplrj`EfbG-gKezJx* zBz+^I{pZ!Ofc>eee!*N+sVJBI>_4h zx3cXhjIrM)ADG4-w*)A$l!VRuTq6@aBG4iDLq~*Tx8q1|@#;(O>-*`90 zJ08s<^fNQ6lmsmk=b?F>8Gf6|n)%!c`Mmx;o8#?{$}FSNtw}^XyF;G0Ll?%$$aiDR zz`dncT1$TYJoeWkS0~D^(Z~Q>6Kqc9@nD@|W59L;%*e~iAmKE8)9A3*ic3y$wzXW9 z-rDoGuRIgAhC7imV_JO{bOQh-z^_#hAe!RW+gieU``VRM8?;63ZixxwP``NRXo z%Eca-m)kbJ-maxGE)D6vnms1XCa`|sU~)V=T467Jqh@=*EMMz-(PHCTVZKyh{c$=^ zN4}h1Rlxe%6p}lPvUl)Rp=L%~Fjd@l@XcnA6wYIuI1RGSuN>qXbjR`#RaDx=*uKW*Qr@!TYTn#$6#em)AHDwUKN(dRSLOtl+t zu(t2a%%e7W`s>|85;rJ|1?z&S?GrkKB3q*=sC^Bg9>wS~Y_E_lgXRRdP zp$>uV>2fO_5IWD4wkGupj=V%7U5p>M7`+ z(m8D8&$g0HcSel0P*xx=4w*3_M98(nu@NefSr?rf-P zS&gk4{CItpc?XR&;-GOecQHPW8J_08ZhLvOwc4O+rgDXFed)a>E)u;zVZ^)oVRMaf z>YzVOH%>6=7KTOVnRZC8eZKrwabkt@>#>Bp>RVF=Eb1DGYcV;B`3`&2MvZcwPkZSZ zRTsji68Mz-`!PHDaHEmOh)D8)qS$nm z_Y$FRq<(1(UAlrB>1&I%?oaZe9Q5Y?Jj{)d)Et`eT6@-(?7K}NmV^^7#Q**x?b&@< zV@h6Jugn!Dr0b4U<+6J|L+)A{ydi%R7cXC|*uoOlGqqlY!0%$V{4!kk9Ww-@!-D&B zH-+I}k>1OU^lZjT2=$S#qT@zY?rjK-u{4h}qm81xn=+^D(OqcmtDbOqTMg&7kum+v z7Y(cBo82Z4FKE{K@2cFkMj;oAmXcz0*a9q9e{~5Irxc;8H zr{ImKXs(dvyMv`Zocy6c%Xdr0>GI<2%KhE2m_7lijTu9P!ti zu_!;^_6(e1_Rm>Xfh;3)+efy#FL{cG2Flh4uKn5_)40Ifk9mnpJl>vo?J$SMxfU@b zA@OZ>yhVy0J9{|TF`-b$16-R(&{w0`8J@v0&s30`53IFn?Cg$rO~z7(;|DV%lM!NW zW09#(?s#lDq6LM?)AJ!7dd9x{F-e$@+MBihlwIb^>ouFrbz4(^$VN{HGi|VB>A6IRYwfi$;WtBf&2-aK2b@-`+;86Gvbg-kaQ;oY7?h(>EOJ{k;t7oL6qXic^<~Uxi zw6>WMIwn$&BP{6ODMCFtZVe{6Q$c}2R|3G+ta=OZtQ)$E)c($qs2x zyKRNzubqFNLN%R9*z|kfx25@RtfS*ty4e4R>uHDi+%-l|r%4~bw7o@IPjZ*$?MzND z{ALeQVKY^J%g=4QBuz%IY%`RH1tTIP%%GXxANTtOab#&M_>qoW`?mRBT=8~I;l$xi ziBa58cK?@M0+l{vXzw5#$Hh_s?o2GkHYQD0`LFiEjP+0p=sOTZ7Y_yddsvWMFFf{@ zx>JwAS_2m=z=GAyf>x?{+4P*AgZ+1|;QWUD&F7C>(2M9E0ikS>;)L)>-0 z`yY9J*42B5^35o*&hr@>jos2#tk~$`gB=if+Crar*7LIH|27fdj1bU>(?TwRc!m{e zZth`sm{=@gFr@fIEYW|+aX;rmTCGE!Ii_2F8|A#_Q1O$vsq>3dVpxBBEBSkZuAcjP zl`tL~Cht_^xxbtC3ZWtkKE0ecZDfL& zjMRSIxD-^}FIG5FgZ#1g8TcaFiBhEDb%?z}tjH-~Vt(&>j>5%BfhsPor0G<>mMunY z_r|MFCA=Su;9+O-{1*_gMtn1mz&v7dzIev4c3!7y>(YKOhvveJcX4=Yr8uFdlI|9Ri3dGfSEP$y z)Y&ipI*X9m5V>>nx#JIbwp-6ov z7tExj(Wl}F(HluS2UK;}{;bDLJuhB=yq96CuPuv~{`;9&LF3_m@ff^IPdWfsL>tTX zy!Z?Ry{w#uuxXP#9b-!Dnn|*ApIe0szTC02u(tCkc;ke8irf$7;qi5eJ+7T@HhLk8 zAb;!&wX&P~BNkwVn~AOJ1FEB>ObV-;QJp~?S6>H0v%Q6XQufnsGyz;H(f%!{G&+872Ve>UCd+1(SN)`-87PuTvP07?KmG{ITB1rnUWTedt02_Si6&2>-SbGV2G8zST=F#VdPKWj4x>8IB#6++a(%emT-{5KAzk?NIu10H+Vv5E2f%=iZ^6 zCWu@MPC}O3)RIA&Pn??L)#L3aQT#_ztbDH5yx{ews^Me=5Bu z-xw`BnjXy-k$g=9=F8V#>EVb~$2eP$JW`ALXU#;1MR0}U#R-IhO^U9lQL#I<(%(G_ z5j!&#%&CNTt<|zsXxDv745BTStoH)wpfNt7lsmblJ!frq$0zt;Pb!t-1XKK;Jz~?W zb!z^VNBEJU{kT&8s(McUvDY}C<|FhDaX;=ywNG;gOPx--+x+@*(Ui1C@fSjqW?CcM zSLmh_fv6hpl&moKSO=TY9lE4yKDH|=hP2zoz|+RU?Qf_JZCfUksY^;=$9{pkynQ-a zaw$y8bh!aOxUeN=$MF{?uFu3-^)!ulA9N8LwuCHkI>=8TiJ>%49 z!Su5fAa01MH56KpsDhI)0N-&LIzx{0sAM)J=Jfoi=_%p+VVU+l2uM#`aZM@ zY|z*8%F5B<;bbE2nUm928#^pHA%WfJl25Z5ulLLK(ZU=hx1~6Jayy*(U?m!He6WeC zldiq#u`w=~qf(qQ9c$t{RyPwD8#{D7wwhs>$u3LD-W-h$IufU%gJkjrufzQDOFCN) zwzi??ZexEu4`wFh~k%n;GSn;eRYYEBZ8cehF_Qr>%&9ftqRI&4en;97y^rF@L|2EM;$p)^ju8xkI ztA}GFBdU&^OPVXum{i}QGJiOdbYPR}*PR(a>TL=qi`6r^6el~ou;ub} z%T~_`s{|9j-L`c3N{0!Td8^E61$?^vR(0lndt?A{;rDdMRLwA+m>_YOuG8%?{Nt&B ze|QELxE}uSTCBD2-o*XFDcNnrP}0l=M4j*9R*Jl5@JuLq`Tp&h63dPU0Sa=I$@^j9 z;W>}Lo>#a)x-f`N`?m=Q+Mxq)JKEdJ@1krzNSpt9*XIqFVEMFM6~e@Gj^_~z&RJq~ z-3dbO{G!MB_3ig7@%8ohh#~#Dt|qwcz7t%|RA-*B0w@_Qmr5jfNI7F~KEQF2Lfx3) z{W=mek=Ye3A(XKIfJaqz(E2chB*DqncGYNRFWzygQZ-`uT$`H+aPH?$elNg41|?LaB|sOLlcwr3 zpK6oJ*{6c}gD#?x5nAj+6<#vBufT`sOyTpzrdO1gZ=jJ%VdJkC+EuUmrHY-`w8(rK zfZclVx%VlZ=L{$6!?mhGF7Cl?ig0oLpR$yhrKP1o`_#=rda?SS;c7u-(A`4aG;#mz z1=ZDl|AR6>-GLH0LYQ#Thpp)vSIlaNWlGN9mmj#j+F$LsmR61NV|pZ4vWvsgcI8YD zafX?Ax{PTVJoTHeLz&Pc6cfbca{8FBs1v5Ar~4Kf<<_O$%6DZ0yf8QQ2O(LZk-mju zEq@80=5V?DnX>?WC&GeK*pl(CLK77zk%J(vOi)P3XR~7LWIDIig)RhvF^TSZD> zdgmz)Xtp3RDTyVtwd-&ZgHiJgHub{Dqw&}w?T}A`q#-5J&}~IPe&@Ahj|XqNJsZ|d z1h(#wF08Di+9*%mlWg7+jF1Hj zJMVe8g|hbTOIHb*!DdV2UcF*^c@6xig7fo=ii)StP22`mQ!;<`ZKhz5gV>KV#ARvi z6ghXGZeaLs{l0@_>s0S+d84ik&wOdpT*B=> zdEcL}rF|kTf4w_97-3T9k@YNhK;HLYBa|HT`Sa)G-9f-x6}h?Bd&u|A&Vb%n$eNXr zOD&QYCCl*f*=?aLn$t{Jufy%Ld`C^E4Gj$|;b{L#fji`*g~6(_SH0&jE`6W7yGO(B zNr(Mr;WNZ^=?$i&t7Uw3#zenD8LA_sjt{q$Toxa)QzbvXMk~(1_U{o>1kCiFKWAfQ zRjtb8wA9qpT&WN&S0v6c!(O+sj*N^v6;oYU5^Tz7PrYf`?%J;~W`M+?n`l-;l!)dW zSF{!Ji(MLZnM&a{qmwlTau+G7et*vnrQK=D@RZKr@9*#DyJ^uHD$m8mWn^TO8L}Z9 zd>1t9LNl}2p02H_s%;1DPs=R;%5y$OAv#;z(-XcKL76!wuDz0&{WpWggx1N8J_Hiu z8Z-LD&1d4k?5n44j%EADP#1|>lNT?>y{)lQYIpD6#bl=F`)_^y(eAl1{x|B2ocr2O z^A@>n*_%7ty}~pNi$l!NEaNq;EtZ`F%G zJJ>wlbRA4XeSZ|IRa53F;4G#Z5WvKzIE~jyez*>_y4toHXoDGuo}l8mB0GV&rU>Gq z3MW+U0O{($;9oLgV#TGUw_c{iYDX!it)FF{KX1ctdDaZrXNz=Zty&P42L1*;Hvg|F z(swd9MU7$qM5YlVBl<_;8;@i8YUke^Xs?52V8xQX)4s9h!$w74B{7bcd;QM9Lud^@ zEB-R5o?b}6SoZbrp9^TX>`(jN?BHC!7e9|^;v|*4_9kdVa?mWG>ERXd`GSZUdm4G7 z*A;ILYi@a8^4^`Aad|ZNGaT(e%48R99Sx4xmHiY_G4DC3i9fg6Sr;OJi$hvVXcO?`D=}9Gq7Zu_5jPxc@Ayg7 zPW<`o;K0e&>#yCAGf)Qu07pQ)jxuUZ>L&XeC+kSFKdOvlWJsV}AG#TP(!DMI zoyUtjIHELZ;`wdMCZoK-VePo6!N=HiQ=4);$TwE`?mixy9|@yd{p<_QW%{3H4l{5Y z2Q+y6FFhSHr%!X(2%F-Yo8#M|*SN-;+h4@%{{feTwICXr&AMWT@gb~TV0GR_572rN zc-Q>@#}fa4dvkLX@Y638%F&L4KY)ww#Mse5aqx^&i^3ZT;0g3dRqKAyJ+n9e7Z}L? Ang9R* literal 0 HcmV?d00001 diff --git a/docs/assets/dashboard-heatmap.png b/docs/assets/dashboard-heatmap.png new file mode 100644 index 0000000000000000000000000000000000000000..9d2a2c0e5a8e0e264b1cb355554a4a477d02ca5d GIT binary patch literal 30386 zcmdSAcQl;e-!?iTA_*di7SR%f=tPU2MD!rhMjuh46GS%#i5`Rq(W6Iiqn8m~L^rw_ zy)*jgb8f!*y}##q&U4mz*L&V|);azeYwX+J_b%7x`drs%2ftR4Be{A1CI|!~QILPB z1_BWPzg~HM{R;3A^C)5o1iB4Uc==qzEph#f^tGOA^3vJ)ZPw+NlGopoE6H9pZItD^ zN_L-rRN43iv8~6CkHYho10!W?Q{8{|>MLEXFp{3x6%`dOOYW;VXW%lYY!`mbv1<3^ z?nyT?)pm(@#-dexzeZWUUyC=oKY7K3SitcmV-sJ8Rj}lDjvKg#Lg7j9ZiXZ*2lqmI z?8R9 zQUhkZi!n9Et0Ap%OE_m{Zk5xSHQZyqH+LckDU2@u_H2|Y8|Qo`b%=#J8Izpu>%n%V z&`?|l&I<>tJCkZPktVoz(N3tUuLXBHH-hVG^s36S>GnR*TX2SHV2!Q(MxV|$;bbi1 zS?Fp0d3PH(+4P+P*merqtyL1c(!!&1x}Sn8i>$`&6g4bO@=_m?LU4hhp`ohc+bvRO ztbr{i(x7WVCqAQJ9()c#R_}_z_IMTpa7Qp$tvx1%LeKv@jpz{s?S?f+Y-MkXY_;Ro zX5O0}t_H()12K=tMb1pXi0i@KGDWBT3?rtjsB(UEs|oCE1S165%Ov;0yr0LLjm4ZV zL$LKb=<^7R!eqqGX3F`p*AWfuyxvxJyS7X_>U!?ZjWif~xm_;zdT z!Sve6uGG<>>lxvBa|uH1*gO-AJ66TD)-oB(S(>f3^m@;b_%qx-o~2qsyw{@4mJHeB zNc(DZ7K=E{Lmc8xk&}rKaiOb57tYbh5?%s&A`H!hXHByC>aaPK(?eZG) zr5cYcCTV}NN=nSsoj9AZWzO@S@FPfKinb*7oBj0s%8FmJ6nk$8drmL?p?OhegkW;r z6MI;R=*eJ3L*9(O#MB61(s`1-t@q2LScT$?{y^xLRgG&@!3Ql@aY#!{_7cuyzY;fB zP+O{wNnQ|n$d_omw1_zMLdchhk&)R+K5%w=(Sad-G!M)3YWHOL;H7&$`+5XfkSW6! zVe^^qtI_2~Y-sp`jYVv+Y>viRX}n-dP(L{BN1q<~E%8Ss7%01WL-i$y`{ly z31LpC5|Ky-YHD#Q*Zt6=mL#^BijrwhG1M7@Ei|~9EXNGC_w!gP6d8Xtl)LX@%(tZ% zSwzo|dCpIG1|jZd=4tYyT3U%aU9LFmIhmB8t8?e6%hyp`>212IFJ*U+1MlOGr&f}R zqBG^1Y?!agiOFLg;Qeff#YJmYfhFTamsLh zhP^v+K3l$JfIbq&oHx}D;g5g+d5Ah0gv#KiAp-X2b1)oF8kv}V&zSmctJ6^2F%hDe z^0X6n1a_YND0Ql`fB}XV4coKFHCv0t{;m%eyS1)-YK+o3x5;(;q-B0Lzv}%ror~!J zeVAJfmO5)rAwO+};hy&hFE`WPKVpFGAV|+X;!YV5yN@{71Y(9wlV@bVl!eS=rWpM-Fo~l56DzKCZV#s%^G?zi zu}1=-Q!iZQpFx00*iydZ!ytI2*bexRNFp=>RgwN(@Z~Fbml5Cx(LYe+!+qQ5@-){x zy98~}|3U-#i>(y&2B)V#IpT4O(9rvNi|LZ3% z;OrK?BRD?DfN}7OcDVYBXR2@=S10SP<-{*{^Zie(LJO?yt))janB7@y#a^pN6>xqQ z6i!Z5(eUG6-FFw^N0;-?9Q=H4Vx`mFARan?j*>G8(Md>tb4C<%x>MsA$S7d%#!9FD zT=}a_traO{7QvKUc=bZGo3lp`Pn1&bZzbC8)z{e0@EZ_|=bfB6T)oj*+=(`;bh|_k z^^~mHi9XfN3&mqDQ9LZnx(IzWgYr9c$ z*jEWMrZM;Q3I8db#5K55UPP6p@KEp`k2|3rcT4OrBF6jQlW{^ zvIO(gLE&id_Rh@xxzXT&inmBj+VH;J_>Jo0j|cB>OSt5ZB-wNHUDNZBSxFt)iLOqb z>)wza+ozs+Mg1kENHWo6$5}vVPlaOil>{AG%+9z+;Ony&k)PhdzYXJ_Co!_<(a`X` zqVN)~JU$9eZ!SCrSL!uwJHKCX!)G$DW##`3gO6|z^|#F}c@E?$W_Y@6%0p7M^8LLA zD~_ALspuRA9;fRK)rP;#2`$euw6fe6;mEbiMypDt7&K%TsKyb1jGEJ~5F3eH;r+qj z+ZihPD6fycGgLM-?TB0Dkc-<~;82I3PHm3DzK=y=n^Ml|kT1$cjXZSl?X7lAv#%LB z50&pRWt!;0ol1?J>{jU#&ck-czm??V#p0~yS^F`OQRID1c-jgs<4vmMJ1ps2SN)>6 zCLYpRTMdU0DecnDY(yd1BOog)2+=32yENt>bysh;)9h6jg{&L&_jFO#b`4DN^MRw? zdwZJ0JX2`S_0~I7^t=P3-A4FcS~MhQ1uLv66`4=6F@1Hzdk}MPGiMW3w5mjKuMZ!1 zBO0ZTHko3vO@bx2Q6HS?aIb)+s*^vbwC$XIK6bCSL3`4rRI#zj!VK=|kf_-LF(vCR*AdDT z%cnbrdQa(DZ1V~`E98k*KOWqr#Yhem<^K9q7a!kIrK9A=TF@R|H^@!%D7`!*t8NuDH`y;9;;i$ZWHyWvu0OhWIC=Qb!c2)R0{uD{zCwo zxa}fa8}P)LyGHjPNiPbwDa2e9ibi z0G~7ousW=1Agi1C$NuU{xIi;AfU;UX_N?Poffo2dwi-To(+O!;PJh0^8|GqfjG(8O z)~52wiM_16Ff zTd#G7^2g6U&O?Ds&3&I4nOtU258h4D^w&*OaKiJIWnR_IHAWEfcz|hKBN(2485sI-t{nVLrvD zJPwOE6v~X%d75)g=o%{TBAn_(?${-2>ytY;1hlrc>TOglZ`_;!j|Va820?ZWYdUdD<% znbgH#7a8R_Yzc1<{@t0v?I5|`je=3ny!rtbya!o-pJ1ez{?>NP-F@`up3yV=ziqEc zXI%BdYNIpHdoMQkzy3x4&8-bV?>YK3k_Rz=AD<<55a!<;@Ia7S5rO!+AApQD4g=%S zZcBcudNbVJo--YK%1{mz>&w67ruAda$8P(Eku_vPTje!S%jfq5PfKs9dBIutq~QcU z7pJgM^pQ>_rpWPe%r~hlIPdoB=+9F~wrwrR4bZdLG(2Ioxv+d^^0*_BC&O4R&&&;A zUa~)d=;P6YjU}CYtB;U1Evphrzcx2` z(_rVYW4_#{SOTo~)jR?vmi4`Sg&K)4>f|ZnaGgX|W~qD-DtA@!|77YY3DK$L(AiOi z3mCBd%yU$Me#kIEu@icYT+vl?aqgx+DJLSymK8q_X0op)%4JugYvWe zPxwR6wkfMcpeu?A_` z{ba$uHwF$K#A@0_WyQOjtY-})T25Y!^e#wW({OIUn?6#Xb} zpwve3@L`C8fJzmj3|SRS$%sqX|SYMBfjd4 z#4Vh6tEpLi07s<-P4D{?x_ZaVa*34=&M)ZQEu$NFrtV~dP<`B%@D{wgFHo+kOQgiV zIrODD*!7434>a1b`56PP();Ith|?-t=sxvD9qA5I+L1OR86oIzO0llII#uSbM7q4k za6PftJOYR*-k)}km={ zfz8?v212GRQ*$~kf^{kuz~Jb0^dpa+u%9oDKtVDNGc}t73?FCSj$e893Xpw(0mSX* zHb}VF?S~0BG0`fQjcmutYKJq5 z68?MeW71q84kYpj(#h<8q|)LqgU3>(8YAA}jC?>2scQX(mvH(SHYluG{w7tBbUj==j|uj&UGyxr_}ZUAL){ zmfUj&Y7ZMLK>q`U1D5YiXX>BuUL-jG#1Lys%H;&))Upwp5RTU(aH&KZYfUEI+%o+cX_w0IhIVtB_3gS|jHYh4CX6r%+_W4c` zN8rJyHbs)HFP%8!Yx_T9O>Gl};hmgqSx+$@$zm=51^x~ceuXw^YIVl$0{;IFsOJhD z^%{-An#J}*U^@OadTvS4hLPJ;)F>LN>Uxnuq}mDU0{#D)ERM~0Ba1v!ZBu0n!vJLu zv!ich?>|lU?cyIVC5dMNff`LHa<&7Q-h1s0zh64=p7> zqPJC3GMtEoI!z6Z`7V0rR1KNk0fDC9+=z8y$~))g21YLjJ~nWvZ91ShlGsq*fU5ic z;1$6(Xy{pd`!5tv_$G)3kHX04+oW?j+f#{5#o&;=1;Y0VH$gsf?E0e=4z6}{vzsW# z5#V2Cn@7*aGn_3m;EI25uC3B-6X6y%g(Aui_8Dl|iJg-jPnMAcHn*=D8SP)qo?=Cx zObAERBB$3oTB)tig)C2BoDe`>1GC@Wd|SC<1~*GWiZl$4iUfao8zP{l#-;Z7r~iMY zgm-z1`Ci*!(Ryx_kv<_Iv4-HD|1fgiWV^FB{fu{vZELi;JZ*YNSxyv@_~VqZTg ze&@(vA<|2Nj^1;o;)nULn)ZG=&fHn&_p7dc#7SqXNLK3=(DWVtmrJ!zUVhzD6SLC+ z78aYGlWt4r)KkyR0nYa&39aNQzWLeG|K!$2$L9LIxWrYjHRVVHO@mm9Wm-~8-9|A< zzO?(HtA<(ua?GemK*3Xkn&)uX-Rn<-re*)^@5I)0GyP;o;!sxYq z@3|+XLSkDpswlU)sY=N@0g=)3g9J?7(YZflf66@NxqX)yY=W2EtpiHPlj44I_(;=B zodYV??K-njDd~6ukZ`PX8H!_b$SoF)s)fpozv>Ol>uu0Iy$Yi?yW`6v#K(7#ZtS7iFE31vlr88!OFxN3jD}C#vO*}HEn0#C8&87AO zoAcI*_sCqbG9H@&??jLJVm&iwS4W7x`omcI8wMoJtlzXsb{l()CgH-n=waRVgrLT# zszdoK!RyUw+WOwL8kl<56UCFhI)eh%z@_=#UuqWVmo$v zZs6SedjbrDhN54FYT>GR-<^dkD?az!VEd(8RaEi%{!?e!DLuU;3e1MB?eh1VR`I42 z7vdteKk$P2b2Gqvtd7w6H%ok0F?2XpKhJAx#g7M#(e)de0()( z>xMrJK6Ca9o;&p`0o}9AAcgLmKe+MrQRnhV3}cDQ2%mNv21>w~_p~DMDL*e4vq=B` ztkMZToCiAidk?+6PYBAFc6bvi3_hj^cq$FIttz{B@fjlb~g~>r@9hdc@@devNwgMQNUh;+*Y96JdnPPoF zI#mauTeXQ(DT3DXD&Nfc<)$;a-jfd~WB(^v>*s%#wT9-kq@)}^jB4UmUAhzlX;AJb z`-#U4i#n&V_ah+pwlG~~{vIrgA^DxZav&$swwaWj^?Q9kD(jt_c4o1p3(P;I&BGH7 zwYTr(F4RKs%nKdnZBFFXR!AXAu^;gerfPxlh&|={oJR*8aQz)%(lnL>-PRkrJ}-97pFjEqAb{%LLQg?fpas zbC{@ud2q-M)ig*pGXeZPNGmEy$my_%$^OYr)#>MGuDadeoS8_(v8s7GM3SwLX4ge` zqE_6hL7`qeeci2xu^WBrJqTu@S~nA>G9iX2ZcSnCZF&&Er=w4D-)oIB!tlp3f*KpA z&>!Beus)jwLeX8;v~PxQgUF#&45QaButLRy|XKPfqiYwcNo8rZ((J=HHw#^usYIuW0z*8W~0 znj!Q@gmFR_gb6?`iB*U18^ea(VsjfvtpHA@zj#F}GBFXX<9TT09d8`X=%lp}GcQLJ zbw}1(t45PDLa9%9k!(28fbe%^a86z>FFWJ>zQ4eo9e1yDe?kd^Cu4N7&K6BnT&GsQ zCdhOZC=)c@0%Yw_4v4ryW(6TDU89BCbTO=9-7E?BM4%0@uddZ_zhYleSx`}ojiS}+ z9^@Aw>nS_8Oo8aRjhcMn>d81Nd59hmz7g`RferNFphd>>xzs7wWk#v38IMuRZr9MI zO{!6Fh+e07USzITktwmbKqi3u^GtoP#QH^yvZd!iCL5(0*w}o|6EYRBI!&PN;G>CrVeVq+na4nPy3On7oYh zQNA1T7{@8eA9qc3vcvW1@Bh*Q#8Jm>G&7`<>Ib0|gj$DLS3tK-(kP~vUx6b9eHkW! z?ZW7Uz5$8=8)bAAY;?HlFn4>Vrb=x;Gs`F|T*U4riViyRMH^`9fxrJ*_!-9fluPL9 zhIPb|A^$*I)%B96V(daePo1(r!~B5v^KKV6a}sHpOx0Q-lETpEJqw`&qz9_($wu}* zKS{{@Bb}dFIk939N%u6=^^rJuM&tSPO`AV`L;W%89~;ah!oTHD5r%A#L-0TmAAjAZ zDm-{ecoblNIZR+zuO1HUy#bNn|Xt$G5f&M5b3O+_`357IbjapHX#xCIdK zZ`*=OK7)w!O9`A0=a|y_#k~sM=k-4%kn>P?kF}z?yZCmK0psdUNGP{0`N|S!)6v1N z3$E!;Fz_;Vcv1y?if+unE}w-(|qtuuXkd(6nS3;x~6&MOk?8oEG4r!ZVM zYuix9P~UIHo%tVQ{$4kIa76W~>EzA@C)3BHRN^;~7%BjAgtR0HkhpAW2`TJU zIOUJTq3?DM0_Wiz`R!hv!`1B8fFA^{vJY)P`)>VZ^ku8}hJ`Oyod|Xo*pHJF6NRm4 z#2n&v+|Z|G=_GLOk_%MO~ADjM$)al@F+wou`;7o{Hg;tJSSRI$Re(`A8dhT&*>lA7ny0=kX`f))fAx)1$&k|a#6z712 zII7xyb>q1L+Pe30v0V4_{Di#NeHTOx$%6`q0h0VHoj}fU7Qw$ODrx10_@$RfAI&)? z_J=8(=6rx@T}=N`Tx2e=TioUunV=*Nz~O>E)4NQTjga3HJ@amLD%%V0>tXBiJBS=* zXHw%gTy(D6X31R~4|_|q^s43S=)h`%%lJ5#T@z6UMWvKj<3f+dj(CtcQzkJv2kPK9 z@rIOSt0_|_x}V)(o6aZpz~l5H(oaF{Tmc-vX5YyzEo-9_m|)=XTMuF!+DP*K+#WSA zly<}E&Kzo+zuA)O_I)?d*NuvP99r$0wb*l>D81lcdNIlI4-JDRI){tl+yhAmHz)8~ zo?PKAzRzsp=(3%hHrqIF7V)mjRo4!&|5-*5Io9;*m;f|g!rL;XJ1=ZE=7+%A1AqWz za1@+B)0ZOZke+K@o1P4*1o-g<^ZVdk&N?;#ay~H{{Aleqz0B|LFO4_OnLE%OM=Aj` zfTRj9R}w@`KqQMv9=K3n^${dftCuXYA1<Ot1#L`v%1US9VtffOqmlB?xzYdr%s zzdwwv*#SOpEoGGp)G+8%{x~jk*m1$sEfjKnOjlQ3XZA2;%0IL@?7uT}r&z^Uyiy4s zm1R_j?;fsn>XW5yD${Fb{=ebf?I}foN*|%M))7o5bjql5A7FJ&mh8E%lQwrecq^sn7 zW8pp5#hcJw(F`&fjOMQ=X@uFH@Fhn>48z>S1o@LLNt5=azU~KQyrFq@^*X zf=Ka}to2gWOL;;BYHXtjE_bNRtqrXl7sk}~x&Yo0h*;eFJUC`Hd>CPl4D@Z|LC9PO+n zbKn~!$z4;=Bi$?sKy)D^J5TlV?E5y9xQwECK`j|i9jwD3&Cua%RZF9X)}03?z)0JsK83?=+ccq@Ge3TD z;J5e1)9v9&EA+%w$IO?9$U_f>iSEi%q-%Ts{xYrG&b&3SCIdL9*L7dpkl)2zs~>-i z$UF^^baV`H`(t-{iU=0-pVHekn5wS#tewH0n`TaZV(}Ey6XiZqQ#6DLirS_Lt=ZX1 zxI4(I)0!bqLe@D=uiFpB=J4=q#bR>@nJY_NA#+aK>oV3XIkIK^CrM6oe3@4f?3(fH`zv_%+rGM;R$aJ#Xx zvWOEE6>k7<9&uUwosDyJO=#CW@%Sb_>9rm`aWu}I6$KKyipptRja0ArRv(|V$jei| zP2OI3EZxb~o`9B28%D;yTXUz&sEsS-ur6iTOjVS9Yap8n0vX9D#kp8|VK(-??JF@; z4NnwG#1aRC2f9JDJGV}v4Zc~1ycv(Q#?aBKI~6b+88J_@zJU5iI@XKdIG1T^ z8R)HVHch%_Pxk4Vo>5h3}4oweCCEc^n4?#ddfXR5W+(vIQ zJcMMn*L#kP<_aRy?K82G^OSrTn6qulkl=7Q4dDtHnt!c!gUZG75N@x#m2S0Aj&)Rd z5;1X$&_9YYIPtBem%V8!8n9atfLZx3|9*CsSZU7F9>AX`}8I4WGF*tT6kMHO|ZHe_tuFwMF_Fqm1ya^x2rVD`^-xG%6ZV&qm zE-e&bZULMC_>1vy5?^D0**DZl<$ut%HGI+q4NEju>X!~S0ouy>x6nX_{jw?HzlZRh z!aRsdc-J=L{^>NJ{dfJ#KIlwls{dg>;`W~`WZkKH^*N}~KST~xB8qqv^1q?|W zA>f>5&wkW6qZpf(qoWYt3T(DR-Iz8|yUd9Y57P63?p+wj;h zoQnR}hx8&&t_BM(p+Oo6?c`-yJ;%~&;}>${v#wy&c8>f0*`kqV18yiTQx^_9^^*QN z!TKtOIUS-G(h^@;S)OXb{?jtb7QFH6Mc1L6oakbKW1}nA47*Vs1(NKBJAMr!rCz(|wRMj)}drdO*a}Z?DSsxTwC1##s~@{7esP@5N@s zn(N8bnyZvqnpoA9@(Qku%v(ml)N`T!wAA-SCM2UYj+;jY7K|Es;)YTe57tEXrT&zK z#3j!aa7kCxB)~TB+td2##|8sWz#z-=hfY(fOOFUo0CL)!bD(c0Qa%YzqEk71^;R1NP}I+q(^*Y^@2pxLYmt2OWH{^x8?KfDUv z%wzP772Jt@TT;t8B|)}37T5VJR8o*>{_B82slAO=&urY+CoVdhXZkl({8Kjbi!Y7Z zmqtczvYq2}P*Yy_V>qEYHzBh(j1mzUBUM4g=sW#&23L^LZMxMub zlt!0X^jX=U{uIn46)ef}i0y~jbv8CZu2T|bg*Pv^j{RIPeENys@6#XU;XVi*XRx;J z`)5YHyvqY>iGKe^*v`@=Pmc|Y_m3bj{m>#3=jYm>+Xp9jB%!$ye{cJGcT9qLoS2BD zpa|ExCs`&%O5u&89y;B_m3WsEl{-gjr`&R^`A3Aekfx|YiIc!%#OW9R+K7i<*BLK$ zFe(QitJBpN7(}g4&|)uzhdg}{{e!#8W}*C+Z*3;2J`d^DZ?3r2a=ob-EnA55{UxF# zTiW`LjJ{Irdty@QfLDKHc5ej9QQ93{A}f~K=6S^W8>TT=yY-#KsJGCL?Mk(%=3IA= zfOk#q!%`c&){&(Jt?ww3Q|~|oURsa;);TUTxjkdT>%n8VeGRdy0P+dzz1`gyV3D#^ zWwvnwfL<4%22@FPn!Z^StK~j=va`dkSkJ0bKSslVO01|X>{LK&`pC%;S;-nv*|05pDBT`?X58vK3pE?;mP4(_w)jQAQ3G3i`-DlorJ%r- zH#+Q=I}^6cVjZFrbGKwl~a6F*s_{m9v_#8!-kDug z$0S(E9N{plbyB$DFl+$m4M6eY^qt*H_HY$r!+Xgdia+syjJR5;(*#i#nP5(&r+xTI zJwUfF)S>A+>Q8R2J-G>3Dr+zF;Wf4JNlZa5CTXJf?~w-5rcnwiSXrP`1{Y;IFvx~_ zSFs!SE^_3{KqRj@^1iH%rn54DX9`vJAF6e#CQN*o>XYol4(q2HTx$UcuKImhJw*{# z8DMc+Nc`0W+yYf)7lmQ77j-w{>wp0BFQ5)}Z3FQzU_H_Q!MIh5I~Wd;b0D=N0>%z( zu*5uPyCIZ58&*nFujr;?n)@$~Q`1Kq3rs68GBM?5<&fcz{Na}K1BO;Yd>1h2IX_+nA<{Lq zudVd=c~ME)t?yeh{xa16{Y1X8O$KzqXn>(1(n-jC@LI0dJo}%&wKUDw=e2=3wVhk| zavfmZM-u_5KHAFTuh7AWvo2wjgyZ?Z7$KeP-{Pi=w=!n1{*LFCX!?)Gb~jI6o%J{? zij=nSLubQ5Bq418C16E^E(^*bfvXp$6zdIV4;AFA!jKSrk#`CSbQY3Gq5C({Jm#B) zd6Rb4XKu=Le@!hjVh?Tsfql=9+iFqi`Qp%w6~FSDnj!8RIE&_!kCuX(GQ+h}kqvv% z@%Bp!U_BK$sXh^!EJf>7%{En2S2Q{LaBKTe#!*4jT>+>LdWkt4syp^}DrU#!k(yxa zc3WfaM8IY{^fvu+f2#$)0fza&1(Kf{JsVDVJPnX^rvsW z|1J0|&-KO0g)ana8_xa%919SiFHo=9z%rH?b9Y54KO;l;g=7)T#SsWmjxVp|M}JTH zHcgYfgWhGfJ^nrF$Hl{Rm&L4=nJ8UCO75VF6**QCnrkMvT7QJy%)X3>VoYi*F`O>3 zk9eQ(QLWVHGmF{2Q@GfZ*RT0O+}E+RZ8w|#sT2bFXg>#v)r2#GUw=4U_sim%q?gN+ zkwtP_aEL!GbvR-TH|^#uzO_#Rx#QiX#t3axhj+k!EnuyPW~~{3lFf^I1ZO}+?_@SG*|*AX^(#b)%M%8bufJZ^F?Vf?;T122kBp5$J=t| zIuwmZJ>{c`rX^f8&{DYDdh$CfA);=KCzyaOJ)@Afj?3tu7(;uIpsMzSR706gY`YTX zu6WncN@u^}N-gt5MWzBhS9^sqc-}Jwr~qD-a9Hg|9d0FAdUA9<`8=C9!V~q1RFYWE zWqgI-9^w3`C~fE2aBhzl2`~sv%F3#$$DrwlGVcr;)Lw(XncqS1E7KhxYTM8He)P92 z9sL5-_B@oQ+1n@hf$tasJn<5kW!=7agen3;;RzwPVqOl~Ta@P2Z#Yi(;AhOIh1`wx z?|lqRrudV;HNL=6XJT&zX;J;XdiB5aexd>h)DYR)sOc-m-jm1%^h!?ybanCb|I%$p zZTxs_Q)FfQu~)p?>ED6`)Na8)^50v{TNm4E&5s#mgj5*bILV_8F%y*Z`*#H;a5E}G z{RJq(chMS#N#|?LSML3JPN3xNZcr^v3=}q6|6*J`RaDt0Qd1iI<~MFns45sX4@NG< z`wLCD)>)b^xhj@J1dFcxkTJ#)sW!T4vCDue(pkk96&(NFk*`amm;Ov392<9p8h+=a#))IPnqMJ;G1M12;F4 z=)hh_om$RK889?sb0le7c}zQXIsks-`{g{k~zGsUE&#aY~JAj zE5zCrJ(TESEBO>^X7ej1_KreosqszBYlnH$P!9ZLA5n)^`-$YN{KKia|E_S6F!jhI zew?+yKa;v`f|QsQwEzPI7LYLk-o3`TiXZ-=1=u)i#nOyLokdyk)p?7?(&oae5?mNT z_-E_k#L2Y$mu&aEQGPC9bBZzDFV^w2w9XJi`L6O#>`@_h&z?_|1L9rZW=4$>-$5Pm z!A#Y+Ry#7xYLUXr@3?`40z?N?t-Fa|)+0VWx+byd`vtimU)4MJrzeRL0<%wvPh8@) zTpcp)(5eycH47y@UTV(K_zPbTYI=F^#J%9)Z+G1P+VF#`_np`B+zq#O&9RbtxUs9Q zh1CGVSmcGA_jnAUw3kTpCb$vkvLqaUdHUFCH*BQ0CJD zK*{@m?>XQp2IcA4E@P*?_NF35-ikl2e$EKn$QIX&nBRqoBSl@TeFXK6E8-7YBI$==d3000vZ z2y$`ppy45KC4rX0=$R|P0JRNX`CU+n|E(9QXa9YlJN&X{@&D;9#e60}4CfHK622Qb z_Xs-grod1@vX@TiQ}lm(HSYFh2)KB`Am>X04Fdho-KT4*nf~+LT#F2#!x%si@fpoh zKBy2%PI|4-*ofg=-gi5ziv`&j0o48(=b#sp?OD@)lS>{m$jeFh*?twxeKEo;s;eLy zZJ>d+oFBcb((X}DYMe9V?TsAClo2XzY&-Qm5!kPkhT4|5B@)oq~SYnyHs>A7#7gs&289$ISk-= zYoDIW2cSK8ya$eQ*qrp#{InwkQcl`&2%unRjHk^TxwD-Sl~f{Qm>sMBF*&&{xpg+z zZ*Sk@)0=ovw_}{>u*99Z?bgim9t9CGj)lc-tB2j+)DI>9r3E}F$d(@QIy1jdxac=h zcP2r_n!y#Pm#m*sWYaf@bItbNAKTMFw@=xzL6?S|JrZ9e)Vt+op6!(Pa_?83&BjB` zYs~6hH2Vq-Ht9lpKYW@r3GLxrtMOLCYdq_JNDLY#>Xw=#iXU*?NfcV?;KOP=cqB6p z)7fTWG%DkDjyy7@Z?RUUKF99WX{Ri$kN;^L<4}8RmTcyozxGP0JxH~bqOQswX18-J zs=u$;RhyufOLH?>oli}@~;`oSa;})s2>$SnWU9N`0Hk10S(vL;epBoMa%~gJXQ;Kv<+EO*xwsS-- ze7E;5Akctyz^YG{gP%hNHk+yY_txs)!{0I%73Lao1wJ8c>)Fo@pF&bc(icaWM#Epf z4jtlcZ^Iw{?7Wu_Zk9S;3tu%3yn7TzC*lI1N{HTspL+a2tLc`ria8pvd2M9vj^=O- zgh#FJl+!TK?;RxWKW^1@k7T<3xm-w4e`tMXcJ)iGy}9LnVdl@31Y#D^0_({&ZMyZ% z!3xc#KvD&D!d_a-8g}yx-S(nA+usG;z511!)}Sh?Gcd@AuJ6j%*roi|1tc1pQ^+XS zz@6wn!IgWmoPeRXT(y%z?zJU!S2xuf9dw_58fGCJF+hmhC#9NSUkh&?~6hG1XoESm%cmC_D1zq|Ks zo;oOy^wji{9`h3!Sc^u2-=Dtd>Z^lhON{e8-@QfZ5jLYi9WKitoJ_aACW;=YmCPDH z@Eq?wS@%qK0`6i^j?2CD&}nTLkKZZH$Vu)hi>KTQBNa-HSIo<0?+O|jUjHtLZrzkD zDX=lmPTtLOQI*8qqm*M z21la|qc`oOrt%iDvqaX_GKMM&4qoT*d%de`o;*CC4vMegK5kLWf=_->j)YI=Lln77 zxhVP;p~)hV$ZI z)4ZV2s}Du0f2OH9ykuXEhy8|5Is9OR?r2D7+ms(mI|gJN$xg(3ApgW#mF0&NhX^w1 zJaGkQvu4Yjc(?`DInFewxo<5`)NAS0sXm>b);K9?gB+?Oc3;fiN2D`pYqmFjcZV{M9|t>dQXj8t-yMCY@x>T$It#5xZ3C%^5~tnv;N!0+P%u4DT+ zKYY78c;qj(Tfs`uhTlVl_$CR<-1urV<9c&|QN8<~X`V#)X+kH&V3i^6)aD?-vdn$s z-4AzTr(vO!u-+-EA}du1vwPaWxUHmkr zB!@m9n^zYr1s^3Es0~z#-7Pdo(nxtv2|a1VjI|#2N-cY)!P%v{$+;_F{m!$KJm!9= zNDeZq)0k}a;aM4^7eX96v%H9DC@ty}oftgN%L@&U!qgQDD}OFe*|_5`<#wdA6cf#* zdf1bEI98H!KY>ZbZ?Tea`b+3@hCbA>85e_lJ4w{!2Icl~FSgB{3qR0z6lQ>R825 zD8xX}OhmY7A<^wxYh*dUpias6g|)BvDJ+x?TLn?#fxnzuYc1Q$Zk%BnCIrQu&a7}e zJvH_jQYK(x-x+nTq+fe}iO!;589)+%q0s1Ie0_wgtf{)67uPC-&4-%z#AN0|GO`^^ z%9c9+WPkNzN`c>!imaC~+jvr(VKY`UM{S;KeBE2pqaX1I=BZYdCEAZdd78+rI1M*8 z#>rw6aU-#!wJaL$-WTmso$PQBgaiMW(9XQir|O}f+RIqM@dTc;4s zC)nP{^d&FqOlLuvp4QZuLUhKG4PU|0v%IUV0{oNvYpetO$>XyfoNhu>Y3{w}dt|n* z3)llY)mn#@c8trlfzrWfL8;dt+1`9UxJ6iCX{o37#L0CNPQx;;>q*>;M3PfUEeSwT zp6li*x}_EIBl$n9t|MY*&?=o7efUy%D3zEmkc;ojIL#GT&w5I^y){QU>F9ay zR!eQOndR1|3VRR)=A_}}OMQ5S=i|Hr5D$q;&-+VRV(sQK8Jm6+GZ2x$eRM56>2;`;V$xdbNd@yMeZ+{p0vVa@RGB5uX6RlN z@}()<%BxPBD5`a{&NE@LR`d~O)W9e_{gH$7GXPt(il)acj$kM_PRs>yZjmZbt45nNJLKvbIa-USvay+rAu1*uYllu$xZ z5fSMDLAoGCdI?3kf^97*da?J2)6VB77B-T)9#78BQ40Wv zh?~9V$$^H$Ig0RRlNKzotKZRP54 z59i?B(brZkTXgoOPk6l#uKU*Of&1z6MNv$1{=eSj7ZYApCnpN)uxMz-eBlnv!$t~* zSnL|e9Uo=muH~DuYVf|BY6g;!x=iDyWhK7^R_q+p>nd0Y3{ zoD~M7KXxK^N3>KXP$Q@_JM#*`R~7+U8AM60UEcCTIk%syEj# z8I?{IyrbfLXL@;7Vs0u>JgLEN2)omp(9t?dRBfxOaVgv|=x7C>5vz*$PTF*+F@ z@BG0568FSLyu7B*e&MEed(lh^-M5rBN{d`H6q72wS#22K-18oiiIa(k z^QvWBx0z8>4R36(OOz?!YK>K|YK6u}cW2kYa{fX$3?wm=#cF&!nEXbiMGCDVBPGAi zT&H9L{u<_VTM4r%FxF;SXIZA7iH?^2WNB2(H&l(0J$d=~G1A1;IPWBIQ2#;agh*6g z>^mKFL%yh6iqhp^h;AGxAap`SF15PiZlIg0Zag=xhv7cUkT5sv!$Hu#glvLSI| zphfpv@cuQJwN!EedinF_)Z1U_d)jruN@A*u%&exE1%smq{g4x@M?LS`p{0rjOz2T(-Rzs6AQ1UvsRZQi2hRS}RWoqZ zRz}_NE#a)X3R;!-tHDSffxR7iM}(HOVE&Vv^Y+o;#p>ps%~U5qs&e(owb5(w+}w#< z9rdnPCKSDAK-AsjMGsvl#M?-uv+%07R&MtsX$)m%Xp=mMuqb;m9zIHv0*P)z(K|~Y z6!2(A5kK}r9w{t*zQLq6aCnxNBYo)eW2^+?DXrP{^Dx65HrDRM`mPyFA}^8)!`jYH zrib3WDWc5BTy~1f_J%+*I4*n$y(o*x;;p(LFHm00tT+K-VSxZx$*;C?g5a}eQK-&x zy$@_Ts10i2yn_6I#vJ&y1>Ki)E#sq3w%2K{KQCNU5V79eXc1s-^S|5uFf^YO7Ioma z)n-u#kMMq(=kOOg!LFsmnxoao&a4dlg|1pFfsTq_#MAk+xBn(Ga*-f1w7PbjQqVm_ z-gH}fUtNDd>_Z5_4+_|_;Z7@g(pxp0sqLr+$!-t&Sm|s+!OnGv4dTi06cO1b3t;0t};=cWfuKw1}Vo=n4>hn1M zT(Hr9VF9{!564JjgC+9$ALYABcHA@Xw~;5HtV&HN3GB$Ps@(wt?hAVze5IwOfg2?J zQo#@Vp@I3VJiT+Y(w!PsCT2NCJrHlc$Iz_A*!=KO`gm!fWc}cAjg<_pXEvx~YV0LX zjy*)ReT~dlZ_b!E4rOLmX_1+W;c6M+dZx>OPrPjBMUVt3dc#;8p}rpV&a6Xq(Mk!) ziH`F%pOu_zD}2}Hcw{easvXw~mzN4x??GIko`R${q^q~hw|%D+lv;Q|dbZh|P(Z~V z^pVLvwjII2>)T~?)-IVY4V9{F`6jCrM9C*NGNw}HtT6&Z=7k4kqfk1aK7$hjeO!WF zn1SPD@&vn|F#krwBIIde&=S6i$U)sFG4Q%qQ+6qsn&8C0sKa%zT@fpN7!Zbs44m=a z69dD-$tqITCj5 zEz1B1EIhzOXX85pI3&v4=5P4wm@PLEkf*}UBTqlR1GB~{Nw^3VMG-`gKsA|nuIY^PN%M!oAD4VzabcMJ;l;$+x~Pl}4MIJ15&#Qlg=Lvz@U zZpS_iJh?<_{`qlJ+>r>=xDjs1Z0T}Sl>MAZvdYVY6PkK80BnVK_Rna44{n}v#RSWZ zR+!^I6v3al^K>QG4>!wi);+a4ISH}W**mMH{^X*f&*H{35am5M{B)wG{UIq$Z;|S+HUQtN{G*qU zn}*I!hrI1d<9llBAS_y6OAzO>CIxeuvZ#(YJ5wY z#YB|u(jn;ZT12U+Igf8%S?CXAWbjTmmG)|Lnm~T&aG8hV)3v@y-q@pe)g@kE!1tvl zM>tYrX6oo471;M-EmjUm&EzZKtI-)-NAlXmQ{dAWi2`QRJA>Ir*1J$fOrIBU2s<}* z8yU2@Z%F8oh^aNK$1_v?&hPDeXRR``*fV_s3UJ%Z{B05zZ1OO4q6t2X(=qi;7WPXw z8};&zud(aRndEYQ;TOq!os?53ztNT5LD$%G<8n8(EBTRxkz1i+8JnHAX(4%vW?c7E z7y>c8x@x{7_xi{sJ9Egvp6g0y;zuvnjP!%60A9ETkm8O3(O#rpr;>IKtX#?MQM@^r zm{aAM`aoG89`qGCwPk#DtHeX(V-0WAdv#u1sy0lX*W7%GRwA#e9B;=Dr+0{~Dxh^a z&D!(?ORsM|d|4p+$!zbD^Ruq9Z+9||C}T|eUGf!sw}obxEKdiQOoI_Echmf(aSNj# zNpz;5g>~ldK1QIN8}cJ^b^a_e%(5|n z)ucRHe$#HIy|`F<#rJE=;vuu~Pn|`Ziy$I-o#98q#w-&?pyIJFJjd?B9|{pW|Eslg z>ZLmBK~W5g`3h6Iu?#;v%Ys_)bT;d@utId5U0-z*`j&k0ce6GPZBac7P#aypV`mgg z`4`tfXDqFINABlI2WDhq7^B_P>I%07EH(=R{FOxOy!I z5Z->Y-6Mp0S_{g5y9jrZL}sVqaIK92PRY{uGFGT!?~s<&S7k^pu}!pr=4_-jT1c^= zH}>11)>==7ydTsTT;N>17WPjlckc(ljhf{MkXclol6TXdzrn$!5I_jy3f@=Do~b?! zy8^f+gq-)w)b|V^H1vISm0Rk);CtKq-lpD!i$qhxHVR2aMb+vrz~$@CdRJSsFGOan zRrTFd?NMmANYJRtZj-`nDC}qD@$s?3ot(aQz>Hy^)o`^i(IY6NVZQ@( zW>N1{{_KwSbVk;(Zklmco5BKlfS{YhbtXB<+k{rQJY>|icNl2)aTyx&X1-vol z`1k~Y8e%)mXBEh@fMkAv4t*$$%{V)W%4Fu5;l1dLQ@&I6F+|pME1<1v25-MQru8EM zWFC>-gUxF9w4o?@RpL%jxz8Vq2#aM9w!CzkS@C{J!(pbkD{&8q(Mecb^C!3A%^Tp+|?$j|DWt zpX9Ob^!mSo=UaZHHTMrid!+Meh>#zpg*B=%-Khurhg;dL;4v&tq117fPA2h@e= zxa=1$Hg#K*|`SVKHansaB5it?Z7`n!`)B-7s;Rn>rx8$v=? zTi@(8G|i8NU{g$JzO5l-lMxLIHhH(%*-g(4yq-B0^fUa)7-pxf75Vt!`q8_s za%z}){^!C%*##l%4sf-{s%AKz+g+j;F=C5pZ4-DYaZ_|JJ})1l z@GH4Q&0oF-E+2)!O)f~U;3ZS(bUQ1MX{p=?LU|Bil%)X6 zvqC)(Ldk}A*ncs%ZgHhE?fcN2$1}2IAf%on@}ht62^Zt)xn0U0NUC)nU1XG^z^%vO zyp>sH#3N}c^-AMCkiYyBer2Hsa%}s21_=Y>2|+$G3#VAxTOVo;ewC#8Erb~+ z_SKb@1i^&BZq6Wo#ioe4mN_RSEfPdTn>D*mh8~bweUr7aRKm;-O9veFt-OwW)k_#( zt!3kS5pS4b!77@iJmyaxLyfe542aUpItkb^^1@0MnS|5!K`h!=jhxyG-BC^!5x=a4 z+-T^toC(tc$4xR76;q=&)!go{I4eSH_W9-@pR%hhqWiJjuSX2aIK~jZTVnwY;M4+8 zlR9unre={FP<454Yqu$1JVh{GKHn+ZEfRfHFkTq4<)L`|oU}LXquKK->PE%%ZlBZz ztma6!t(pYy1eGnKyC`h01AZO0DH+X6;6mw)i8tAG!BTbkrBWBlS0vgqNKq?lWJkCl>%G_ZZz5(wxB>k4Ur=miilID~YWk7zi^bf1>sY!3w zCCccWf-I`*N)K|hKlpZ+I~}t)@Q9A>T@t zXM_)8z{lO^$f4~y(qUrs;3+QyVywO^Cg|)XW)CW>;$AD1Z?S{!%<4r^7ZbF(VmM)h z=mBkTKqm<03)A%o>1yagitv3J%kI^cX7k{7`@p6&oQ+#_VdqXbIZy&uauo=q=bFzrGEyk6?MHV2H4>Ri=P_d>4>|o+XZ?cc6 zbCq|d@66oiYl|s9PA=89;_15~O47dThZr1;Zpg9?eG!G+W!o5KPD9B3Lv_jIuoe&s;&S0RZ9q22^6hM8UW_YkQz9&8aD@h!=}-dh z9GvbH!{MH`-VT9AdQEGIDj6b;@N#I0NX-egio&SWJ@yT?&Zem9Nw)%aEg5WvFoEcv zu#uu>vph>F?F(|ZH^|W0InVt{u4M=ic~G&V@*Lq-t4Mk0o)&}hapzOjJf8+PWMKZv zVN%QH3Yy*fB;|Q&f7z*YQ#j+N3Y!{7xny(1oH2ZOwbkF@AWbXnjvLOCv}`|PEBsYJ zv`Qy!>hB0m>1?#y0+uwHtjLt}J8>SnHr|$2|Dn^g7xA%PMaQtao@i+B@pFCt-|rVV#GM5_K;ClK8jWvKqzs($1xE?^5N71usK=Y=aCuP0=g{smpQbBz6S7M z7}kR>qC@XV9ORT0VNwkDJ6d4!>pO0nek3=_`_-Q2s;D%fKq0WisBnU-SMhP9IL6VO z_~#oefie4i%St}IzU?1Dj6ScaC^Fyc%Ozze?`?tfvp8tYw_*K^R5#Mxpj#P3Wg&~s ziA>NF)RT*keAU?~*os_T4=wgb6l-C)1G2gR+4;`Zkk_?~Fd*E;)|!*J=7cW}5l!@n ztJ0Qm@9pECCj6Y(B}^{Hciu=Ha1QZsv(fJDubn@uB5A0|1rDY>GdDF{wq$qNIMoXr zWC`KA^Cr*CP$uH>O3ZvbO;It_Zm55n>84xA@hofgMj)hnk51!V zFdd`%n_k=4jHSV$YkqW5+h;o#pL_%gZY}54u$$K9(#KJu>bQ}%R~tnS_1L%{H>EzC6H-WgVmAkKbHSl1`t(D#siL!SR|B~s|nhH8nWJ9)1pO-xBa3~w}owD1rW zVPU@r=$I+`egvXErJ{alV4mz-hCP0_;1t|q3gxyy`NjNUA{u&vq;VzEQqtIUV_n=h zGjHEYV*t6=BI$3viukS7nG@%JPy_}mZi8wxxP^s_yK zb6y>a5UpGUSn0XmrcfCmOp`C1V6I0Qe|PT0_FH}fRM#xS z0$PX}*4)}qZm4XW_Bc5+&CK>VhmfDCB^*T!zMd~|;z2S1@A?FI*Y{VpCWYIG6Vxg@ zo9sk)*v9TdQlhq(L1hJUKE73QZQ~HtHS;66UL^xvEg7s=n3*tOA&K|aY&+#ra`u(; zTb=HBIq>J+VN9pfs+-n7{1rkcrJaQzz#Vgr3+Jp25vu2hzli03B&?$QY}>Cp{K!ZrQx5^xXs@T}mI`iROa80=-G$+~PaQkV#d=KA4>m-VPr!Z_qy?ws zH@IM@i0A8;{o{3^dz)ErGJKjot>2bt%&J1Xx$`d1%x^mo27Q(+eQ?LGAD%Uz?!tuL z2Cg_+p4-BqT|$>y7nS`$Am~N3Jboi@B4mm!tHuM9RlgHi;xQC-z*2G_8WM=12kr*g zAzcd&{Rp3ZpZTCblg*eCR7{)H!(%uldZBl4K2MyN{ky4c4`5uOW3Tbwv7N5LEKc#& zsWtV$$EOj4oQf@#ZP?xFf7s9gOah0pl7q<<^2obg@LJSLf_#5%LsgEvPOM&n)KN7i z7_~f`Ft|P(ao-|JK!91pzS#ZZgl}i8?wpy>V+QYhX0-8O6yUK3jjvW2JKgLoPHgY4 z^m8^Yv@hgp{BgWrwHl6Yq47EioUNtI;Wm@)B+2j;tZXaVq(=tn1}GJ~>DZ-mQwT18 z`GaE50U=*epuNHAQ-Yx%Z>X+~wENO-yz=2n^yk8srQ-Z=3Yc7Ci>XFAl~C%8{-8va zqZBms!d`nZ=Y^PIM5AKZ#)M*G_;Gs`V_4syJN5@)uy*!WZ0h_7b-XU&nqgiJkxJoV z^4x?o`o)rBrk#1Nu{b|JLe$>J{R6AHP2SXRdy}%2ChT&t(3ClbEf!?~OHQ77%ZrKX ztLJpzWUu_lYCp|2_2jlfXH+e~^Ah|8{AthYeF!GlcS$Uh@OApqfE*#ZscU!*Na+Mi}ijzu(5G#ZMDqeR0ywHx#S=m@(79 z3yYltl&Jx9^b+zF)x4s!0HZlAO?~v(7}qDP1AVb!jPKL4b86~FT)pn37K;icK7xp~ zaLcAbA7uUmHB88tH`dE;9 zmiK-_4QUTDa%ed-uCtIswCZO5Y~>xDJ)u2vHpM1OuQ$-y%>8;xNwN2l8@bgq?-y)( zrF%%0ex^_2ZT*ZO>#pEO>bTZ9*V<0f+py8wlNpHGwWlU|^g#zn-NsdXP0jYpEra@T z?xCYbBqIOTZpzsG$L5!@`{2$Sw^-SmXJnQ)ipxvBX4VBH` z1%j;RJml*|_$2hr81LzoMj7#gB}y*7ZypUqa;MK#i%mYfF)*FUdaLqA+Vaa8^rD?Bc z%N(;JaoGW+x83HJX7Nm$iZ8XxzCNhNzouNjeI5dENTckB{P?rT${@|^JY49}$40B@ zijLEJGqi%6$L43shChB3|7JfuTMT?oWb$Z23a2Sd&lS20Vp4^ik(W7TGMm}|{B~VS zMy=8R#r?8xHBu5YX3G;>H0st$i%sJp;;l90P~u3=oI&^eIWZljgpD`TQD39YS8EU6 z?g3$gDlpj%P23Iz_{03qZU)OgfrubMSW1F<_zH#9QXVh^jxt7=dZ20BP{BRkV>xtn&Kof8<>3cYGMZ5MW)OJ-BtTXhH@za@){FDTnX(i5y} z0}PzLy?#dr?falV?}V{4X>*E20j@OJLO+9JyJb}=_61{_xTD4_3YO4Kt3Ni|ik_$* zKCw<@I4I(8y1s&*z)C`WqfeH{b6Jl+mMv}?B6`7}Slv^F_f604SADJSN)6>7<@o;N zt=TJVr?7|!P#Nmgbgfi4=5?R$jI8J3^m}Ao<9MqWU{N9V(GcQr3;}8Cq-7tbk8bm} z;~vwTQkknL$UmQhG+_Xj#_%Ui9eFV6I~&p;Cz4q5t^~$r?=zbkJC%(nX!*U^TC@<*p*^0}!Ze2r?v_+c z^A;4IY+N#X^xMW$;Rdu9o#ig64WR4N>!5k7h4)EU!j@Rk4}0_Bqa8E>dBi6j3#H#p z5pTKqz!?p2dd~8`rpdW!MaFA`r!hq&NN_{>@buW;(ic{>Gz5d;!%z;M5x@SEbs<@MZH?k=U$xjU<|_}u9vG2R~2$qqe! zK3MW#KU>+`M@}Ba+g09aVGvzyTdTumH^owoyn5Zi9u&w@S_+gMvI6WzfYv$mINar> zxtZIZ=4C03W|{?m9p^_KHqT4N8wY%qV+d!9tI;v8rT!$jA$Y~9#n4;T?qGD<8MUPx zaB3jdyC^VNIC~Y?0heefDAr@`dmrw(6ac$E;J}6t*9Vf{a{x(d^Gwrc%qSZq$?EnY zt~7Sp@Z2DGqWwVrI37i~hTo~+ZYnvgbypsWetLzX)FS8;AYE%g3D9<)B`t+n_opj>pV4|&hNaQ7 zfV+GKLx*vpqX08duui8{@bVA$+6v=|Hi7}R7W)Nyz6>CR6cldumO!XN(a4v&eQN|9 zB|ZOFL^lAn7rhPYFeGo${P>Ojiop!+`biLd^+ zt1dI4D!BodJX}U2?3z@Kar^ z|3OiydTZ>1_s1_g!YQuBNL^JB6!W$rPsA9!Uqvo)UkeF7M<>QW0mNK8V?F4%P9J!_ z72wi9QJHhXr+-I91s1Ft6_QUhzKG0@?%h{4ZV}14c8By5?$cynlRVxWUw6nPyr4X{ zJIRnZiY#%p+}!9&wh4UpPKyBvG^O;Vn4zTVL~s!rkW7%gBf)Q(6(TW&d1 z0kgCf7$d&F$A~*^^OoeQNn3o8=Dm`RX{>L-NTUB%HlFe#soQDB3R24l4N`yhs`>Zv`~OVr`n!++(+|G`qJK+2EM8ovpt!jB&f@kOXprjn@o<^q Y=Te4xL9UhW?`LSK=|3rdWb@{K07rISMgRZ+ literal 0 HcmV?d00001 diff --git a/docs/assets/dashboard-overview.png b/docs/assets/dashboard-overview.png new file mode 100644 index 0000000000000000000000000000000000000000..6a6c29573d815e63b78522b3ab6f0bad2bb6d6e4 GIT binary patch literal 19060 zcmeIaXIN8fw>BCJ;t~WKpwe6-ARtAN-gGH~AfTdj2uiO}T4*6EOI?7#0tBQhReC3s zKtP(bL_m5-sDV%tNRhZ$_fb|ZO#;*ijuYZ;63)-!?CIQf&D|%lNm=hai5*N4shp489a8V$x8mey}gmY z5%xi9UZa^j8DU4l&P>g|t>f9aUL})aD~LQwmwor};YTa?n7`E$-zYn!rhVTQ_4xbL zrPuosM?-aU2#v}Z?Ok`*TOu$jQwJrz9?7qJ92lHq9rFcn0R4XH)gch*IM*Rw5a`y` zAL0OLM!_&f;o>fxz&;MAXQK*vC!OXrR%0e^h?FZjy$UnX?Co2A1lE>k|)y%R8R|4}SewoS`$K-qx#WucN zjGz^BI6ID?G`RmUfIf8Pr$-+>y;lv7^^NP8RpPSKYj>8j@D7i|!&&K@(k+;6lWr}| z7@-KKnQ^jjnCrN$Z8mJ)aYf(29b$z#D4xDM=05}9-%+l$L1@?KHppDfQYor(fxv8U zwBD(yWj}P=&LqB7%ZrH2)~{0aK8T6s^n$w_&^(SlJ8eh3DrnpUhizP|^x!W8~e0%yMkgOZ0w@z&A_xoWdM7uY&6W_VSVL z&#cF4-s%sv(2&Xf@YRGw<-n8W#0rwT`sAaVh|4Qd)Jpv-V2&Eah=*`@PYuNDpD7ph zapNlwTgoY4hkec88hM&6Q613TV=knpcY28+)O&dadUY`B%GoIArMBp)0>w@3VAs7n z()e}QjGz*lp(g%n7vZ{k;7$t1~WZMiT89RkdkA6y^ zK&GsGO@ol-8e-T9nClB!+;m#BNuLQGt&jt@VYW8EvvM+iw!Dy`!qI4a8snID_#_CV z6y0UZ=R4I%cOQPkW&ZrO^Ydj2Hm_cBLl-9&@P0c(It)E$LLNiHEooE8sH7Ie&!7j^ z53PTS*3O>BD&KAn2-44gwwuB#@4FP?$BCIqKOyHP03|1wt5#~aIgJvM=>Bp$kr~pp z$HzJ3Z4P-B+WY32#IKc03hr~xjabYX`eZ?GzXRW+ROh-VMl2d9HT$btG^{q&?>Yqv z3l)l|GpW;zhC$xd4XZjfkBtJ%?zYUrpMMQ8Y#>L6{&rATj37iEtMKL!==|^EjJZhu!;Zm$V%REQ zf7ptNDaw$Ib&(S<{ma1x;e1)_uG5s|v}8m86VXsNXkWkZ<(uRAeBAW*Mjl4gkb;#7 ze$n0IMlg%!>a1v;=T;QOlxEsnr{uskF>&Ivs9pEVifR|hs~?^|8OjRi#%N6Kt);}y zM6Rg}06-4{VGC;xJMK$rrN^lT9aFQ;^**juFFxJ)f;x|lwjPc4_cL}I%1~Qg_`|q< z+{dOm)Xhout8Eg>1vUgz(=nOhhs=hyMU0}64co&mJ{b;n+M810U%E0{g2q*X3P(^U zzssk!d@g}oBE)B!0%zE9KDKWz*&H|yDP!zrY-7~OAuku_pU{Tfml3Y^zrRY_e6I%I zoGPah5tDUo6fG};gBj!q=mpPl$40KpQ^SVjp~QZJejert8C)F{ddGl}FfuhJps|9G zo{8Jg{V|yYzU@m#YDmI1Q@nMn79ZMThTNawpK&L7Gl2_I1inbq~f>pVbkTNWB z(`;;8Nm0b;W2&u%PAZ3-ooyd%a%dtV+>KEXlwVP0oL_gvudo;r|1)i|rNO&|r{db< zhmW4xTL-Y~uV~>tz5Buw4C~xS@-s=BeRmg0<+kg}wgWLM6#hAMUf_z}B<1~&?Eb!% zHG@FnM0|cuLda0UhMcy>%Yd1}Y{n)#mziyAX>l;5Xwp1gr40f}+y-^ZpNi@ULXP55Z8%pGw#lc1YFeM@|JEA$OI zD^f33G+W+AhDU%wP!R`|_w1MK!4D7Y86$+XC?a>wXwGd&!M)2uu=Cr6&5X9+X_@Z@ zA1ESwo=pW%3Udzl9^SM=W8?R=+VE8SWZke&vB96~(CvBN#Z+ebaP zgl5`7p_}_F%t*RTk3cH2M-e)K0izSBdfIe~msrIG6sxorj(4TalEhmR*v#H|F?GK} znyJUwo9N8Bi_~~rDX&;>f2wwCT`r`kXVVOV#e0@3f?mzUp9Z|+ad9yD0DpI;kI{LX z{OUysH5f0kt!zT277rdN4mS59Th|h24Az80xs}XC4Q4`f5{w)QY90^0=cB|931Ys= z3cs+rp#^7`KvWUBZB#2ldlqxNOinI>ZJbmQIP&6DmS$LlHU1h2Ezv^D$>|nd!t8%5 zixPV8hkVJYz}=pXFniFw7!_=AnT5BScj3ys@V7~k2mT8~N*a9AC_~)_+E@vXB(N~o zG{Oti&SoSkOuj$CJtOHi))jA(jtYI+ndgPhS;G#8B2{!-UNlBr2Z7oH&$Peb$zO=V zxo@gh&G#KSLi$+^SsZ~K(fQH1o3^aQt?5}Jk=#78H467TSoc_sY?-u}Anq=gXIf@?i~)cW~my;o;SdC{9vaYWl*8e{skc+5v33v?UA zP|isIIZNs^SPa3kOo`@osS5ahWTgE4p2!Bk|L4veKJWWFGrAiCDQ+@4QZ&tvta1%= zH_mDueYLxyqv`cLXvIXs7W#_o3nRt2&D$UEf9wG^ zA}L)$&8O`}C^igAzU1TFpIao9T)+pJtzx-fe%$b)JF)-ZknWgaUE`%dKk{B{uO<^` zP(uxd+=Y%$_*#P}Sjb=}bNm*h?(}T}&;r&Wfg~Jj;z3H3x?i7bed3r8NXECn;Bk zY?n|Pp*}2nJkrs02y4~J7$XAU^TvQC8mDNd0iE-QX|2=7d|^KAQ$xjsi6B*QbG18r znyhU-&T?Jc{^=s_Dx3DfnbwMTp4nT5H#9^{vM&Q1@#f+4?;asvGD{@2Fm}wd7~UY= zEzo6w(zX_lj&A$uY_VNz$o3W_6j`4ix``-_od4+VC9>~5^E1ud#ByKcv8a4(YTV!V zcosWU*2QSJ(Qvlued@QJ1uNNctsf!*8=P_f<%0DUGm%RRm9?fxLt2*CrhfvkH=LVJ zPuAC2*^@FI*KK(YMek7M2!al~e>?|)c*PI#cCJ}aFfP0AwZOA=)eU4jJa*^35i>SV zY6e#Zxo@}K-*Du9!JEvQ=tSF=;L-Yf9wn?F<83d=tC3^=F?mx{wU(8d;X8dgX$Phv z(=`!LdJk7>ypw^kS6g*)A=&AEcf2n=)ZA4zFra57IX;kouh3?_p=`6&)h{01EN{Dh zws>wb`=l9ZYWpn&Vr!qR-RLsbLsI!_sWU#+#uf(4#sK*rR_v$}Z;>H8KU(H(LMAr* zZEB@ox);)gOzKQ~`6H%6`sQEG$l5%n-|b?`Ri~YA-NEM4UK2X4Ej~Yr3Z{c}Xg}lD z&nIU$2lNsWCk61G`b7%TGeJI92)WVurrK;P75Iq$?w{tO++I1j@gHZC5^4-A71GlD z`$SeJ|Mbe_WV)Bk))YaoVWc<2G@jGTPZikHLUD?$XU48YcNBf6Y`0_S-Khu1E@@Uj zXLLo-7W-JO61~IUYIZyXW<_V>Z;Q74n-(zL+{&S&H@OjQRh6|gs;PCAqftJ!9WhQp zr;^l>&-xbK#v}ZvJ^(Ce<~8WuvvYE#xW9LaVc#1VD$ykAVy@LvnRZT2HclKD{wzD> z;nw%urlgTO+oP$KCB6=@i0ieN#-1>jENv~zzGgmjxJWi#w(@w?Rp$<@-OI1$$D`HC zz?9(4u?qd4jYK(R)svK40ptENC=E48yL8|Lc|?U-soE&qC!hT|9X3>i?1_dr%0hBC zomum9iKplCd|-;tuX^2*X6n&*-fQt88_UV2kEjbJF{YH*WW~_jwXx-u-uGG#Gdq9S z)0>;<9V-19P5X+j3ZxJf`58;SwmdSX`tw?!{%9E6>hjxUW6mt$Alb}UGk{%2KsU#( zFBs==pru8P3`J1;_cQHEn)&OrsctVmJC==BwM@LvKH8~cNcrQ-pK(Wov$aoQ|Be{S zXf|p!5HBlE3-zd28M3PjN6$Z9ak~Z`CwsYSkHy|Rk)49djX_6tL14wlJCnKAN0NOamPfjixhKZB z%muZppVN_F6j#R@yr@FZE6u1UH>XB4YyBtuF}~$w<_ie{V@ri5Z+1}7d|9Or+jVcJ zaw3}(X)H3m|FeH?X5G{1rV^;#@N~P?RP<@8(69276_>#XGeT;oG+cNDit+j+G*jT8$EU2iu%f4in=F9C3Pn==tjvG$p z;lq9BFPUVhvHFCv`Ln}I_N*XBAf-B1O(XRQAke*N&^z0WKMn$Z`qS$4O&;4jn?u7p(hdiN)c!MC;EMsY4cnjKVJ2Y=D}=q$G@28>JtLYXS(~fh zdIVX#^mg>GJeOqS9J{KkVt2VQGlsZ)8B2ETotoAH_K)nW7TUxZf@(}Vh*tdhbC7q> zce0WEjm+mKYpH{B?GP{s*zM1;!_Tx26U~rw{p|puy$|aAem_uE4ahC-;|9Bh$Awe` z2eY+gLfJbyM`D%)Q&X4qbBEm>u3e0=FX&C&Mh9yTCx}t6o$tSFRZM6V1_u|*jOjFK zrVGcqf(MKS65bHUS|=2mp9G=1YBHlz+ZwZpKUPqFp;qbw&IXlUZ(f$vM_~5sx%tFA z-8UM;yENe)1ad_A$_+z)lc9L|kP^4e(t~~jAxj}?({VLB>Z-DkMGm%65855AXyRw$ z2zr%p<<+yLyMECz{_j)|V*{jKQaeq`GWE7g!U7fFP&YH0j$6QsNnAPW@uH|U^Vd9L zX%7A-sZ#xM7T4H>!q3c`gaEc+S4G$hmBb8TAKie)?GmrQmQBDl&4o?_uOq4Qfiv0V zGlIO<=uB@X6kwwVpw7Sg%I%t6Hcw#=vHNp<-e>q?q7LK4E%})?UlkFqYz2<+Fnx&JCPU_XDRDcwDo~i z_cj%?i9-IhD=S!OogPZ5P9e-qJGZa_gfKh{xVZ1MM*GASTn%h!(N;Lp>s z=Jv`UQD{6OuIqC3NQq>8N`RVY&E|K~k(Jb2roEdb`pK!8>~!y$7LUa^+*rB4Tp+ey~l{+JcJbM3Pe!18`fb_H`G~UAM)TU*8LR5TNa1E){E3BgD=k|0@5%i;XiA>+dDZOS^+6o`VR4MMs zH3v<|h>Czy{(eVE+U%YF;_=9^HB%P{XY>aHS14vQ4zV!{+dt@#9<)(X9<3|hW9V2m zXZi)P^X~)VSKcSkyVFTpE5eGg?HcVucMBD!tC~U=<|MMWBiGxi!Z8At?~J-LY(tNz zqts~&PvdH1N-WE@o0ryGniWMqmf-iPMNXK?G|P{g&9Lqod4n+Xo3svrQlXD_qidGL z@$#?M2hOzq!6=Eo&cLGSW`w>mLD4}Ip>%wjh8P#ESWk<``d2RxPWmkjojeH)3{Woe zvVbv-z6`(+PWHY(6R}D4>F_D3t!^uzUBl(xZ%U0_5JZRiZ7vWb)381>2tujbTL0pE zzMHg@T`3lNh!W2Q6@nrNWXFAoS7K#wUQ|((kK_(mKO0P!`O7rRLHoyxpt3STQ@xLI z4j@^=6kk&6BfM11KB1nm7Ca=vNWhoBY|n${Gz)N=b-Q!)3nRld{W5POH!Q81_R|WM zhm9Ku=ukp;0l$FFwG*IMwq3`SK72Z@$-Vzg_F*xBWvfaAswea(xB(Z~``S$`3>oo^`o_R@HdQ(j5NLt`l54h5o+rO; zcvDeLT1`=t*R82=VKZw}G;8~k+d}?9hn8xx$VSA%enn~=_k=W%U=*6@l-dT$U6t=6 zxqDztQ6KIa7`S=)kM`yDuE(p06ayr5{*q`f)F(7(qL8%ve!$4sD8?8-oDp%$Yl496 zyZzaHdN9{Y=|*Dlz<5j?qIGIg^}FmyF@b>lW+r(tUR8v6{t8h-qq{&q1RVl|+24Rs znjGwpkC{;=7nU{mbhO;?wH+$=MXa7aZCXo=<=RF3toSsJI@fYmO@KFt#9%Q>W?PN@ z1pqm@#GX-LVhMBT_|R>pkd+K`A05r$1K(`G4*O9aGT=<6Pd|9;8&vGX8OXSA_;x+o z@aDyVxcu}mA?yjwbuUC&<$4qEWBjCFuE$D>i9wh`WI&x-OPIw)uhmxG$7CMU`i<$D z;%dCKpiFr>c(MuxG5#B-YquDCvb9pl=KLn^JOtMibl=D6#Z&o~aqW7qD(<||oMKo+ z#4{iTE(fxv{ky)yNZfZ2_^OIKr!W!554i214C zV&p5W8z^lZ(ieL9D@w!4mlH={IdzW;=~Q1v1chySnG|1gLyyXN_2*7?lB6GSU&aTZ1|=t&Aj^*q;T8g}1PX+=6;&Ni%r21s4jzrLb%1oYt)kTpqZ zalWK?^K(gD4*U{o&UzO1OPb<%2a*&LBF? z>QOy1w}h~YwjYul^H=7MGPS7+4lI*NBXwdkk;sFEk=$>kg$_aNx)2p2ClF)i>+9+5 zYb^cTKwF?sKK+SBwzb%`mH>;PZ}S|i^&zFf#Bvh@%#}C7XXQTKC{{Z2 z;qzXnqW1PioOuGQx2@?Jk%HX#_1SCxGT-?EQ(sv4eBok8tc*Ql)+fg3SOq$Gpp9 zkoF1s%p6H?M^Ec8*KAJ+>Y;pnBVMo`O9M0??%j{nFpuVmmNMx-O$31`Zb#3*v%ex} z!arO1<`N>;)7PWU&9;O(m)&em^trtqx$CyayV)a5BbSgjaHxQwKz&m}Q>(B<_2lPH z)OOG2dd^HpAv4W91xB8UH_=47)r4S12gWyz^{U8INbE{ZW}1J!<;A)Qa#^|8{gaf3 zHH+VP+;IqM_xJ5@ubDRzZ8b|naUUn!E9;B*(kjQZqXi#)SIlm5A39sR(+pl=m+NW& zIDHhd>FZuzCW|C_$Et14!alcc-(s~<6nQ-xd>z8U*MMAB7r^2n$1|#u#4c0G+gGId;g%0FAgYWyw6?9 z36F4MT$+6v67sVqjvyGM|J&k{j_CeE zAOV}37C)GiA_P1spf$d;haz2l1M6-E+3l^SrSvEZL5%8DR2_mo?7x~y6*JetV*TYb zo!DKGX0YbwXYH&T+#1cqQhwj&054)5ezswwX1QKLL^3hU182CDduYKhADZZYMeRoZ zm&NfrTC{9o;y)6t!K0X{xJ;qi9>(dBP=3`6l&Hz)J5Dl6FiXPKI#*Z1yjB2WRbMJTBZ8fyQs zE8G7OHJ8XW#EkSx%?hJ8boIe(C)DZ$cqZ` zoss$a32dJR_juGR?1>QjZpSMNH8lw&c1!e3(fQ@m!Au(;C)%a0Lh@5xRQTNQ%N^@% z&HOhV1kWQNCD|t4GrT!Z)^V|Bkd~eM<;8J?l(_7&fvks#VavM~PLe967Isz=vQrJv zAxq26bq94##$12ughls7ns=pHmtS7Bm8tKBtbCS{)Pg8L!=g57N^tr`<`Ccg{@gFu z-URA%@t{~cXP1Te=Yz!TnENAx&DO)jQba^;=&PS*B2!yq-D#B$59{Z{(1${>YZhw z(qlF+WN$1I;T)me8!?V5tyN2Mw!)(`{gxBD@MlY^Qp+JvlR+>3c?VL93$5 zjlkhGF3HF^|K7rDA+E$eePMpdH%}JP(kTJD)j`Xt6*WkUg(1<51~(k!_E_TES!w=W ziPqE)(Ra(ZPJ(XzQfs9jTUoQ5UGTF%ZCl25b{`LP7iAS{qWPIp>G7&vihcD~kA_N; zQ^7)r)`)?#r3f|d$ndc*YGCKa3HX|JP2g;_N$ZzvEibC_m}T6-#sb}ITvgtotyTlJ zGWhyrJZq|>Kn~ULB6PkTBFkEjl*f2$W?y)zko71VRxzMEW{W0S4>{+uVqt@Y5+vtB`}{Hm3jiON2= zO-e$pj#fxCna5PjADw*CNsw-9X8rM(lnqAlsgB`GgjzV+3)5zWpF;eOP@ise8eq(= zMN8jJH*>-LvGT>lOs6If=}H`t^fy)rsh{>*qv_6Pg65$ii|T^5MO&*S7E26&eMGN60NR)`_zlhjqT#D`}MoVaUdI`S|Xk3ky^j8$N&;hz3 zc=)>gTotTDZbJDqmojZ1r-bzh8`tM}t52sECgf`P`|*){4UgUtlF77T)lXZ8ARr>Hs`TgEcu@e>c0V7C(nf^j;)?oJJH-Q1U$SmUqyS7mkyBSrL{Rc#6 zH&)ycgJWbIPw%SxbiKdWmc7FvtGy2G;ofv9i8aR{$EP6X zkCyXd#4`fgI+LWm;y1K#EaO*icG6vp$yyxjzhnn7Acj}~!sx9lh_C9j6P+DG3Vh*B z8w(|Ywx!JwNp8(7b=BoQ(Ll3fY6>s?CQ{0K-yy?1OqSoUX%cV!Q+;L2;*wJ+yzTeM zrB~NrSE=KDyNKNGQ8)B8R26Svm?uQjLQyO)R=_OnCnrKP(Gm{p{DKDfUqu2it%{od z-3T@G(1q~#XiH1@aujT0A~?$+Xj5Y{n3^ziDG_`f!oCv;Zo8`6wJrpWL0hSF+Kg8o z8%U=l^Va$2HdV{}Zy-lwFi6s2Nx@WYa zhln8FsV>X|2oG;EU`-k2HDQ^sfaSzAjYz&j(pvn!wupN6#llPcl!9Nl3D#-nUFk}w z2U{O>HXO=61JO#mU>duWx~HYDq3Xe2E8rHJo&5Vmx@XRQC@DtHP1VzUJjZr5h38k)757GeDONjEOgEp?zz zg8En*3W3f4NG`e+e`{D$D{jK=ot1x#l;9pbB;h`fMTJMW=NKvF((u-T6F9t@_WF&6 z+{Cr37j{-5D;@IdXmGBKjYF+tze*Met0TywAb7;RWhycqKI#0FE?58(!JCRfd6u6K zgB;HR(%fO+0hu4ggl3j>ShE(w+kzQ27%4|!O+T8rD{>Ubxgm-9AH+``P4tEiikLqolcSvH{{7B(5a@&0FMS0l&Q<6bF!i07!wqxOb)q320}L3< zUPvbUIvh86z!T$P&;IZPA61=2Z9z*edS zZfJK9cv)lki5up(_NbnThDw-E7{j9Ov-9JZ5zlT%Tx$snEKTRC4K;XYzDW%mhfLO0 zlBe|(WMak@U|$hcNq|s(MVhP`RtGt{e?rJ7+F6LOa_lx?NZ+_LJ3N$laQ_&P_*nOqH|joOVQ(r$e_tGPDc?wourNvV&9NfR1WdVT$98O%Dh!!! za#^p4JV`BLh?lx|Ohc{_5$+a2JNpcM3`zZL-_TH0mn}LI{8safu+f>7AsYn3ch)7t zytvJoEG85ufC0CLgv$QaHInRb6$Xsk7#Me7oM4Kn%w}WMfVFoool;2 zoT!gyqIYUaD-LJ5lMMw@l7#|pX5L?ZQei`1dUKFB6edW^E}N0CvAYh#33(Epreunj z3}ruN4MGX>I&ILI@oCt5Ma>)yO&4E%vs~%3L$SSzC3tswZav;gS9v8jHA&5@@9b0V zRI;`|;am3bg{R6qzMr2c4t^jt2@IerpMOu40Z@hl6xh6mU*)PUzs8N1go9<}6}|EBxv}xYCY{~V z0YL@(teKA=md^<;2??g26S~JO5;ug-xWS29S`^ckO=sz@Di<{5gy@W3Z&;o3O@xwKR23>1)c1;85)Y zsZ1YUP`mrq7k=ISOQ7>oeGO{y1yUC0?0^3wY4c=Dka)a7Zc`Jz+r^wDC$jVO?IF;d z%|9Ec=Q{zs1O55x|KpvCH9(j8y>C%xrP*vUhdr34yt%m<5)$(E?OV=L&Oy@sBq=Vn zIdS&9_y~v9C9Qk=b8j>Cmvd z3Gv}q8V?bVu(#gpdAqwe9(?f_EoAM&*_g=nk?HB_k&(oW#VX&;?(5WShB+4+%aAYl!rl_NMC!@Gf*a#=L z(ni!q;Bj<19R^<5MBq)uwP9KpW8;L?6hQZWHM^gmghcFS?CtFhAfN#O`7<}N)H(v` zQ<#X9V!Vpicz$n^Vh9DyS%+tdwIVreW@X?-|7Y=oR_m)`2OGj>?7Q)w-4>v(BV=R7zTsU)>-I};M~FJGQ9~s!Vu)kuiQLrw6tM2v6Q)P+Gb`lGV6ovlOMP=a`5Z!iZ*WX>^axz%2Ha=ld9{_bwKd}l zCDo9_`t4V-{}4z4u64ITh6vP=X&FHB1)d9i*jDHht4L4TveXt{cs%E&hR3E zP0ff)0lc`t00yTIfD6e40>Ly}o4z`nN4(d64y__!c@GboTlPxW^SK}IcI?jIX^gOC zkg1R#VPo8}&X5+)kaxp{G0Fo21G6;zr9`XGNCYHPc4GCB%lca$J|s4 zA^8Trqo_GUIGeKwKZsw~8r9O%(}N9mh_Uv2^&;LHyq=c=taL2d^mD*RHl;`p2ADoF zdurIc$kYU)9kR2u9OHCyc(U3_c`BS<1BeRmY78IaqbgI)4BMQIrOs7i`z(X{J}k7g zOlS&)UQJbOE*TWQFU@+dcQ8Nb2To(044vLDF~eN*PZqmnn(FCFFJ;FU^6!V~jNIt} z+#M%=&>?QO9*rkHTawdG=bHna*AV9e@`AfJFS5oeBa(~ppWS5gD3fz1Jc|&t4_=60%@|^A+>rNTv=`^vg2fr45TWo6O z;VfLD*rnDEnIABTI<>~*<0jTRa{kl)p)S1_;)U4*?J(mN8oK%+|q&&j=LsmmGJ^&t|YS^X28`lP6D_ z>9Ht|%mWsl18cUa49X`RYg;?zg0FvbFP5^B!}k-43n?O(v-S`u6SWuGVrCFz+*#-- zYVoR!%zeDOr>9ZDPTOHv$${E2$RF3}^4=kk{5UgEl4|wUCUpQX&VPt)tbLURwC`t9N<8YN13ex!lE~ z!U)1!VI2BEy*7VJ#iT$C0A@Ef;OW~Z4rLl%0n)g0fOji>9Xw$LAoZ~OO_-KW&*b>H zAhs&NoizZbD;Ak%O;1jKA{9`!{50D~Vs-opgE_1Xw={-m=n=dTXA{LjIR1%bGuBx} zK}DQvBTnUMNd=>wD3;>ka5@@c#XtxXFJb!oFKPvO<9ncZ7>z~4o9QhfpP#=CJmK8s zRIq77eSKh6*63nqN8o+SAFVnHQ*0)R!$i@CAbcX~r|g`L(Bw~uQkKA)MF#j~xUz!c z%=#xnR>f#MY{Q8t0_>*%e!L%W(iii@S?+}4!Q`Auv*w<7alP!eM-C3Uu7jTjQ`!!3 zgf(uwhzw^Q0@*nOV1znYR#f%~a3Lnrf6efmYlaromw`$?!?GE>H ze1;^;P0eE@-&oZZeYEX%K|ukq2pkB;lmhB66v^i<9hGRmZRX0|ew_Ye)#0o-XpXNQ zO2$##m+DPloAMtnlJmN1`KWo30=^x3b>gfT;F|3|#{b+IrS_~q3kdLytlu1g!yo?} z_DlU&mP>$m^uKiP*WUslXk$e!YJXTx&o^|1b?G}AJ~#SwwATQWg=?iIpS5@Db>^~$ z3g{frttE$lS=1ENnWgCE-RGvu+}dx6td(|%pvRugLAp(Ua<<&te~;Rq+Ib0ZtiOMM zH6NEMFhssEx2eet3Y)MGp%0hMVVNs$zMseAL7*S;;(tH8xvI&mot@zq(PCD=M;&~Z z?m227wbTT)fm3$CVvTZ@`)y3O*$g$;t(9#0a^(@wT*ITS?Uo~s!QrdFw#&VGG>;|b zc*B=Bi^pmQdb!+Y@WRTz-V-y@dm9W7N^MSQ`ro1eNnwt{XU=2A*t;Bui=fU<*&j-w zADY}2mRhFGGD8)Q*YqsU>PX4SX|}ohk(isJl=y@l>Yb(T!`NBnqBezfHBJU@aIyNbfpicCFSl5+R}Sh%UR}i{2FlWqUVjf zF0u0TK{s(RIP}F{*MK7Gg^7Ixj}g!;R~QuGf=1d z-zp@2U0Slg3N-uSh3^MB*k)BkBG{lfx; zC7vnbNBVu9-=v|roMM15y(0E~Koc5D8@DtKh!sr|aGrfi*;;W+z6%)RxW*w~DDrM^ zW$x&Te3p{x_U=R*2aI!<%~IAd404Hw@5kiGivtxi&;*q2u%)Ll;w_v`=M#2j%uPkr7bj zga9y)lfOAO?gp~fzE@17_v%^cD5Km;{4JS%IX5#+gYcTne*v0T4FG8S#lN7>=RJ+= z@gcrXr;5tqt*PlDU_RMjIGN7gOMd6yS4&26qA-KSA~^uS-dY7NKI*cJvD>1irFtV# zq2E|Q5cu&Q`PhHK9_9aONBPe-PdC18#(6>Dm0qv76Mgh_hJWG!D}p-jfjY}a1NVgV zLaAz`NUK)zqPFr0O8I5QTDMIMCh=wAQwb;+D|S1Y0!HetKy z8s4Cdp-BLutAyT*PI&>=Pdz4>?Y!{Ge|rys_YK{xZ?@mA0Mt=BM(2iyH*is2X+VX( zWy!>3Ouuk zi>F5rbFhff`CH70Wo>A9)<{3h`=R4n6c6O!tO;<7E3=OLl62AAT zBCKe+&2J&V2w;4#a-!x1gUMZOD2?l+*VeSUpIHheB48_aT9ng>1Y#3)QCUIo*IHjT zVGSN}!}B#a`>tl(&_QQ5mad*1uJ9*I)simG%#K)34Y$;cKwI8aM~dzbD#ArUA0D%# zj(pbpW&11Dce%$vpx@}f-WLEL*?(v0KqocDe~4-b=fsxv0R+C2R>z`_i%Op%$1{Hx zm5S|uuSUV-Am}CG3xxpviD{19UYM#K$?Nx-*y(x`j~4}DohK;dA4)rBSRZl_z4Y*( zf3AZ%x3nfAHnuR+k?{eo%y=?#b{pM9OUJ7ZAE24TBiyOFZCmp#rpSXGxK0?k%q$+m zS26SNR{o!q+ciDdLbrhSJ zz^Ng7k1r`kG_7`3h@&DxpO5{ye*P%QQws?<5jG?+inWaRWfVbKD0o@oDokzc-5?z8l0bX@XOl!hm2i zmjaX_NNrJUh!?Z8Zzrq<40!;=@AfT!J|*6k z>rMLfl?cWTjYF8UE(hjR0zjgwfs2-ut#5BR>p36zas1by0B@D(KIh)3Qqu$T+*v{~ z`L)=&#Y`M$l#&zHz@`M#;BdbKNDBwAlf%Xf>ioTuaR%7${XVB2ur7ld)d|TwZX|dB z$>C#U9noHg_eZeEiM0fivDROYcktt6C&_LmO1#x?wTl@NB7Px*oubTh5fCeZI_vy+ zAZeD?gXK(d0*%J2EhB*QAc?!6&KCn0X~o=Pe9dmR4Qi*?9dNTchp{{24rY3P!!UYC zWnJKNzP|x0&1ihqd`13KXkSmQ9+TTY7tn2g*{TLdUJKO&KJ+dJSR{X4RBK}F!cWz4 zWS^Y}UcK-x4Y>6L)gf%A5`8kIam93EGon@IH1P3GBVb7x*B!Xi;xo`E_qGg1I4j;q zfhqNkSbJxCYvC}kxkULt;5eqnf7*U2%_Ayud`3U>yK&e{z{Sox0Q>UZmSGb)bnsxB zi0Eo{mtq5_H|f{B4?cMP>qQvy@(XMp=VNhGsJFD#bfvgW)-Zf(NU4p9XLFaDIudgcA{^Pv}U;c+^sm^`=-s>ej zu~whYI!YDq)F%KK=y z2>Lq6@#Bt}2nfiu&%dqNk%E&3^qqQQH$$0S+4gXAA(q|PpBo)P3j6!BhuTd54MgnV zEOe&Do?Qa^{n?3|AB?OQcSvGM$pWI_n;JCMUL36%STx7A0J>`SpRkf}H+Xf}M0|!8 zIw(_%#VUs+bNVuOuHPQTw~f3|2-(Y`>20TJBrn$fVs$zIt806i6TSq7?JStkEX^8! zc@g6}7vP!yUGGo%&$!7cfchW%U(?H3b9gQ;_29jl182A+ryc+bInML7MiOMbVlM~T zz#`k6>6lFtPqz;w_lB1Of%CNEc`n@>6;;kQ#NHkXj;uLTxAG8qu&ZLM2cF!6R_3VB z{34A2uRG6GlS^NW>Fs5_3prbaGXfH+rR^yv5|E{nD$O!Yf2GQy1pRNTqrmca9=CVO zBWC=&a0)}@{L0*4J?htc38F~9`hDYsZUhwcBB`DW!OiKK-SbUF$=)*j1h81M@lP!^ zEjSq2j^S!~W82q~QGpLU1bPK}_vor-#PhnmVxqU@#8iqC8LJZ#eg&ArAIjH`LfdLl zldbE$L!xe4=qxsS9MR8lMwk62`IJN1oP|?!;*KU9T&R_b-^~J4RiIhn^}jqK(IMf#7*GvKdv@F)BitNLIT8{JWSJQ3!=?0*K6Zb*`@W0-1mBVrB?vwzx5rM z*VyJn`>927_JSLroB-$Ef!+;&U1#~yhMeLMn<6bWeuUb?8UVuL+X-f5=+ASN_F!d^$Ee^j;!bXO=5ZoH}Fi`rRWL zFFV|gqUZ{u0-dW# zDy|QHP!N?jGmW!hD=f2cSJUPFjbC#GVC&U$Sufz!KmU>80=YMU&i}99pft|`ti$18 k83KL4O^*E{fk!}pefh)kWuTl6a0YZ=-}G+9AC7PSFUnB$4*&oF literal 0 HcmV?d00001 diff --git a/docs/technical/retrieval_architecture.md b/docs/technical/retrieval_architecture.md index d56c545..82a1c32 100644 --- a/docs/technical/retrieval_architecture.md +++ b/docs/technical/retrieval_architecture.md @@ -150,15 +150,79 @@ What the fake embedding provider **cannot** provide: - Meaningful ranking of results by relevance - Any real-world retrieval precision or recall +## Hybrid Reranking (Post-RRF) + +After RRF fusion, an optional **Hybrid Reranker** applies multi-signal weighted fusion +to improve Top-K ranking quality. This replaces the previous simple embedding tiebreaker. + +**Source:** `src/ticketpilot/retrieval/hybrid_reranker.py` + +### Signals + +| Signal | Default Weight | Description | +|--------|---------------|-------------| +| RRF score | 0.40 | Min-max normalized RRF fusion score | +| Embedding similarity | 0.25 | Cosine similarity (auto-disabled with FakeEmbedding) | +| Intent metadata boost | 0.20 | Intent → doc_type matching (e.g., refund→policy +0.15) | +| Content quality | 0.15 | Length appropriateness + keyword density | + +When FakeEmbeddingProvider is detected, the embedding signal weight is redistributed +proportionally among the other 3 signals (so weights always sum to 1.0). + +### Configuration + +Weights and parameters are configurable via `config/reranker.yaml`: + +```yaml +weights: + rrf_score: 0.40 + embedding_similarity: 0.25 + intent_metadata_boost: 0.20 + content_quality: 0.15 +``` + +**Source:** `src/ticketpilot/retrieval/reranker_config.py` + +## Multi-Query Expansion + +An optional **MultiQueryExpander** generates query variants using LLM to improve recall. +When enabled, the original query + N variants are each run through the retrieval pipeline +independently, then merged before hybrid reranking. + +**Source:** `src/ticketpilot/retrieval/query_expander.py` + +### Merge Strategies + +Results from multiple query variants are merged using: + +- **sum_score** (default): RRF scores summed per chunk_id — docs found by multiple + variants get higher scores (multi-path validation) +- **max_score**: Keep highest RRF score per chunk_id +- **rrf_again**: Apply second-level RRF across variant rankings + +**Source:** `src/ticketpilot/retrieval/result_merger.py` + +### Pipeline Integration + +``` +Query → (optional) MultiQueryExpander → N variants + → per-variant: keyword + vector + RRF + → merge (sum_score) + → HybridReranker (4-signal fusion) + → Top-K output + RetrievalTrace +``` + +Both features are backward-compatible: `enable_query_expansion=False` by default, +and `intent=None` disables intent boost. All new fields in RetrievalTrace are Optional. + ## Deferred Items The following retrieval refinements are explicitly deferred: -- **Real embedding provider** — Two tiers planned: small (384-d) and quality (768-d). The `EmbeddingProvider` protocol interface is ready for integration. - **Realistic enterprise data pack** — Current 36-document seed set is synthetic. A real data pack with actual FAQ, policy, and case documents is needed. - **SourceRouter implementation** — Intent-to-source routing (e.g., refund tickets search only FAQ + Policy) was designed but not implemented. - **Persistent retrieval traces** — `RetrievalTrace` is in-memory only. The `retrieval_traces` DB table migration is deferred. -- **Retrieval evaluation** — No golden question-answer pairs, no precision/recall/mRR metrics, no evaluation harness. - **BM25 or alternative keyword retrieval** — PostgreSQL FTS is sufficient for MVP. BM25 may improve keyword ranking. - **Embedding fine-tuning** — No support ticket data available for fine-tuning. - **Evidence scoring threshold tuning** — RRF scores have no absolute meaning; threshold tuning deferred until evaluation data exists. +- **Cross-encoder reranker** — Would require sentence-transformers dependency; deferred in favor of lightweight multi-signal fusion. diff --git a/openspec/changes/add-hybrid-retrieval-reranking/design.md b/openspec/changes/add-hybrid-retrieval-reranking/design.md new file mode 100644 index 0000000..2ec515d --- /dev/null +++ b/openspec/changes/add-hybrid-retrieval-reranking/design.md @@ -0,0 +1,306 @@ +# Design: Hybrid Retrieval Reranking + +## Architecture Overview + +``` + ┌─────────────────────────────────────┐ + │ MultiQueryExpander │ + │ (LLM 生成 2 个查询变体 + 原始查询) │ + └──────────────┬──────────────────────┘ + │ 3 queries + ┌──────────────▼──────────────────────┐ + │ Parallel Retrieval (3 路) │ + │ keyword_search + vector_search │ + │ per query → RRF fusion │ + └──────────────┬──────────────────────┘ + │ 3 × top_2k fused results + ┌──────────────▼──────────────────────┐ + │ Result Merger + Dedup │ + │ (chunk_id 去重, 取最高 RRF score) │ + └──────────────┬──────────────────────┘ + │ merged candidates + ┌──────────────▼──────────────────────┐ + │ HybridReranker │ + │ │ + │ signal_1: rrf_score (w1) │ + │ signal_2: embedding_sim (w2) │ + │ signal_3: intent_meta_boost (w3) │ + │ signal_4: content_quality (w4) │ + │ │ + │ final_score = Σ(wi × normalized_i) │ + └──────────────┬──────────────────────┘ + │ reranked top_k + ┌──────────────▼──────────────────────┐ + │ RetrievalTrace │ + │ (所有信号 + 权重 + 最终分数) │ + └─────────────────────────────────────┘ +``` + +## Component Design + +### 1. MultiQueryExpander + +**File**: `src/ticketpilot/retrieval/query_expander.py` + +```python +class MultiQueryExpander: + """Generate query variants using LLM for improved recall.""" + + def __init__(self, llm_client=None, num_variants: int = 2): + self._llm = llm_client # Reuse DraftAgent's LLM config + self._num_variants = num_variants + + def expand(self, query: str, intent: str = "") -> list[str]: + """Return [original_query, variant_1, variant_2, ...]. + + On LLM failure, returns [original_query] only. + """ +``` + +**LLM Prompt**: +``` +你是一个搜索查询优化器。给定一个客服工单查询，生成 {n} 个不同角度的搜索关键词变体。 +要求：每个变体 5-15 个字，覆盖不同语义角度（同义词、上位词、具体化）。 +只输出 JSON 数组，不要解释。 + +查询：{query} +意图：{intent} +``` + +**输出**: `["退款到账时间", "退款进度查询"]` + +**Fallback chain**: +1. LLM 成功 → 返回变体 +2. LLM 超时/报错 → 返回 `[original_query]` + 日志警告 +3. 无 API key → 跳过扩展，返回 `[original_query]` + +### 2. ResultMerger + +**File**: `src/ticketpilot/retrieval/result_merger.py` + +```python +def merge_retrieval_results( + result_sets: list[list[FusedResult]], + strategy: str = "max_score", # or "sum_score", "rrf_again" +) -> list[FusedResult]: + """Merge multiple retrieval result sets, deduplicating by chunk_id. + + strategy="max_score": keep highest RRF score per chunk_id + strategy="sum_score": sum RRF scores across queries (boosts docs found by multiple queries) + strategy="rrf_again": treat each query as a ranker, apply second-level RRF + """ +``` + +**推荐策略**: `sum_score` — 被多个查询变体命中的文档得分更高（类似 PageIndex 的多路径验证思想）。 + +### 3. HybridReranker + +**File**: `src/ticketpilot/retrieval/hybrid_reranker.py`（替代现有 `reranker.py`） + +```python +@dataclass +class RerankSignal: + """One scoring signal with its weight and raw/normalized values.""" + name: str + weight: float + raw_value: float + normalized_value: float + contribution: float # weight * normalized_value + +@dataclass +class RerankResult: + """Reranked result with signal breakdown.""" + chunk_id: UUID + final_score: float + signals: list[RerankSignal] + rank: int + +class HybridReranker: + """Multi-signal reranker combining RRF, embedding, intent, and content signals.""" + + def __init__(self, config: RerankerConfig | None = None): + self._config = config or RerankerConfig.default() + + def rerank( + self, + candidates: list[FusedResult], + query: str, + query_embedding: list[float] | None, + intent: IntentClass | None, + top_k: int = 10, + ) -> list[RerankResult]: + """Rerank candidates using weighted multi-signal fusion.""" +``` + +#### Signal 1: RRF Score (weight: 0.4) +- 直接使用现有 RRF score +- Min-max normalization: `(score - min) / (max - min)` + +#### Signal 2: Embedding Similarity (weight: 0.25) +- cosine_similarity(query_embedding, doc_embedding) +- 需要真实 embedding 才有意义 +- FakeEmbedding 时此信号权重自动降为 0，重新分配 + +#### Signal 3: Intent Metadata Boost (weight: 0.2) +- 基于 `IntentClass` → `doc_type` 匹配表 +- 匹配: +1.0, 不匹配: 0.0 +- 二值信号，不做连续打分 + +#### Signal 4: Content Quality (weight: 0.15) +- `length_score`: 内容长度适中（200-800字）得分最高，太短/太长扣分 +- `keyword_density`: 查询关键词在内容中的命中比例 +- 两个子信号取平均 + +#### Weight Auto-adjustment +```python +def _adjust_weights(self, has_real_embedding: bool) -> dict[str, float]: + """Redistribute weights when signals are unavailable.""" + weights = self._config.weights.copy() + if not has_real_embedding: + # Remove embedding signal, redistribute proportionally + embedding_weight = weights.pop("embedding_similarity") + total = sum(weights.values()) + weights = {k: v / total * 1.0 for k, v in weights.items()} + return weights +``` + +### 4. RerankerConfig + +**File**: `src/ticketpilot/retrieval/reranker_config.py` + +```python +@dataclass +class RerankerConfig: + weights: dict[str, float] # signal_name -> weight + intent_boost_table: dict[str, dict[str, float]] # intent -> {doc_type: boost} + content_quality: ContentQualityConfig + enable_llm_scoring: bool = False # Phase 2: LLM-based relevance + + @classmethod + def default(cls) -> "RerankerConfig": + """Default config with balanced weights.""" + + @classmethod + def from_yaml(cls, path: str) -> "RerankerConfig": + """Load from config file for A/B experiments.""" +``` + +**Config file**: `config/reranker.yaml` +```yaml +weights: + rrf_score: 0.40 + embedding_similarity: 0.25 + intent_metadata_boost: 0.20 + content_quality: 0.15 + +intent_boost: + refund: + policy: 0.15 + faq: 0.10 + complaint: + case: 0.15 + policy: 0.10 + # ... + +content_quality: + optimal_length_min: 200 + optimal_length_max: 800 + keyword_density_weight: 0.5 +``` + +### 5. RetrievalTrace 扩展 + +现有 `RetrievalTrace` 新增字段： + +```python +@dataclass +class RetrievalTrace: + # ... existing fields ... + + # New fields for hybrid reranking + query_variants: list[str] | None = None # 扩展查询列表 + expansion_latency_ms: int = 0 + merged_result_count: int = 0 # 去重后候选数 + rerank_signals: list[dict] | None = None # 每个结果的信号分解 + reranker_weights: dict[str, float] | None = None # 实际使用的权重 + has_real_embedding: bool = False # 是否使用真实 embedding +``` + +### 6. Pipeline 集成 + +修改 `src/ticketpilot/retrieval/pipeline.py`: + +```python +def hybrid_retrieval( + query: str, + top_k: int = 10, + intent: IntentClass | None = None, # NEW: 用于 intent boost + # ... existing params ... + enable_query_expansion: bool = True, # NEW: 多查询扩展开关 + reranker_config: RerankerConfig | None = None, # NEW: 重排配置 +) -> RetrievalTrace: + """Enhanced hybrid retrieval with multi-query expansion and hybrid reranking.""" + + # Step 0: Query expansion (optional) + if enable_query_expansion: + expander = MultiQueryExpander() + queries = expander.expand(query, intent.value if intent else "") + else: + queries = [query] + + # Step 1-3: Parallel retrieval per query + all_fused = [] + for q in queries: + trace_q = _single_query_retrieval(q, top_k, doc_types, ...) + all_fused.append(trace_q.fused_results) + + # Step 4: Merge + dedup + merged = merge_retrieval_results(all_fused, strategy="sum_score") + + # Step 5: Hybrid rerank + reranker = HybridReranker(config=reranker_config) + reranked = reranker.rerank( + candidates=merged, + query=query, + query_embedding=query_embedding, + intent=intent, + top_k=top_k, + ) + + # Step 6: Build trace + return RetrievalTrace(...) +``` + +## File Manifest + +| File | Action | Description | +|------|--------|-------------| +| `src/ticketpilot/retrieval/query_expander.py` | NEW | MultiQueryExpander | +| `src/ticketpilot/retrieval/result_merger.py` | NEW | Result merge + dedup | +| `src/ticketpilot/retrieval/hybrid_reranker.py` | NEW | 多信号混合重排器 | +| `src/ticketpilot/retrieval/reranker_config.py` | NEW | 重排配置 dataclass | +| `config/reranker.yaml` | NEW | 默认权重配置 | +| `src/ticketpilot/retrieval/pipeline.py` | MODIFY | 集成 expansion + hybrid rerank | +| `src/ticketpilot/retrieval/traces.py` | MODIFY | 新增 trace 字段 | +| `src/ticketpilot/retrieval/retrieve_evidence.py` | MODIFY | 传递 intent 参数 | +| `tests/unit/test_query_expander.py` | NEW | 查询扩展单元测试 | +| `tests/unit/test_result_merger.py` | NEW | 结果合并单元测试 | +| `tests/unit/test_hybrid_reranker.py` | NEW | 混合重排单元测试 | +| `tests/unit/test_pipeline_retrieval.py` | MODIFY | 更新 pipeline 测试 | +| `reports/retrieval/hybrid_rerank_comparison.md` | NEW | before/after 对比报告 | + +## Compatibility + +- `retrieve_evidence()` 接口向后兼容：新增 `intent` 参数，可选 +- `hybrid_retrieval()` 接口向后兼容：新增参数均有默认值 +- FakeEmbeddingProvider 继续作为无网络 fallback +- 现有 RetrievalTrace 消费者（dashboard, evaluation）不受影响（新字段都是 Optional） +- `reranker.py` 保留不删除（`rerank_with_embeddings` 和 `rerank_with_cross_encoder`），新 reranker 是独立模块 + +## Safety Constraints + +- 多查询扩展 LLM 调用失败必须 graceful fallback（返回原始查询） +- HybridReranker 权重总和必须 = 1.0（运行时校验） +- 无真实 embedding 时自动降级（embedding 信号权重归零重新分配） +- 所有配置通过 YAML 文件管理，不硬编码 +- 质量门必须通过 diff --git a/openspec/changes/add-hybrid-retrieval-reranking/proposal.md b/openspec/changes/add-hybrid-retrieval-reranking/proposal.md new file mode 100644 index 0000000..3507212 --- /dev/null +++ b/openspec/changes/add-hybrid-retrieval-reranking/proposal.md @@ -0,0 +1,118 @@ +# Proposal: Hybrid Retrieval Reranking + +## Executive Summary + +TicketPilot 当前检索管线：keyword FTS + pgvector HNSW → RRF fusion → embedding tiebreaker rerank。Phase 8 已实现 OpenAICompatibleProvider 但默认仍是 FakeEmbeddingProvider，reranker 只用 embedding 相似度做 tiebreaker，cross-encoder 是空 TODO。 + +本次改造引入**混合重排器（Hybrid Reranker）**：在 RRF fusion 之后，用多信号加权融合替代单一 embedding tiebreaker，显著提升 Top-K 排序质量。同时加入**多查询扩展**提升召回率。 + +灵感来源：PageIndex 的 LLM 树搜索思路，但适配 TicketPilot 的客服场景——不需要建树（文档短），而是用 LLM 做查询扩展和相关性评分。 + +## Baseline (Current State) + +### Retrieval Pipeline +``` +Query → build_retrieval_query (静态意图词映射, _INTENT_TERMS 硬编码) + → keyword_search (PostgreSQL FTS 'simple' + 32个业务词 LIKE 兜底) + → vector_search (pgvector HNSW, FakeEmbedding 384-dim) + → RRF fusion (k=60) + → rerank_with_embeddings (embedding相似度作tiebreaker, 无实际语义) + → top_k output +``` + +### Known Gaps +1. **FakeEmbedding 无语义**: 向量搜索路输出随机排序，RRF 有一半输入是噪声 +2. **静态查询扩展**: `_INTENT_TERMS` 硬编码，"退款"↔"退钱"↔"返还费用" 无法覆盖 +3. **Reranker 弱**: 仅 embedding tiebreaker，cross-encoder 是空 TODO +4. **无意图感知排序**: 退款工单检索到物流文档不会被降权 +5. **无查询多样性**: 单一查询表达，语义变体覆盖不足 + +## Goal + +1. 实现 **HybridReranker**：多信号加权融合（RRF score + embedding similarity + intent metadata boost + content quality signal） +2. 实现 **MultiQueryExpander**：基于 LLM 生成 2-3 个查询变体，并行检索后合并去重 +3. 接入真实 embedding 作为默认（保留 FakeEmbedding 用于无网络测试） +4. 所有新信号记录到 RetrievalTrace，支持调试和评估 +5. 用现有 101 eval tickets 做 before/after 对比 + +## Non-goals + +- ❌ 不做 PageIndex 树搜索（TicketPilot 文档平均 500-1000 字，无需层级目录） +- ❌ 不做 cross-encoder reranker（需要 sentence-transformers 重依赖，与 uv 轻量原则冲突） +- ❌ 不改知识库 schema（不加文档摘要字段，留到下个迭代） +- ❌ 不改 DraftAgent 内部逻辑 +- ❌ 不做 embedding fine-tuning +- ❌ 不做生产部署 +- ❌ 不 commit API key + +## Key Design Decisions + +### A. Hybrid Reranker 信号融合 + +| 信号 | 权重范围 | 来源 | 说明 | +|------|----------|------|------| +| RRF score | 0.3-0.5 | 现有 | keyword + vector 融合排名 | +| Embedding similarity | 0.2-0.3 | 现有(需真实embedding) | query-document 语义相似度 | +| Intent metadata boost | 0.1-0.2 | 新增 | 意图分类→文档类型匹配加分 | +| Content quality signal | 0.05-0.1 | 新增 | 内容长度、关键词密度等启发式 | + +权重通过配置文件管理，支持 A/B 实验。 + +### B. Intent Metadata Boost 逻辑 + +| IntentClass | 优先 doc_type | 加分 | +|-------------|--------------|------| +| REFUND | policy, faq | +0.15 | +| RETURN_EXCHANGE | policy, faq | +0.15 | +| COMPLAINT | case, policy | +0.15 | +| TECHNICAL_ISSUE | faq, case | +0.1 | +| ACCOUNT_ISSUE | policy, faq | +0.1 | +| LOGISTICS | faq, case | +0.1 | +| PRODUCT_CONSULTING | faq | +0.15 | +| OTHER | (无加分) | 0 | + +### C. Multi-Query Expansion + +``` +原始查询: "我买的东西退款一直没到账" + ↓ LLM expansion (DeepSeek-chat, temperature=0.5) +扩展查询: ["退款到账时间", "退款进度查询", "退款未收到怎么办"] + ↓ 并行检索 (3路) + ↓ RRF merge + dedup + ↓ HybridReranker +最终 Top-K +``` + +- 每次扩展生成 2 个变体（控制 token 消耗） +- DeepSeek-chat 调用，单次 ~200 tokens，成本可忽略 +- 扩展失败时 graceful fallback 到原始查询 + +### D. 真实 Embedding 策略 + +| 决策 | 选择 | +|------|------| +| 默认 provider | `EMBEDDING_PROVIDER` env var 控制，默认 `openai_compatible` | +| 无网络 fallback | 自动降级到 FakeEmbeddingProvider + 日志警告 | +| 模型 | `text-embedding-3-small` (1536-dim) 或 DashScope `text-embedding-v4` (1024-dim) | +| 索引重建 | 维度变更时自动检测 + 提示重建 | + +## Proposed Metrics + +| Metric | Definition | Baseline Target | +|--------|-----------|----------------| +| Top-3 hit rate | Top-3 检索命中 golden expected doc | 提升 10%+ | +| Top-5 hit rate | Top-5 检索命中 | 提升 8%+ | +| MRR | Mean Reciprocal Rank | 提升 15%+ | +| Intent-aware precision | 检索结果 doc_type 与意图匹配率 | 新指标 | +| Reranker latency | 单次重排耗时 | < 50ms (无LLM) / < 500ms (含LLM) | +| Query expansion coverage | 扩展查询召回原始查询未命中的文档比例 | 新指标 | + +## Constraints + +- FakeEmbeddingProvider 必须保留为无网络环境的 fallback +- HybridReranker 权重必须可配置（支持 A/B） +- 多查询扩展的 LLM 调用失败必须 graceful fallback +- 所有新信号必须记录到 RetrievalTrace +- 现有 101 eval tickets 不修改 +- 质量门必须通过（ruff + tests + openspec） +- 无 API key commit diff --git a/openspec/changes/add-hybrid-retrieval-reranking/specs/hybrid-reranking/spec.md b/openspec/changes/add-hybrid-retrieval-reranking/specs/hybrid-reranking/spec.md new file mode 100644 index 0000000..b5bc101 --- /dev/null +++ b/openspec/changes/add-hybrid-retrieval-reranking/specs/hybrid-reranking/spec.md @@ -0,0 +1,159 @@ +# hybrid-reranking Specification + +## Purpose +Define the hybrid reranking system that combines multiple scoring signals (RRF score, embedding similarity, intent metadata boost, content quality) to produce higher-quality Top-K retrieval results for customer support ticket evidence retrieval. + +## Requirements + +### Requirement: HybridReranker multi-signal fusion +The system SHALL implement a HybridReranker that combines at least 4 scoring signals with configurable weights. + +#### Scenario: HybridReranker produces ranked results +- **WHEN** HybridReranker.rerank() is called with candidates, query, and config +- **THEN** returns a list of RerankResult sorted by final_score descending + +#### Scenario: Signal weights sum to 1.0 +- **WHEN** RerankerConfig is loaded +- **THEN** all signal weights sum to 1.0 (validated at load time) + +#### Scenario: Weight auto-adjustment on missing signals +- **WHEN** a signal is unavailable (e.g., fake embedding) +- **THEN** its weight is redistributed proportionally among available signals + +### Requirement: RRF Score Signal +The system SHALL use the existing RRF fusion score as the primary reranking signal. + +#### Scenario: RRF score normalization +- **WHEN** RRF scores are processed +- **THEN** scores are min-max normalized to [0, 1] range within the candidate set + +### Requirement: Embedding Similarity Signal +The system SHALL compute cosine similarity between query embedding and document embedding as a reranking signal. + +#### Scenario: Real embedding similarity +- **WHEN** real embedding provider is active +- **THEN** embedding_similarity signal contributes to final score per configured weight + +#### Scenario: Fake embedding auto-downgrade +- **WHEN** FakeEmbeddingProvider is detected +- **THEN** embedding_similarity weight is set to 0 and redistributed to other signals + +### Requirement: Intent Metadata Boost Signal +The system SHALL boost documents whose doc_type matches the classified intent. + +#### Scenario: Intent-doc_type match +- **WHEN** intent is REFUND and doc_type is "policy" +- **THEN** intent_metadata_boost score = 1.0 (boost applied) + +#### Scenario: Intent-doc_type mismatch +- **WHEN** intent is REFUND and doc_type is "case" +- **THEN** intent_metadata_boost score = 0.0 (no boost) + +#### Scenario: No intent available +- **WHEN** intent is None +- **THEN** intent_metadata_boost score = 0.0 for all candidates + +### Requirement: Content Quality Signal +The system SHALL score documents based on content length appropriateness and keyword density. + +#### Scenario: Optimal length content scores highest +- **WHEN** content length is between 200-800 characters +- **THEN** length_score is at its peak + +#### Scenario: Very short content scores lower +- **WHEN** content length < 50 characters +- **THEN** length_score is significantly below peak + +#### Scenario: Keyword density scoring +- **WHEN** query contains "退款" and document contains "退款" 3 times +- **THEN** keyword_density is higher than a document containing it 0 times + +### Requirement: MultiQueryExpander +The system SHALL generate query variants using LLM to improve recall. + +#### Scenario: Successful expansion +- **WHEN** expand("退款没到账", intent="refund") is called with LLM available +- **THEN** returns ["退款没到账", variant_1, variant_2] (3 queries total) + +#### Scenario: LLM failure fallback +- **WHEN** LLM call fails (timeout, error, no API key) +- **THEN** returns ["退款没到账"] (original query only) with warning logged + +#### Scenario: Variant quality control +- **WHEN** LLM returns variants longer than 50 characters or empty +- **THEN** invalid variants are filtered out + +### Requirement: ResultMerger +The system SHALL merge results from multiple query retrievals with deduplication. + +#### Scenario: Sum-score merge +- **WHEN** chunk X appears in query_1 results (score=0.3) and query_2 results (score=0.2) +- **THEN** merged score for chunk X = 0.5 (sum) + +#### Scenario: Deduplication +- **WHEN** same chunk_id appears in multiple result sets +- **THEN** only one entry in merged results with aggregated score + +#### Scenario: Empty result sets +- **WHEN** all result sets are empty +- **THEN** returns empty list + +### Requirement: RetrievalTrace extension +The system SHALL record all hybrid reranking signals in the retrieval trace. + +#### Scenario: Trace records query variants +- **WHEN** query expansion is used +- **THEN** trace.query_variants contains all query strings used + +#### Scenario: Trace records per-result signals +- **WHEN** hybrid reranking completes +- **THEN** trace.rerank_signals contains signal breakdown for each result + +#### Scenario: Trace records actual weights used +- **WHEN** weights are auto-adjusted +- **THEN** trace.reranker_weights reflects the adjusted weights + +#### Scenario: Trace records embedding provider status +- **WHEN** pipeline completes +- **THEN** trace.has_real_embedding indicates if real embedding was used + +### Requirement: Pipeline backward compatibility +The system SHALL maintain backward compatibility with existing callers. + +#### Scenario: Existing retrieve_evidence call without intent +- **WHEN** retrieve_evidence() is called without intent parameter +- **THEN** works identically to before (intent=None, no intent boost) + +#### Scenario: Existing hybrid_retrieval call without new params +- **WHEN** hybrid_retrieval() is called without enable_query_expansion and reranker_config +- **THEN** uses defaults (expansion enabled, default reranker config) + +### Requirement: RerankerConfig from YAML +The system SHALL support loading reranker configuration from YAML files. + +#### Scenario: Load default config +- **WHEN** RerankerConfig.default() is called +- **THEN** returns config with balanced weights (0.40, 0.25, 0.20, 0.15) + +#### Scenario: Load from YAML file +- **WHEN** RerankerConfig.from_yaml("config/reranker.yaml") is called +- **THEN** loads and validates weights from file + +#### Scenario: Invalid YAML weights +- **WHEN** YAML weights sum to 0.8 (not 1.0) +- **THEN** raises ValueError with descriptive message + +### Requirement: Graceful degradation +The system SHALL degrade gracefully when components are unavailable. + +#### Scenario: No LLM for query expansion +- **WHEN** no LLM API key configured +- **THEN** query expansion is skipped, pipeline continues with original query + +#### Scenario: No real embedding provider +- **WHEN** FakeEmbeddingProvider is the only available provider +- **THEN** reranker runs with embedding signal weight = 0, other signals adjusted + +#### Scenario: RerankerConfig file missing +- **WHEN** config/reranker.yaml does not exist +- **THEN** falls back to RerankerConfig.default() diff --git a/openspec/changes/add-hybrid-retrieval-reranking/tasks.md b/openspec/changes/add-hybrid-retrieval-reranking/tasks.md new file mode 100644 index 0000000..e9a0a4b --- /dev/null +++ b/openspec/changes/add-hybrid-retrieval-reranking/tasks.md @@ -0,0 +1,111 @@ +# Tasks: Hybrid Retrieval Reranking + +## Phase 1: RerankerConfig + 配置文件 (30 min) + +### Task 1.1: Create `reranker_config.py` +- [ ] `RerankerConfig` dataclass: weights, intent_boost_table, content_quality +- [ ] `ContentQualityConfig` dataclass: optimal_length_min/max, keyword_density_weight +- [ ] `RerankerConfig.default()` class method +- [ ] `RerankerConfig.from_yaml(path)` class method +- [ ] `validate()` 方法：校验权重总和 = 1.0 +- [ ] Unit tests: 默认配置、YAML 加载、权重校验 + +### Task 1.2: Create `config/reranker.yaml` +- [ ] 默认权重: rrf_score=0.40, embedding_similarity=0.25, intent_metadata_boost=0.20, content_quality=0.15 +- [ ] Intent boost 表: 8 个 IntentClass × 优先 doc_type +- [ ] Content quality 参数: optimal_length_min=200, optimal_length_max=800 +- [ ] Unit test: YAML 可解析且权重合法 + +## Phase 2: HybridReranker 核心 (45 min) + +### Task 2.1: Create `hybrid_reranker.py` — Signal 1 (RRF Score) +- [ ] `RerankSignal` dataclass: name, weight, raw_value, normalized_value, contribution +- [ ] `RerankResult` dataclass: chunk_id, final_score, signals, rank +- [ ] `HybridReranker.rerank()` 骨架 +- [ ] Signal 1: RRF score min-max normalization +- [ ] Unit test: 单信号 rerank 结果 = RRF 排序 + +### Task 2.2: Signal 2 (Embedding Similarity) +- [ ] cosine_similarity 计算（复用 reranker.py 现有函数） +- [ ] 从 DB 获取 doc embedding（复用 `_get_document_embedding`） +- [ ] FakeEmbedding 检测：自动降级（权重归零重分配） +- [ ] Unit test: 真实 embedding 时相似度参与打分；fake 时自动降级 + +### Task 2.3: Signal 3 (Intent Metadata Boost) +- [ ] `intent_boost_table` 查询逻辑 +- [ ] IntentClass → doc_type 匹配 → 加分 +- [ ] Unit test: 退款意图 + policy 文档 → +0.15；退款意图 + logistics 文档 → 0 + +### Task 2.4: Signal 4 (Content Quality) +- [ ] `length_score`: 正态分布曲线，optimal_length 中心峰值最高 +- [ ] `keyword_density`: 查询词在 content 中的命中比例 +- [ ] 两个子信号取平均作为 content_quality signal +- [ ] Unit test: 短/中/长内容得分差异；高/低关键词密度得分差异 + +### Task 2.5: Weight Auto-adjustment + Integration +- [ ] `_adjust_weights()`: 信号不可用时重新分配权重 +- [ ] 4 信号加权求和 → final_score +- [ ] 结果按 final_score 降序排列，赋 rank +- [ ] Unit test: 权重总和始终 = 1.0；4 信号融合正确 + +## Phase 3: MultiQueryExpander (30 min) + +### Task 3.1: Create `query_expander.py` +- [ ] `MultiQueryExpander` class +- [ ] LLM prompt 模板（中文，JSON 输出） +- [ ] `expand()` → `[original, variant_1, variant_2]` +- [ ] JSON 解析 + 校验（长度、数量） +- [ ] Fallback: LLM 失败 → `[original]` + 日志警告 +- [ ] Unit test: 正常扩展、LLM 失败 fallback、无 API key 跳过 + +## Phase 4: ResultMerger (20 min) + +### Task 4.1: Create `result_merger.py` +- [ ] `merge_retrieval_results(result_sets, strategy="sum_score")` +- [ ] `sum_score`: 同一 chunk_id 在多路结果中得分求和 +- [ ] `max_score`: 取最高 RRF score +- [ ] `rrf_again`: 对多路排名做二次 RRF +- [ ] Dedup by chunk_id，保留最高分版本的 content +- [ ] Unit test: 3 路结果合并去重；sum_score 加分逻辑 + +## Phase 5: Pipeline 集成 (30 min) + +### Task 5.1: 修改 `traces.py` +- [ ] 新增字段: query_variants, expansion_latency_ms, merged_result_count +- [ ] 新增字段: rerank_signals, reranker_weights, has_real_embedding +- [ ] 所有新字段 Optional，默认 None/0/False +- [ ] 现有测试不受影响 + +### Task 5.2: 修改 `pipeline.py` +- [ ] `hybrid_retrieval()` 新增参数: intent, enable_query_expansion, reranker_config +- [ ] Step 0: query expansion (if enabled) +- [ ] Step 1-3: 并行检索 per query variant +- [ ] Step 4: merge_retrieval_results +- [ ] Step 5: HybridReranker.rerank (替代现有 rerank_with_embeddings) +- [ ] Step 6: 构建扩展 trace +- [ ] 向后兼容：新参数都有默认值，现有调用不受影响 + +### Task 5.3: 修改 `retrieve_evidence.py` +- [ ] 新增 `intent` 参数传递到 `hybrid_retrieval()` +- [ ] 向后兼容：intent 默认 None + +### Task 5.4: 更新现有测试 +- [ ] `test_pipeline_retrieval.py`: 新增 hybrid rerank 路径测试 +- [ ] `test_retrieve_evidence.py`: 新增 intent 传递测试 +- [ ] 所有现有测试必须继续通过 + +## Phase 6: Evaluation + Report (30 min) + +### Task 6.1: Before/After 对比 +- [ ] 用现有 101 eval tickets 跑 before（当前 pipeline） +- [ ] 跑 after（hybrid reranker） +- [ ] 对比指标: Top-3 hit rate, Top-5 hit rate, MRR +- [ ] 输出 `reports/retrieval/hybrid_rerank_comparison.md` + +### Task 6.2: 质量门 +- [ ] `ruff check` 通过 +- [ ] `pytest` 全部通过 +- [ ] `openspec validate --all` 通过 +- [ ] Secret scan 通过 + +## Total Estimated Time: ~3 hours diff --git a/src/ticketpilot/retrieval/hybrid_reranker.py b/src/ticketpilot/retrieval/hybrid_reranker.py new file mode 100644 index 0000000..1fe4be8 --- /dev/null +++ b/src/ticketpilot/retrieval/hybrid_reranker.py @@ -0,0 +1,300 @@ +"""Hybrid reranker combining multiple scoring signals. + +Signals: +1. RRF score (from keyword + vector fusion) +2. Embedding similarity (cosine similarity, requires real embedding) +3. Intent metadata boost (intent -> doc_type matching) +4. Content quality (length appropriateness + keyword density) + +All signals are normalized to [0, 1] and combined with configurable weights. +""" +from __future__ import annotations + +import math +from dataclasses import dataclass, field +from typing import Any, Optional +from uuid import UUID + +from ticketpilot.retrieval.reranker_config import RerankerConfig +from ticketpilot.retrieval.traces import FusedResult + + +# --------------------------------------------------------------------------- +# Output dataclasses +# --------------------------------------------------------------------------- + +@dataclass +class RerankSignal: + """One scoring signal with its weight and raw/normalized values.""" + name: str + weight: float + raw_value: float + normalized_value: float + contribution: float # weight * normalized_value + + +@dataclass +class RerankResult: + """Reranked result with signal breakdown.""" + chunk_id: UUID + doc_id: UUID + doc_type: str + content: str + final_score: float + signals: list[RerankSignal] = field(default_factory=list) + rank: int = 0 + # Preserve original RRF info + rrf_score: float = 0.0 + keyword_rank: Optional[int] = None + keyword_contribution: Optional[float] = None + vector_rank: Optional[int] = None + vector_contribution: Optional[float] = None + sources: list[str] = field(default_factory=list) + + def to_fused_result(self) -> FusedResult: + """Convert back to FusedResult for downstream compatibility.""" + from ticketpilot.retrieval.schema.knowledge import DocType # noqa: PLC0415 + return FusedResult( + chunk_id=self.chunk_id, + doc_id=self.doc_id, + doc_type=DocType(self.doc_type) if isinstance(self.doc_type, str) else self.doc_type, + content=self.content, + rrf_score=self.rrf_score, + keyword_rank=self.keyword_rank, + keyword_contribution=self.keyword_contribution, + vector_rank=self.vector_rank, + vector_contribution=self.vector_contribution, + sources=self.sources + ["hybrid_rerank"], + ) + + +# --------------------------------------------------------------------------- +# Signal computations +# --------------------------------------------------------------------------- + +def _cosine_similarity(vec1: list[float], vec2: list[float]) -> float: + """Compute cosine similarity between two vectors.""" + dot = sum(a * b for a, b in zip(vec1, vec2)) + norm1 = math.sqrt(sum(a * a for a in vec1)) + norm2 = math.sqrt(sum(b * b for b in vec2)) + if norm1 == 0 or norm2 == 0: + return 0.0 + return dot / (norm1 * norm2) + + +def _length_score(length: int, opt_min: int, opt_max: int) -> float: + """Score content length on a bell curve centered on [opt_min, opt_max]. + + Returns 1.0 at the optimal midpoint, decaying for shorter/longer content. + """ + if length <= 0: + return 0.0 + midpoint = (opt_min + opt_max) / 2 + # Gaussian-like decay: sigma = (opt_max - opt_min) / 2 + sigma = max((opt_max - opt_min) / 2, 1) + return math.exp(-0.5 * ((length - midpoint) / sigma) ** 2) + + +def _keyword_density(query: str, content: str) -> float: + """Compute what fraction of query terms appear in content. + + Splits query by whitespace, checks each term's presence. + """ + terms = [t.strip() for t in query.split() if t.strip()] + if not terms or not content: + return 0.0 + hits = sum(1 for t in terms if t in content) + return hits / len(terms) + + +def _normalize_minmax(values: list[float]) -> list[float]: + """Min-max normalize a list of values to [0, 1].""" + if not values: + return [] + lo = min(values) + hi = max(values) + if hi - lo < 1e-12: + return [1.0] * len(values) + return [(v - lo) / (hi - lo) for v in values] + + +# --------------------------------------------------------------------------- +# HybridReranker +# --------------------------------------------------------------------------- + +class HybridReranker: + """Multi-signal reranker combining RRF, embedding, intent, and content signals.""" + + def __init__( + self, + config: RerankerConfig | None = None, + embedding_provider: Any | None = None, + ) -> None: + self._config = config or RerankerConfig.default() + self._embedding_provider = embedding_provider + + def rerank( + self, + candidates: list[FusedResult], + query: str, + query_embedding: list[float] | None = None, + intent: str | None = None, + top_k: int = 10, + ) -> list[RerankResult]: + """Rerank candidates using weighted multi-signal fusion. + + Args: + candidates: Fused results from RRF fusion. + query: Original query text (for keyword density). + query_embedding: Query embedding vector (for embedding similarity). + intent: Classified intent string (for intent boost). + top_k: Number of results to return. + + Returns: + Reranked list of RerankResult, sorted by final_score descending. + """ + if not candidates: + return [] + + # Determine which signals are available + has_embedding = ( + query_embedding is not None + and len(query_embedding) > 0 + and self._embedding_provider is not None + ) + # Check if using real (non-fake) embedding + is_real_embedding = has_embedding and _is_real_embedding_provider( + self._embedding_provider + ) + + unavailable: set[str] = set() + if not is_real_embedding: + unavailable.add("embedding_similarity") + + weights = self._config.adjust_weights_for_missing_signals(unavailable) + + # Compute raw signal values for all candidates + rrf_scores = [c.rrf_score for c in candidates] + norm_rrf = _normalize_minmax(rrf_scores) + + # Pre-compute doc embeddings if needed + doc_embeddings: dict[UUID, list[float]] = {} + if is_real_embedding: + doc_embeddings = self._load_doc_embeddings( + [c.chunk_id for c in candidates] + ) + + # Build rerank results + results: list[RerankResult] = [] + for i, cand in enumerate(candidates): + signals: list[RerankSignal] = [] + + # Signal 1: RRF score + w = weights.get("rrf_score", 0.0) + raw = rrf_scores[i] + norm = norm_rrf[i] + signals.append(RerankSignal( + name="rrf_score", weight=w, + raw_value=raw, normalized_value=norm, + contribution=w * norm, + )) + + # Signal 2: Embedding similarity + w = weights.get("embedding_similarity", 0.0) + if is_real_embedding and cand.chunk_id in doc_embeddings: + sim = _cosine_similarity(query_embedding, doc_embeddings[cand.chunk_id]) + else: + sim = 0.0 + signals.append(RerankSignal( + name="embedding_similarity", weight=w, + raw_value=sim, normalized_value=sim, # already in [0,1] + contribution=w * sim, + )) + + # Signal 3: Intent metadata boost + w = weights.get("intent_metadata_boost", 0.0) + boost = self._config.get_intent_boost(intent, cand.doc_type) + # Normalize: boost is already a small positive value, cap at 1.0 + norm_boost = min(boost, 1.0) + signals.append(RerankSignal( + name="intent_metadata_boost", weight=w, + raw_value=boost, normalized_value=norm_boost, + contribution=w * norm_boost, + )) + + # Signal 4: Content quality + w = weights.get("content_quality", 0.0) + cq = self._config.content_quality + len_score = _length_score( + len(cand.content), cq.optimal_length_min, cq.optimal_length_max + ) + kd = _keyword_density(query, cand.content) + content_score = ( + (1 - cq.keyword_density_weight) * len_score + + cq.keyword_density_weight * kd + ) + signals.append(RerankSignal( + name="content_quality", weight=w, + raw_value=content_score, normalized_value=content_score, + contribution=w * content_score, + )) + + # Final score + final = sum(s.contribution for s in signals) + + results.append(RerankResult( + chunk_id=cand.chunk_id, + doc_id=cand.doc_id, + doc_type=cand.doc_type.value if hasattr(cand.doc_type, 'value') else str(cand.doc_type), + content=cand.content, + final_score=final, + signals=signals, + rrf_score=cand.rrf_score, + keyword_rank=cand.keyword_rank, + keyword_contribution=cand.keyword_contribution, + vector_rank=cand.vector_rank, + vector_contribution=cand.vector_contribution, + sources=list(cand.sources), + )) + + # Sort by final_score descending + results.sort(key=lambda r: r.final_score, reverse=True) + + # Assign ranks + for i, r in enumerate(results[:top_k], 1): + r.rank = i + + return results[:top_k] + + def _load_doc_embeddings( + self, chunk_ids: list[UUID] + ) -> dict[UUID, list[float]]: + """Load document embeddings from DB for the given chunk IDs.""" + embeddings: dict[UUID, list[float]] = {} + try: + from ticketpilot.retrieval.db.connection import get_db_connection # noqa: PLC0415 + + with get_db_connection() as conn: + with conn.cursor() as cur: + for cid in chunk_ids: + cur.execute( + "SELECT embedding FROM knowledge_chunks WHERE id = %s", + (str(cid),), + ) + row = cur.fetchone() + if row and row[0]: + emb_str = row[0] + if isinstance(emb_str, str): + emb_str = emb_str.strip("[]") + embeddings[cid] = [float(x) for x in emb_str.split(",")] + elif isinstance(emb_str, list): + embeddings[cid] = [float(x) for x in emb_str] + except Exception: + pass # Graceful degradation + return embeddings + + +def _is_real_embedding_provider(provider: Any) -> bool: + """Check if the embedding provider is a real (non-fake) provider.""" + name = getattr(provider, "provider_name", "unknown") + return name not in ("fake", "unknown", "") diff --git a/src/ticketpilot/retrieval/pipeline.py b/src/ticketpilot/retrieval/pipeline.py index db4610a..01d1f78 100644 --- a/src/ticketpilot/retrieval/pipeline.py +++ b/src/ticketpilot/retrieval/pipeline.py @@ -1,17 +1,67 @@ -"""Hybrid retrieval pipeline combining keyword and vector search with RRF fusion.""" +"""Hybrid retrieval pipeline combining keyword and vector search with RRF fusion. +Enhanced with: +- Multi-query expansion (LLM-generated query variants) +- Hybrid reranking (multi-signal weighted fusion) +""" import time from typing import Optional from ticketpilot.retrieval.keyword_search import keyword_search from ticketpilot.retrieval.providers.fake_embedding import FakeEmbeddingProvider, get_fake_embedding_provider -from ticketpilot.retrieval.reranker import rerank_with_embeddings +from ticketpilot.retrieval.reranker_config import RerankerConfig +from ticketpilot.retrieval.hybrid_reranker import HybridReranker, RerankResult +from ticketpilot.retrieval.query_expander import MultiQueryExpander +from ticketpilot.retrieval.result_merger import merge_retrieval_results from ticketpilot.retrieval.rrf import DEFAULT_RRF_K, rrf_fusion from ticketpilot.retrieval.schema.knowledge import DocType -from ticketpilot.retrieval.traces import RetrievalTrace +from ticketpilot.retrieval.traces import FusedResult, RetrievalTrace from ticketpilot.retrieval.vector_search import get_hnsw_params, vector_search +def _single_query_retrieval( + query: str, + query_embedding: list[float], + top_k: int, + doc_types: Optional[list[DocType]], + exclude_business_domains: Optional[list[str]], + embedding_provider, + rrf_k: int, +) -> tuple[list[FusedResult], list, str, int, list, int]: + """Run keyword + vector + RRF for a single query. + + Returns (fused_results, keyword_results, search_method, keyword_latency, + vector_results, vector_latency). + """ + provider_name = getattr(embedding_provider, "provider_name", "unknown") + + # Keyword search + kw_start = time.perf_counter() + kw_results, kw_method = keyword_search( + query=query, + top_k=top_k * 2, + doc_types=doc_types, + exclude_business_domains=exclude_business_domains, + ) + kw_latency = int((time.perf_counter() - kw_start) * 1000) + + # Vector search + vec_start = time.perf_counter() + vec_results, _ = vector_search( + query_embedding=query_embedding, + top_k=top_k * 2, + doc_types=doc_types, + exclude_business_domains=exclude_business_domains, + embedding_provider_name=provider_name, + ) + vec_latency = int((time.perf_counter() - vec_start) * 1000) + + # RRF fusion + fused = rrf_fusion(keyword_results=kw_results, vector_results=vec_results, k=rrf_k) + + return fused, kw_results, kw_method, kw_latency, vec_results, vec_latency + + def hybrid_retrieval( query: str, top_k: int = 10, @@ -19,18 +69,21 @@ def hybrid_retrieval( exclude_business_domains: Optional[list[str]] = None, embedding_provider: Optional[FakeEmbeddingProvider] = None, rrf_k: int = DEFAULT_RRF_K, - enable_reranking: bool = True, # Enabled with improved strategy + enable_reranking: bool = True, + # New params for hybrid reranking (backward compatible) + intent: Optional[str] = None, + enable_query_expansion: bool = False, + reranker_config: Optional[RerankerConfig] = None, ) -> RetrievalTrace: """ Perform hybrid retrieval combining keyword and vector search. Pipeline: - 1. Generate query embedding using the embedding provider - 2. Run keyword search (FTS + LIKE fallback) - 3. Run vector search (HNSW) - 4. Fuse results using RRF - 5. Re-rank top results using embedding similarity (optional) - 6. Return complete trace for debugging and audit + 1. (Optional) Expand query into variants via LLM + 2. For each query variant: keyword search + vector search + RRF fusion + 3. Merge results from all variants (sum_score dedup) + 4. (Optional) Hybrid rerank with multi-signal fusion + 5. Return complete trace for debugging and audit Args: query: Search query string @@ -39,98 +92,177 @@ def hybrid_retrieval( embedding_provider: Embedding provider (default: FakeEmbeddingProvider) rrf_k: RRF k parameter (default: 60) enable_reranking: Enable re-ranking step (default: True) + intent: Classified intent string (for intent-aware reranking) + enable_query_expansion: Enable LLM-based query expansion (default: False) + reranker_config: Custom reranker config (default: from YAML or built-in) Returns: RetrievalTrace with complete pipeline information """ total_start_time = time.perf_counter() - # Use provided embedding provider or default (matching DB dimension) + # Use provided embedding provider or default if embedding_provider is None: - from ticketpilot.retrieval.vector_search import _detect_embedding_dim + from ticketpilot.retrieval.vector_search import _detect_embedding_dim # noqa: PLC0415 dim = _detect_embedding_dim() embedding_provider = get_fake_embedding_provider(dimension=dim) - # Generate query embedding + provider_name = getattr(embedding_provider, "provider_name", "unknown") + is_real = provider_name not in ("fake", "unknown", "") + + # Generate query embedding for the original query query_embedding = embedding_provider.embed(query) - # Keyword search - keyword_start = time.perf_counter() - keyword_results, keyword_search_method = keyword_search( - query=query, - top_k=top_k * 2, # Fetch more to account for fusion - doc_types=doc_types, - exclude_business_domains=exclude_business_domains, - ) - keyword_latency_ms = int((time.perf_counter() - keyword_start) * 1000) + # --- Step 0: Query Expansion --- + expansion_start = time.perf_counter() + query_variants = [query] + expansion_latency = 0 + if enable_query_expansion: + try: + expander = MultiQueryExpander() + query_variants = expander.expand(query, intent or "") + except Exception: + query_variants = [query] + expansion_latency = int((time.perf_counter() - expansion_start) * 1000) - # Get provider name for trace (handles both FakeEmbeddingProvider and others) - provider_name = getattr(embedding_provider, "provider_name", "unknown") + # --- Step 1-3: Per-query retrieval + RRF --- + all_fused: list[list[FusedResult]] = [] + # Use the first query's keyword/vector results for the trace + first_kw_results = [] + first_kw_method = "fts" + first_kw_latency = 0 + first_vec_results = [] + first_vec_latency = 0 - # Vector search - vector_start = time.perf_counter() - vector_results, vector_latency_ms = vector_search( - query_embedding=query_embedding, - top_k=top_k * 2, # Fetch more to account for fusion - doc_types=doc_types, - exclude_business_domains=exclude_business_domains, - embedding_provider_name=provider_name, - ) - vector_latency_ms = int((time.perf_counter() - vector_start) * 1000) - - # RRF Fusion - fusion_start = time.perf_counter() - fused_results = rrf_fusion( - keyword_results=keyword_results, - vector_results=vector_results, - k=rrf_k, - ) - fusion_latency_ms = int((time.perf_counter() - fusion_start) * 1000) + for i, q in enumerate(query_variants): + # Generate embedding for variant (reuse original for first query) + if i == 0: + q_emb = query_embedding + else: + q_emb = embedding_provider.embed(q) - # Re-ranking (optional) - rerank_latency_ms = 0 - if enable_reranking and fused_results: + fused, kw_res, kw_meth, kw_lat, vec_res, vec_lat = _single_query_retrieval( + query=q, + query_embedding=q_emb, + top_k=top_k, + doc_types=doc_types, + exclude_business_domains=exclude_business_domains, + embedding_provider=embedding_provider, + rrf_k=rrf_k, + ) + all_fused.append(fused) + + if i == 0: + first_kw_results = kw_res + first_kw_method = kw_meth + first_kw_latency = kw_lat + first_vec_results = vec_res + first_vec_latency = vec_lat + + # --- Step 4: Merge results from all query variants --- + merge_start = time.perf_counter() + if len(all_fused) > 1: + merged = merge_retrieval_results(all_fused, strategy="sum_score") + else: + merged = all_fused[0] if all_fused else [] + merge_latency = int((time.perf_counter() - merge_start) * 1000) + merged_count = len(merged) + + # --- Step 5: Reranking --- + rerank_latency = 0 + rerank_signals_data = None + reranker_weights_data = None + final_fused: list[FusedResult] = [] + + if enable_reranking and merged: rerank_start = time.perf_counter() - - # Take top 20 for re-ranking (more than final top_k) - candidates = fused_results[:20] - - # Re-rank using embedding similarity - fused_results = rerank_with_embeddings( + + # Load reranker config + if reranker_config is None: + try: + reranker_config = RerankerConfig.from_yaml("config/reranker.yaml") + except Exception: + reranker_config = RerankerConfig.default() + + # Take top candidates for reranking + candidates = merged[: max(top_k * 3, 20)] + + reranker = HybridReranker( + config=reranker_config, + embedding_provider=embedding_provider, + ) + reranked: list[RerankResult] = reranker.rerank( + candidates=candidates, + query=query, query_embedding=query_embedding, - fused_results=candidates, + intent=intent, top_k=top_k, - embedding_provider=embedding_provider, ) - rerank_latency_ms = int((time.perf_counter() - rerank_start) * 1000) + + # Convert RerankResult back to FusedResult for downstream compatibility + final_fused = [r.to_fused_result() for r in reranked] + + # Extract trace data + if reranked: + rerank_signals_data = [] + for r in reranked: + sig_data = { + "chunk_id": str(r.chunk_id), + "final_score": round(r.final_score, 6), + "signals": [ + { + "name": s.name, + "weight": round(s.weight, 4), + "raw": round(s.raw_value, 6), + "normalized": round(s.normalized_value, 6), + "contribution": round(s.contribution, 6), + } + for s in r.signals + ], + } + rerank_signals_data.append(sig_data) + + # Get actual weights from the first result's signals + if reranked[0].signals: + reranker_weights_data = { + s.name: round(s.weight, 4) for s in reranked[0].signals + } + + rerank_latency = int((time.perf_counter() - rerank_start) * 1000) else: - # Limit to top_k without re-ranking - fused_results = fused_results[:top_k] + final_fused = merged[:top_k] - final_evidence_ids = [r.chunk_id for r in fused_results] + final_evidence_ids = [r.chunk_id for r in final_fused] # Total latency - total_latency_ms = int((time.perf_counter() - total_start_time) * 1000) + total_latency = int((time.perf_counter() - total_start_time) * 1000) # Build trace trace = RetrievalTrace( query=query, query_embedding=query_embedding, - keyword_results=keyword_results, - keyword_latency_ms=keyword_latency_ms, - keyword_search_method=keyword_search_method, - vector_results=vector_results, - vector_latency_ms=vector_latency_ms, - fused_results=fused_results, - fusion_latency_ms=fusion_latency_ms, + keyword_results=first_kw_results, + keyword_latency_ms=first_kw_latency, + keyword_search_method=first_kw_method, + vector_results=first_vec_results, + vector_latency_ms=first_vec_latency, + fused_results=final_fused, + fusion_latency_ms=merge_latency, rrf_k=rrf_k, final_evidence_ids=final_evidence_ids, - total_latency_ms=total_latency_ms, + total_latency_ms=total_latency, embedding_provider=provider_name, hnsw_params=get_hnsw_params(), top_k=top_k, - rerank_latency_ms=rerank_latency_ms, + rerank_latency_ms=rerank_latency, reranking_enabled=enable_reranking, + # Hybrid reranking fields + query_variants=query_variants if len(query_variants) > 1 else None, + expansion_latency_ms=expansion_latency, + merged_result_count=merged_count, + rerank_signals=rerank_signals_data, + reranker_weights=reranker_weights_data, + has_real_embedding=is_real, ) return trace @@ -145,14 +277,6 @@ def simple_retrieval( Simple retrieval interface returning just content. Convenience function for cases where trace is not needed. - - Args: - query: Search query string - top_k: Maximum number of results - doc_types: Optional filter by document types - - Returns: - List of content strings for top-k results """ trace = hybrid_retrieval(query, top_k, doc_types) - return [r.content for r in trace.fused_results] \ No newline at end of file + return [r.content for r in trace.fused_results] diff --git a/src/ticketpilot/retrieval/query_expander.py b/src/ticketpilot/retrieval/query_expander.py new file mode 100644 index 0000000..bde8262 --- /dev/null +++ b/src/ticketpilot/retrieval/query_expander.py @@ -0,0 +1,136 @@ +"""Multi-query expansion using LLM for improved retrieval recall. + +Generates query variants by asking an LLM to rephrase the original query +from different semantic angles. Falls back to original query on failure. +""" +from __future__ import annotations + +import json +import logging +import os +import re + +logger = logging.getLogger(__name__) + +_EXPANSION_PROMPT = """\ +你是一个搜索查询优化器。给定一个客服工单查询，生成 {n} 个不同角度的搜索关键词变体。 +要求： +- 每个变体 5-15 个字 +- 覆盖不同语义角度（同义词、上位词、具体场景） +- 不要重复原始查询 +- 只输出 JSON 数组，不要解释 + +查询：{query} +意图：{intent} + +输出格式：["变体1", "变体2"] +""" + + +class MultiQueryExpander: + """Generate query variants using LLM for improved recall. + + Uses the same LLM endpoint as DraftAgent (configured via env vars). + Falls back to returning only the original query on any failure. + """ + + def __init__( + self, + num_variants: int = 2, + base_url: str | None = None, + api_key: str | None = None, + model: str | None = None, + timeout: int = 15, + ) -> None: + self._num_variants = num_variants + self._base_url = ( + base_url + or os.environ.get("TICKETPILOT_LLM_BASE_URL", "https://api.deepseek.com") + ).rstrip("/") + self._api_key = api_key or os.environ.get("TICKETPILOT_LLM_API_KEY", "") + self._model = model or os.environ.get("TICKETPILOT_LLM_MODEL", "deepseek-chat") + self._timeout = timeout + + def expand(self, query: str, intent: str = "") -> list[str]: + """Return [original_query, variant_1, variant_2, ...]. + + On any failure, returns [original_query] only. + """ + if not self._api_key: + logger.debug("No LLM API key, skipping query expansion") + return [query] + + try: + variants = self._call_llm(query, intent) + # Validate variants + valid = [v for v in variants if self._is_valid_variant(v, query)] + result = [query] + valid[: self._num_variants] + logger.info( + "Query expansion: '%s' -> %d variants: %s", + query, len(valid), valid[: self._num_variants], + ) + return result + except Exception as e: + logger.warning("Query expansion failed, using original: %s", e) + return [query] + + def _call_llm(self, query: str, intent: str) -> list[str]: + """Call LLM to generate query variants.""" + import urllib.request # noqa: PLC0415 + + prompt = _EXPANSION_PROMPT.format( + n=self._num_variants, query=query, intent=intent + ) + payload = { + "model": self._model, + "messages": [{"role": "user", "content": prompt}], + "max_tokens": 200, + "temperature": 0.5, + } + req = urllib.request.Request( + f"{self._base_url}/chat/completions", + data=json.dumps(payload).encode("utf-8"), + headers={ + "Authorization": f"Bearer {self._api_key}", + "Content-Type": "application/json", + }, + method="POST", + ) + with urllib.request.urlopen(req, timeout=self._timeout) as resp: + result = json.loads(resp.read().decode("utf-8")) + + content = result.get("choices", [{}])[0].get("message", {}).get("content", "") + return self._parse_variants(content) + + def _parse_variants(self, text: str) -> list[str]: + """Extract JSON array from LLM response.""" + # Try markdown code fence + match = re.search(r"```(?:json)?\s*\n?(.*?)\n?```", text, re.DOTALL) + if match: + try: + parsed = json.loads(match.group(1).strip()) + if isinstance(parsed, list): + return [str(v) for v in parsed] + except json.JSONDecodeError: + pass + + # Try raw JSON array + match = re.search(r"\[.*\]", text, re.DOTALL) + if match: + try: + parsed = json.loads(match.group(0)) + if isinstance(parsed, list): + return [str(v) for v in parsed] + except json.JSONDecodeError: + pass + + return [] + + def _is_valid_variant(self, variant: str, original: str) -> bool: + """Check if a variant is valid: non-empty, different from original, reasonable length.""" + v = variant.strip() + if not v or len(v) > 50: + return False + if v == original.strip(): + return False + return True diff --git a/src/ticketpilot/retrieval/reranker_config.py b/src/ticketpilot/retrieval/reranker_config.py new file mode 100644 index 0000000..5eaff48 --- /dev/null +++ b/src/ticketpilot/retrieval/reranker_config.py @@ -0,0 +1,150 @@ +"""Configuration for the hybrid reranker. + +Defines signal weights, intent boost tables, and content quality parameters. +Supports loading from YAML files for A/B experiments. +""" +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + + +@dataclass +class ContentQualityConfig: + """Parameters for the content quality scoring signal.""" + optimal_length_min: int = 200 + optimal_length_max: int = 800 + keyword_density_weight: float = 0.5 + + +@dataclass +class RerankerConfig: + """Configuration for the hybrid reranker. + + Attributes: + weights: Signal name -> weight mapping. Must sum to 1.0. + intent_boost_table: IntentClass value -> {doc_type: boost_value}. + content_quality: Content quality scoring parameters. + num_query_variants: Number of LLM-generated query variants. + """ + + weights: dict[str, float] = field(default_factory=dict) + intent_boost_table: dict[str, dict[str, float]] = field(default_factory=dict) + content_quality: ContentQualityConfig = field(default_factory=ContentQualityConfig) + num_query_variants: int = 2 + + # --- Validation --- + + def validate(self) -> None: + """Validate config. Raises ValueError on issues.""" + if not self.weights: + raise ValueError("weights cannot be empty") + total = sum(self.weights.values()) + if abs(total - 1.0) > 1e-6: + raise ValueError( + f"weights must sum to 1.0, got {total:.4f}: {self.weights}" + ) + for name, w in self.weights.items(): + if w < 0: + raise ValueError(f"weight '{name}' must be >= 0, got {w}") + + # --- Factories --- + + @classmethod + def default(cls) -> RerankerConfig: + """Default config with balanced weights.""" + cfg = cls( + weights={ + "rrf_score": 0.40, + "embedding_similarity": 0.25, + "intent_metadata_boost": 0.20, + "content_quality": 0.15, + }, + intent_boost_table={ + "refund": {"policy": 0.15, "faq": 0.10}, + "return_exchange": {"policy": 0.15, "faq": 0.10}, + "account_issue": {"policy": 0.10, "faq": 0.10}, + "technical_issue": {"faq": 0.10, "case": 0.10}, + "product_consulting": {"faq": 0.15}, + "logistics": {"faq": 0.10, "case": 0.10}, + "complaint": {"case": 0.15, "policy": 0.10}, + "other": {}, + }, + content_quality=ContentQualityConfig( + optimal_length_min=200, + optimal_length_max=800, + keyword_density_weight=0.5, + ), + num_query_variants=2, + ) + cfg.validate() + return cfg + + @classmethod + def from_yaml(cls, path: str | Path) -> RerankerConfig: + """Load config from a YAML file. + + Falls back to default if file not found. + """ + import yaml # noqa: PLC0415 + + path = Path(path) + if not path.exists(): + cfg = cls.default() + return cfg + + with path.open("r", encoding="utf-8") as f: + data: dict[str, Any] = yaml.safe_load(f) or {} + + weights = data.get("weights", {}) + intent_boost = data.get("intent_boost", {}) + cq_data = data.get("content_quality", {}) + num_variants = data.get("num_query_variants", 2) + + cq = ContentQualityConfig( + optimal_length_min=cq_data.get("optimal_length_min", 200), + optimal_length_max=cq_data.get("optimal_length_max", 800), + keyword_density_weight=cq_data.get("keyword_density_weight", 0.5), + ) + + cfg = cls( + weights=weights, + intent_boost_table=intent_boost, + content_quality=cq, + num_query_variants=num_variants, + ) + cfg.validate() + return cfg + + # --- Helpers --- + + def get_intent_boost(self, intent: str | None, doc_type: str) -> float: + """Get boost value for an intent+doc_type combination.""" + if intent is None: + return 0.0 + # Normalize doc_type to lowercase for table lookup + dt_lower = doc_type.lower() if doc_type else "" + return self.intent_boost_table.get(intent, {}).get(dt_lower, 0.0) + + def adjust_weights_for_missing_signals( + self, unavailable_signals: set[str] + ) -> dict[str, float]: + """Redistribute weights when signals are unavailable. + + Returns a new weight dict with unavailable signals removed + and remaining weights renormalized to sum to 1.0. + """ + if not unavailable_signals: + return dict(self.weights) + + available = { + k: v for k, v in self.weights.items() if k not in unavailable_signals + } + total = sum(available.values()) + if total <= 0: + # Fallback: equal weight on all available + n = len(available) or 1 + return {k: 1.0 / n for k in available} + + return {k: v / total for k, v in available.items()} diff --git a/src/ticketpilot/retrieval/result_merger.py b/src/ticketpilot/retrieval/result_merger.py new file mode 100644 index 0000000..716c7bb --- /dev/null +++ b/src/ticketpilot/retrieval/result_merger.py @@ -0,0 +1,146 @@ +"""Merge retrieval results from multiple query variants. + +Supports three merge strategies: +- max_score: keep highest RRF score per chunk_id +- sum_score: sum RRF scores across queries (boosts docs found by multiple queries) +- rrf_again: treat each query as a ranker, apply second-level RRF +""" +from __future__ import annotations + +from collections import defaultdict +from uuid import UUID + +from ticketpilot.retrieval.traces import FusedResult + + +def merge_retrieval_results( + result_sets: list[list[FusedResult]], + strategy: str = "sum_score", +) -> list[FusedResult]: + """Merge multiple retrieval result sets, deduplicating by chunk_id. + + Args: + result_sets: List of FusedResult lists, one per query variant. + strategy: Merge strategy - "max_score", "sum_score", or "rrf_again". + + Returns: + Merged and deduplicated list of FusedResult, sorted by score descending. + """ + if not result_sets: + return [] + + # Flatten and filter empty sets + non_empty = [rs for rs in result_sets if rs] + if not non_empty: + return [] + if len(non_empty) == 1: + return list(non_empty[0]) + + if strategy == "sum_score": + return _merge_sum_score(non_empty) + elif strategy == "max_score": + return _merge_max_score(non_empty) + elif strategy == "rrf_again": + return _merge_rrf_again(non_empty) + else: + return _merge_sum_score(non_empty) + + +def _merge_sum_score( + result_sets: list[list[FusedResult]], +) -> list[FusedResult]: + """Sum RRF scores for the same chunk_id across query variants. + + Docs found by multiple queries get higher scores (multi-path validation). + """ + best: dict[UUID, FusedResult] = {} + score_sums: dict[UUID, float] = defaultdict(float) + + for result_set in result_sets: + for r in result_set: + score_sums[r.chunk_id] += r.rrf_score + # Keep the version with most info (prefer one with both keyword+vector) + if r.chunk_id not in best or len(r.sources) > len(best[r.chunk_id].sources): + best[r.chunk_id] = r + + # Build merged results with summed scores + merged: list[FusedResult] = [] + for cid, representative in best.items(): + merged.append(FusedResult( + chunk_id=cid, + doc_id=representative.doc_id, + doc_type=representative.doc_type, + content=representative.content, + rrf_score=score_sums[cid], + keyword_rank=representative.keyword_rank, + keyword_contribution=representative.keyword_contribution, + vector_rank=representative.vector_rank, + vector_contribution=representative.vector_contribution, + sources=representative.sources + ["multi_query"], + )) + + merged.sort(key=lambda r: r.rrf_score, reverse=True) + return merged + + +def _merge_max_score( + result_sets: list[list[FusedResult]], +) -> list[FusedResult]: + """Keep the highest RRF score per chunk_id.""" + best: dict[UUID, FusedResult] = {} + + for result_set in result_sets: + for r in result_set: + if r.chunk_id not in best or r.rrf_score > best[r.chunk_id].rrf_score: + best[r.chunk_id] = r + + merged = list(best.values()) + merged.sort(key=lambda r: r.rrf_score, reverse=True) + return merged + + +def _merge_rrf_again( + result_sets: list[list[FusedResult]], +) -> list[FusedResult]: + """Apply second-level RRF: treat each query variant as a ranker. + + Uses RRF k=60 on the rank positions within each query's results. + """ + k = 60 + # Build per-query rank maps + rank_maps: list[dict[UUID, int]] = [] + representative: dict[UUID, FusedResult] = {} + + for result_set in result_sets: + rank_map: dict[UUID, int] = {} + for i, r in enumerate(result_set, 1): + rank_map[r.chunk_id] = i + if r.chunk_id not in representative: + representative[r.chunk_id] = r + rank_maps.append(rank_map) + + # Compute second-level RRF scores + rrf_scores: dict[UUID, float] = defaultdict(float) + for rm in rank_maps: + for cid, rank in rm.items(): + rrf_scores[cid] += 1.0 / (k + rank) + + # Build merged results + merged: list[FusedResult] = [] + for cid, score in rrf_scores.items(): + rep = representative[cid] + merged.append(FusedResult( + chunk_id=cid, + doc_id=rep.doc_id, + doc_type=rep.doc_type, + content=rep.content, + rrf_score=score, + keyword_rank=rep.keyword_rank, + keyword_contribution=rep.keyword_contribution, + vector_rank=rep.vector_rank, + vector_contribution=rep.vector_contribution, + sources=rep.sources + ["rrf_again"], + )) + + merged.sort(key=lambda r: r.rrf_score, reverse=True) + return merged diff --git a/src/ticketpilot/retrieval/retrieve_evidence.py b/src/ticketpilot/retrieval/retrieve_evidence.py index f193c67..5692777 100644 --- a/src/ticketpilot/retrieval/retrieve_evidence.py +++ b/src/ticketpilot/retrieval/retrieve_evidence.py @@ -6,6 +6,7 @@ from ticketpilot.retrieval.pipeline import hybrid_retrieval from ticketpilot.retrieval.providers.fake_embedding import FakeEmbeddingProvider from ticketpilot.retrieval.query_builder import build_retrieval_query +from ticketpilot.retrieval.reranker_config import RerankerConfig from ticketpilot.retrieval.schema.knowledge import DocType from ticketpilot.retrieval.traces import RetrievalTrace from ticketpilot.schema.evidence import EvidenceCandidate @@ -19,14 +20,26 @@ def retrieve_evidence( top_k: int = 10, doc_types: list[DocType] | None = None, embedding_provider: Optional[FakeEmbeddingProvider] = None, + # New params for hybrid reranking (backward compatible) + enable_query_expansion: bool = False, + reranker_config: Optional[RerankerConfig] = None, ) -> tuple[list[EvidenceCandidate], RetrievalTrace]: """Retrieve evidence candidates from the knowledge base. Constructs a retrieval query from ticket state, runs hybrid - retrieval, and maps fused results to evidence candidates. + retrieval with optional query expansion and hybrid reranking, + and maps fused results to evidence candidates. Always returns a RetrievalTrace, even when no results are found. """ query = build_retrieval_query(normalized_text, intent, risk_flags) - trace = hybrid_retrieval(query=query, top_k=top_k, doc_types=doc_types, embedding_provider=embedding_provider) + trace = hybrid_retrieval( + query=query, + top_k=top_k, + doc_types=doc_types, + embedding_provider=embedding_provider, + intent=intent.value if intent else None, + enable_query_expansion=enable_query_expansion, + reranker_config=reranker_config, + ) candidates = map_fused_to_evidence(trace.fused_results) return candidates, trace diff --git a/src/ticketpilot/retrieval/traces.py b/src/ticketpilot/retrieval/traces.py index af25f29..ff17bd2 100644 --- a/src/ticketpilot/retrieval/traces.py +++ b/src/ticketpilot/retrieval/traces.py @@ -1,6 +1,6 @@ """Retrieval trace schema for debugging and explainability.""" -from datetime import datetime, timezone, timezone +from datetime import datetime, timezone from typing import Any, Optional from uuid import UUID @@ -191,6 +191,34 @@ class RetrievalTrace(BaseModel): description="Whether re-ranking was enabled", ) + # Hybrid reranking metadata + query_variants: Optional[list[str]] = Field( + default=None, + description="Query variants used for multi-query expansion", + ) + expansion_latency_ms: int = Field( + default=0, + ge=0, + description="Query expansion latency in milliseconds", + ) + merged_result_count: int = Field( + default=0, + ge=0, + description="Number of results after merge+dedup, before reranking", + ) + rerank_signals: Optional[list[dict[str, Any]]] = Field( + default=None, + description="Per-result signal breakdown from hybrid reranker", + ) + reranker_weights: Optional[dict[str, float]] = Field( + default=None, + description="Actual weights used by hybrid reranker (may be adjusted)", + ) + has_real_embedding: bool = Field( + default=False, + description="Whether a real (non-fake) embedding provider was used", + ) + def get_result_by_chunk_id(self, chunk_id: UUID) -> Optional[FusedResult]: """Get fused result by chunk ID.""" for result in self.fused_results: diff --git a/tests/unit/test_hybrid_reranker.py b/tests/unit/test_hybrid_reranker.py new file mode 100644 index 0000000..cc8e083 --- /dev/null +++ b/tests/unit/test_hybrid_reranker.py @@ -0,0 +1,169 @@ +"""Unit tests for HybridReranker.""" +import math +from uuid import uuid4 + +import pytest + +from ticketpilot.retrieval.hybrid_reranker import ( + HybridReranker, + RerankResult, + _cosine_similarity, + _keyword_density, + _length_score, + _normalize_minmax, +) +from ticketpilot.retrieval.reranker_config import RerankerConfig +from ticketpilot.retrieval.schema.knowledge import DocType +from ticketpilot.retrieval.traces import FusedResult + + +def _make_fused( + chunk_id=None, doc_type="FAQ", content="test content", rrf_score=0.5 +) -> FusedResult: + return FusedResult( + chunk_id=chunk_id or uuid4(), + doc_id=uuid4(), + doc_type=DocType(doc_type), + content=content, + rrf_score=rrf_score, + keyword_rank=1, + keyword_contribution=0.016, + vector_rank=2, + vector_contribution=0.015, + sources=["keyword", "vector"], + ) + + +class TestCosineSimilarity: + def test_identical_vectors(self): + v = [1.0, 0.0, 0.0] + assert abs(_cosine_similarity(v, v) - 1.0) < 1e-9 + + def test_orthogonal_vectors(self): + assert abs(_cosine_similarity([1, 0], [0, 1])) < 1e-9 + + def test_opposite_vectors(self): + assert abs(_cosine_similarity([1, 0], [-1, 0]) - (-1.0)) < 1e-9 + + def test_zero_vector(self): + assert _cosine_similarity([0, 0], [1, 0]) == 0.0 + + +class TestLengthScore: + def test_optimal_length_scores_highest(self): + score = _length_score(500, 200, 800) + assert score > 0.9 + + def test_very_short_scores_low(self): + score = _length_score(10, 200, 800) + assert score < 0.3 + + def test_very_long_scores_lower(self): + score_optimal = _length_score(500, 200, 800) + score_long = _length_score(3000, 200, 800) + assert score_optimal > score_long + + def test_zero_length(self): + assert _length_score(0, 200, 800) == 0.0 + + +class TestKeywordDensity: + def test_all_terms_present(self): + assert _keyword_density("退款 到账", "退款一直没到账怎么办") == 1.0 + + def test_partial_terms(self): + assert _keyword_density("退款 到账 物流", "退款政策说明") == pytest.approx(1 / 3) + + def test_no_terms(self): + assert _keyword_density("退款", "物流发货说明") == 0.0 + + def test_empty_query(self): + assert _keyword_density("", "some content") == 0.0 + + +class TestNormalizeMinMax: + def test_uniform_values(self): + result = _normalize_minmax([5, 5, 5]) + assert result == [1.0, 1.0, 1.0] + + def test_normal_range(self): + result = _normalize_minmax([0, 5, 10]) + assert result[0] == 0.0 + assert result[1] == pytest.approx(0.5) + assert result[2] == 1.0 + + def test_empty(self): + assert _normalize_minmax([]) == [] + + +class TestHybridReranker: + def test_empty_candidates(self): + reranker = HybridReranker() + assert reranker.rerank([], "test") == [] + + def test_single_candidate(self): + c = _make_fused(rrf_score=0.5, content="退款政策说明 退款条件") + reranker = HybridReranker() + results = reranker.rerank([c], "退款", intent="refund", top_k=5) + assert len(results) == 1 + assert results[0].rank == 1 + assert results[0].final_score > 0 + + def test_ranking_order(self): + c1 = _make_fused(rrf_score=0.3, content="退款政策说明") + c2 = _make_fused(rrf_score=0.8, content="退款退款退款退款退款") + reranker = HybridReranker() + results = reranker.rerank([c1, c2], "退款", top_k=10) + assert len(results) == 2 + # c2 has higher RRF + more keyword hits, should rank first + assert results[0].chunk_id == c2.chunk_id + assert results[0].rank == 1 + + def test_intent_boost_effect(self): + policy_doc = _make_fused( + doc_type="POLICY", rrf_score=0.3, content="退款政策 退款条件" + ) + case_doc = _make_fused( + doc_type="CASE", rrf_score=0.3, content="退款案例 退款处理" + ) + reranker = HybridReranker() + results = reranker.rerank( + [policy_doc, case_doc], "退款", intent="refund", top_k=10 + ) + # Policy doc should rank higher due to intent boost for refund→policy + policy_result = next(r for r in results if r.chunk_id == policy_doc.chunk_id) + case_result = next(r for r in results if r.chunk_id == case_doc.chunk_id) + assert policy_result.final_score > case_result.final_score + + def test_signals_recorded(self): + c = _make_fused(content="退款政策说明 退款流程") + reranker = HybridReranker() + results = reranker.rerank([c], "退款", top_k=5) + assert len(results[0].signals) == 4 + signal_names = {s.name for s in results[0].signals} + assert signal_names == { + "rrf_score", "embedding_similarity", + "intent_metadata_boost", "content_quality", + } + + def test_fake_embedding_weight_redistribution(self): + """With fake embedding provider, embedding weight should be 0.""" + cfg = RerankerConfig.default() + # No embedding_provider = fake + reranker = HybridReranker(config=cfg, embedding_provider=None) + c = _make_fused(content="test content") + results = reranker.rerank([c], "test", top_k=5) + # embedding_similarity signal should have weight=0 + emb_signal = next( + s for s in results[0].signals if s.name == "embedding_similarity" + ) + assert emb_signal.weight == 0.0 + + def test_to_fused_result_conversion(self): + c = _make_fused(content="退款政策") + reranker = HybridReranker() + results = reranker.rerank([c], "退款", top_k=5) + fused = results[0].to_fused_result() + assert isinstance(fused, FusedResult) + assert "hybrid_rerank" in fused.sources + assert fused.chunk_id == c.chunk_id diff --git a/tests/unit/test_query_expander.py b/tests/unit/test_query_expander.py new file mode 100644 index 0000000..ed23696 --- /dev/null +++ b/tests/unit/test_query_expander.py @@ -0,0 +1,67 @@ +"""Unit tests for MultiQueryExpander.""" +import json +from unittest.mock import MagicMock, patch + +import pytest + +from ticketpilot.retrieval.query_expander import MultiQueryExpander + + +class TestMultiQueryExpander: + def test_no_api_key_returns_original(self, monkeypatch): + monkeypatch.delenv("TICKETPILOT_LLM_API_KEY", raising=False) + expander = MultiQueryExpander(api_key="") + result = expander.expand("退款没到账", "refund") + assert result == ["退款没到账"] + + def test_parse_json_array(self): + expander = MultiQueryExpander(api_key="fake") + variants = expander._parse_variants('["退款进度", "退款到账时间"]') + assert variants == ["退款进度", "退款到账时间"] + + def test_parse_markdown_fence(self): + expander = MultiQueryExpander(api_key="fake") + text = '```json\n["退款进度", "退款到账时间"]\n```' + variants = expander._parse_variants(text) + assert variants == ["退款进度", "退款到账时间"] + + def test_parse_invalid_returns_empty(self): + expander = MultiQueryExpander(api_key="fake") + assert expander._parse_variants("not json") == [] + + def test_is_valid_variant_normal(self): + expander = MultiQueryExpander(api_key="fake") + assert expander._is_valid_variant("退款进度查询", "退款没到账") + + def test_is_valid_variant_empty(self): + expander = MultiQueryExpander(api_key="fake") + assert not expander._is_valid_variant("", "query") + + def test_is_valid_variant_same_as_original(self): + expander = MultiQueryExpander(api_key="fake") + assert not expander._is_valid_variant("退款没到账", "退款没到账") + + def test_is_valid_variant_too_long(self): + expander = MultiQueryExpander(api_key="fake") + assert not expander._is_valid_variant("a" * 51, "query") + + @patch("ticketpilot.retrieval.query_expander.MultiQueryExpander._call_llm") + def test_expand_success(self, mock_llm): + mock_llm.return_value = ["退款进度", "退款到账时间"] + expander = MultiQueryExpander(api_key="fake-key") + result = expander.expand("退款没到账", "refund") + assert result == ["退款没到账", "退款进度", "退款到账时间"] + + @patch("ticketpilot.retrieval.query_expander.MultiQueryExpander._call_llm") + def test_expand_llm_failure_fallback(self, mock_llm): + mock_llm.side_effect = RuntimeError("API error") + expander = MultiQueryExpander(api_key="fake-key") + result = expander.expand("退款没到账", "refund") + assert result == ["退款没到账"] + + @patch("ticketpilot.retrieval.query_expander.MultiQueryExpander._call_llm") + def test_expand_filters_invalid_variants(self, mock_llm): + mock_llm.return_value = ["", "a" * 51, "有效变体"] + expander = MultiQueryExpander(api_key="fake-key") + result = expander.expand("退款没到账") + assert result == ["退款没到账", "有效变体"] diff --git a/tests/unit/test_reranker_config.py b/tests/unit/test_reranker_config.py new file mode 100644 index 0000000..8170b73 --- /dev/null +++ b/tests/unit/test_reranker_config.py @@ -0,0 +1,101 @@ +"""Unit tests for RerankerConfig.""" +import tempfile +from pathlib import Path + +import pytest + +from ticketpilot.retrieval.reranker_config import ContentQualityConfig, RerankerConfig + + +class TestRerankerConfigValidation: + def test_default_config_is_valid(self): + cfg = RerankerConfig.default() + cfg.validate() # should not raise + + def test_weights_sum_to_one(self): + cfg = RerankerConfig.default() + total = sum(cfg.weights.values()) + assert abs(total - 1.0) < 1e-6 + + def test_empty_weights_raises(self): + cfg = RerankerConfig(weights={}) + with pytest.raises(ValueError, match="weights cannot be empty"): + cfg.validate() + + def test_weights_not_summing_raises(self): + cfg = RerankerConfig(weights={"a": 0.3, "b": 0.3}) + with pytest.raises(ValueError, match="must sum to 1.0"): + cfg.validate() + + def test_negative_weight_raises(self): + cfg = RerankerConfig(weights={"a": 1.5, "b": -0.5}) + with pytest.raises(ValueError, match="must be >= 0"): + cfg.validate() + + +class TestRerankerConfigFromYaml: + def test_load_from_yaml(self, tmp_path): + yaml_content = """\ +weights: + rrf_score: 0.50 + embedding_similarity: 0.30 + intent_metadata_boost: 0.10 + content_quality: 0.10 + +intent_boost: + refund: + policy: 0.20 + +content_quality: + optimal_length_min: 100 + optimal_length_max: 600 + keyword_density_weight: 0.3 + +num_query_variants: 3 +""" + p = tmp_path / "reranker.yaml" + p.write_text(yaml_content) + cfg = RerankerConfig.from_yaml(str(p)) + assert cfg.weights["rrf_score"] == 0.50 + assert cfg.num_query_variants == 3 + assert cfg.content_quality.optimal_length_min == 100 + + def test_missing_file_returns_default(self): + cfg = RerankerConfig.from_yaml("/nonexistent/path.yaml") + assert cfg.weights == RerankerConfig.default().weights + + +class TestRerankerConfigHelpers: + def test_get_intent_boost_match(self): + cfg = RerankerConfig.default() + boost = cfg.get_intent_boost("refund", "policy") + assert boost == 0.15 + + def test_get_intent_boost_no_match(self): + cfg = RerankerConfig.default() + boost = cfg.get_intent_boost("refund", "case") + assert boost == 0.0 + + def test_get_intent_boost_none_intent(self): + cfg = RerankerConfig.default() + boost = cfg.get_intent_boost(None, "policy") + assert boost == 0.0 + + def test_adjust_weights_no_missing(self): + cfg = RerankerConfig.default() + adjusted = cfg.adjust_weights_for_missing_signals(set()) + assert adjusted == cfg.weights + + def test_adjust_weights_embedding_missing(self): + cfg = RerankerConfig.default() + adjusted = cfg.adjust_weights_for_missing_signals({"embedding_similarity"}) + assert "embedding_similarity" not in adjusted + total = sum(adjusted.values()) + assert abs(total - 1.0) < 1e-6 + + def test_adjust_weights_preserves_proportions(self): + cfg = RerankerConfig.default() + adjusted = cfg.adjust_weights_for_missing_signals({"embedding_similarity"}) + # rrf was 0.40, now should be 0.40/0.75 ≈ 0.533 + expected_ratio = 0.40 / 0.75 + assert abs(adjusted["rrf_score"] - expected_ratio) < 1e-6 diff --git a/tests/unit/test_result_merger.py b/tests/unit/test_result_merger.py new file mode 100644 index 0000000..262e2ca --- /dev/null +++ b/tests/unit/test_result_merger.py @@ -0,0 +1,81 @@ +"""Unit tests for result_merger.""" +from uuid import uuid4 + +import pytest + +from ticketpilot.retrieval.result_merger import merge_retrieval_results +from ticketpilot.retrieval.schema.knowledge import DocType +from ticketpilot.retrieval.traces import FusedResult + + +def _fused(chunk_id=None, rrf_score=0.5, content="test", sources=None): + return FusedResult( + chunk_id=chunk_id or uuid4(), + doc_id=uuid4(), + doc_type=DocType.FAQ, + content=content, + rrf_score=rrf_score, + keyword_rank=1, + keyword_contribution=0.016, + sources=sources or ["keyword"], + ) + + +class TestMergeRetrievalResults: + def test_empty_input(self): + assert merge_retrieval_results([]) == [] + + def test_all_empty_sets(self): + assert merge_retrieval_results([[], []]) == [] + + def test_single_set_passthrough(self): + r = _fused() + result = merge_retrieval_results([[r]]) + assert len(result) == 1 + assert result[0].chunk_id == r.chunk_id + + def test_sum_score_dedup(self): + cid = uuid4() + r1 = _fused(chunk_id=cid, rrf_score=0.3, sources=["keyword"]) + r2 = _fused(chunk_id=cid, rrf_score=0.2, sources=["vector"]) + merged = merge_retrieval_results([[r1], [r2]], strategy="sum_score") + assert len(merged) == 1 + assert merged[0].rrf_score == pytest.approx(0.5) + + def test_sum_score_different_chunks(self): + c1 = uuid4() + c2 = uuid4() + r1 = _fused(chunk_id=c1, rrf_score=0.3) + r2 = _fused(chunk_id=c2, rrf_score=0.5) + merged = merge_retrieval_results([[r1], [r2]], strategy="sum_score") + assert len(merged) == 2 + # c2 should rank first (higher score) + assert merged[0].chunk_id == c2 + + def test_max_score_strategy(self): + cid = uuid4() + r1 = _fused(chunk_id=cid, rrf_score=0.3) + r2 = _fused(chunk_id=cid, rrf_score=0.7) + merged = merge_retrieval_results([[r1], [r2]], strategy="max_score") + assert len(merged) == 1 + assert merged[0].rrf_score == pytest.approx(0.7) + + def test_rrf_again_strategy(self): + c1 = uuid4() + c2 = uuid4() + # c1 ranked #1 in both queries, c2 ranked #2 + r1q1 = _fused(chunk_id=c1, rrf_score=0.5) + r1q2 = _fused(chunk_id=c1, rrf_score=0.4) + r2q1 = _fused(chunk_id=c2, rrf_score=0.3) + r2q2 = _fused(chunk_id=c2, rrf_score=0.6) + merged = merge_retrieval_results( + [[r1q1, r2q1], [r1q2, r2q2]], strategy="rrf_again" + ) + assert len(merged) == 2 + # c1 ranked higher in both, should be first + assert merged[0].chunk_id == c1 + + def test_multi_query_marker(self): + r = _fused() + merged = merge_retrieval_results([[r], [r]]) + assert "multi_query" in merged[0].sources From 127c9eef5ed40529b346187084c80263bba59941 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 8 Jun 2026 12:28:45 +0000 Subject: [PATCH 3/7] =?UTF-8?q?fix:=20context=20management=20=E2=80=94=20e?= =?UTF-8?q?vidence=20cap=20+=20LLM-guided=20search=20merge?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. Evidence list capped at _MAX_EVIDENCE=15, sorted by score. Prevents unbounded context growth across reformulate iterations. 2. _llm_guided_search now MERGES with existing evidence instead of replacing it. Same dedup-by-chunk_id pattern as _reformulate_search. Previously, LLM-guided search would discard all prior good evidence. 3. Both paths now log total evidence count after modification. --- src/ticketpilot/drafting/draft_agent.py | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/src/ticketpilot/drafting/draft_agent.py b/src/ticketpilot/drafting/draft_agent.py index 51efc9a..8e9a662 100644 --- a/src/ticketpilot/drafting/draft_agent.py +++ b/src/ticketpilot/drafting/draft_agent.py @@ -39,6 +39,8 @@ # Minimum RRF score threshold for evidence to be considered "good" _EVIDENCE_SCORE_THRESHOLD = 0.01 +# Maximum evidence items to keep (prevents unbounded context growth) +_MAX_EVIDENCE = 15 # Maximum agent loop iterations (safety bound) _MAX_ITERATIONS = 5 # Safe fallback when agent cannot produce a grounded reply @@ -790,9 +792,14 @@ def _reformulate_search( state.evidence.append(c) existing_ids.add(c.chunk_id) + # Cap evidence by score to prevent unbounded context growth + state.evidence.sort(key=lambda e: e.score, reverse=True) + state.evidence = state.evidence[:_MAX_EVIDENCE] + logger.info( - "DraftAgent: reformulated search added %d new results", + "DraftAgent: reformulated search added %d new results (total %d)", len(new_candidates), + len(state.evidence), ) def _llm_guided_search( @@ -832,9 +839,19 @@ def _llm_guided_search( if query and query not in state.search_queries_used: state.search_queries_used.append(query) raw_results = _search_knowledge(query) - state.evidence = self._raw_results_to_candidates(raw_results) + # Merge with existing evidence (don't replace — preserve good earlier results) + new_candidates = self._raw_results_to_candidates(raw_results) + existing_ids = {c.chunk_id for c in state.evidence} + for c in new_candidates: + if c.chunk_id not in existing_ids: + state.evidence.append(c) + existing_ids.add(c.chunk_id) + # Cap by score + state.evidence.sort(key=lambda e: e.score, reverse=True) + state.evidence = state.evidence[:_MAX_EVIDENCE] logger.info( - "DraftAgent: LLM-guided search returned %d results", + "DraftAgent: LLM-guided search added %d new results (total %d)", + len(new_candidates), len(state.evidence), ) except Exception as e: From 9283af09b5a81dedd107a802018eb6c2af1073a0 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 8 Jun 2026 13:40:22 +0000 Subject: [PATCH 4/7] =?UTF-8?q?fix:=20OCR=20review=20findings=20=E2=80=94?= =?UTF-8?q?=207=20files,=2010=20issues=20resolved?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hybrid_reranker: - N+1 query → batch WHERE id IN (...) for _load_doc_embeddings - bare except:pass → logger.warning for embedding load failures - keyword density: CJK substring + Latin word boundary matching - _normalize_minmax: uniform values return 0.5 (neutral) not 1.0 - _is_real_embedding_provider: check embed/encode methods first query_expander: - log message no longer leaks query content (PII risk) pipeline: - remove redundant outer try/except around query expansion - add logger.warning for YAML config load failures reranker_config: - ContentQualityConfig.__post_init__ validates length/density constraints - from_yaml logs warning when file not found result_merger: - unknown strategy logs warning before fallback - multi_query marker deduplication - hardcoded k=60 → DEFAULT_RRF_K constant - representative selection by rrf_score, not sources length draft_agent: - initial evidence capped to _MAX_EVIDENCE (sorted by score) --- src/ticketpilot/drafting/draft_agent.py | 5 ++- src/ticketpilot/retrieval/hybrid_reranker.py | 39 ++++++++++++++------ src/ticketpilot/retrieval/pipeline.py | 13 ++++--- src/ticketpilot/retrieval/query_expander.py | 4 +- src/ticketpilot/retrieval/reranker_config.py | 18 ++++++++- src/ticketpilot/retrieval/result_merger.py | 11 ++++-- tests/unit/test_hybrid_reranker.py | 2 +- 7 files changed, 64 insertions(+), 28 deletions(-) diff --git a/src/ticketpilot/drafting/draft_agent.py b/src/ticketpilot/drafting/draft_agent.py index 8e9a662..42d07ba 100644 --- a/src/ticketpilot/drafting/draft_agent.py +++ b/src/ticketpilot/drafting/draft_agent.py @@ -357,9 +357,10 @@ def generate_draft( } ) - # Seed state with any pre-retrieved evidence + # Seed state with any pre-retrieved evidence (capped to prevent context overflow) if evidence_candidates: - state.evidence = list(evidence_candidates) + sorted_candidates = sorted(evidence_candidates, key=lambda e: e.score, reverse=True) + state.evidence = sorted_candidates[:_MAX_EVIDENCE] try: result = self._run_agent_loop( diff --git a/src/ticketpilot/retrieval/hybrid_reranker.py b/src/ticketpilot/retrieval/hybrid_reranker.py index 1fe4be8..6f32bc4 100644 --- a/src/ticketpilot/retrieval/hybrid_reranker.py +++ b/src/ticketpilot/retrieval/hybrid_reranker.py @@ -11,6 +11,8 @@ from __future__ import annotations import math +import re +import logging from dataclasses import dataclass, field from typing import Any, Optional from uuid import UUID @@ -18,6 +20,7 @@ from ticketpilot.retrieval.reranker_config import RerankerConfig from ticketpilot.retrieval.traces import FusedResult +logger = logging.getLogger(__name__) # --------------------------------------------------------------------------- # Output dataclasses @@ -99,11 +102,18 @@ def _keyword_density(query: str, content: str) -> float: """Compute what fraction of query terms appear in content. Splits query by whitespace, checks each term's presence. + Uses word boundary for Latin text, substring for CJK. """ terms = [t.strip() for t in query.split() if t.strip()] if not terms or not content: return 0.0 - hits = sum(1 for t in terms if t in content) + def _term_in_content(term: str) -> bool: + # CJK characters: substring match (no word boundaries in Chinese/Japanese) + if any('\u4e00' <= ch <= '\u9fff' for ch in term): + return term in content + # Latin text: word boundary match to avoid "art" matching "smart" + return bool(re.search(r'\b' + re.escape(term) + r'\b', content, re.IGNORECASE)) + hits = sum(1 for t in terms if _term_in_content(t)) return hits / len(terms) @@ -114,7 +124,7 @@ def _normalize_minmax(values: list[float]) -> list[float]: lo = min(values) hi = max(values) if hi - lo < 1e-12: - return [1.0] * len(values) + return [0.5] * len(values) return [(v - lo) / (hi - lo) for v in values] @@ -271,30 +281,35 @@ def _load_doc_embeddings( ) -> dict[UUID, list[float]]: """Load document embeddings from DB for the given chunk IDs.""" embeddings: dict[UUID, list[float]] = {} + if not chunk_ids: + return embeddings try: from ticketpilot.retrieval.db.connection import get_db_connection # noqa: PLC0415 with get_db_connection() as conn: with conn.cursor() as cur: - for cid in chunk_ids: - cur.execute( - "SELECT embedding FROM knowledge_chunks WHERE id = %s", - (str(cid),), - ) - row = cur.fetchone() - if row and row[0]: - emb_str = row[0] + placeholders = ",".join(["%s"] * len(chunk_ids)) + cur.execute( + f"SELECT id, embedding FROM knowledge_chunks WHERE id IN ({placeholders})", + [str(cid) for cid in chunk_ids], + ) + for row in cur.fetchall(): + cid = UUID(row[0]) + emb_str = row[1] + if emb_str: if isinstance(emb_str, str): emb_str = emb_str.strip("[]") embeddings[cid] = [float(x) for x in emb_str.split(",")] elif isinstance(emb_str, list): embeddings[cid] = [float(x) for x in emb_str] - except Exception: - pass # Graceful degradation + except Exception as e: + logger.warning("Failed to load document embeddings: %s", e) return embeddings def _is_real_embedding_provider(provider: Any) -> bool: """Check if the embedding provider is a real (non-fake) provider.""" + if not hasattr(provider, 'embed') and not hasattr(provider, 'encode'): + return False name = getattr(provider, "provider_name", "unknown") return name not in ("fake", "unknown", "") diff --git a/src/ticketpilot/retrieval/pipeline.py b/src/ticketpilot/retrieval/pipeline.py index 01d1f78..36b8d7c 100644 --- a/src/ticketpilot/retrieval/pipeline.py +++ b/src/ticketpilot/retrieval/pipeline.py @@ -4,6 +4,7 @@ - Multi-query expansion (LLM-generated query variants) - Hybrid reranking (multi-signal weighted fusion) """ +import logging import time from typing import Optional @@ -13,6 +14,8 @@ from ticketpilot.retrieval.hybrid_reranker import HybridReranker, RerankResult from ticketpilot.retrieval.query_expander import MultiQueryExpander from ticketpilot.retrieval.result_merger import merge_retrieval_results + +logger = logging.getLogger(__name__) from ticketpilot.retrieval.rrf import DEFAULT_RRF_K, rrf_fusion from ticketpilot.retrieval.schema.knowledge import DocType from ticketpilot.retrieval.traces import FusedResult, RetrievalTrace @@ -118,11 +121,8 @@ def hybrid_retrieval( query_variants = [query] expansion_latency = 0 if enable_query_expansion: - try: - expander = MultiQueryExpander() - query_variants = expander.expand(query, intent or "") - except Exception: - query_variants = [query] + expander = MultiQueryExpander() + query_variants = expander.expand(query, intent or "") expansion_latency = int((time.perf_counter() - expansion_start) * 1000) # --- Step 1-3: Per-query retrieval + RRF --- @@ -181,7 +181,8 @@ def hybrid_retrieval( if reranker_config is None: try: reranker_config = RerankerConfig.from_yaml("config/reranker.yaml") - except Exception: + except Exception as e: + logger.warning("Failed to load reranker config from YAML, using default: %s", e) reranker_config = RerankerConfig.default() # Take top candidates for reranking diff --git a/src/ticketpilot/retrieval/query_expander.py b/src/ticketpilot/retrieval/query_expander.py index bde8262..b3cc156 100644 --- a/src/ticketpilot/retrieval/query_expander.py +++ b/src/ticketpilot/retrieval/query_expander.py @@ -66,8 +66,8 @@ def expand(self, query: str, intent: str = "") -> list[str]: valid = [v for v in variants if self._is_valid_variant(v, query)] result = [query] + valid[: self._num_variants] logger.info( - "Query expansion: '%s' -> %d variants: %s", - query, len(valid), valid[: self._num_variants], + "Query expansion: original_len=%d -> %d variants (total valid: %d)", + len(query), len(valid[: self._num_variants]), len(valid), ) return result except Exception as e: diff --git a/src/ticketpilot/retrieval/reranker_config.py b/src/ticketpilot/retrieval/reranker_config.py index 5eaff48..b22c7f8 100644 --- a/src/ticketpilot/retrieval/reranker_config.py +++ b/src/ticketpilot/retrieval/reranker_config.py @@ -17,6 +17,18 @@ class ContentQualityConfig: optimal_length_max: int = 800 keyword_density_weight: float = 0.5 + def __post_init__(self) -> None: + if self.optimal_length_min > self.optimal_length_max: + raise ValueError( + f"optimal_length_min ({self.optimal_length_min}) must be <= " + f"optimal_length_max ({self.optimal_length_max})" + ) + if not 0 <= self.keyword_density_weight <= 1: + raise ValueError( + f"keyword_density_weight must be between 0 and 1, " + f"got {self.keyword_density_weight}" + ) + @dataclass class RerankerConfig: @@ -87,12 +99,14 @@ def from_yaml(cls, path: str | Path) -> RerankerConfig: Falls back to default if file not found. """ + import logging # noqa: PLC0415 import yaml # noqa: PLC0415 + logger = logging.getLogger(__name__) path = Path(path) if not path.exists(): - cfg = cls.default() - return cfg + logger.warning("Reranker config file not found: %s, using defaults", path) + return cls.default() with path.open("r", encoding="utf-8") as f: data: dict[str, Any] = yaml.safe_load(f) or {} diff --git a/src/ticketpilot/retrieval/result_merger.py b/src/ticketpilot/retrieval/result_merger.py index 716c7bb..3cc422f 100644 --- a/src/ticketpilot/retrieval/result_merger.py +++ b/src/ticketpilot/retrieval/result_merger.py @@ -7,11 +7,14 @@ """ from __future__ import annotations +import logging from collections import defaultdict from uuid import UUID from ticketpilot.retrieval.traces import FusedResult +logger = logging.getLogger(__name__) + def merge_retrieval_results( result_sets: list[list[FusedResult]], @@ -43,6 +46,7 @@ def merge_retrieval_results( elif strategy == "rrf_again": return _merge_rrf_again(non_empty) else: + logger.warning("Unknown merge strategy '%s', falling back to 'sum_score'", strategy) return _merge_sum_score(non_empty) @@ -60,7 +64,7 @@ def _merge_sum_score( for r in result_set: score_sums[r.chunk_id] += r.rrf_score # Keep the version with most info (prefer one with both keyword+vector) - if r.chunk_id not in best or len(r.sources) > len(best[r.chunk_id].sources): + if r.chunk_id not in best or r.rrf_score > best[r.chunk_id].rrf_score: best[r.chunk_id] = r # Build merged results with summed scores @@ -76,7 +80,7 @@ def _merge_sum_score( keyword_contribution=representative.keyword_contribution, vector_rank=representative.vector_rank, vector_contribution=representative.vector_contribution, - sources=representative.sources + ["multi_query"], + sources=representative.sources + (["multi_query"] if "multi_query" not in representative.sources else []), )) merged.sort(key=lambda r: r.rrf_score, reverse=True) @@ -106,7 +110,8 @@ def _merge_rrf_again( Uses RRF k=60 on the rank positions within each query's results. """ - k = 60 + from ticketpilot.retrieval.rrf import DEFAULT_RRF_K # noqa: PLC0415 + k = DEFAULT_RRF_K # Build per-query rank maps rank_maps: list[dict[UUID, int]] = [] representative: dict[UUID, FusedResult] = {} diff --git a/tests/unit/test_hybrid_reranker.py b/tests/unit/test_hybrid_reranker.py index cc8e083..dd9e4ec 100644 --- a/tests/unit/test_hybrid_reranker.py +++ b/tests/unit/test_hybrid_reranker.py @@ -84,7 +84,7 @@ def test_empty_query(self): class TestNormalizeMinMax: def test_uniform_values(self): result = _normalize_minmax([5, 5, 5]) - assert result == [1.0, 1.0, 1.0] + assert result == [0.5, 0.5, 0.5] def test_normal_range(self): result = _normalize_minmax([0, 5, 10]) From 559fd4291356fc167be44f3348738214930a38ae Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 8 Jun 2026 13:52:29 +0000 Subject: [PATCH 5/7] test: add 27 edge-case tests from OCR review findings hybrid_reranker (+10): - TestIsRealEmbeddingProvider: 6 cases (real/fake/none/encode/unknown) - TestKeywordDensityEdgeCases: 4 cases (Latin boundary, CJK, mixed) - TestHybridReranker: top_k truncation + overflow query_expander (+5): - default intent, num_variants limit, whitespace variant, non-string parse result_merger (+5): - unknown strategy fallback, highest-rrf representative, multi_query dedup, rrf_again precise score verification reranker_config (+7): - all signals missing, ContentQualityConfig validation (3 cases), malformed YAML, empty YAML validation --- tests/unit/test_hybrid_reranker.py | 77 ++++++++++++++++++++++++++++++ tests/unit/test_query_expander.py | 28 +++++++++++ tests/unit/test_reranker_config.py | 42 ++++++++++++++++ tests/unit/test_result_merger.py | 49 +++++++++++++++++++ 4 files changed, 196 insertions(+) diff --git a/tests/unit/test_hybrid_reranker.py b/tests/unit/test_hybrid_reranker.py index dd9e4ec..5e95323 100644 --- a/tests/unit/test_hybrid_reranker.py +++ b/tests/unit/test_hybrid_reranker.py @@ -1,5 +1,6 @@ """Unit tests for HybridReranker.""" import math +from unittest.mock import MagicMock from uuid import uuid4 import pytest @@ -167,3 +168,79 @@ def test_to_fused_result_conversion(self): assert isinstance(fused, FusedResult) assert "hybrid_rerank" in fused.sources assert fused.chunk_id == c.chunk_id + + def test_top_k_less_than_candidates(self): + """top_k truncates results.""" + candidates = [_make_fused(rrf_score=0.1 * i) for i in range(5)] + reranker = HybridReranker() + results = reranker.rerank(candidates, "test", top_k=2) + assert len(results) == 2 + assert results[0].rank == 1 + assert results[1].rank == 2 + + def test_top_k_more_than_candidates(self): + """top_k > len(candidates) returns all candidates.""" + candidates = [_make_fused(rrf_score=0.5)] + reranker = HybridReranker() + results = reranker.rerank(candidates, "test", top_k=10) + assert len(results) == 1 + + +class TestIsRealEmbeddingProvider: + def test_real_provider(self): + from ticketpilot.retrieval.hybrid_reranker import _is_real_embedding_provider + provider = MagicMock() + provider.embed = MagicMock() + provider.provider_name = "openai" + assert _is_real_embedding_provider(provider) is True + + def test_fake_provider(self): + from ticketpilot.retrieval.hybrid_reranker import _is_real_embedding_provider + provider = MagicMock() + provider.embed = MagicMock() + provider.provider_name = "fake" + assert _is_real_embedding_provider(provider) is False + + def test_no_embed_or_encode(self): + from ticketpilot.retrieval.hybrid_reranker import _is_real_embedding_provider + provider = MagicMock(spec=[]) # no attributes + assert _is_real_embedding_provider(provider) is False + + def test_unknown_provider_name(self): + from ticketpilot.retrieval.hybrid_reranker import _is_real_embedding_provider + provider = MagicMock() + provider.encode = MagicMock() + # No provider_name attribute → getattr returns "unknown" + del provider.provider_name + assert _is_real_embedding_provider(provider) is False + + def test_none_provider(self): + from ticketpilot.retrieval.hybrid_reranker import _is_real_embedding_provider + assert _is_real_embedding_provider(None) is False + + def test_encode_method_sufficient(self): + from ticketpilot.retrieval.hybrid_reranker import _is_real_embedding_provider + provider = MagicMock(spec=["encode", "provider_name"]) + provider.encode = MagicMock() + provider.provider_name = "bge" + assert _is_real_embedding_provider(provider) is True + + +class TestKeywordDensityEdgeCases: + def test_latin_word_boundary_no_false_positive(self): + """'art' should NOT match inside 'smart'.""" + assert _keyword_density("art", "smart car") == 0.0 + + def test_latin_word_boundary_exact_match(self): + """'art' matches standalone 'Art'.""" + assert _keyword_density("art", "Art of war") == 1.0 + + def test_cjk_substring_match(self): + """CJK terms use substring matching.""" + assert _keyword_density("退款", "退款政策说明") == 1.0 + + def test_mixed_cjk_latin(self): + """Mixed query: CJK substring + Latin word boundary.""" + assert _keyword_density("退款 policy", "退款 policy 说明") == 1.0 + # 'policy' inside 'policyholder' should not match + assert _keyword_density("policy", "policyholder agreement") == 0.0 diff --git a/tests/unit/test_query_expander.py b/tests/unit/test_query_expander.py index ed23696..4deadcf 100644 --- a/tests/unit/test_query_expander.py +++ b/tests/unit/test_query_expander.py @@ -65,3 +65,31 @@ def test_expand_filters_invalid_variants(self, mock_llm): expander = MultiQueryExpander(api_key="fake-key") result = expander.expand("退款没到账") assert result == ["退款没到账", "有效变体"] + + @patch("ticketpilot.retrieval.query_expander.MultiQueryExpander._call_llm") + def test_expand_default_intent(self, mock_llm): + """Test that expand works with default intent parameter (empty string).""" + mock_llm.return_value = ["退款进度"] + expander = MultiQueryExpander(api_key="fake-key") + result = expander.expand("退款没到账") # no intent arg + assert result == ["退款没到账", "退款进度"] + + @patch("ticketpilot.retrieval.query_expander.MultiQueryExpander._call_llm") + def test_expand_respects_num_variants_limit(self, mock_llm): + """When LLM returns more variants than num_variants, truncate.""" + mock_llm.return_value = ["变体1", "变体2", "变体3"] + expander = MultiQueryExpander(api_key="fake-key", num_variants=2) + result = expander.expand("original query") + assert len(result) == 3 # original + 2 variants + assert "变体3" not in result + + def test_is_valid_variant_with_whitespace(self): + """Variant that equals original after strip should be invalid.""" + expander = MultiQueryExpander(api_key="fake") + assert not expander._is_valid_variant(" 退款没到账 ", "退款没到账") + + def test_parse_non_string_elements(self): + """JSON array with non-string elements should convert to strings.""" + expander = MultiQueryExpander(api_key="fake") + variants = expander._parse_variants('[123, true, "正常"]') + assert variants == ["123", "True", "正常"] diff --git a/tests/unit/test_reranker_config.py b/tests/unit/test_reranker_config.py index 8170b73..27357aa 100644 --- a/tests/unit/test_reranker_config.py +++ b/tests/unit/test_reranker_config.py @@ -99,3 +99,45 @@ def test_adjust_weights_preserves_proportions(self): # rrf was 0.40, now should be 0.40/0.75 ≈ 0.533 expected_ratio = 0.40 / 0.75 assert abs(adjusted["rrf_score"] - expected_ratio) < 1e-6 + + def test_adjust_weights_all_signals_missing(self): + """When all signals are removed, return empty dict.""" + cfg = RerankerConfig.default() + all_signals = set(cfg.weights.keys()) + adjusted = cfg.adjust_weights_for_missing_signals(all_signals) + assert isinstance(adjusted, dict) + assert len(adjusted) == 0 + + +class TestContentQualityConfig: + def test_valid_config(self): + cq = ContentQualityConfig(optimal_length_min=100, optimal_length_max=500) + assert cq.optimal_length_min == 100 + + def test_min_greater_than_max_raises(self): + with pytest.raises(ValueError, match="optimal_length_min.*must be <="): + ContentQualityConfig(optimal_length_min=800, optimal_length_max=200) + + def test_density_weight_out_of_range_raises(self): + with pytest.raises(ValueError, match="keyword_density_weight must be between 0 and 1"): + ContentQualityConfig(keyword_density_weight=1.5) + + def test_negative_density_weight_raises(self): + with pytest.raises(ValueError, match="keyword_density_weight must be between 0 and 1"): + ContentQualityConfig(keyword_density_weight=-0.1) + + +class TestRerankerConfigFromYamlEdgeCases: + def test_malformed_yaml_raises(self, tmp_path): + """Invalid YAML syntax should raise.""" + p = tmp_path / "bad.yaml" + p.write_text(":\n invalid: [yaml\n") + with pytest.raises(Exception): + RerankerConfig.from_yaml(str(p)) + + def test_empty_yaml_raises_validation(self, tmp_path): + """Empty YAML file produces empty weights → validation error.""" + p = tmp_path / "empty.yaml" + p.write_text("") + with pytest.raises(ValueError, match="weights cannot be empty"): + RerankerConfig.from_yaml(str(p)) diff --git a/tests/unit/test_result_merger.py b/tests/unit/test_result_merger.py index 262e2ca..de6a7d9 100644 --- a/tests/unit/test_result_merger.py +++ b/tests/unit/test_result_merger.py @@ -79,3 +79,52 @@ def test_multi_query_marker(self): r = _fused() merged = merge_retrieval_results([[r], [r]]) assert "multi_query" in merged[0].sources + + def test_unknown_strategy_defaults_to_sum_score(self): + """Unknown strategy string falls back to sum_score.""" + cid = uuid4() + r1 = _fused(chunk_id=cid, rrf_score=0.3) + r2 = _fused(chunk_id=cid, rrf_score=0.2) + merged = merge_retrieval_results([[r1], [r2]], strategy="unknown_strategy") + assert len(merged) == 1 + assert merged[0].rrf_score == pytest.approx(0.5) # sum_score behavior + + def test_sum_score_prefers_highest_rrf_representative(self): + """When same chunk appears multiple times, representative has highest rrf_score.""" + cid = uuid4() + r1 = _fused(chunk_id=cid, rrf_score=0.1, sources=["keyword"]) + r2 = _fused(chunk_id=cid, rrf_score=0.8, sources=["vector"]) + merged = merge_retrieval_results([[r1], [r2]], strategy="sum_score") + assert len(merged) == 1 + # Representative should be r2 (higher rrf_score) + assert "vector" in merged[0].sources + + def test_multi_query_marker_no_duplicate(self): + """Same chunk from 3 queries should have only one 'multi_query' marker.""" + cid = uuid4() + r1 = _fused(chunk_id=cid, rrf_score=0.3, sources=["keyword"]) + r2 = _fused(chunk_id=cid, rrf_score=0.2, sources=["keyword"]) + r3 = _fused(chunk_id=cid, rrf_score=0.1, sources=["keyword"]) + merged = merge_retrieval_results([[r1], [r2], [r3]], strategy="sum_score") + assert len(merged) == 1 + assert merged[0].sources.count("multi_query") == 1 + + def test_rrf_again_precise_scores(self): + """Verify exact RRF scores with k=60.""" + c1 = uuid4() + c2 = uuid4() + r1q1 = _fused(chunk_id=c1, rrf_score=0.5) + r1q2 = _fused(chunk_id=c1, rrf_score=0.4) + r2q1 = _fused(chunk_id=c2, rrf_score=0.3) + r2q2 = _fused(chunk_id=c2, rrf_score=0.6) + merged = merge_retrieval_results( + [[r1q1, r2q1], [r1q2, r2q2]], strategy="rrf_again" + ) + k = 60 + expected_c1 = 2 * (1 / (k + 1)) # Both rank 1 + expected_c2 = 2 * (1 / (k + 2)) # Both rank 2 + assert len(merged) == 2 + assert merged[0].chunk_id == c1 + assert merged[0].rrf_score == pytest.approx(expected_c1) + assert merged[1].chunk_id == c2 + assert merged[1].rrf_score == pytest.approx(expected_c2) From fdf075c160515370e40daf5187664e864ea74ecc Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 8 Jun 2026 13:58:14 +0000 Subject: [PATCH 6/7] ci: add AI code review workflow (OCR + MiMo-V2.5) GitHub Actions workflow that runs on every PR: - Installs Open Code Review CLI - Configures MiMo-V2.5 as LLM backend - Reviews PR diff and posts results as PR comment - Requires MIMO_API_KEY and MIMO_BASE_URL repo secrets --- .github/workflows/code-review.yml | 87 +++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 .github/workflows/code-review.yml diff --git a/.github/workflows/code-review.yml b/.github/workflows/code-review.yml new file mode 100644 index 0000000..de812b9 --- /dev/null +++ b/.github/workflows/code-review.yml @@ -0,0 +1,87 @@ +name: AI Code Review + +on: + pull_request: + types: [opened, synchronize] + +permissions: + contents: read + pull-requests: write + +jobs: + review: + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 # Full history for diff + + - uses: actions/setup-node@v4 + with: + node-version: '20' + + - name: Install Open Code Review + run: npm install -g @alibaba-group/open-code-review + + - name: Configure OCR with MiMo + env: + MIMO_API_KEY: ${{ secrets.MIMO_API_KEY }} + MIMO_BASE_URL: ${{ secrets.MIMO_BASE_URL }} + run: | + ocr config set llm.url "$MIMO_BASE_URL" + ocr config set llm.auth_token "$MIMO_API_KEY" + ocr config set llm.model "mimo-v2.5" + ocr config set llm.use_anthropic false + ocr config set language "English" + + - name: Run Code Review + id: review + env: + GH_TOKEN: ${{ github.token }} + PR_TITLE: ${{ github.event.pull_request.title }} + PR_NUMBER: ${{ github.event.pull_request.number }} + BASE_SHA: ${{ github.event.pull_request.base.sha }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + run: | + echo "Reviewing: $BASE_SHA..$HEAD_SHA" + + # Run OCR and capture output + REVIEW=$(ocr review \ + --from "$BASE_SHA" \ + --to "$HEAD_SHA" \ + --audience agent \ + --background "PR #${PR_NUMBER}: ${PR_TITLE}" \ + 2>&1) || true + + # Save to file for the comment step + echo "$REVIEW" > /tmp/ocr-review.txt + + # Check if there are actual comments + if echo "$REVIEW" | grep -q "comment(s)"; then + echo "has_issues=true" >> "$GITHUB_OUTPUT" + else + echo "has_issues=false" >> "$GITHUB_OUTPUT" + fi + + - name: Post Review Comment + if: always() + env: + GH_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ github.event.pull_request.number }} + run: | + REVIEW=$(cat /tmp/ocr-review.txt) + + gh pr comment "$PR_NUMBER" \ + --body "## 🤖 AI Code Review (Open Code Review + MiMo) + +
+ Review Results + + \`\`\` + ${REVIEW} + \`\`\` + +
+ + _Automated review by [Open Code Review](https://github.com/alibaba/open-code-review) + Xiaomi MiMo-V2.5_" From 8ba4f65426b3cba1a8b32d39c180844cbffb96a6 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 8 Jun 2026 14:11:56 +0000 Subject: [PATCH 7/7] docs: rewrite README for GitHub star optimization - Chinese-first hook with quantified metrics (60% auto-send, 40% human review) - 3-column screenshot grid at top - 3-step quick start (down from 7) - Chinese mermaid architecture diagram - Bilingual feature comparison table - Reduced from 254 to 183 lines (-28%) --- README.md | 313 +++++++++++++++++++++--------------------------------- 1 file changed, 121 insertions(+), 192 deletions(-) diff --git a/README.md b/README.md index 2add7d3..61a2aa3 100644 --- a/README.md +++ b/README.md @@ -1,253 +1,182 @@ -# TicketPilot +# 🎫 TicketPilot -[![Tests](https://img.shields.io/badge/tests-1%2C662-brightgreen)]() +**中文客服工单 AI 分拣系统 — 确定性管线，零 LLM 调用，全链路可追溯** + +> 跨境电商客服 Copilot：意图分类 → 风险评估 → 混合检索 → 证据化草稿 → 人工审核台 +> 60% 工单自动发送，40% 路由到人工，0% 关键工单遗漏 + +[![Tests](https://img.shields.io/badge/tests-1%2C760-brightgreen)]() [![Coverage](https://img.shields.io/badge/coverage-87%25-brightgreen)]() [![Python](https://img.shields.io/badge/python-3.11%2B-blue)]() [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![Docker](https://img.shields.io/badge/docker-compose-blue?logo=docker)]() -**AI Customer Service Copilot for cross-border e-commerce — deterministic pipeline, full-chain traceability, hybrid retrieval.** - -TicketPilot triages customer tickets through intent classification, risk assessment, hybrid evidence retrieval, and draft generation — then routes only the ~20% that need human judgment to a review console. The pipeline is fully deterministic (zero LLM calls), with every decision traceable from answer → citation → chunk → document. - -> This is a portfolio project demonstrating production-grade RAG architecture patterns. All data is synthetic. - --- -## Why I Built This - -Most AI customer service demos hide the hard parts: how do you know the LLM isn't hallucinating? How do you decide which tickets need a human? How do you trace a wrong answer back to its source? - -TicketPilot answers these questions with engineering, not prompts: +## 为什么做这个项目 -- **No black boxes** — every retrieval, classification, and confidence score is explainable -- **No hallucination risk in the pipeline** — LLM is only used for draft generation, with 8-category forbidden-promise detection -- **Hybrid retrieval that works** — keyword FTS + vector HNSW → RRF fusion → multi-signal reranking -- **Confidence you can calibrate** — 4-dimensional scoring with isotonic regression, not arbitrary thresholds +大多数 AI 客服 demo 回避了最难的问题：**你怎么知道 LLM 没有在胡说？哪些工单需要人来判断？错误的回答怎么追溯到源头？** ---- +TicketPilot 用工程手段回答这些问题： -## Screenshots +- **管线内零 LLM 调用** — 分类、风险、检索、评分全部确定性执行，结果可复现 +- **混合检索而非纯向量** — 关键词 FTS + pgvector HNSW → RRF 融合 → 4 信号混合重排序 +- **4 层置信度路由** — HIGH/MEDIUM 自动发送，LOW 人工审核，CRITICAL 强制转人工 +- **8 类禁止承诺检测** — 退款金额、法律威胁、隐私承诺等，AI 草稿不会越线 -### Confidence Monitoring Dashboard -![Dashboard Overview](docs/assets/dashboard-overview.png) +> Portfolio demo project. All data is synthetic. -### Tier & Agent Routing Distribution -![Dashboard Charts](docs/assets/dashboard-charts.png) +--- -### Intent × Risk Label Heatmap -![Dashboard Heatmap](docs/assets/dashboard-heatmap.png) +## 截图 -## Architecture +| 监控大盘 | 置信度分布 | 意图×风险热力图 | +|:---:|:---:|:---:| +| ![Dashboard](docs/assets/dashboard-overview.png) | ![Charts](docs/assets/dashboard-charts.png) | ![Heatmap](docs/assets/dashboard-heatmap.png) | -```mermaid -graph TD - A[Ticket Input] --> B[Intent Classifier] - B --> C[Risk Assessor] - C --> D[Hybrid Retrieval
FTS + pgvector → RRF → Hybrid Rerank] - D --> E[Multi-Agent Router] - E --> F[Refund Agent] - E --> G[Complaint Agent] - E --> H[Logistics Agent] - E --> I[Technical Agent] - E --> J[Default Agent] - F --> K[Draft Generator] - G --> K - H --> K - I --> K - J --> K - K --> L[Claim Guard] - K --> M[Citation Validator] - L --> N[Confidence Scorer
4 dimensions] - M --> N - N --> O{Confidence Tier} - O -->|HIGH| P[Auto-Send] - O -->|MEDIUM| P - O -->|LOW| Q[Human Review] - O -->|CRITICAL| R[Force Escalation] - Q --> S{Decision} - S -->|Approve| P - S -->|Edit| T[Revise] - T --> P - P --> U[Feedback Loop] - U --> V[Isotonic Calibrator] - V --> B -``` +--- -## What Makes It Different +## 30 秒上手 -| Feature | Typical RAG | TicketPilot | -|---------|------------|-------------| -| Retrieval | Single vector search | Keyword FTS + Vector HNSW → RRF → **4-signal hybrid reranking** | -| Confidence | Binary (confident / not) | 4-dimensional weighted: retrieval + classification + citation + evidence density | -| Routing | All-auto or all-human | 4-tier degradation: AUTO → CAUTIOUS → HUMAN_REVIEW → ESCALATION | -| Hallucination guard | None or keyword filter | 8-category forbidden promise detection (refund amounts, legal threats, etc.) | -| Traceability | None | Full chain: answer → citation → chunk → document | -| Agent architecture | Single agent | Multi-agent orchestrator with intent-based routing to 5 specialists | -| Pipeline determinism | LLM-dependent | Rule-driven, zero LLM calls in pipeline | -| Calibration | Static thresholds | Feedback loop with isotonic regression + reliability diagrams | -| Self-reflection | None | Skill seed learning from successful draft patterns | +```bash +git clone https://github.com/lennney/ticketpilot.git && cd ticketpilot -## Hybrid Retrieval Pipeline +pip install uv && uv sync # 安装依赖 +docker compose up -d db # 启动 PostgreSQL + pgvector +uv run python scripts/ingest_knowledge.py # 灌入知识库 +uv run uvicorn ticketpilot.api:app --port 8000 # 启动 API ``` -Query → LLM Query Expansion (2 variants) - → Parallel Retrieval (keyword FTS + vector HNSW per variant) - → RRF Fusion (k=60) - → Multi-variant Merge (sum_score dedup) - → Hybrid Reranker (4-signal weighted fusion): - ├── RRF score (weight: 0.40) - ├── Embedding similarity (weight: 0.25) - ├── Intent metadata boost (weight: 0.20) - └── Content quality (weight: 0.15) - → Top-K Evidence + Full RetrievalTrace -``` - -The reranker is fully configurable via `config/reranker.yaml` — weights, intent boost tables, and content quality parameters are all externalized for A/B experimentation. - -## Key Modules - -### Confidence & Routing -- **ConfidenceScorer** — 4-dimensional scoring (retrieval 35%, classification 25%, citation 25%, evidence density 15%) -- **DegradationRouter** — 4-tier routing based on confidence level -- **Claim Guard** — Forbidden promise detection, citation coverage, risk acknowledgment -### Multi-Agent System -- **Orchestrator** — Intent-based routing to specialized agents -- **5 Specialists** — RefundAgent, ComplaintAgent, LogisticsAgent, TechnicalAgent, DefaultAgent -- **Self-Reflection Skills** — Agents learn from successful draft patterns - -### Retrieval -- **Hybrid search** — PostgreSQL FTS + pgvector HNSW → RRF fusion -- **Hybrid Reranker** — Multi-signal weighted fusion (RRF + embedding + intent + content quality) -- **Query Expansion** — LLM-generated query variants for improved recall -- **RetrievalTrace** — Full explainability with per-signal breakdown +```bash +# 一键 demo（灌数据 + 启动服务 + 跑评测） +bash scripts/demo.sh +``` -### Feedback & Calibration -- **FeedbackCollector** — Records (confidence, action, was_correct) from human reviews -- **IsotonicCalibrator** — Pure Python PAV algorithm for confidence calibration -- **ReliabilityDiagram** — ASCII art visualization for terminal +```bash +# 人工审核台 +uv run streamlit run src/ticketpilot/review/console.py --server.port 8501 +``` -### Evaluation -- **NLI Scorer** — Sentence decomposition, synonym expansion, negation detection -- **Retrieval Metrics** — Precision@K, Recall@K, MRR, NDCG -- **A/B Experiment Framework** — Same tickets, two configs, comparison report +--- -## Quick Start +## 架构 -```bash -git clone https://github.com/lennney/ticketpilot.git -cd ticketpilot +```mermaid +graph TD + A[工单输入] --> B[意图分类
8 类 + 置信度] + B --> C[风险评估
8 标记 × 3 级别] + C --> D[混合检索
FTS + pgvector → RRF] + D --> E[多 Agent 路由] + E --> F[退款 / 投诉 / 物流 / 技术 / 默认] + F --> G[草稿生成] + G --> H[Claim Guard
禁止承诺检测] + H --> I[置信度评分
4 维加权] + I --> J{路由决策} + J -->|HIGH / MEDIUM| K[自动发送] + J -->|LOW| L[人工审核] + J -->|CRITICAL| M[强制转人工] + L -->|通过| K + K --> N[反馈回路] + N --> O[等距校准器] + O --> B +``` -pip install uv -uv sync +--- -cp .env.example .env.local -# Edit .env.local with your API keys (optional — pipeline works without LLM keys) +## 和普通 RAG 的区别 -docker compose up -d db +| 维度 | 典型 RAG | TicketPilot | +|------|---------|-------------| +| 检索 | 单路向量搜索 | 关键词 FTS + 向量 HNSW → RRF → **4 信号混合重排序** | +| 置信度 | 二元（自信/不自信） | 4 维加权：检索 35% + 分类 25% + 引用 25% + 证据密度 15% | +| 路由 | 全自动或全人工 | 4 层降级：AUTO → CAUTIOUS → HUMAN_REVIEW → ESCALATION | +| 幻觉防护 | 无 | 8 类禁止承诺检测（退款金额、法律威胁等） | +| 可追溯性 | 无 | 全链路：回答 → 引用 → chunk → 文档 | +| Agent 架构 | 单 Agent | 5 个专职 Agent + 意图路由 | +| 管线确定性 | 依赖 LLM | 规则驱动，管线内零 LLM 调用 | +| 校准 | 静态阈值 | 反馈回路 + 等距回归 + 可靠性图 | -uv run python scripts/ingest_knowledge.py +--- -uv run uvicorn ticketpilot.api:app --host 0.0.0.0 --port 8000 -``` +## API -### One-Click Demo +| 端点 | 方法 | 说明 | +|------|------|------| +| `/api/tickets` | POST | 提交工单处理 | +| `/api/chat` | POST | 对话式 Copilot | +| `/api/chat/stream` | POST | SSE 流式响应 | +| `/api/reviews` | POST | 提交人工审核决策 | +| `/api/evaluation` | GET | 评测指标 | -```bash -bash scripts/demo.sh -``` +--- -### Run Tests +## 测试 ```bash -# Unit tests (no database required) +# 单元测试（无需数据库） TICKETPILOT_SKIP_DB_TESTS=1 uv run pytest tests/ --ignore=tests/integration -q -# Full quality gate (lint + tests + integration + openspec + secret scan) +# 完整质量门禁（lint + 测试 + 集成 + openspec + 密钥扫描） bash scripts/run_quality_gate.sh ``` -### Review Console & Dashboard - -```bash -# Human review interface -uv run streamlit run src/ticketpilot/review/console.py --server.port 8501 - -# Metrics dashboard -uv run python scripts/run_dashboard.py +``` +1,760 tests passing · 87% coverage · ≥ 70% enforced ``` -## API Endpoints - -| Endpoint | Method | Description | -|----------|--------|-------------| -| `/api/health` | GET | Health check | -| `/api/chat` | POST | Chat with AI copilot | -| `/api/chat/stream` | POST | Streaming chat (SSE) | -| `/api/tickets` | POST | Process ticket | -| `/api/reviews` | POST | Submit review decision | -| `/api/evaluation` | GET | Get evaluation metrics | +--- -## Project Structure +## 项目结构 ``` src/ticketpilot/ -├── api/ # FastAPI endpoints + SSE streaming -├── classification/ # Intent classifier (deterministic, 8 classes) -├── confidence/ # 4-dimensional confidence scorer -├── degradation/ # 4-tier response router -├── drafting/ # DraftAgent, claim guard, citation validator -├── evaluation/ # NLI scorer, retrieval metrics, A/B experiments -├── experiment/ # A/B experiment framework -├── feedback/ # Feedback collector, calibrator, threshold advisor -├── guardrails/ # PII detection, security scanning -├── intake/ # Ticket normalization, entity extraction -├── multi_agent/ # Orchestrator + 5 specialized agents -├── retrieval/ # Hybrid retrieval (FTS + HNSW → RRF → hybrid rerank) -│ ├── hybrid_reranker.py # Multi-signal weighted reranking -│ ├── query_expander.py # LLM query expansion -│ ├── result_merger.py # Multi-variant result merging -│ └── reranker_config.py # YAML-configurable weights -├── review/ # Streamlit review console -├── risk/ # Risk assessor (8 flag types, 3 severity) -├── schema/ # Pydantic data models -├── tracing/ # Provenance tracking -└── triggers/ # CLI + webhook entry points +├── api/ # FastAPI + SSE 流式 +├── classification/ # 意图分类（确定性，8 类） +├── confidence/ # 4 维置信度评分 +├── degradation/ # 4 层响应路由 +├── drafting/ # 草稿生成 + Claim Guard + 引用验证 +├── evaluation/ # NLI 评分、检索指标、A/B 实验 +├── feedback/ # 反馈收集、等距校准、阈值顾问 +├── guardrails/ # PII 检测、安全扫描 +├── multi_agent/ # 编排器 + 5 专职 Agent +├── retrieval/ # 混合检索（FTS + HNSW → RRF → 混合重排序） +│ ├── hybrid_reranker.py # 多信号加权重排序 +│ ├── query_expander.py # LLM 查询扩展 +│ ├── result_merger.py # 多变体结果合并 +│ └── reranker_config.py # YAML 可配置权重 +├── review/ # Streamlit 人工审核台 +├── risk/ # 风险评估（8 标记，3 级别） +├── schema/ # Pydantic 数据模型 +└── tracing/ # 来源追溯 ``` -## Test Coverage - -``` -1,662 tests passing -├── Unit tests (no DB): 1,662 -├── Integration tests (DB required): separate -└── Coverage: 87% (>= 70% enforced) -``` +--- -## Contributing +## 参与贡献 -Contributions welcome! Good first issues: +欢迎 PR！适合入门的方向： -- 📝 **Documentation** — Improve Chinese/English docs, add usage examples -- 🧪 **Test coverage** — Add edge case tests for retrieval or classification -- 🔧 **Bug fixes** — Check [Issues](https://github.com/lennney/ticketpilot/issues) for open bugs -- 🌐 **Internationalization** — Add multi-language support for the review console +- 📝 **文档** — 中英文使用示例、架构说明 +- 🧪 **测试** — 检索/分类的边界 case +- 🔧 **Bug 修复** — 看 [Issues](https://github.com/lennney/ticketpilot/issues) +- 🌐 **国际化** — 审核台多语言支持 ```bash -# Setup dev environment -uv sync --group dev +uv sync --group dev # 安装开发依赖 uv run pytest tests/ -v - -# Run quality gate before submitting PR -bash scripts/run_quality_gate.sh +bash scripts/run_quality_gate.sh # 提 PR 前跑一下 ``` -## Technical Docs +--- + +## 技术文档 + +- [检索架构](docs/technical/retrieval_architecture.md) — 混合检索管线详解 +- [质量门禁](docs/technical/quality_gate.md) — 测试和验证规则 +- [项目 Portfolio](docs/portfolio/index.md) — 指标和 elevator pitch -- [Retrieval Architecture](docs/technical/retrieval_architecture.md) — Hybrid retrieval pipeline deep dive -- [Validation Policy](docs/technical/validation_policy.md) — Testing and quality gate rules -- [Portfolio](docs/portfolio/index.md) — Project elevator pitch and key metrics +--- ## License