Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
LLM_PROVIDER=auto
ANTHROPIC_API_KEY=
ANTHROPIC_MODEL=claude-sonnet-4-6
OPENAI_API_KEY=
OPENAI_MODEL=gpt-4o-mini
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ node_modules/
venv/
__pycache__/
*.pyc
.cache/
.pytest_cache/
.mypy_cache/
.ruff_cache/
Expand Down
85 changes: 85 additions & 0 deletions code/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Support Triage Agent — HackerRank Orchestrate (May 2026)

Terminal-based RAG agent that triages support tickets across HackerRank, Claude, and Visa using only the local corpus in `../data/`.

## Architecture

```
ticket ──► pre-rules (regex) ──► early return (escalate / out-of-scope)
──► company inference (None→best by retrieval)
──► language gate (non-English → escalate for None/Visa)
──► dense retrieval (MiniLM, single index, metadata filter)
──► coverage-floor check ──► early escalate
──► LLM (Anthropic primary, OpenAI fallback, tool-forced JSON)
──► extractive fallback when no LLM key/quota is available
──► verifier (citation + sentence grounding)
──► post-rules (confidence/citation/area allow-list)
──► RowOutput
```

## Setup

```bash
cd code/
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # set ANTHROPIC_API_KEY and/or OPENAI_API_KEY
```

Recommended model: `OPENAI_MODEL=gpt-4o` (tuned against this) or `ANTHROPIC_MODEL=claude-sonnet-4-6`. `gpt-4o-mini` over-escalates sensitive-but-answerable tickets.

## Run

```bash
# Full pipeline against the real tickets:
python main.py

# Eval against the 10-row gold sample:
python eval.py

# Resume after partial run:
python main.py --resume

# No embeddings (TF-IDF fallback, fully offline):
python main.py --no-embeddings

# Dry run (skip LLM, only pre-rule decisions):
python main.py --dry-run --limit 5
```

Reads input from `../support_tickets/support_tickets.csv` and writes to
`../support_tickets/output.csv` with the exact 8-column header
`issue,subject,company,response,product_area,status,request_type,justification`.

## Design decisions

- **Single dense index, not BM25 + RRF.** Corpus is small (~3k chunks); MiniLM cosine is enough and avoids index-mismatch bugs under time pressure.
- **Rules before LLM, rules after LLM.** Pre-rules catch sensitive cases (fraud, legal, refund demands, score appeals, prompt injection, outage). Post-rules enforce confidence floor, mandatory citations, and product_area allow-list. The LLM is only trusted on grounded support questions.
- **Coverage floor check.** If retrieval similarity is below threshold we escalate before calling the LLM — this is the main hallucination defense.
- **Grounding verifier.** Cited chunk IDs must come from retrieval; unsupported response sentences are stripped or the row is escalated.
- **Determinism.** `temperature=0`, fixed `seed=42`, sorted tie-breaks, stable cosine sort, deterministic chunking.
- **Secrets.** Read only from env (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, model/provider settings). Local `.env`, `.env.local`, `code/.env`, and `code/.env.local` are supported.
- **`None`-company area handling.** When `company=None` and a pre-rule short-circuits the row, `product_area` is left blank for pleasantries and pure escalations, but `off_topic_trivia` is tagged `conversation_management` (Claude's catch-all for off-topic chat) since None-company invalid questions belong to that bucket.

## Files

| File | Purpose |
|---|---|
| `main.py` | CLI entry point |
| `agent.py` | Orchestrator |
| `retriever.py` | Dense retriever + TF-IDF fallback |
| `corpus.py` | Markdown loader + chunker |
| `escalation.py` | Pre/post rule tables |
| `verifier.py` | Citation grounding check |
| `prompts.py` | System prompt + few-shots |
| `llm_client.py` | Anthropic/OpenAI tool-call wrapper |
| `schemas.py` | Pydantic models |
| `io_csv.py` | CSV reader/writer with strict header |
| `eval.py` | Accuracy harness vs. gold sample |
| `config.py` | Constants |

## Known limits

- Gold sample is only 10 rows — accuracy numbers are noisy.
- Visa corpus is 14 docs; Visa tickets escalate often by design.
- We do not split multi-intent tickets; the LLM handles them in a single call.
Loading