feat: Cross-cutting agent infrastructure and ingestion pipeline library

[barney-w]

The system has two kinds of agent today: a coordinator (routes questions) and specialists (HR, IT, Website — answer questions). This issue adds a third kind: cross-cutting agents — shared helpers that any specialist can call mid-conversation, like "index this PDF for me."

We're not building any specific helper here. We're building the shelf to put them on — registration, auto-discovery, and builder wiring — so that #2 (Document Agent) has solid ground to build on.

We're also making the ingestion pipeline (currently CLI-only) callable as a library, so a future Document Agent can index files on the fly during a chat.

Nothing visible to users changes.

Blocks: #2

---

What already exists (don't rebuild)

• Per-agent model config — DomainAgent.model_id property (_base.py:83-85), 3-tier resolution in builder (builder.py:375-380)
• Attachment handling — PDF tiering in builder.py:97-143, attachments flow via current_attachments context var (builder.py:30-31)
• PDF extraction — pdf.py with count_pages(), extract_text(), size guards

---

Done when

• CrossCuttingAgent ABC exists and is importable
• AgentTier enum: coordinator, specialist, cross_cutting
• AgentRegistry has register_cross_cutting() and get_cross_cutting_agents()
• discover_agents() scans agents/cross_cutting/*/agent.py for CrossCuttingAgent subclasses
• build_orchestrator() wires cross-cutting tools into domain agents
• IngestionPipeline class in ingestion/src/pipeline/api.py with ingest_pdf() and check_exists()
• ingestion added as path dependency of api/
• Embedding client initialised at startup, injectable into IngestionPipeline
• All 378 existing unit tests pass
• New unit tests for all of the above

---

Architecture

Today:

Coordinator ──handoff──► Domain Agent
                              └── Tool: search_knowledge_base

After this issue + #2:

Coordinator ──handoff──► Domain Agent
                              ├── Tool: search_knowledge_base     (existing)
                              └── Tool: find_and_index_document   (wired here, built in #2)

Same handoff chain. Zero regression risk. We're just expanding each specialist's toolbox.

---

Part 1 — Cross-cutting agent infrastructure

New files:

api/src/agents/
├── _cross_cutting.py          # CrossCuttingAgent ABC
├── cross_cutting/__init__.py  # package marker (agent impl in #2)

Modified files: _base.py (add AgentTier), _registry.py (add cross-cutting section), _discovery.py (add second scan pass)

CrossCuttingAgent — unlike DomainAgent, these are not handoff targets. They expose a single tool via create_tool() that specialists call.

Discovery — _discovery.py uses pkgutil.iter_modules + importlib to find agents. Extension adds a second scan inside cross_cutting/ for CrossCuttingAgent subclasses, registered via AgentRegistry.register_cross_cutting().

Builder wiring — after building domain agents, iterate cross-cutting agents, call create_tool() on each, and append those tools to every domain agent's tool list alongside the existing RAG tool.

---

Part 2 — Ingestion pipeline as library

The ingestion/ package is CLI-only today. The Document Agent needs to call it within a request lifecycle, async.

Key adaptations needed:

• extract_text_from_pdf() takes Path, needs a bytes variant (follow pdf.py's fitz.open(stream=...) pattern)
• generate_embeddings() and upload_chunks() are sync — need async wrappers

New interface:

class IngestionPipeline:
    async def ingest_pdf(self, content: bytes, metadata: DocumentMetadata) -> IngestionResult: ...
    async def check_exists(self, document_id: str) -> bool: ...

Add ingestion as a path dependency in api/pyproject.toml (brings PyMuPDF ~15 MB into Docker image — acceptable).

---

Open questions

1. Docker image size — measure delta after adding ingestion dependency. If >100 MB, consider shared package.
2. Async embedding client — confirm where it should live in the API's DI setup.

---

Where to start

1. api/src/agents/_base.py, _registry.py — current agent pattern
2. api/src/agents/_discovery.py:14-30 — how discovery works
3. api/src/orchestrator/builder.py:97-143 and :339-413 — attachment handling and agent wiring
4. ingestion/src/pipeline/ — existing sync pipeline
5. Start with Part 1. No external dependencies, easy to test in isolation.