Distinguished Engineer, AI & ML Systems · UBS London
Building production AI agents, LLM evaluation frameworks, and MLOps infrastructure for Tier-1 financial institutions. Current focus: making AI systems that are explainable, auditable, and actually deployable in regulated environments — not just impressive in demos.
- Multi-agent systems — three-layer orchestration architecture: intent routing, specialist agents, reconciliation with human-in-the-loop escalation
- LLM evaluation — golden dataset frameworks, model-as-judge scoring, CI/CD regression gates aligned to FCA model risk standards
- MLOps & CI/CD for ML — Vertex AI pipelines, shadow deployments, automated rollback, full audit trails — cut model release cycles by 60%
- RAG at scale — hierarchical chunking, query rewriting, cross-encoder re-ranking, uncertainty thresholding for sub-second inference on millions of documents
- Enterprise AI adoption — bridging the gap between what AI can do in a lab and what Risk, Legal and Compliance will actually sign off on
| Repo | Description |
|---|---|
| llm-failure-taxonomy | 6-class system-level LLM production failure taxonomy — 50 labeled incidents, rule-based + Claude API classifiers, failure budget calculator. Companion code for Beyond Hallucination (arXiv / MLSys 2026) |
| enterprise-agent-framework | Production-grade multi-agent orchestration for regulated environments — router, specialist agents, reconciler, JSONL audit trail |
| llm-evaluation-toolkit | Golden dataset evaluation harness, model-as-judge scoring, CI/CD regression gates for production LLM releases |
| mlops-cicd-templates | GitHub Actions + Vertex AI ML pipelines — shadow deploys, canary rollouts, rollback scripts, BigQuery audit logging |
| rag-production-patterns | Hierarchical chunking, query rewriting, re-ranking, uncertainty thresholding — RAG patterns that survive production |
| financial-doc-intelligence | Document classification and extraction for financial documents with plain-English interpretability outputs |
| gcp-vertex-ai-accelerators | Vertex AI, BigQuery ML and Cloud Run utility scripts — reusable accelerators from real enterprise deployments |
| gpu-llm-profiler | NVIDIA GPU inference profiler — speculative decoding acceptance-rate sweep (γ 1–8), KV cache hit rate vs context length (with/without prefix caching), AWQ / GPTQ / FP16 throughput comparison. Built on vLLM. 14 unit tests run GPU-free. |
Beyond Hallucination: A System-Level Failure Taxonomy for Production LLMs · Zenodo, 2026 · Sole author
Most LLM safety work focuses on hallucination. This paper argues that's the wrong frame — hallucination is one of six distinct failure classes, and roughly half of production failures are silent (no error signal, no flagged output). Introduces the FC-A/B/C/D failure budget framework with risk-tiered governance, a multi-label classifier (~88% accuracy), and a failure-budget calculator. Dataset: 50 annotated real-world production incidents.
| Project | Contribution | Status |
|---|---|---|
| Anthropic claude-cookbooks | 08_The_compliance_aware_agent.ipynb — 44-cell notebook: FC-A/B/C/D failure budget framework, three-layer multi-agent orchestration, confidence-based escalation, human-in-the-loop for FCA/MiFID II/Basel III regulated environments | PR #783 · open |
| Anthropic claude-cookbooks | evals/model_as_judge/ — model-as-judge evaluation pipeline: golden dataset design, rubric scoring (0.99 judge-human correlation), regression gates, shadow deployment eval, CI entry point | PR #786 · open |
| LangChain / LangGraph | compliance_checkpoint_fca_mifid2.ipynb — compliance-aware HITL checkpoint: 4-node StateGraph, append-only SQLite audit trail (tamper-proof triggers), write-intent-before-execute pattern, uuid5 idempotency key, 5 runnable scenarios — FCA SYSC / MiFID II Art. 25 / SR 11-7 | PR #8422 · awaiting maintainer assignment |
| NVIDIA NeMo Guardrails | financial_services_compliance_guardrails.ipynb — GLiNER PII detection (IBAN, sort code, account number, NI number), MiFID II topic control via NIM, FCA COBS 4 disclaimer enforcement, Consumer Duty / SR 11-7 audit trail — 5 runnable scenarios with assertions | PR #2216 · open |
| HuggingFace evaluate | financial_llm_faithfulness — rule-based metric for regulated financial AI: numerical faithfulness (% / bps / monetary values vs reference), FCA COBS 4 / MiFID II disclaimer detection, composite compliance risk score (0–1). First financial metric in the library. Deterministic — no external APIs. | PR #783 · open |
| OpenHands | Financial document intelligence utility — classifies earnings reports, loan agreements, audit reports and regulatory filings with plain-English risk outputs. No external APIs. | PR #14271 · under review |
| vLLM | benchmarks/benchmark_spec_decode_analysis.py — speculative decoding analysis script: sweeps gamma (1–8), records acceptance rate (α = (speedup−1)/γ), tok/s, p50/p95/p99 latency, GPU memory. First spec decode analysis script in vllm/benchmarks/. SSH-signed. | PR #49825 · open |
| vLLM | silu_and_mul_dynamic_per_token_quant CUDA kernel — fused SiLU+gating + dynamic per-token FP8 quantization kernel. Two-phase: Phase 1 computes per-token absmax via warp_max + shared memory reduction; Phase 2 quantizes to FP8. Fills gap left by existing per-tensor silu_and_mul_quant. Addresses Q3 2026 SIG-Quantization roadmap item #48168. | PR #49828 · open |
| flash-attention | tests/test_flash_attn_gqa_decode.py — 6 test groups for production LLM serving gaps: GQA extreme ratios (8:1/16:1/32:1 — Llama 3 70B patterns), speculative decode verify phase (seqlen_q=2–8), chunked prefill+decode varlen mixed batches, paged KV small block sizes [16,32,128], softcap+GQA decode, long context (8k/16k) GQA backward. SSH-signed. | PR #2730 · open |
