diff --git a/README.md b/README.md index a18ac33..355a933 100644 --- a/README.md +++ b/README.md @@ -135,6 +135,50 @@ Tracked across every debate session: - Disagreement persistence across rounds --- +## Getting Started + +### Prerequisites + +Ensure you have Python 3.10+ installed and set your OpenAI API key: + +```bash +export OPENAI_API_KEY=sk-... +``` + +Install the dependencies: + +```bash +pip install -r requirements.txt +``` + +### 1. Ingest Data + +Before running a debate, the agents need a retrieval corpus (FAISS index). ArgumentLab includes a sample corpus to get started instantly: + +```bash +python setup/ingest_corpus.py --sample +``` + +You can also ingest your own `.txt` or `.pdf` documents: + +```bash +python setup/ingest_corpus.py --docs path/to/your/documents/ +``` + +### 2. Run a Debate + +Execute a full, structured debate by providing a proposition. The debate streams live to the console, printing argument blocks and judge scores round-by-round. + +```bash +python setup/debate.py \ + --proposition "Companies should replace legacy infrastructure with AI-driven systems." \ + --session-id my_debate_001 +``` + +Once finished, the debate state is automatically exported to `local_data/results/my_debate_001.json` and a human-readable `my_debate_001.md` report. + +--- + ## Demo Flow 1. Input a real-world question (e.g., *"Should companies replace legacy infrastructure with AI-driven systems?"*) diff --git a/docs/design.md b/docs/design.md index c58065e..6d45e36 100644 --- a/docs/design.md +++ b/docs/design.md @@ -32,10 +32,16 @@ Implements the core LangGraph agent nodes. Both agents follow a strict, determin ### 4. `core/retriever.py` (RAG Interface) A thin abstraction over the vector database (e.g. FAISS). It exposes `retrieve_multi()` which aggregates search results for multiple queries and deduplicates them by `source_id`, guaranteeing the best chunks are surfaced to the agent. -### 5. `core/prompts.py` -Isolates all LangChain `ChatPromptTemplate` strings. It handles formatting debate histories and chunk excerpts, making it easy to iterate on prompt wording without touching workflow logic. +### 5. `core/evaluation.py` (Parallel Evaluators) +Contains the three concurrent evaluation nodes that run after the agents: +- **`judge_node`**: Uses an LLM to score both arguments across four dimensions, detects convergence/stalemate, updates the debate `status`, and increments the round. +- **`hallucination_check`**: Validates that cited sources explicitly support the claims. Appends failing claim IDs to `hallucination_flags`. +- **`contradiction_check`**: Compares current arguments against the agent's historical claims to detect goalpost shifting. Appends offending claim IDs to `contradiction_flags`. -### 6. `orchestrator/graph.py` (Workflow Topology) +### 6. `core/prompts.py` & `core/eval_prompts.py` +Isolate all LangChain `ChatPromptTemplate` strings. They handle formatting debate histories, chunk excerpts, and evaluation logic, making it easy to iterate on prompt wording without touching workflow logic. + +### 7. `orchestrator/graph.py` (Workflow Topology) This file compiles the `StateGraph` that controls the execution flow. It is heavily parallelized to reduce latency: - **Agent Fan-out**: The `start_round` node branches unconditionally to `proponent_node` and `opponent_node`, running them concurrently. - **Evaluation Sync & Fan-out**: Both agents join at a dummy node (`start_evaluation`). From there, the graph fans out again to three concurrent evaluation nodes: `judge_node`, `hallucination_check`, and `contradiction_check`. diff --git a/local_data/sample_corpus.json b/local_data/sample_corpus.json new file mode 100644 index 0000000..8c14ac2 --- /dev/null +++ b/local_data/sample_corpus.json @@ -0,0 +1,82 @@ +[ + { + "title": "McKinsey Global Institute — The Economic Potential of Generative AI (2023)", + "chunks": [ + "Generative AI could add between $2.6 trillion and $4.4 trillion annually across 63 use cases analyzed. Customer operations, marketing and sales, software engineering, and R&D account for roughly 75% of this total potential value.", + "Legacy IT systems represent one of the most significant structural barriers to AI adoption in large enterprises. Organizations with fragmented, siloed data architectures report 40–60% longer AI deployment timelines than those operating on modernized infrastructure.", + "The total cost of maintaining legacy infrastructure in financial services, insurance, and telecommunications averages 70–80% of annual IT budgets, leaving limited capital available for AI investment or experimentation.", + "Enterprises that completed cloud migration and data platform modernization before beginning AI adoption report 2.5x higher rates of successful AI deployment at scale compared to those attempting AI on top of legacy stacks.", + "AI-driven automation could affect 60–70% of employees' time across occupations. The near-term impact is primarily augmentation rather than full replacement — workers shift toward higher-judgment tasks while AI handles routine processing.", + "Technical debt in legacy systems compounds over time. Organizations that defer modernization face transition costs that grow at an estimated 15–20% annually, making earlier action significantly cheaper in present-value terms.", + "Cyber vulnerabilities in legacy systems account for 72% of critical infrastructure incidents reported to CISA in 2022. AI-native architectures with continuous monitoring and automated patch management reduce this attack surface structurally.", + "The productivity gains from AI are not uniformly distributed. Organizations in the top quartile of AI maturity — characterized by clean data infrastructure, strong MLOps practices, and executive alignment — capture 3–4x more value than median adopters.", + "Industries with the highest proportions of legacy infrastructure — banking, insurance, and utilities — also face the greatest competitive threat from AI-native entrants who do not carry the same technical debt burden." + ] + }, + { + "title": "Harvard Business Review — Why AI Transformations Fail (2023)", + "chunks": [ + "More than 80% of enterprise AI projects fail to reach production. The most commonly cited root causes are data quality issues, misaligned organizational incentives, and insufficient change management — not algorithmic or model limitations.", + "Big-bang infrastructure replacement projects have a 70% failure rate in large enterprises, driven by scope creep, budget overruns, and organizational change fatigue. Incremental modernization strategies succeed at nearly twice the rate of full replacements.", + "Replacing legacy infrastructure destroys tacit operational knowledge that staff have accumulated over years or decades. This knowledge is rarely captured in documentation and is consistently underestimated during migration planning.", + "Organizations that attempt full legacy replacement simultaneously with major AI deployment report a compounding failure rate. Technical complexity and organizational disruption interact to derail both initiatives at once.", + "Middleware and API abstraction layers — so-called 'strangler fig' patterns — can expose legacy system capabilities to AI tools without requiring full replacement. This approach achieves 60–80% of the productivity gains at a fraction of the transition cost and risk.", + "The average enterprise AI transformation requires 3–5 years to generate measurable ROI. Executive and board impatience with this timeline is one of the leading causes of premature program cancellation before value is captured.", + "Vendor lock-in risk is acute in AI infrastructure decisions. Organizations that build on proprietary AI cloud platforms report significantly higher switching costs and reduced negotiating leverage when performance expectations are not met.", + "Staff resistance to AI adoption correlates more strongly with the disruption caused by infrastructure change than with the AI systems themselves. Employees in organizations that retained familiar workflows while layering AI on top report higher adoption rates and satisfaction.", + "The business case for legacy replacement is frequently overstated. Consulting-led transformation programs have a documented pattern of front-loading projected benefits while deferring transition risks and costs to later project phases." + ] + }, + { + "title": "Gartner — IT Modernization and AI Readiness Report (2024)", + "chunks": [ + "By 2026, Gartner projects that 75% of enterprises will have deployed AI in at least one major business function, but fewer than 30% will have achieved AI readiness at the underlying infrastructure level.", + "The true cost of full legacy replacement must account for hidden transition expenses: data migration, staff retraining, regulatory re-certification, parallel system operation, and a productivity dip averaging 12–18 months during the transition period.", + "AI systems require clean, labeled, high-volume, and continuously updated data. Legacy systems frequently store data in proprietary formats, on-premises databases, and siloed line-of-business applications that are structurally incompatible with modern ML pipelines.", + "Organizations in regulated industries — banking, healthcare, energy, and utilities — face additional constraints on infrastructure modernization, including regulatory approval processes that can extend transition timelines by 2–4 years beyond private-sector peers.", + "A hybrid architecture strategy, in which AI capabilities are layered incrementally on top of selectively modernized legacy components, is rated as the most pragmatic approach by 68% of enterprise CIOs surveyed in Gartner's 2024 CIO Agenda study.", + "Technical staff attrition accelerates during major infrastructure transitions. Enterprises report 20–35% higher voluntary turnover among senior engineers during multi-year modernization programs, creating a dangerous knowledge drain at the moment institutional knowledge matters most.", + "AI-native companies that have never operated legacy infrastructure consistently outperform legacy-burdened incumbents on AI deployment speed, model iteration cycles, and data quality metrics — but also on unit economics, with AI-native firms spending 40% less per AI workload.", + "The gap between AI aspiration and AI readiness is widening. In Gartner's 2024 survey, 89% of enterprise leaders identified AI as a top-three strategic priority, but only 23% rated their data infrastructure as ready to support production AI at scale.", + "Composable architecture — building modular, API-first systems that can be assembled and reassembled — is emerging as the dominant post-legacy strategy, allowing organizations to retire legacy components gradually while maintaining operational continuity." + ] + }, + { + "title": "MIT Sloan Management Review — Legacy Modernization Strategies (2023)", + "chunks": [ + "The 'strangle the monolith' pattern — gradually replacing legacy components with microservices while keeping the core system operational — has emerged as the dominant risk-mitigation strategy for enterprises that cannot tolerate operational disruption during transition.", + "Organizations that fully replaced legacy ERP systems with AI-native alternatives reported an average productivity improvement of 23% over a three-year post-transition period, but also experienced an average of 14 months of disruption during and immediately after the migration.", + "Legacy systems often encode decades of undocumented business logic. AI systems trained on the outputs of these legacy systems may inadvertently perpetuate historical biases, regulatory workarounds, and operational inefficiencies embedded in the original design.", + "Cloud-native AI infrastructure enables elasticity, global distribution, and continuous model retraining at cost structures that are structurally unavailable to on-premises legacy architectures — a permanent competitive disadvantage for organizations that delay.", + "The risk of catastrophic failure during legacy migration is not evenly distributed across the project lifecycle. It is concentrated in the data migration phase, where data loss, corruption, or field-mapping errors can have irreversible downstream effects on model training and decision-making.", + "Companies that operate parallel legacy and AI-native systems during transition incur dramatically higher operational costs — typically 30–50% above steady-state — but report significantly lower rates of critical business disruption than organizations that pursue hard cutover strategies.", + "AI governance frameworks increasingly require full auditability of data lineage, model decisions, and system access logs. Many legacy systems cannot provide the provenance and logging capabilities that modern AI governance standards demand, creating regulatory exposure.", + "The productivity argument for legacy replacement is time-horizon dependent. On a 2-year horizon, replacement costs almost always exceed benefits. On a 7–10 year horizon, the compound productivity advantage of AI-native infrastructure typically justifies the investment.", + "Middle management resistance is a more reliable predictor of legacy modernization failure than technical complexity. Managers whose authority derives from expertise in legacy processes have structural incentives to delay transitions that would commoditize their knowledge." + ] + }, + { + "title": "Brookings Institution — AI and the Future of Industrial Infrastructure (2024)", + "chunks": [ + "The United States has an estimated $2.1 trillion in aging enterprise infrastructure across manufacturing, finance, healthcare, and logistics — much of it running on software platforms that predate the commercial internet and were not designed for data interoperability.", + "AI adoption in manufacturing has demonstrated measurable gains in predictive maintenance, yield optimization, and supply chain resilience. However, 61% of manufacturers report that sensor and control system incompatibility with modern data platforms is their primary barrier to scaling.", + "The national security implications of legacy infrastructure vulnerabilities are significant. Critical infrastructure sectors — energy grids, water systems, financial clearing networks — are disproportionately dependent on aging systems with known, unpatched vulnerabilities that AI-enhanced cyber attacks can exploit systematically.", + "Public-sector organizations face a distinct set of constraints on AI adoption: procurement rules, multi-year budget cycles, civil service hiring limitations, and political risk aversion create structural inertia that private-sector modernization timelines do not account for.", + "The environmental cost of maintaining legacy data center infrastructure is substantial. Legacy on-premises systems consume an estimated 3–5x more energy per compute unit than modern cloud infrastructure, a gap that widens as AI workloads increase in intensity.", + "Workforce displacement from AI-driven infrastructure modernization is not uniform across the economy. High-skill technical roles show strong net job growth; administrative, data entry, and routine processing roles show the steepest displacement, concentrated in regions already facing economic stress.", + "International competitiveness data shows that countries with higher rates of enterprise infrastructure modernization — South Korea, Singapore, Estonia — also show faster AI productivity growth, suggesting infrastructure readiness is a macroeconomic variable, not just a firm-level one." + ] + }, + { + "title": "IEEE Transactions on Software Engineering — Technical Debt and AI Readiness (2023)", + "chunks": [ + "Technical debt is not a binary condition. The IEEE Software Engineering Body of Knowledge classifies it across four dimensions: code debt, data debt, architectural debt, and test debt. AI readiness requires addressing all four — organizations that modernize code while leaving data debt unresolved consistently fail to deploy AI at scale.", + "COBOL-based systems still process an estimated $3 trillion in daily transactions in the global banking system. The average age of a production COBOL application is 43 years. Replacing these systems introduces operational risk that quantitatively exceeds the risk of continuing to operate them for most institutions.", + "The 'rewrite from scratch' anti-pattern in software engineering is well-documented. Systems built from scratch to replace working legacy software have a higher defect rate in the first 3 years of operation than the systems they replaced, due to the loss of implicit correctness encoded in the legacy codebase.", + "Containerization and API-first design patterns allow AI inference services to be deployed alongside legacy systems without requiring full replacement. This architectural pattern, sometimes called 'AI wrapping,' has shown 70–85% of the performance gains of full replacement at 20–30% of the cost in case studies across financial services.", + "Data schema drift — the gradual divergence between how a system stores data and what downstream consumers expect — is the most common technical failure mode in AI deployments on legacy infrastructure. Without schema versioning and data contracts, AI models trained on historical data rapidly become miscalibrated.", + "The empirical evidence on microservices migrations is mixed. Organizations with fewer than 500 engineers show no statistically significant productivity gain from microservices architecture over well-maintained monoliths. The benefits of microservices are realized primarily at organizational scale, not technical scale.", + "AI model governance requires model versioning, data lineage tracking, and reproducible training pipelines. Legacy infrastructure built before MLOps was a recognized discipline cannot support these requirements without significant re-engineering — creating a compliance gap that regulators are beginning to close." + ] + } +] \ No newline at end of file diff --git a/setup/debate.py b/setup/debate.py new file mode 100644 index 0000000..d50cf1a --- /dev/null +++ b/setup/debate.py @@ -0,0 +1,376 @@ +#!/usr/bin/env python3 +""" +scripts/run_debate.py + +Entry point for running a full ArgumentLab debate from the command line. + +Streams each node's output as it arrives via graph.stream(), printing +arguments and judge scores round-by-round. At the end, prints a structured +summary table and exports the full results to JSON + Markdown. + +Usage: + python setup/debate.py \ + --proposition "Companies should replace legacy infrastructure with AI-driven systems." \ + --session-id my_debate_001 + + # Optional flags: + --index-path local_data/faiss_index (default) + --output-dir local_data/results (default) + --top-k 4 (chunks retrieved per query) + +Prerequisites: + 1. export OPENAI_API_KEY=sk-... + 2. python setup/ingest_corpus.py --sample +""" + +import argparse +import os +import sys +import textwrap +import uuid +from datetime import datetime, timezone +from pathlib import Path + +# --------------------------------------------------------------------------- +# Path setup — allow running from repo root without pip install +# --------------------------------------------------------------------------- +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from argument_lab.core.faiss_index import FaissIndex +from argument_lab.core.retriever import Retriever +from argument_lab.core.state import DebateState, MAX_ROUNDS +from argument_lab.core.exporter import export_debate +from argument_lab.orchestrator.graph import build_graph + + +# --------------------------------------------------------------------------- +# ANSI colour helpers (degrade gracefully on Windows without colorama) +# --------------------------------------------------------------------------- + +def _supports_colour() -> bool: + return sys.stdout.isatty() and os.name != "nt" + + +RESET = "\033[0m" if _supports_colour() else "" +BOLD = "\033[1m" if _supports_colour() else "" +DIM = "\033[2m" if _supports_colour() else "" +CYAN = "\033[36m" if _supports_colour() else "" +GREEN = "\033[32m" if _supports_colour() else "" +YELLOW = "\033[33m" if _supports_colour() else "" +RED = "\033[31m" if _supports_colour() else "" +BLUE = "\033[34m" if _supports_colour() else "" + + +def _hr(char: str = "─", width: int = 72) -> str: + return char * width + + +# --------------------------------------------------------------------------- +# Streaming printer — handles each node update as it arrives +# --------------------------------------------------------------------------- + +def _print_agent_update(node_name: str, update: dict) -> None: + """ + Called when a proponent or opponent node emits a state update. + Prints the new argument in a readable format. + """ + new_args = update.get("arguments", []) + if not new_args: + return + + arg = new_args[-1] # the argument just produced + role_colour = CYAN if arg.agent == "proponent" else YELLOW + label = f"{role_colour}{BOLD}{arg.agent.upper()}{RESET}" + + print(f"\n {label} — Round {arg.round}") + print(f" {DIM}{_hr('·', 68)}{RESET}") + + # Wrap claim text for readability + claim_lines = textwrap.wrap(arg.claim, width=64) + print(f" {BOLD}Claim{RESET} (confidence: {arg.confidence_score:.2f})") + for line in claim_lines: + print(f" {line}") + + print(f" {BOLD}Evidence{RESET}") + for e in arg.evidence: + print(f" [{e.source_id}] (reliability: {e.reliability_score:.2f})") + excerpt_lines = textwrap.wrap(e.excerpt, width=60) + for line in excerpt_lines: + print(f" {DIM}{line}{RESET}") + + if arg.counterpoints_addressed: + ids = ", ".join(arg.counterpoints_addressed) + print(f" {BOLD}Addresses:{RESET} {DIM}{ids}{RESET}") + + if arg.assumptions: + print(f" {BOLD}Assumptions:{RESET}") + for a in arg.assumptions: + print(f" • {a}") + + +def _print_judge_update(update: dict) -> None: + """ + Called when the judge node emits a state update. + Prints the score table for the round just evaluated. + """ + scores = update.get("scores", []) + if not scores: + return + + score = scores[-1] + p = score.proponent_score + o = score.opponent_score + + verdict = "" + if score.convergence_detected: + verdict = f" {GREEN}{BOLD}✅ CONVERGENCE DETECTED{RESET}" + elif score.stalemate_detected: + verdict = f" {RED}{BOLD}⚖️ STALEMATE DETECTED{RESET}" + + print(f"\n {BLUE}{BOLD}JUDGE — Round {score.round}{RESET}") + print(f" {DIM}{_hr('·', 68)}{RESET}") + print(f" {'Dimension':<24} {'Proponent':>10} {'Opponent':>10}") + print(f" {DIM}{_hr('·', 46)}{RESET}") + + dims = [ + ("Logical Consistency", p.logical_consistency, o.logical_consistency), + ("Evidence Support", p.evidence_support, o.evidence_support), + ("Relevance", p.relevance, o.relevance), + ("Completeness", p.completeness, o.completeness), + ] + for name, pv, ov in dims: + print(f" {name:<24} {pv:>10.2f} {ov:>10.2f}") + + print(f" {DIM}{_hr('·', 46)}{RESET}") + print(f" {'Composite (weighted)':<24} {p.composite:>10.3f} {o.composite:>10.3f}") + + if verdict: + print(verdict) + + print(f"\n {BOLD}Judge's note:{RESET}") + for line in textwrap.wrap(score.explanation, width=64): + print(f" {line}") + + +def _print_hallucination_update(update: dict) -> None: + flags = update.get("hallucination_flags", []) + if flags: + print(f"\n {RED}⚠ Hallucination flags:{RESET} {', '.join(flags)}") + + +def _print_contradiction_update(update: dict) -> None: + flags = update.get("contradiction_flags", []) + if flags: + print(f"\n {RED}⚠ Contradiction flags:{RESET} {', '.join(flags)}") + + +# --------------------------------------------------------------------------- +# Summary table (printed at the end of all rounds) +# --------------------------------------------------------------------------- + +def _print_summary(final_state: DebateState) -> None: + scores = final_state.get("scores", []) + status = final_state.get("status", "unknown") + + status_display = { + "converged": f"{GREEN}✅ Converged{RESET}", + "stalemate": f"{YELLOW}⚖️ Stalemate{RESET}", + "terminated": f"{BLUE}🏁 Terminated (max rounds){RESET}", + "in_progress": f"{DIM}⏳ Still in progress{RESET}", + }.get(status, status) + + print(f"\n{BOLD}{_hr('═')}{RESET}") + print(f"{BOLD} DEBATE SUMMARY{RESET}") + print(f"{BOLD}{_hr('═')}{RESET}\n") + print(f" Status: {status_display}") + print(f" Rounds: {len(scores)} / {MAX_ROUNDS} completed\n") + + if scores: + print(f" {'Round':<8} {'Proponent':>12} {'Opponent':>12} {'Verdict'}") + print(f" {DIM}{_hr('·', 52)}{RESET}") + for s in sorted(scores, key=lambda x: x.round): + verdict = "" + if s.convergence_detected: + verdict = f"{GREEN}Converged{RESET}" + elif s.stalemate_detected: + verdict = f"{YELLOW}Stalemate{RESET}" + print( + f" {s.round:<8} " + f"{s.proponent_score.composite:>12.3f} " + f"{s.opponent_score.composite:>12.3f} " + f"{verdict}" + ) + + h_count = len(final_state.get("hallucination_flags", [])) + c_count = len(final_state.get("contradiction_flags", [])) + i_count = len(final_state.get("ignored_claims", [])) + + print(f"\n {BOLD}Evaluation flags{RESET}") + print(f" {'Hallucinations:':<22} {h_count}") + print(f" {'Contradictions:':<22} {c_count}") + print(f" {'Ignored claims:':<22} {i_count}") + + # Agent confidence drift + print(f"\n {BOLD}Confidence trajectories{RESET}") + for agent, positions in final_state.get("agent_positions", {}).items(): + trajectory = " → ".join(f"{v:.2f}" for v in positions) + print(f" {agent.capitalize():<14} {trajectory}") + + print(f"\n{BOLD}{_hr('═')}{RESET}\n") + + +# --------------------------------------------------------------------------- +# Entry point +# --------------------------------------------------------------------------- + +def main() -> None: + parser = argparse.ArgumentParser( + description="Run an ArgumentLab structured debate.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=textwrap.dedent(""" + Examples: + python setup/debate.py \ + --proposition "Companies should replace legacy infrastructure with AI." + + python setup/debate.py \ + --proposition "Remote work improves engineering productivity." \ + --session-id remote_work_001 \ + --top-k 6 + """), + ) + parser.add_argument( + "--proposition", + required=True, + help="The debate proposition. Agents will argue FOR and AGAINST this.", + ) + parser.add_argument( + "--session-id", + default=None, + help="Unique session identifier for output filenames. " + "Defaults to a timestamp-based ID.", + ) + parser.add_argument( + "--index-path", + default="local_data/faiss_index", + help="Path to the FAISS index directory (default: local_data/faiss_index)", + ) + parser.add_argument( + "--output-dir", + default="local_data/results", + help="Directory for JSON + Markdown output (default: local_data/results)", + ) + parser.add_argument( + "--top-k", + type=int, + default=4, + help="Number of evidence chunks to retrieve per query (default: 4)", + ) + args = parser.parse_args() + + # ── Pre-flight checks ────────────────────────────────────────────────── + if not os.environ.get("OPENAI_API_KEY"): + print(f"{RED}Error: OPENAI_API_KEY is not set.{RESET}") + print(" export OPENAI_API_KEY=sk-...") + sys.exit(1) + + session_id = args.session_id or ( + "debate_" + datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") + ) + + # ── Header ──────────────────────────────────────────────────────────── + print(f"\n{BOLD}{_hr('═')}{RESET}") + print(f"{BOLD} ARGUMENTLAB{RESET}") + print(f"{BOLD}{_hr('═')}{RESET}") + print(f"\n {BOLD}Proposition:{RESET}") + for line in textwrap.wrap(args.proposition, width=64): + print(f" {line}") + print(f"\n {BOLD}Session:{RESET} {session_id}") + print(f" {BOLD}Rounds:{RESET} {MAX_ROUNDS}") + print(f" {BOLD}Top-k:{RESET} {args.top_k}\n") + print(f"{BOLD}{_hr('═')}{RESET}\n") + + # ── Load index ───────────────────────────────────────────────────────── + print(f" Loading FAISS index from {args.index_path}...") + try: + faiss_index = FaissIndex.load(args.index_path) + except FileNotFoundError as e: + print(f"\n{RED}Error:{RESET} {e}") + sys.exit(1) + + retriever = Retriever(index=faiss_index, top_k=args.top_k) + + # ── Build graph ──────────────────────────────────────────────────────── + print(f" Building debate graph...\n") + debate_graph = build_graph(retriever) + + # ── Initial state ────────────────────────────────────────────────────── + initial_state: DebateState = { + "proposition": args.proposition, + "current_round": 1, + "arguments": [], + "claims_registry": {}, + "addressed_claims": set(), + "ignored_claims": set(), + "agent_positions": {}, + "repetition_flags": [], + "status": "in_progress", + "hallucination_flags": [], + "contradiction_flags": [], + "scores": [], + } + + # ── Stream the debate ────────────────────────────────────────────────── + current_round_printed = 0 + final_state: DebateState | None = None + + for chunk in debate_graph.stream(initial_state): + for node_name, update in chunk.items(): + # Print round header when we first see a new round's arguments + new_args = update.get("arguments", []) + if new_args: + round_num = new_args[-1].round + if round_num != current_round_printed: + current_round_printed = round_num + print(f"\n{BOLD}{_hr()}{RESET}") + print(f"{BOLD} ROUND {round_num}{RESET}") + print(f"{BOLD}{_hr()}{RESET}") + + # Dispatch to per-node printers + if node_name in ("proponent", "opponent"): + _print_agent_update(node_name, update) + elif node_name == "judge": + _print_judge_update(update) + elif node_name == "hallucination_check": + _print_hallucination_update(update) + elif node_name == "contradiction_check": + _print_contradiction_update(update) + + # Accumulate the last known full state + # LangGraph's stream() yields (node_name, state_delta) tuples; + # the final full state is available via invoke() but we reconstruct + # it from the last graph_update node output which holds the full state. + if node_name == "graph_update": + final_state = update # graph_update passthrough holds full state + + # Fallback: run invoke() to guarantee we have the final state + if final_state is None or "proposition" not in final_state: + print(f"\n {DIM}Retrieving final state...{RESET}") + final_state = debate_graph.invoke(initial_state) + + # ── Summary ──────────────────────────────────────────────────────────── + _print_summary(final_state) + + # ── Export ───────────────────────────────────────────────────────────── + print(f" Exporting results...") + json_path, md_path = export_debate( + state=final_state, + session_id=session_id, + output_dir=args.output_dir, + ) + print(f"\n {GREEN}✓{RESET} JSON: {json_path}") + print(f" {GREEN}✓{RESET} Markdown: {md_path}") + print(f"\n{BOLD}{_hr('═')}{RESET}\n") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/setup/ingest_corpus.py b/setup/ingest_corpus.py new file mode 100644 index 0000000..292b59d --- /dev/null +++ b/setup/ingest_corpus.py @@ -0,0 +1,273 @@ +#!/usr/bin/env python3 +""" +scripts/ingest_corpus.py + +Chunks source documents, embeds them with OpenAI, and saves a FAISS index +to disk so the debate agents have a real retrieval corpus. + +Supports two modes: + --sample Use the bundled sample corpus (local_data/sample_corpus.json). + Good for testing and demos. No document prep required. + + --docs PATH Ingest a folder of .txt or .pdf files for production use. + Each file is chunked, embedded, and added to the index. + +The index is always saved to local_data/faiss_index/ and will overwrite +any existing index at that path. + +Usage: + # Sample corpus (fast, for testing): + python setup/ingest_corpus.py --sample + + # Real documents: + python setup/ingest_corpus.py --docs path/to/your/documents/ + + # Both (sample + real docs merged into one index): + python setup/ingest_corpus.py --sample --docs path/to/your/documents/ +""" + +import argparse +import json +import os +import sys +import textwrap +from pathlib import Path + +# --------------------------------------------------------------------------- +# Path setup — allow running from repo root without pip install +# --------------------------------------------------------------------------- +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from argument_lab.core.faiss_index import ChunkRecord, FaissIndex + + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +SAMPLE_CORPUS_PATH = Path(__file__).resolve().parent.parent / "local_data" / "sample_corpus.json" +INDEX_OUTPUT_PATH = Path(__file__).resolve().parent.parent / "local_data" / "faiss_index" + +CHUNK_SIZE = 400 # target characters per chunk +CHUNK_OVERLAP = 80 # character overlap between adjacent chunks + + +# --------------------------------------------------------------------------- +# Chunking +# --------------------------------------------------------------------------- + +def chunk_text(text: str, source_id_prefix: str) -> list[ChunkRecord]: + """ + Splits text into overlapping fixed-size chunks. Each chunk becomes one + row in the FAISS index. + + For the sample corpus, source documents are already pre-chunked at a + sentence level, so this function is primarily used for raw .txt/.pdf + ingestion where documents arrive as continuous text. + """ + text = text.strip() + chunks: list[ChunkRecord] = [] + start = 0 + chunk_idx = 0 + + while start < len(text): + end = start + CHUNK_SIZE + excerpt = text[start:end].strip() + + # Don't create a chunk that's just whitespace or too short to be useful + if len(excerpt) >= 40: + chunks.append(ChunkRecord( + source_id=f"{source_id_prefix}_chunk_{chunk_idx:03d}", + excerpt=excerpt, + doc_title=source_id_prefix, + )) + chunk_idx += 1 + + start = end - CHUNK_OVERLAP # overlap for context continuity + + return chunks + + +# --------------------------------------------------------------------------- +# Sample corpus loader +# --------------------------------------------------------------------------- + +def load_sample_corpus() -> list[ChunkRecord]: + """ + Loads the bundled sample_corpus.json. Each document entry contains a + title and a list of pre-chunked excerpts, so we don't re-chunk them — + they're already at the right granularity. + """ + if not SAMPLE_CORPUS_PATH.exists(): + raise FileNotFoundError( + f"Sample corpus not found at {SAMPLE_CORPUS_PATH}. " + "Ensure local_data/sample_corpus.json is present in the repo." + ) + + with open(SAMPLE_CORPUS_PATH, "r", encoding="utf-8") as f: + documents = json.load(f) + + records: list[ChunkRecord] = [] + for doc_idx, doc in enumerate(documents): + title = doc.get("title", f"doc_{doc_idx:03d}") + # Sanitise title for use as a source_id prefix + prefix = f"doc_{doc_idx:03d}" + for chunk_idx, excerpt in enumerate(doc.get("chunks", [])): + records.append(ChunkRecord( + source_id=f"{prefix}_chunk_{chunk_idx:03d}", + excerpt=excerpt.strip(), + doc_title=title, + )) + + print(f"[ingest] Sample corpus: {len(documents)} documents → {len(records)} chunks") + return records + + +# --------------------------------------------------------------------------- +# Real document loader +# --------------------------------------------------------------------------- + +def load_docs_folder(docs_path: Path) -> list[ChunkRecord]: + """ + Ingests all .txt and .pdf files from a folder. Each file is chunked + with overlapping windows and turned into ChunkRecords. + + PDF support requires pypdf (`pip install pypdf`). If pypdf is not + installed, PDF files are skipped with a warning. + """ + records: list[ChunkRecord] = [] + files = sorted(docs_path.glob("*")) + supported = {".txt", ".pdf"} + + for file_idx, filepath in enumerate(files): + suffix = filepath.suffix.lower() + if suffix not in supported: + print(f"[ingest] Skipping unsupported file type: {filepath.name}") + continue + + prefix = f"user_{file_idx:03d}_{filepath.stem[:30]}" + text = _extract_text(filepath) + + if not text: + print(f"[ingest] Warning: no text extracted from {filepath.name}") + continue + + file_chunks = chunk_text(text, source_id_prefix=prefix) + records.extend(file_chunks) + print(f"[ingest] {filepath.name}: {len(file_chunks)} chunks") + + print(f"[ingest] User docs: {len(files)} files → {len(records)} chunks") + return records + + +def _extract_text(filepath: Path) -> str: + if filepath.suffix.lower() == ".txt": + return filepath.read_text(encoding="utf-8", errors="ignore") + + if filepath.suffix.lower() == ".pdf": + try: + from pypdf import PdfReader + except ImportError: + print( + f"[ingest] Warning: pypdf not installed — skipping {filepath.name}. " + "Install with: pip install pypdf" + ) + return "" + reader = PdfReader(str(filepath)) + return "\n".join( + page.extract_text() or "" for page in reader.pages + ) + + return "" + + +# --------------------------------------------------------------------------- +# Deduplication +# --------------------------------------------------------------------------- + +def deduplicate(records: list[ChunkRecord]) -> list[ChunkRecord]: + """ + Removes exact-duplicate excerpts that arise when sample corpus and + user docs overlap. Keeps the first occurrence. + """ + seen: set[str] = set() + unique: list[ChunkRecord] = [] + for record in records: + key = record.excerpt.strip().lower() + if key not in seen: + seen.add(key) + unique.append(record) + removed = len(records) - len(unique) + if removed: + print(f"[ingest] Removed {removed} duplicate chunks") + return unique + + +# --------------------------------------------------------------------------- +# Entry point +# --------------------------------------------------------------------------- + +def main() -> None: + parser = argparse.ArgumentParser( + description="Build the ArgumentLab FAISS retrieval index.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=textwrap.dedent(""" + Examples: + python setup/ingest_corpus.py --sample + python setup/ingest_corpus.py --docs ./my_documents/ + python setup/ingest_corpus.py --sample --docs ./my_documents/ + """), + ) + parser.add_argument( + "--sample", + action="store_true", + help="Include the bundled sample corpus (local_data/sample_corpus.json)", + ) + parser.add_argument( + "--docs", + type=Path, + metavar="PATH", + help="Path to a folder of .txt or .pdf files to ingest", + ) + args = parser.parse_args() + + if not args.sample and not args.docs: + parser.error("Specify at least one of --sample or --docs PATH") + + if not os.environ.get("OPENAI_API_KEY"): + print( + "[ingest] Warning: OPENAI_API_KEY is not set. " + "Embedding calls will fail unless you export it first.\n" + " export OPENAI_API_KEY=sk-..." + ) + + # Collect all chunks + all_records: list[ChunkRecord] = [] + + if args.sample: + all_records.extend(load_sample_corpus()) + + if args.docs: + docs_path = args.docs + if not docs_path.is_dir(): + print(f"[ingest] Error: '{docs_path}' is not a directory.") + sys.exit(1) + all_records.extend(load_docs_folder(docs_path)) + + if not all_records: + print("[ingest] No chunks produced. Check your inputs.") + sys.exit(1) + + all_records = deduplicate(all_records) + print(f"[ingest] Total chunks to embed: {len(all_records)}") + + # Build and save the index + index = FaissIndex.build(all_records) + index.save(INDEX_OUTPUT_PATH) + + print(f"\n[ingest] Done. Index saved to: {INDEX_OUTPUT_PATH}") + print(f"[ingest] Run a debate with: python setup/debate.py --proposition \"...\"") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/src/argument_lab/core/agents.py b/src/argument_lab/core/agents.py index fefc672..b19bbb0 100644 --- a/src/argument_lab/core/agents.py +++ b/src/argument_lab/core/agents.py @@ -22,7 +22,7 @@ from argument_lab.core.models import Argument, Claim, EvidenceRef from argument_lab.core.retriever import Retriever, RetrieverError -from argument_lab.core.state import DebateState +from argument_lab.core.state import DebateState, MAX_ROUNDS from argument_lab.core.prompts import ( QUERY_FORMULATION_SYSTEM, QUERY_FORMULATION_USER, @@ -77,7 +77,7 @@ def _formulate_queries( "stance": stance, "history": history, "current_round": current_round, - "round_goal": ROUND_GOALS[min(current_round, 3)], + "round_goal": ROUND_GOALS[min(current_round, MAX_ROUNDS)], }) queries = result.get("queries", []) if isinstance(queries, list) and all(isinstance(q, str) for q in queries): @@ -134,7 +134,7 @@ def _generate_argument( "role": role, "stance": "FOR" if role == "Proponent" else "AGAINST", "proposition": proposition, - "counterpoint_rule": COUNTERPOINT_RULES[min(current_round, 3)], + "counterpoint_rule": COUNTERPOINT_RULES[min(current_round, MAX_ROUNDS)], "evidence_context": evidence_context, }) @@ -209,7 +209,7 @@ def _enforce_counterpoint_rule( "role": role, "stance": "FOR" if role == "Proponent" else "AGAINST", "proposition": proposition, - "counterpoint_rule": COUNTERPOINT_RULES[min(current_round, 3)], + "counterpoint_rule": COUNTERPOINT_RULES[min(current_round, MAX_ROUNDS)], "evidence_context": evidence_context, })), ("user", AGENT_USER_TEMPLATE.format_map({ diff --git a/src/argument_lab/core/eval_prompts.py b/src/argument_lab/core/eval_prompts.py new file mode 100644 index 0000000..d48a9b5 --- /dev/null +++ b/src/argument_lab/core/eval_prompts.py @@ -0,0 +1,253 @@ +""" +argument_lab/core/eval_prompts.py + +Prompt templates for the three parallel evaluation nodes: + - Judge (scoring + convergence/stalemate detection) + - Hallucination checker (evidence grounding verification) + - Contradiction checker (cross-round consistency auditing) + +Kept separate from agent prompts so each can be tuned and versioned +independently without touching orchestration code. +""" + +from argument_lab.core.models import Argument, JudgeEvaluation + + +# --------------------------------------------------------------------------- +# Shared formatting helpers +# --------------------------------------------------------------------------- + +def format_argument_for_eval(arg: Argument) -> str: + """ + Renders a single Argument as a clearly labelled block for evaluation + prompts. Includes all fields the evaluator needs to do its job. + """ + evidence_lines = "\n".join( + f" [{e.source_id}] \"{e.excerpt}\" (reliability: {e.reliability_score:.2f})" + for e in arg.evidence + ) + addressed = ", ".join(arg.counterpoints_addressed) or "none" + return ( + f"Agent: {arg.agent.upper()}\n" + f"Claim ID: {arg.id}\n" + f"Claim: {arg.claim}\n" + f"Evidence:\n{evidence_lines}\n" + f"Assumptions: {', '.join(arg.assumptions) or 'none'}\n" + f"Counterpoints addressed: {addressed}\n" + f"Confidence declared: {arg.confidence_score:.2f}" + ) + + +def format_prior_scores(scores: list[JudgeEvaluation]) -> str: + """ + Renders the composite score trajectory for both agents across prior + rounds. Injected into the judge prompt so the stalemate detector has + the numbers it needs. + """ + if not scores: + return "No prior rounds scored yet — this is Round 1." + lines = [] + for s in scores: + p = s.proponent_score.composite + o = s.opponent_score.composite + lines.append( + f" Round {s.round}: " + f"Proponent={p:.3f} Opponent={o:.3f}" + ) + return "\n".join(lines) + + +def format_prior_args_for_agent(args: list[Argument], agent: str) -> str: + """ + Returns all prior arguments from a single agent, formatted for the + contradiction checker. Ordered chronologically so the LLM can track + how the agent's position evolved. + """ + agent_args = sorted( + [a for a in args if a.agent == agent], + key=lambda a: a.round, + ) + if not agent_args: + return "No prior arguments from this agent." + return "\n\n".join(format_argument_for_eval(a) for a in agent_args) + + +# --------------------------------------------------------------------------- +# Judge prompts +# --------------------------------------------------------------------------- + +JUDGE_SYSTEM = """\ +You are an impartial debate judge evaluating structured arguments in a \ +multi-round AI debate. You do not take sides. Your only job is to score \ +each argument objectively on four dimensions and determine whether the \ +debate has reached a meaningful conclusion. + +Scoring rubric — all scores in [0.0, 1.0]: + + logical_consistency (weight 30%) + Does the conclusion follow from the premises? + Are there internal contradictions within the argument itself? + + evidence_support (weight 30%) + Are the claims backed by the cited sources? + Does the evidence actually say what the agent claims it says? + Penalise heavily if the agent asserts facts not present in the evidence. + + relevance (weight 20%) + Does the argument address the stated proposition directly? + Penalise arguments that pivot to related but different claims. + + completeness (weight 20%) + Does the argument meaningfully engage with the opponent's strongest \ +prior point? + An argument that ignores a strong counterpoint scores low here. + +Convergence rule: + Set convergence_detected=true ONLY if both agents have explicitly \ +conceded or accepted a shared core claim in the arguments you are evaluating. \ +A high score for both agents does NOT constitute convergence. + +Stalemate rule: + Set stalemate_detected=true if BOTH of the following are true: + (a) The debate is past Round 1. + (b) Neither agent's composite score has improved by more than 0.05 \ +compared to their score in the immediately prior round. + If no prior scores exist, stalemate_detected must be false. + +You must respond using the required JSON schema exactly. +No preamble. No prose outside the schema fields. +""" + +JUDGE_USER = """\ +Proposition: "{proposition}" +Round being evaluated: {current_round} + +Prior round score history: +{prior_scores} + +Arguments to evaluate this round: + +--- PROPONENT --- +{proponent_arg} + +--- OPPONENT --- +{opponent_arg} + +Score both arguments on all four rubric dimensions. Determine convergence \ +and stalemate status per the rules above. Provide a concise 2-4 sentence \ +justification in the explanation field covering the key reasons for your \ +scores and your verdict. +""" + + +# --------------------------------------------------------------------------- +# Hallucination checker prompts +# --------------------------------------------------------------------------- + +HALLUCINATION_SYSTEM = """\ +You are a strict evidence auditor for a structured AI debate. Your only job \ +is to verify that every factual claim in an argument is explicitly and \ +directly supported by the evidence the agent cited. + +You are NOT evaluating argument quality, logic, or persuasiveness. +You are ONLY checking: does the cited text actually say what the agent \ +claims it says? + +Flag a claim if ANY of the following are true: + - The claim states a specific fact (number, statistic, name, date, causal \ +relationship) that does not appear in the cited evidence excerpts. + - The claim makes a logical leap that goes materially beyond what the \ +evidence states — even if the leap seems reasonable. + - The agent's declared confidence is materially higher than the evidence \ +warrants (e.g., claims certainty when the evidence only shows correlation). + +Severity guide: + high A specific verifiable fact is directly contradicted by the \ +evidence, or is entirely absent from all cited sources. + medium The evidence is related and plausible but does not clearly or \ +explicitly support the specific claim being made. + low The connection is reasonable but the evidence is indirect, \ +thin, or only partially relevant. + +If ALL claims in the argument are well-grounded in the cited evidence, \ +return an empty flags list. Do not manufacture flags. + +Respond using the required JSON schema only. No prose outside the schema. +""" + +HALLUCINATION_USER = """\ +Proposition under debate: "{proposition}" + +Evaluate the following argument for hallucinated evidence connections: + +{argument_block} + +For each claim in this argument, verify whether the cited evidence \ +explicitly supports it. Return a flag only for claims that fail this check. \ +Each flag must reference the claim_id shown above. +""" + + +# --------------------------------------------------------------------------- +# Contradiction checker prompts +# --------------------------------------------------------------------------- + +CONTRADICTION_SYSTEM = """\ +You are a logical consistency auditor for a structured multi-round AI debate. \ +Your job is to detect whether an agent is contradicting their own prior \ +arguments — either directly or subtly across rounds. + +You are NOT evaluating argument quality or whether the agent is right or wrong. \ +You are ONLY checking internal consistency within a single agent's argument \ +history. + +Contradiction types to detect: + + direct_negation + The current claim explicitly states the opposite of a prior claim. + + weakened_commitment + The agent previously asserted X with high confidence but now qualifies \ +or walks back X without acknowledging the shift or citing new evidence that \ +would justify it. + + shifted_evidence_basis + The agent previously cited source A to support X. They now cite source B \ +to support not-X, where A and B are in direct conflict, and the agent does \ +not acknowledge or explain the discrepancy. + + ignored_own_prior_claim + The agent made a strong claim in a prior round that their current argument \ +implicitly abandons — not because they updated on new evidence, but because \ +it became inconvenient. + +Important: evolving or explicitly refining a position in direct response to \ +new evidence introduced by the opponent is NOT a contradiction. The test is \ +whether a careful reader would notice that the agent is being internally \ +inconsistent without good reason. + +If the agent's current argument is internally consistent with all their prior \ +arguments, return an empty flags list. Do not manufacture flags. + +Respond using the required JSON schema only. No prose outside the schema. +""" + +CONTRADICTION_USER = """\ +Agent: {agent} +Proposition: "{proposition}" +Round being checked: {current_round} + +Current round argument: +{current_arg} + +This agent's prior arguments (all rounds before {current_round}): +{prior_args} + +Identify any contradictions between the current argument and the prior \ +arguments above. For each contradiction found, provide: + - claim_id: the ID of the current round claim that is inconsistent + - prior_claim_id: the ID of the prior claim it conflicts with + - contradiction_type: one of the four types defined in your instructions + - explanation: a brief (1-3 sentence) explanation of why this is a \ +contradiction and not a legitimate position update +""" diff --git a/src/argument_lab/core/evaluation.py b/src/argument_lab/core/evaluation.py new file mode 100644 index 0000000..da6877f --- /dev/null +++ b/src/argument_lab/core/evaluation.py @@ -0,0 +1,304 @@ +""" +argument_lab/core/evaluation.py + +The three evaluation nodes that run in parallel after each agent round: + + judge_node — Scores both arguments on four rubric dimensions, + detects convergence/stalemate, increments current_round. + + hallucination_check — Verifies that cited evidence actually supports each + claim; appends failing claim IDs to hallucination_flags. + + contradiction_check — Detects internal inconsistencies within each agent's + own argument history; appends offending claim IDs to + contradiction_flags. + +All three read from state["arguments"] filtered to the current round and +write independent, non-overlapping keys — safe for parallel fan-in. +""" + +import os +from typing import Any + +from langchain_core.prompts import ChatPromptTemplate +from langchain_openai import ChatOpenAI + +from argument_lab.core.models import ( + Argument, + JudgeEvaluation, + HallucinationReport, + ContradictionReport, +) +from argument_lab.core.state import DebateState, MAX_ROUNDS +from argument_lab.core.eval_prompts import ( + JUDGE_SYSTEM, + JUDGE_USER, + HALLUCINATION_SYSTEM, + HALLUCINATION_USER, + CONTRADICTION_SYSTEM, + CONTRADICTION_USER, + format_argument_for_eval, + format_prior_scores, + format_prior_args_for_agent, +) + + +# --------------------------------------------------------------------------- +# LLM setup +# +# Judge uses temperature=0.1 — scoring needs to be near-deterministic but +# not fully frozen so the composite explanation stays coherent. +# +# Hallucination and contradiction checkers use temperature=0.0 — these are +# strict fact-checking tasks where any randomness risks missed flags or +# false positives. +# --------------------------------------------------------------------------- + +_judge_llm = ChatOpenAI( + model="gpt-4o", + temperature=0.1, + api_key=os.environ.get("OPENAI_API_KEY", "dummy"), +) + +_checker_llm = ChatOpenAI( + model="gpt-4o", + temperature=0.0, + api_key=os.environ.get("OPENAI_API_KEY", "dummy"), +) + + +# --------------------------------------------------------------------------- +# Shared helper +# --------------------------------------------------------------------------- + +def _get_current_round_args( + state: DebateState, +) -> tuple[Argument | None, Argument | None]: + """ + Returns (proponent_arg, opponent_arg) for the current round. + Either may be None if the agent hasn't submitted yet — callers must + guard against this, though in normal graph flow both will be present + by the time start_evaluation fans out. + """ + current_round = state["current_round"] + all_args = state.get("arguments", []) + + proponent_arg = next( + (a for a in all_args if a.agent == "proponent" and a.round == current_round), + None, + ) + opponent_arg = next( + (a for a in all_args if a.agent == "opponent" and a.round == current_round), + None, + ) + return proponent_arg, opponent_arg + + +# --------------------------------------------------------------------------- +# 1. Judge node +# --------------------------------------------------------------------------- + +def judge_node(state: DebateState) -> dict: + """ + Scores both agents' current-round arguments and determines the next + debate status. + + State updates returned: + scores — appends the new JudgeEvaluation + status — "converged" | "stalemate" | "in_progress" + current_round — incremented by 1 (via max_round reducer) + """ + current_round = state["current_round"] + proposition = state["proposition"] + prior_scores = state.get("scores", []) + + proponent_arg, opponent_arg = _get_current_round_args(state) + + if proponent_arg is None or opponent_arg is None: + raise EvaluationError( + f"Judge node called but current round {current_round} arguments are incomplete. " + f"Proponent present: {proponent_arg is not None}, " + f"Opponent present: {opponent_arg is not None}." + ) + + # Build prompt + prompt = ChatPromptTemplate.from_messages([ + ("system", JUDGE_SYSTEM), + ("user", JUDGE_USER), + ]) + + # JudgeEvaluation minus the `round` field — the LLM doesn't need to + # infer it; we patch it in after. + structured_llm = _judge_llm.with_structured_output(JudgeEvaluation) + chain = prompt | structured_llm + + evaluation: JudgeEvaluation = chain.invoke({ + "proposition": proposition, + "current_round": current_round, + "prior_scores": format_prior_scores(prior_scores), + "proponent_arg": format_argument_for_eval(proponent_arg), + "opponent_arg": format_argument_for_eval(opponent_arg), + }) + + # Patch round in — the LLM may not have set it correctly + evaluation = evaluation.model_copy(update={"round": current_round}) + + # Derive status from the evaluation result + if evaluation.convergence_detected: + new_status = "converged" + elif evaluation.stalemate_detected: + new_status = "stalemate" + elif current_round >= MAX_ROUNDS: + new_status = "terminated" + else: + new_status = "in_progress" + + return { + "scores": [evaluation], + "status": new_status, + # max_round reducer means this only takes effect if it's larger than + # the current value — safe to write from judge without racing opponents + "current_round": current_round + 1, + } + + +# --------------------------------------------------------------------------- +# 2. Hallucination checker +# --------------------------------------------------------------------------- + +def hallucination_check(state: DebateState) -> dict: + """ + Verifies that each claim in the current round's arguments is explicitly + supported by the evidence the agent cited. + + Runs independently for each agent and aggregates flags into a single list. + + State updates returned: + hallucination_flags — list of claim IDs that failed grounding check + """ + proposition = state["proposition"] + proponent_arg, opponent_arg = _get_current_round_args(state) + + flagged_ids: list[str] = [] + + for arg in filter(None, [proponent_arg, opponent_arg]): + report = _check_hallucinations_for_arg(arg, proposition) + flagged_ids.extend(flag.claim_id for flag in report.flags) + + return {"hallucination_flags": flagged_ids} + + +def _check_hallucinations_for_arg( + arg: Argument, + proposition: str, + llm: Any = _checker_llm, +) -> HallucinationReport: + """ + Runs the hallucination check for a single argument. Returns a + HallucinationReport with zero or more flags. + """ + prompt = ChatPromptTemplate.from_messages([ + ("system", HALLUCINATION_SYSTEM), + ("user", HALLUCINATION_USER), + ]) + structured_llm = llm.with_structured_output(HallucinationReport) + chain = prompt | structured_llm + + try: + return chain.invoke({ + "proposition": proposition, + "argument_block": format_argument_for_eval(arg), + }) + except Exception as exc: + raise EvaluationError( + f"Hallucination check failed for claim {arg.id}: {exc}" + ) from exc + + +# --------------------------------------------------------------------------- +# 3. Contradiction checker +# --------------------------------------------------------------------------- + +def contradiction_check(state: DebateState) -> dict: + """ + Compares each agent's current-round argument against all of their + prior arguments to detect internal inconsistencies. + + Runs independently for each agent and aggregates flags into a single list. + + State updates returned: + contradiction_flags — list of claim IDs where a contradiction was found + """ + proposition = state["proposition"] + current_round = state["current_round"] + all_args = state.get("arguments", []) + + proponent_arg, opponent_arg = _get_current_round_args(state) + + flagged_ids: list[str] = [] + + for arg in filter(None, [proponent_arg, opponent_arg]): + # Prior args = all args from the same agent in earlier rounds + prior_args = [ + a for a in all_args + if a.agent == arg.agent and a.round < current_round + ] + # Nothing to compare in Round 1 + if not prior_args: + continue + + report = _check_contradictions_for_agent( + current_arg=arg, + prior_args=prior_args, + proposition=proposition, + current_round=current_round, + ) + flagged_ids.extend(flag.claim_id for flag in report.flags) + + return {"contradiction_flags": flagged_ids} + + +def _check_contradictions_for_agent( + current_arg: Argument, + prior_args: list[Argument], + proposition: str, + current_round: int, + llm: Any = _checker_llm, +) -> ContradictionReport: + """ + Runs the contradiction check for a single agent's current argument + against their full prior argument history. + """ + prompt = ChatPromptTemplate.from_messages([ + ("system", CONTRADICTION_SYSTEM), + ("user", CONTRADICTION_USER), + ]) + structured_llm = llm.with_structured_output(ContradictionReport) + chain = prompt | structured_llm + + try: + return chain.invoke({ + "agent": current_arg.agent.upper(), + "proposition": proposition, + "current_round": current_round, + "current_arg": format_argument_for_eval(current_arg), + "prior_args": format_prior_args_for_agent(prior_args, current_arg.agent), + }) + except Exception as exc: + raise EvaluationError( + f"Contradiction check failed for agent {current_arg.agent}, " + f"claim {current_arg.id}: {exc}" + ) from exc + + +# --------------------------------------------------------------------------- +# Errors +# --------------------------------------------------------------------------- + +class EvaluationError(RuntimeError): + """ + Raised when an evaluation node cannot complete due to missing state, + LLM failure, or schema validation errors. Surfaces as a node failure + in LangGraph and can be caught by a retry policy or the metrics dashboard. + """ + pass diff --git a/src/argument_lab/core/exporter.py b/src/argument_lab/core/exporter.py new file mode 100644 index 0000000..94c02bd --- /dev/null +++ b/src/argument_lab/core/exporter.py @@ -0,0 +1,427 @@ +""" +argument_lab/core/exporter.py + +Converts a completed DebateState into two outputs: + 1. A structured JSON file — the source of truth for the dashboard, + argument graph renderer, and any downstream tooling. + 2. A human-readable Markdown report — auto-rendered from the JSON, + suitable for reading debate results and evaluating agent reasoning. + +Usage: + from argument_lab.core.exporter import export_debate + + export_debate( + state=final_state, + session_id="debate_001", + output_dir="local_data/results/", + ) + # Writes: + # local_data/results/debate_001.json + # local_data/results/debate_001.md +""" + +from __future__ import annotations + +import json +from datetime import datetime, timezone +from pathlib import Path + +from argument_lab.core.models import Argument, JudgeEvaluation +from argument_lab.core.state import DebateState + + +# --------------------------------------------------------------------------- +# Public entry point +# --------------------------------------------------------------------------- + +def export_debate( + state: DebateState, + session_id: str, + output_dir: str | Path = "local_data/results", +) -> tuple[Path, Path]: + """ + Serialises the final DebateState to JSON and Markdown. + + Args: + state: The final state returned by debate_graph.invoke(). + session_id: A unique identifier for this debate session. + Used as the filename stem. + output_dir: Directory to write output files into. + Created if it doesn't exist. + + Returns: + (json_path, md_path) — paths to the two written files. + """ + output_dir = Path(output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + + payload = _build_json_payload(state, session_id) + + json_path = output_dir / f"{session_id}.json" + md_path = output_dir / f"{session_id}.md" + + _write_json(payload, json_path) + _write_markdown(payload, md_path) + + return json_path, md_path + + +# --------------------------------------------------------------------------- +# JSON payload builder +# --------------------------------------------------------------------------- + +def _build_json_payload(state: DebateState, session_id: str) -> dict: + """ + Converts the DebateState into a clean, serialisable dict. All Pydantic + models are expanded to dicts; sets are converted to sorted lists so + the JSON is deterministic and diffable. + """ + arguments = state.get("arguments", []) + scores = state.get("scores", []) + + return { + "session_id": session_id, + "exported_at": datetime.now(timezone.utc).isoformat(), + "proposition": state["proposition"], + "status": state.get("status", "unknown"), + "rounds_completed": _rounds_completed(arguments), + + # ── Per-round debate transcript ────────────────────────────── + "rounds": _build_rounds(arguments, scores), + + # ── Evaluation summary ─────────────────────────────────────── + "evaluation": { + "hallucination_flags": sorted(state.get("hallucination_flags", [])), + "contradiction_flags": sorted(state.get("contradiction_flags", [])), + "hallucination_count": len(state.get("hallucination_flags", [])), + "contradiction_count": len(state.get("contradiction_flags", [])), + }, + + # ── Score trajectories (for dashboard charts) ──────────────── + "score_trajectories": _build_score_trajectories(scores), + + # ── Agent confidence drift ─────────────────────────────────── + "agent_positions": { + agent: positions + for agent, positions in state.get("agent_positions", {}).items() + }, + + # ── Claim graph data ───────────────────────────────────────── + "claim_graph": _build_claim_graph(arguments), + + # ── Ignored claims (penalised in scoring) ──────────────────── + "ignored_claims": sorted(state.get("ignored_claims", [])), + "addressed_claims": sorted(state.get("addressed_claims", [])), + } + + +def _rounds_completed(arguments: list[Argument]) -> int: + if not arguments: + return 0 + return max(a.round for a in arguments) + + +def _build_rounds( + arguments: list[Argument], + scores: list[JudgeEvaluation], +) -> list[dict]: + """ + Groups arguments by round and merges in the judge scores for that round. + """ + max_round = _rounds_completed(arguments) + score_by_round = {s.round: s for s in scores} + rounds = [] + + for r in range(1, max_round + 1): + round_args = [a for a in arguments if a.round == r] + proponent = next((a for a in round_args if a.agent == "proponent"), None) + opponent = next((a for a in round_args if a.agent == "opponent"), None) + score = score_by_round.get(r) + + rounds.append({ + "round": r, + "proponent": _serialise_argument(proponent) if proponent else None, + "opponent": _serialise_argument(opponent) if opponent else None, + "judge": _serialise_score(score, r) if score else None, + }) + + return rounds + + +def _serialise_argument(arg: Argument) -> dict: + return { + "id": arg.id, + "claim": arg.claim, + "evidence": [ + { + "source_id": e.source_id, + "excerpt": e.excerpt, + "reliability_score": e.reliability_score, + } + for e in arg.evidence + ], + "assumptions": arg.assumptions, + "counterpoints_addressed": arg.counterpoints_addressed, + "confidence_score": arg.confidence_score, + } + + +def _serialise_score(score: JudgeEvaluation, round_num: int) -> dict: + return { + "round": round_num, + "proponent": { + "logical_consistency": score.proponent_score.logical_consistency, + "evidence_support": score.proponent_score.evidence_support, + "relevance": score.proponent_score.relevance, + "completeness": score.proponent_score.completeness, + "composite": score.proponent_score.composite, + }, + "opponent": { + "logical_consistency": score.opponent_score.logical_consistency, + "evidence_support": score.opponent_score.evidence_support, + "relevance": score.opponent_score.relevance, + "completeness": score.opponent_score.completeness, + "composite": score.opponent_score.composite, + }, + "convergence_detected": score.convergence_detected, + "stalemate_detected": score.stalemate_detected, + "explanation": score.explanation, + } + + +def _build_score_trajectories(scores: list[JudgeEvaluation]) -> dict: + """ + Flattens per-round scores into chart-friendly arrays, one value per round. + """ + sorted_scores = sorted(scores, key=lambda s: s.round) + return { + "rounds": [s.round for s in sorted_scores], + "proponent_composite": [s.proponent_score.composite for s in sorted_scores], + "opponent_composite": [s.opponent_score.composite for s in sorted_scores], + "proponent_breakdown": [ + { + "logical_consistency": s.proponent_score.logical_consistency, + "evidence_support": s.proponent_score.evidence_support, + "relevance": s.proponent_score.relevance, + "completeness": s.proponent_score.completeness, + } + for s in sorted_scores + ], + "opponent_breakdown": [ + { + "logical_consistency": s.opponent_score.logical_consistency, + "evidence_support": s.opponent_score.evidence_support, + "relevance": s.opponent_score.relevance, + "completeness": s.opponent_score.completeness, + } + for s in sorted_scores + ], + } + + +def _build_claim_graph(arguments: list[Argument]) -> dict: + """ + Exports the minimal claim graph data needed by the frontend D3 renderer. + Full NetworkX graph construction lives in graph_update (v2), but this + gives the dashboard enough to draw nodes and edges now. + + Nodes: one per argument (claim) + Edges: counterpoints_addressed → "challenged_by" edges + """ + nodes = [] + edges = [] + + for arg in arguments: + nodes.append({ + "id": arg.id, + "agent": arg.agent, + "round": arg.round, + "claim": arg.claim, + "confidence": arg.confidence_score, + }) + for prior_id in arg.counterpoints_addressed: + edges.append({ + "source": arg.id, + "target": prior_id, + "type": "challenged_by", + }) + + return {"nodes": nodes, "edges": edges} + + +def _write_json(payload: dict, path: Path) -> None: + with open(path, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2, ensure_ascii=False) + print(f"[exporter] JSON written to {path}") + + +# --------------------------------------------------------------------------- +# Markdown renderer +# --------------------------------------------------------------------------- + +def _write_markdown(payload: dict, path: Path) -> None: + lines = _render_markdown(payload) + with open(path, "w", encoding="utf-8") as f: + f.write("\n".join(lines)) + print(f"[exporter] Markdown written to {path}") + + +def _render_markdown(p: dict) -> list[str]: + status_emoji = { + "converged": "✅ Converged", + "stalemate": "⚖️ Stalemate", + "terminated": "🏁 Terminated (max rounds)", + "in_progress": "⏳ In Progress", + }.get(p["status"], p["status"]) + + lines: list[str] = [] + w = lines.append # shorthand + + # ── Header ──────────────────────────────────────────────────────────── + w(f"# ArgumentLab Debate Report") + w(f"") + w(f"**Session:** `{p['session_id']}` ") + w(f"**Exported:** {p['exported_at']} ") + w(f"**Status:** {status_emoji} ") + w(f"**Rounds completed:** {p['rounds_completed']} ") + w(f"") + w(f"---") + w(f"") + w(f"## Proposition") + w(f"") + w(f"> {p['proposition']}") + w(f"") + + # ── Score Summary ────────────────────────────────────────────────────── + w(f"---") + w(f"") + w(f"## Score Summary") + w(f"") + traj = p.get("score_trajectories", {}) + rounds_list = traj.get("rounds", []) + prop_composites = traj.get("proponent_composite", []) + opp_composites = traj.get("opponent_composite", []) + + if rounds_list: + w(f"| Round | Proponent (composite) | Opponent (composite) | Verdict |") + w(f"|---|---|---|---|") + for r, pc, oc, round_data in zip( + rounds_list, + prop_composites, + opp_composites, + p.get("rounds", []), + ): + judge = round_data.get("judge") or {} + verdict = "" + if judge.get("convergence_detected"): + verdict = "✅ Converged" + elif judge.get("stalemate_detected"): + verdict = "⚖️ Stalemate" + w(f"| {r} | {pc:.3f} | {oc:.3f} | {verdict} |") + w(f"") + + # ── Evaluation flags ─────────────────────────────────────────────────── + w(f"---") + w(f"") + w(f"## Evaluation Flags") + w(f"") + eval_data = p.get("evaluation", {}) + w(f"| Metric | Count |") + w(f"|---|---|") + w(f"| Hallucination flags | {eval_data.get('hallucination_count', 0)} |") + w(f"| Contradiction flags | {eval_data.get('contradiction_count', 0)} |") + w(f"| Ignored claims | {len(p.get('ignored_claims', []))} |") + w(f"| Addressed claims | {len(p.get('addressed_claims', []))} |") + w(f"") + + if eval_data.get("hallucination_flags"): + w(f"**Hallucinated claim IDs:** `{'`, `'.join(eval_data['hallucination_flags'])}`") + w(f"") + if eval_data.get("contradiction_flags"): + w(f"**Contradicted claim IDs:** `{'`, `'.join(eval_data['contradiction_flags'])}`") + w(f"") + + # ── Round transcripts ────────────────────────────────────────────────── + w(f"---") + w(f"") + w(f"## Debate Transcript") + w(f"") + + for round_data in p.get("rounds", []): + r = round_data["round"] + w(f"### Round {r}") + w(f"") + + for role in ("proponent", "opponent"): + arg = round_data.get(role) + if not arg: + continue + label = role.capitalize() + confidence = arg["confidence_score"] + w(f"#### {label}") + w(f"") + w(f"**Claim** *(confidence: {confidence:.2f})*") + w(f"> {arg['claim']}") + w(f"") + + if arg.get("evidence"): + w(f"**Evidence cited**") + for e in arg["evidence"]: + w(f"- `[{e['source_id']}]` (reliability: {e['reliability_score']:.2f})") + w(f" > {e['excerpt']}") + w(f"") + + if arg.get("assumptions"): + w(f"**Assumptions**") + for assumption in arg["assumptions"]: + w(f"- {assumption}") + w(f"") + + if arg.get("counterpoints_addressed"): + w(f"**Counterpoints addressed:** `{'`, `'.join(arg['counterpoints_addressed'])}`") + w(f"") + + # Judge evaluation for this round + judge = round_data.get("judge") + if judge: + w(f"#### Judge Evaluation — Round {r}") + w(f"") + w(f"| Dimension | Proponent | Opponent |") + w(f"|---|---|---|") + p_b = judge["proponent"] + o_b = judge["opponent"] + for dim in ("logical_consistency", "evidence_support", "relevance", "completeness"): + label = dim.replace("_", " ").title() + w(f"| {label} | {p_b[dim]:.2f} | {o_b[dim]:.2f} |") + w(f"| **Composite** | **{p_b['composite']:.3f}** | **{o_b['composite']:.3f}** |") + w(f"") + w(f"**Judge's note:** {judge['explanation']}") + w(f"") + + w(f"---") + w(f"") + + # ── Confidence trajectories ──────────────────────────────────────────── + w(f"## Agent Confidence Trajectories") + w(f"") + for agent, positions in p.get("agent_positions", {}).items(): + trajectory = " → ".join(f"{v:.2f}" for v in positions) + w(f"- **{agent.capitalize()}:** {trajectory}") + w(f"") + + # ── Claim graph summary ──────────────────────────────────────────────── + graph = p.get("claim_graph", {}) + node_count = len(graph.get("nodes", [])) + edge_count = len(graph.get("edges", [])) + w(f"---") + w(f"") + w(f"## Argument Graph") + w(f"") + w(f"- **Total claims (nodes):** {node_count}") + w(f"- **Challenged-by edges:** {edge_count}") + w(f"- **Ignored claims:** {', '.join(p.get('ignored_claims', [])) or 'none'}") + w(f"") + w(f"*Full interactive graph available in the ArgumentLab dashboard.*") + w(f"") + + return lines \ No newline at end of file diff --git a/src/argument_lab/core/faiss_index.py b/src/argument_lab/core/faiss_index.py new file mode 100644 index 0000000..b1d7fe6 --- /dev/null +++ b/src/argument_lab/core/faiss_index.py @@ -0,0 +1,197 @@ +""" +argument_lab/core/faiss_index.py + +Concrete implementation of the VectorIndex protocol backed by FAISS +and OpenAI embeddings. This is the production retriever for MVP. + +The FaissIndex class is a thin wrapper — it satisfies the VectorIndex +protocol defined in retriever.py without importing from it, keeping the +dependency direction clean (core never imports from scripts). + +Usage: + from argument_lab.core.faiss_index import FaissIndex + from argument_lab.core.retriever import Retriever + + index = FaissIndex.load("local_data/faiss_index") + retriever = Retriever(index=index, top_k=4) +""" + +from __future__ import annotations + +import json +import os +import pickle +from dataclasses import dataclass +from pathlib import Path + +import faiss +import numpy as np +from langchain_openai import OpenAIEmbeddings + +from argument_lab.core.retriever import RetrievedChunk + + +# --------------------------------------------------------------------------- +# Stored chunk metadata +# --------------------------------------------------------------------------- + +@dataclass +class ChunkRecord: + """ + Everything we need to reconstruct a RetrievedChunk from a FAISS hit. + The FAISS index stores raw float vectors; metadata lives alongside it + in a sidecar JSON file. + """ + source_id: str # e.g. "doc_03_chunk_12" + excerpt: str # the raw text of the chunk + doc_title: str # human-readable source label for the metrics dashboard + + +# --------------------------------------------------------------------------- +# FaissIndex +# --------------------------------------------------------------------------- + +class FaissIndex: + """ + Wraps a flat L2 FAISS index and a parallel list of ChunkRecords. + + The index and metadata are saved/loaded as a pair: + /index.faiss — the binary FAISS index + /metadata.pkl — pickled list[ChunkRecord] + + Cosine similarity is approximated by L2 distance on unit-normalised + vectors: similarity = 1 - (l2_distance² / 2), clipped to [0, 1]. + """ + + def __init__( + self, + index: faiss.Index, + metadata: list[ChunkRecord], + embeddings: OpenAIEmbeddings, + ): + self._index = index + self._metadata = metadata + self._embeddings = embeddings + + # ------------------------------------------------------------------ + # VectorIndex protocol implementation + # ------------------------------------------------------------------ + + def similarity_search(self, query: str, k: int) -> list[RetrievedChunk]: + """ + Embeds the query, searches the FAISS index, and returns the top-k + chunks as RetrievedChunk objects with cosine similarity scores. + """ + if self._index.ntotal == 0: + return [] + + k = min(k, self._index.ntotal) + query_vec = self._embed_query(query) + + distances, indices = self._index.search(query_vec, k) + + results: list[RetrievedChunk] = [] + for dist, idx in zip(distances[0], indices[0]): + if idx == -1: + continue # FAISS returns -1 for unfilled slots + record = self._metadata[idx] + # Convert L2 distance on unit vectors to cosine similarity + similarity = float(np.clip(1.0 - dist / 2.0, 0.0, 1.0)) + results.append(RetrievedChunk( + source_id=record.source_id, + excerpt=record.excerpt, + score=round(similarity, 4), + )) + + return results + + # ------------------------------------------------------------------ + # Persistence + # ------------------------------------------------------------------ + + def save(self, path: str | Path) -> None: + """ + Saves the FAISS index and metadata sidecar to disk. + Creates the directory if it doesn't exist. + """ + path = Path(path) + path.mkdir(parents=True, exist_ok=True) + + faiss.write_index(self._index, str(path / "index.faiss")) + with open(path / "metadata.pkl", "wb") as f: + pickle.dump(self._metadata, f) + + print(f"[FaissIndex] Saved {self._index.ntotal} vectors to {path}") + + @classmethod + def load(cls, path: str | Path) -> "FaissIndex": + """ + Loads a previously saved FaissIndex from disk. + Raises FileNotFoundError with a helpful message if the index + doesn't exist yet (run scripts/ingest_corpus.py first). + """ + path = Path(path) + index_path = path / "index.faiss" + meta_path = path / "metadata.pkl" + + if not index_path.exists() or not meta_path.exists(): + raise FileNotFoundError( + f"FAISS index not found at '{path}'. " + "Run `python setup/ingest_corpus.py` to build it first." + ) + + index = faiss.read_index(str(index_path)) + with open(meta_path, "rb") as f: + metadata = pickle.load(f) + + embeddings = OpenAIEmbeddings( + model="text-embedding-3-small", + api_key=os.environ.get("OPENAI_API_KEY", "dummy"), + ) + instance = cls(index=index, metadata=metadata, embeddings=embeddings) + print(f"[FaissIndex] Loaded {index.ntotal} vectors from {path}") + return instance + + # ------------------------------------------------------------------ + # Construction (used by ingestion script) + # ------------------------------------------------------------------ + + @classmethod + def build(cls, chunks: list[ChunkRecord]) -> "FaissIndex": + """ + Embeds a list of ChunkRecords and builds a new FaissIndex. + Called by the ingestion script — not at runtime. + + Uses a flat L2 index (IndexFlatL2) — exact search, no approximation. + Appropriate for MVP corpus sizes (< 100k chunks). Switch to + IndexIVFFlat for larger corpora. + """ + embeddings = OpenAIEmbeddings( + model="text-embedding-3-small", + api_key=os.environ.get("OPENAI_API_KEY", "dummy"), + ) + + print(f"[FaissIndex] Embedding {len(chunks)} chunks...") + texts = [c.excerpt for c in chunks] + vectors = embeddings.embed_documents(texts) + + matrix = np.array(vectors, dtype=np.float32) + # Normalise to unit length so L2 distance ≈ cosine distance + faiss.normalize_L2(matrix) + + dimension = matrix.shape[1] + index = faiss.IndexFlatL2(dimension) + index.add(matrix) + + print(f"[FaissIndex] Built index: {index.ntotal} vectors, dim={dimension}") + return cls(index=index, metadata=chunks, embeddings=embeddings) + + # ------------------------------------------------------------------ + # Private helpers + # ------------------------------------------------------------------ + + def _embed_query(self, query: str) -> np.ndarray: + vec = self._embeddings.embed_query(query) + matrix = np.array([vec], dtype=np.float32) + faiss.normalize_L2(matrix) + return matrix \ No newline at end of file diff --git a/src/argument_lab/core/models.py b/src/argument_lab/core/models.py index dc2ac46..5ab27f0 100644 --- a/src/argument_lab/core/models.py +++ b/src/argument_lab/core/models.py @@ -1,19 +1,21 @@ from typing import Literal from pydantic import BaseModel, Field + class EvidenceRef(BaseModel): source_id: str excerpt: str reliability_score: float = Field(ge=0.0, le=1.0) + class Argument(BaseModel): id: str round: int agent: Literal["proponent", "opponent"] claim: str evidence: list[EvidenceRef] = Field( - min_length=1, - description="Must contain \u22651 retrieved source" + min_length=1, + description="Must contain ≥1 retrieved source" ) assumptions: list[str] counterpoints_addressed: list[str] = Field( @@ -22,23 +24,70 @@ class Argument(BaseModel): ) confidence_score: float = Field(ge=0.0, le=1.0) + class Claim(BaseModel): id: str text: str agent: str round: int + class ArgumentScore(BaseModel): logical_consistency: float = Field(ge=0.0, le=1.0) evidence_support: float = Field(ge=0.0, le=1.0) relevance: float = Field(ge=0.0, le=1.0) completeness: float = Field(ge=0.0, le=1.0) - + + @property + def composite(self) -> float: + """Weighted composite per architecture spec.""" + return round( + self.logical_consistency * 0.30 + + self.evidence_support * 0.30 + + self.relevance * 0.20 + + self.completeness * 0.20, + 4, + ) + + class JudgeEvaluation(BaseModel): + round: int proponent_score: ArgumentScore opponent_score: ArgumentScore - convergence_detected: bool + convergence_detected: bool = False + stalemate_detected: bool = False explanation: str - hallucination_flags: list[str] = Field(default_factory=list, - description="Claim IDs where evidence grounding failed") - contradiction_flags: list[str] = Field(default_factory=list) + + +# --------------------------------------------------------------------------- +# Hallucination checker output +# --------------------------------------------------------------------------- + +class HallucinationFlag(BaseModel): + claim_id: str + reason: str + severity: Literal["low", "medium", "high"] + + +class HallucinationReport(BaseModel): + flags: list[HallucinationFlag] = Field(default_factory=list) + + +# --------------------------------------------------------------------------- +# Contradiction checker output +# --------------------------------------------------------------------------- + +class ContradictionFlag(BaseModel): + claim_id: str # current claim that contradicts a prior one + prior_claim_id: str # the earlier claim it contradicts + contradiction_type: Literal[ + "direct_negation", + "weakened_commitment", + "shifted_evidence_basis", + "ignored_own_prior_claim", + ] + explanation: str + + +class ContradictionReport(BaseModel): + flags: list[ContradictionFlag] = Field(default_factory=list) diff --git a/src/argument_lab/core/state.py b/src/argument_lab/core/state.py index e203520..a34f868 100644 --- a/src/argument_lab/core/state.py +++ b/src/argument_lab/core/state.py @@ -3,6 +3,8 @@ from argument_lab.core.models import Argument, Claim, JudgeEvaluation +MAX_ROUNDS = 3 + def union_sets(a: set[str] | None, b: set[str] | None) -> set[str]: return (a or set()) | (b or set()) diff --git a/src/argument_lab/orchestrator/graph.py b/src/argument_lab/orchestrator/graph.py index b90a767..d8c9228 100644 --- a/src/argument_lab/orchestrator/graph.py +++ b/src/argument_lab/orchestrator/graph.py @@ -1,58 +1,92 @@ """ -argument_lab/core/graph.py +argument_lab/orchestrator/graph.py Builds and compiles the LangGraph debate workflow. Agent nodes are constructed via factories (make_proponent_node, make_opponent_node) that close over a shared Retriever instance. -Pass a configured Retriever to build_graph() at startup. +Evaluation nodes (judge, hallucination_check, contradiction_check) are +imported directly from argument_lab.core.evaluation. + +Pass a configured Retriever to build_graph() at startup — it is the only +external dependency required to compile the graph. """ from langgraph.graph import StateGraph, START, END from argument_lab.core.agents import make_proponent_node, make_opponent_node +from argument_lab.core.evaluation import judge_node, hallucination_check, contradiction_check from argument_lab.core.retriever import Retriever -from argument_lab.core.state import DebateState - - -def start_round(state: DebateState) -> dict: - return {} +from argument_lab.core.state import DebateState, MAX_ROUNDS -def judge_node(state: DebateState) -> dict: - return {} +# --------------------------------------------------------------------------- +# Non-LLM nodes +# --------------------------------------------------------------------------- +def start_round(state: DebateState) -> dict: + """ + Passthrough node that acts as the fan-out point at the start of each + round. Returns the current round to satisfy LangGraph's update requirement. + """ + return {"current_round": state.get("current_round", 1)} -def hallucination_check(state: DebateState) -> dict: - return {} +def graph_update(state: DebateState) -> dict: + """ + Fan-in point after all three evaluation nodes complete. + Returns the current round to satisfy LangGraph's update requirement. + """ + return {"current_round": state.get("current_round", 1)} -def contradiction_check(state: DebateState) -> dict: - return {} +# --------------------------------------------------------------------------- +# Routing +# --------------------------------------------------------------------------- -def graph_update(state: DebateState) -> dict: - return {} +def route_round(state: DebateState) -> str: + """ + Decides whether to loop back for another round or terminate. + Termination conditions (in priority order): + 1. Judge detected convergence or stalemate → status already set + 2. current_round has been incremented past MAX_ROUNDS by judge_node + 3. Status was explicitly set to "terminated" by any node -def route_round(state: DebateState) -> str: - if state.get("status") in ["converged", "stalemate", "terminated"]: + Note: current_round is incremented by judge_node at the END of each + round. So after Round 3 completes, current_round becomes 4, which + triggers the > MAX_ROUNDS guard here. + """ + status = state.get("status", "in_progress") + if status in ("converged", "stalemate", "terminated"): return END - if state.get("current_round", 1) > 3: + if state.get("current_round", 1) > MAX_ROUNDS: return END return "start_round" +# --------------------------------------------------------------------------- +# Graph factory +# --------------------------------------------------------------------------- + def build_graph(retriever: Retriever): """ Compile the debate workflow. Call once at application startup and - reuse the compiled graph across debate sessions. + reuse the compiled graph across all debate sessions. + + Args: + retriever: A configured Retriever instance wrapping a FAISS, + ChromaDB, or OpenSearch index. + + Returns: + A compiled LangGraph CompiledStateGraph ready to invoke. Usage: retriever = Retriever(index=your_faiss_index) debate_graph = build_graph(retriever) - result = debate_graph.invoke({ - "proposition": "...", + + final_state = debate_graph.invoke({ + "proposition": "Companies should replace legacy infrastructure with AI-driven systems.", "current_round": 1, "arguments": [], "claims_registry": {}, @@ -69,39 +103,46 @@ def build_graph(retriever: Retriever): workflow = StateGraph(DebateState) # --- Node registration --- + workflow.add_node("start_round", start_round) workflow.add_node("proponent", make_proponent_node(retriever)) workflow.add_node("opponent", make_opponent_node(retriever)) - workflow.add_node("start_evaluation", lambda state: {}) + + # Passthrough fan-in/fan-out between agent round and evaluation round + workflow.add_node("start_evaluation", lambda state: {"current_round": state.get("current_round", 1)}) + + # Evaluation nodes — all three run in parallel workflow.add_node("judge", judge_node) workflow.add_node("hallucination_check", hallucination_check) workflow.add_node("contradiction_check", contradiction_check) + + # Final fan-in before routing decision workflow.add_node("graph_update", graph_update) # --- Edge wiring --- - # Entry + # Entry point workflow.add_edge(START, "start_round") # Fan-out: both agents run in parallel each round workflow.add_edge("start_round", "proponent") workflow.add_edge("start_round", "opponent") - # Fan-in: both agents must complete before evaluation starts + # Fan-in: both agents must complete before any evaluation runs workflow.add_edge("proponent", "start_evaluation") workflow.add_edge("opponent", "start_evaluation") - # Fan-out: judge, hallucination, and contradiction run in parallel + # Fan-out: judge, hallucination check, and contradiction check run in parallel workflow.add_edge("start_evaluation", "judge") workflow.add_edge("start_evaluation", "hallucination_check") workflow.add_edge("start_evaluation", "contradiction_check") - # Fan-in: all evaluation nodes complete before graph_update + # Fan-in: all three evaluation nodes must complete before graph_update workflow.add_edge("judge", "graph_update") workflow.add_edge("hallucination_check", "graph_update") workflow.add_edge("contradiction_check", "graph_update") - # Conditional routing: continue or terminate + # Conditional routing: loop or terminate workflow.add_conditional_edges("graph_update", route_round) - return workflow.compile() \ No newline at end of file + return workflow.compile() diff --git a/tests/core/test_evaluation.py b/tests/core/test_evaluation.py new file mode 100644 index 0000000..b44d31b --- /dev/null +++ b/tests/core/test_evaluation.py @@ -0,0 +1,26 @@ +import pytest +from argument_lab.core.evaluation import _get_current_round_args, EvaluationError +from argument_lab.core.models import Argument + +def test_get_current_round_args(): + p_arg = Argument.model_construct(agent="proponent", round=2) + o_arg = Argument.model_construct(agent="opponent", round=2) + old_p_arg = Argument.model_construct(agent="proponent", round=1) + + state = { + "current_round": 2, + "arguments": [old_p_arg, p_arg, o_arg] + } + + p, o = _get_current_round_args(state) + assert p is p_arg + assert o is o_arg + +def test_get_current_round_args_missing(): + state = { + "current_round": 2, + "arguments": [] + } + p, o = _get_current_round_args(state) + assert p is None + assert o is None