veriact refactoring and agent-harness workspace#1
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors VeriAct into a CLI-driven ReAct agent that interacts with three command-line tools (verify, run_spec_harness, task_complete) instead of executing Python in-process, and adds an agent_harness/ scaffolding system to run external coding agents in per-task workspaces.
Changes:
- Introduces a new CLI tool dispatcher (
veriact/tools/cli.py) plus tool descriptors for prompt rendering, and a new CLI-based agent loop (veriact/agent/cli_agent.py). - Adds
--no-harnessrunner mode and an offline scorer (veriact/run/score_no_harness.py) to evaluate no-harness runs post-hoc. - Adds
agent_harness/to scaffold self-contained task directories (vendored tools + wrappers + drivers) for comparing external agents.
Reviewed changes
Copilot reviewed 43 out of 52 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| veriact/tools/verifier_tool.py | New OpenJML verifier wrapper + error parsing/classification used by CLI verify. |
| veriact/tools/descriptors.py | New prompt-facing tool descriptors (stubs) for CLI-dispatched tools. |
| veriact/tools/default_tools.py | Fixes Tool base import path for task_complete tool. |
| veriact/tools/cli.py | New CLI entrypoint exposing verify/harness/submit as subprocess-driven tools. |
| veriact/tools/base.py | Trims Tool base class to only the CLI-ReAct needs; removes serialization helpers. |
| veriact/tools/init.py | Package marker for tools module. |
| veriact/tools.py | Removes prior in-process Tool suite (verify/analyze/harness wrappers). |
| veriact/tool_validation.py | Removes AST-based Tool validation used by CodeAct mode. |
| veriact/run/score_no_harness.py | New offline scorer to post-hoc run spec-harness over no-harness outputs. |
| veriact/run/run_single.py | Updates imports; adds --no-harness flag and passes through to agent. |
| veriact/run/run_batch.py | Updates imports; adds --no-harness flag and passes through to per-task agents. |
| veriact/README.md | Updates documentation to describe the CLI-ReAct architecture and workflows. |
| veriact/prompts/veriact_prompt.yaml | Removes old CodeAct (Python-exec) system prompt. |
| veriact/prompts/veriact_cli_prompt.yaml | Adds CLI-ReAct system prompt (verify + harness + task_complete). |
| veriact/prompts/veriact_cli_no_harness_prompt.yaml | Adds CLI-ReAct no-harness prompt (verify + task_complete only). |
| veriact/core/utility.py | New consolidated utility module (errors, parsing, serialization helpers, etc.). |
| veriact/core/monitoring.py | Updates imports to new veriact.core.utility location. |
| veriact/core/models.py | Updates Tool + utility imports to new locations. |
| veriact/core/memory.py | Updates imports to new veriact.core.* module structure. |
| veriact/core/file_utility.py | New JSON/file I/O helpers module. |
| veriact/core/data_types.py | New shared domain types/constants module (Task/TestCase/threshold). |
| veriact/core/agent_types.py | New AgentType wrappers and input/output coercion helpers. |
| veriact/core/init.py | Package marker for new core module. |
| veriact/config.py | Changes dotenv loading strategy (search cwd + repo config/.env). |
| veriact/codeact.py | Removes old CodeAct agent implementation (in-process Python exec loop). |
| veriact/agent/cli_agent.py | New CLI-driven MultiStepAgent + CLIAgent implementation. |
| veriact/agent/agent.py | New VeriActAgent entrypoint that sets up per-task workspace + runs CLIAgent. |
| veriact/agent/init.py | Exports CLI agent and entrypoint. |
| veriact/agent.py | Removes old CodeAct mode entrypoint. |
| veriact/_function_type_hints_utils.py | Removes type-hints schema generation helpers used by old tool serialization. |
| veriact/init.py | Updates package exports/import paths to new agent/core layout. |
| scripts/run_spec_harness_loop.sh | Adds a helper script for running spec-harness eval over response lists. |
| pyproject.toml | Updates console script entry to veriact.tools.cli:main. |
| agent_harness/templates/verify.sh | New wrapper script template for verify tool in scaffolded task dirs. |
| agent_harness/templates/verifier_tool.py | Vendored verifier implementation for scaffolded task dirs. |
| agent_harness/templates/submit.sh | New wrapper script template for submit tool in scaffolded task dirs. |
| agent_harness/templates/score_all.py | Post-hoc scoring template for entire out-root (esp. no-harness arm). |
| agent_harness/templates/run_specharness.sh | New wrapper script template for harness tool in scaffolded task dirs. |
| agent_harness/templates/run_agents.py | Template driver to fan out agent sessions over task dirs. |
| agent_harness/templates/root_README.md.tmpl | Template README for scaffolded out-roots. |
| agent_harness/templates/requirements.txt | Template pip requirements for scaffolded harness venv bootstrap. |
| agent_harness/templates/collect.py | Template aggregator to build comparison tables from submissions. |
| agent_harness/templates/cli.py | Vendored multi-tool CLI for scaffolded task dirs (verify/harness/submit). |
| agent_harness/templates/AGENTS.md.tmpl | Template agent prompt/instructions (with harness). |
| agent_harness/templates/AGENTS_no_harness.md.tmpl | Template agent prompt/instructions (no-harness arm). |
| agent_harness/templates/_env.sh | Template env/bootstrap script to find Python + install javalang if needed. |
| agent_harness/scaffold.py | New scaffolder to generate per-task workspaces and drop templates. |
| agent_harness/README.md | Documentation for agent_harness usage and experiment workflow. |
| agent_harness/init.py | Package marker and high-level module docs for agent_harness. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+59
to
+78
| def extract_errors(error_message: str) -> list[tuple[str, str]]: | ||
| pattern = r"(/tmp/[^:]+:\d+: )(\w+):" | ||
| matches = re.split(r"(?=/tmp/[^:]+:\d+: \w+:)", error_message) | ||
| errors = [] | ||
| for match in matches: | ||
| if match.strip(): | ||
| level_match = re.search(pattern, match) | ||
| if level_match: | ||
| error_level = level_match.group(2) | ||
| if error_level != "warning": | ||
| if ( | ||
| "Associated declaration:" in match | ||
| or "Associated method exit" in match | ||
| ): | ||
| if len(errors) == 0: | ||
| continue | ||
| errors[-1] = (errors[-1][0], errors[-1][1] + match) | ||
| else: | ||
| errors.append((error_level, match.strip())) | ||
| return errors |
Comment on lines
+199
to
+208
| def return_verification_result(raw_output: str, return_code: int) -> VerificationResult: | ||
| classified_errors = [] | ||
| if return_code != 0 and raw_output.strip(): | ||
| try: | ||
| errors = extract_errors(raw_output) | ||
| for error_level, error_msg in errors: | ||
| error_type = classify_failures(error_level, error_msg) | ||
| classified_errors.append({"type": error_type, "raw": error_msg[:500]}) | ||
| except Exception: | ||
| classified_errors.append({"type": "ParseError", "raw": raw_output[:500]}) |
| | `memory.py` | `ActionStep`, `AgentMemory` — records each agent step for logging and replay | | ||
| | `agent_types.py` | `AgentText` type wrapper for agent I/O | | ||
| | `prompts/veriact_prompt.yaml` | System prompt: role, tool descriptions, workflow, timeout strategy | | ||
| The three tools are the subcommands of [`cli.py`](veriact/cli.py): |
Comment on lines
+15
to
+20
| # 2) repo config/.env relative to this package, if it exists | ||
| for base in (Path(__file__).resolve().parent, *Path(__file__).resolve().parents): | ||
| candidate = base / "config" / ".env" | ||
| if candidate.exists(): | ||
| load_dotenv(dotenv_path=str(candidate)) | ||
| break |
Comment on lines
+7
to
+11
| basepath="/home/mdrh/experiments/all_approaches_result_jsonl" | ||
| benchmark_path="/home/mdrh/code/VeriAct/benchmarks/specgenbench/sgb.json" | ||
| openjml="openjml" | ||
| output_root="/home/mdrh/experiments/spec_harness_sgb_results_updated" | ||
| threads=12 |
| --verbose | ||
|
|
||
| sleep 5 | ||
| pkill -f cvc4-1.6 || true |
Comment on lines
+350
to
+353
| p = argparse.ArgumentParser( | ||
| prog="harness/cli.py", | ||
| description="VeriAct tools: verify | analyze | harness | submit.", | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--no-harnessprompt and flag to run veract w/o spec-harnessagent-harnessthat prepare seperate task workspace for CLI coding agents (codex, claude code)