Skip to content

Add fix agent human /fs-fix functional eval #182

Description

@ascerra

Summary

Add a functional eval under eval/fix/ that runs the real fix agent pipeline against an existing PR fixture (human /fs-fix via env) and asserts the post-script pushes a new commit touching calc.py.

Motivation

Regression guard for the fix pipeline (human trigger path). Code eval (#180 / PR #177) covers issue→PR; fix needs existing-PR→new-commit.

Acceptance criteria

  • eval/fix/ with at least one human /fs-fix case
  • Shared eval scripts supply fix env (HUMAN_INSTRUCTION, PRE_AGENT_HEAD, empty REVIEW_BODY_FILE) and checkout PR head
  • Capture includes head_sha, files, pre_agent_head for judges
  • Functional-tests CI selects and runs fix

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions