Conversation
…to 7aed6b2 Revert the swe_rebench data source back to dyyyyyyyy/swe-rebench-filtered (split=train) and restore the verl submodule to 7aed6b2 (drop the 5a38699 bump). Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Code Review
This pull request updates data preprocessing scripts for SWE-rebench, adjusts agent interaction parameters in run_infer.sh, moves logging initialization inside the semaphore block in agent_loop.py, and adds a safety check for empty test files in swe_rebench.py rewards. The review feedback suggests two improvements: defensively creating the output directory in agent_loop.py to avoid a potential FileNotFoundError before adding the file handler, and using shlex.quote and -- in git checkout within swe_rebench.py to prevent shell injection and option parsing issues.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if test_files: | ||
| reset_tests_command = f"git checkout {base_commit} {' '.join(test_files)}" | ||
| else: | ||
| reset_tests_command = "echo 'skip reset'" |
There was a problem hiding this comment.
To prevent potential shell injection vulnerabilities or command execution failures, filenames should be properly escaped using shlex.quote when constructing shell commands. Additionally, using -- before the file list in git checkout is a best practice to prevent filenames starting with a dash (-) from being interpreted as command-line options.
| if test_files: | |
| reset_tests_command = f"git checkout {base_commit} {' '.join(test_files)}" | |
| else: | |
| reset_tests_command = "echo 'skip reset'" | |
| if test_files: | |
| import shlex | |
| reset_tests_command = f"git checkout {base_commit} -- {' '.join(shlex.quote(f) for f in test_files)}" | |
| else: | |
| reset_tests_command = "echo 'skip reset'" |
| self.logger.info(f"output_dir: {self.output_dir}") | ||
|
|
||
| async with self._semaphore: | ||
| add_file_handler(self.output_dir / "run.log", self.run_id) |
There was a problem hiding this comment.
The directory self.output_dir is not guaranteed to exist when add_file_handler is called. If the logging handler tries to open the log file before the directory is created, it can raise a FileNotFoundError. Creating the directory defensively before adding the file handler prevents this potential failure.
| add_file_handler(self.output_dir / "run.log", self.run_id) | |
| self.output_dir.mkdir(parents=True, exist_ok=True) | |
| add_file_handler(self.output_dir / "run.log", self.run_id) |
Summary
Minor data-pipeline / eval fixes and housekeeping for the SWE-agent examples.
Data preprocessing
swe_rebench.py: add aSKIP_INSTANCEShook and drop instances without an image; remove thegit checkout <base_commit>/git cleansteps from the per-instance reset; print the final instance count.swe_rebench_v2.py: filter out instances whose deployment image isNone.Eval / reward
reward/swe_rebench.py: skip the test-file reset when a task has no modified test files (avoids runninggit checkout <commit>with no paths).Agent loop
agent_loop.py: move the per-run log handler and config logging inside the concurrency semaphore.Misc
run_infer.sh: point the example at the modal parquet/config and bump--nnodes+ sampling params.README.md: add supervisor note to the citation.verlsubmodule.