Skip to content
This repository was archived by the owner on Apr 16, 2026. It is now read-only.

REPRODUCE.md

Latest commit

 

History

History
316 lines (242 loc) · 8.02 KB

REPRODUCE.md

File metadata and controls

316 lines (242 loc) · 8.02 KB
Raw

Reproducing CodeStruct Results on SWE-Bench

This guide provides step-by-step instructions to reproduce the results from the CodeStruct paper using the SWE-Bench Verified dataset.

Quick Start

1. Prerequisites

  • Python >= 3.11
  • OpenAI API key (for GPT models) OR AWS credentials (for Bedrock models)
  • At least 16GB RAM recommended
  • GPU not required

2. Installation (Automated)

The recommended way is to use the setup script which creates a venv and installs everything:

cd CodeStruct

# Run setup (creates venv, installs all dependencies, verifies AST tools)
chmod +x setup.sh
./setup.sh

# Activate the virtual environment
source venv/bin/activate

The setup script will:

  • Find Python >= 3.11 on your system
  • Create a virtual environment at venv/
  • Install tree-sitter with correct pinned versions (0.21.3 + tree-sitter-languages 1.10.2)
  • Install SWE-agent and swebench
  • Verify all components work (including AST tool tests)

2b. Manual Installation (Alternative)

If you prefer to install manually:

cd CodeStruct

# Create venv with Python 3.11+
python3.11 -m venv venv
source venv/bin/activate

# CRITICAL: Install tree-sitter with pinned versions FIRST
pip install "tree-sitter==0.21.3" "tree-sitter-languages==1.10.2"

# Install SWE-agent
cd SWE-agent
mkdir -p trajectories
pip install -e .
cd ..

# Install SWE-Bench for evaluation
pip install swebench

⚠️ Important: tree-sitter versions must be pinned. tree-sitter >= 0.22 has an incompatible API with tree-sitter-languages, causing AST tools to silently fail.

3. Set Up API Keys

For OpenAI models:

export OPENAI_API_KEY=sk-your-key-here

For AWS Bedrock models:

export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key
export AWS_REGION_NAME=us-east-1

Reproducing Results

Option A: Quick Test (5 instances)

Test on a small subset to verify setup:

cd CodeStruct/SWE-agent

# CodeStruct configuration (with AST tools)
python -m sweagent run-batch \
  --instances.type swe_bench \
  --instances.subset verified \
  --instances.split test \
  --config config/default_with_readcode_batch.yaml \
  --agent.model.name gpt-5-mini \
  --agent.model.per_instance_cost_limit 3.00 \
  --agent.model.top_p null \
  --agent.model.temperature 1 \
  --instances.slice :5 \
  --num_workers 5

Option B: Full Evaluation (All 500 instances)

Step 1: Run CodeStruct Configuration

With OpenAI GPT-5-mini:

cd CodeStruct/SWE-agent

python -m sweagent run-batch \
  --instances.type swe_bench \
  --instances.subset verified \
  --instances.split test \
  --config config/default_with_readcode_batch.yaml \
  --agent.model.name gpt-5-mini \
  --agent.model.per_instance_cost_limit 3.00 \
  --agent.model.top_p null \
  --agent.model.temperature 1 \
  --num_workers 10

With AWS Bedrock Qwen3-Coder 480B:

cd CodeStruct/SWE-agent

python -m sweagent run-batch \
  --instances.type swe_bench \
  --instances.subset verified \
  --instances.split test \
  --config config/default_with_readcode_batch.yaml \
  --agent.model.name bedrock/qwen.qwen3-coder-480b-a35b-v1:0 \
  --agent.model.per_instance_cost_limit 0.00 \
  --agent.model.temperature 0 \
  --num_workers 15

Step 2: Evaluate Results

After the batch run completes, find the predictions file in the trajectories directory and evaluate:

cd CodeStruct/SWE-agent

# Find your predictions file path
# It will be in: trajectories/username/config_name__model_name__*/preds.json

python -m swebench.harness.run_evaluation \
  --predictions_path trajectories/$USER/default_with_readcode_batch__gpt-5-mini__*/preds.json \
  -d princeton-nlp/SWE-bench_Verified \
  -s test \
  --run_id gpt5mini_codestruct_reproduction \
  --max_workers 15

The evaluation will generate a results JSON file with:

  • resolved: Number of issues successfully resolved
  • total: Total number of issues evaluated
  • resolve_rate: Percentage of issues resolved

Option C: Run All Model Variants

To reproduce all model results from the paper:

cd CodeStruct/SWE-agent

# GPT-5 (if available)
python -m sweagent run-batch \
  --instances.type swe_bench \
  --instances.subset verified \
  --instances.split test \
  --config config/default_with_readcode_batch.yaml \
  --agent.model.name gpt-5 \
  --agent.model.per_instance_cost_limit 5.00 \
  --agent.model.top_p null \
  --agent.model.temperature 1 \
  --num_workers 10

# GPT-5-mini
python -m sweagent run-batch \
  --instances.type swe_bench \
  --instances.subset verified \
  --instances.split test \
  --config config/default_with_readcode_batch.yaml \
  --agent.model.name gpt-5-mini \
  --agent.model.per_instance_cost_limit 3.00 \
  --agent.model.top_p null \
  --agent.model.temperature 1 \
  --num_workers 10

# GPT-5-nano
python -m sweagent run-batch \
  --instances.type swe_bench \
  --instances.subset verified \
  --instances.split test \
  --config config/default_with_readcode_batch.yaml \
  --agent.model.name gpt-5-nano \
  --agent.model.per_instance_cost_limit 1.00 \
  --agent.model.top_p null \
  --agent.model.temperature 1 \
  --num_workers 10

Comparing with Baseline (Original)

To compare CodeStruct with the baseline (without AST tools), you'll need to:

  1. Get the original SWE-agent configuration files (without AST tools)
  2. Run the same experiments with those configurations
  3. Compare the results

The baseline configuration should:

  • NOT include tools/ast_read bundle
  • NOT include tools/ast_edit bundle
  • Use standard bash tool instead of safeBash
  • Use standard string-based editing tools

Understanding the Results

Result Files

After evaluation, you'll find result files in:

  • Predictions: trajectories/username/config_model__*/preds.json
  • Evaluation: princeton-nlp/SWE-bench_Verified/test/run_id.json

Key Metrics

  • Resolved: Number of test instances where the patch passed all tests
  • Resolved Rate: Percentage of resolved instances
  • Applied: Number of patches that were successfully applied
  • Applied Rate: Percentage of successfully applied patches

Trajectory Files

Each instance generates a trajectory file showing:

  • Agent's thought process
  • Tool calls made
  • Code changes applied
  • Terminal output

These are stored in: trajectories/username/config_model__*/

Troubleshooting

Issue: "Module not found"

Solution: Make sure you ran pip install -e . from the SWE-agent directory

Issue: "API key not found"

Solution: Verify your environment variables are set:

echo $OPENAI_API_KEY  # Should show your key
# or
echo $AWS_ACCESS_KEY_ID  # Should show your key

Issue: "Rate limit exceeded"

Solution: Reduce the number of workers:

--num_workers 5  # Instead of 10 or 15

Issue: "Out of memory"

Solution:

  1. Reduce num_workers
  2. Close other applications
  3. Use a machine with more RAM

Issue: Slow execution

Solution:

  1. Check your internet connection
  2. Reduce workers for API rate limits
  3. Consider using a faster model for testing

Advanced Options

Custom Subsets

Run on specific instances:

--instances.slice 0:100  # First 100 instances
--instances.slice 100:200  # Instances 100-200

Cost Control

Set cost limits per instance:

--agent.model.per_instance_cost_limit 2.00  # Max $2 per instance

Temperature and Sampling

Adjust model behavior:

--agent.model.temperature 1  # Higher = more creative
--agent.model.top_p null     # Disable nucleus sampling

Results Storage

The ../results/SWE-Bench/ directory contains pre-computed results:

  • CodeStruct results (with AST tools):

    • gpt5_codestruct.json
    • gpt5mini_codestruct.json
    • gpt5nano_codestruct.json
    • qwen3_8b_codestruct.json
    • qwen3_32b_codestruct.json
    • qwen3coder480b_codestruct.json

  • Original results (baseline, no AST tools):

    • gpt5_original.json
    • gpt5mini_original.json
    • gpt5nano_original.json
    • qwen3_8b_original.json
    • qwen3_32b_original.json
    • qwen3coder480b_original.json

You can compare your reproduced results with these files.