Ran the canonical command on tag v0.1.0 (093a329), no code changes, data-v0.1.0:
python -m bench_env.run --split test --agent generic_v2 \ --env-url http://localhost:4173 \ --model-base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \ --model-name qwen3.6-plus \ --parallel 8 --isolation pages --headless
|
L1 |
L2 |
L3 |
L4 |
SR |
FC |
| Reported |
100.0 |
78.1 |
44.6 |
3.8 |
45.7 |
34.0 |
| Ours |
95.0 |
53.4 |
21.7 |
8.8 |
32.4 |
17.2 |
The profile differs qualitatively: we score higher on L4 but much lower on L2/L3, and FC is half yours (most of our failures are MAX_STEPS loops, not premature completion). Enabling enable_thinking only moved SR to 34.0, so that isn't the cause.
Question: Which dataset tag matches the leaderboard — the root README points to data-v0.1.0, while docs/getting-started.md points to data-v1.0.
Happy to share logs/trajectories.
Ran the canonical command on tag v0.1.0 (093a329), no code changes, data-v0.1.0:
python -m bench_env.run --split test --agent generic_v2 \ --env-url http://localhost:4173 \ --model-base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \ --model-name qwen3.6-plus \ --parallel 8 --isolation pages --headlessThe profile differs qualitatively: we score higher on L4 but much lower on L2/L3, and FC is half yours (most of our failures are
MAX_STEPSloops, not premature completion). Enablingenable_thinkingonly moved SR to 34.0, so that isn't the cause.Question: Which dataset tag matches the leaderboard — the root README points to data-v0.1.0, while docs/getting-started.md points to data-v1.0.
Happy to share logs/trajectories.