Skip to content

Convert the judge / evaluator hot path to async #393

@franconicola

Description

Problem.
attacks/evaluator/evaluation_step.py (~1.35 kLOC) runs judge calls via ThreadPoolExecutor. With N goals × M judges × K iterations and blocking HTTP, throughput is GIL/thread-bound. httpx, openai, and litellm all support async.

Actions.

  • Introduce an httpx.AsyncClient instance scoped to the run (with the timeout from chore(ci)(deps): bump codecov/codecov-action from 4 to 5 #2).
  • Rewrite the judge fan-out as asyncio.gather with a bounded asyncio.Semaphore for rate-limit pressure (configurable, default e.g. 10).
  • Benchmark: pick one representative attack (e.g. pair with 50 goals × 1 judge) and capture before/after wall clock + tokens/s.
  • Keep a thin sync façade so callers don't need to be async.
  • Update docs/ with a "Concurrency" page documenting the new knob.

Files:
attacks/evaluator/evaluation_step.py, attacks/evaluator/base.py, possibly router/router.py.

Acceptance:
≥2× speedup on the chosen benchmark; no regression in tests/unit; a new integration test confirms ordering of results is stable.

Metadata

Metadata

Labels

bugSomething isn't working

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions