This repo reproduces and extends an evaluation pipeline for assesing chemical synthesis safety, built on Inspect AI, using only free-tier model providers (Gemini, Groq).
Source: ChemSafetyBench paper (arXiv:2411.16736) - repo not available anymore
The original benchmark uses a single GPT-4o judge for both safety and quality scoring for every task. A single judge can't tell you whether its score reflects the response or the judge's own idiosyncrasies (self-preference, verbosity bias, position bias). Running two independent judges and measuring agreement (Cohen's kappa / correlation) is a methodologically stronger setup than the original paper.
The secondary aim of the project is to provide an open source, free to use evaluation pipeline. This could be adapted, depending on personal budgets.
-
Get free API keys:
- Google AI Studio: https://aistudio.google.com
- Groq: https://console.groq.com
-
Set environment variables:
export GOOGLE_API_KEY="..." export GROQ_API_KEY="..."
-
Install deps:
pip install inspect-ai google-genai groq <any other LLM API SDK>
-
Run a small smoke test before spending your daily quota:
inspect eval task.py -T limit=10 --model google/gemini-2.5-flash -
Run the full sweep (see
RATE_LIMIT_NOTES.mdfor batching guidance):inspect eval task.py --model groq/llama-3.3-70b-versatile inspect eval task.py --model google/gemini-2.5-flash
build_dataset.py,dataset.py- creates the dataset, loads prompts into InspectSamplessolvers.py- harness variants (plain generate, CoT, name-hack)scorer.py- dual-judge safety + quality scorer (Gemini + Groq judges)task.py- wires dataset + solver + scorer into an InspectTaskRATE_LIMIT_NOTES.md— free-tier quota budgeting for a full run
3 models under test (Gemini Flash, Llama-3.3-70B via Groq, one more Groq model e.g. Qwen) x 2 harness conditions (plain, CoT) x ~200 sampled Synthesis prompts = 1,200 generations, each scored by 2 judges = 2,400 judge calls. Comfortably within free daily quotas if split over ~2 days.