NiceEval

Progressive, agent-native evals tool for AI agents, with excellent DX

中文 | Deutsch | Español | français | 日本語 | 한국어 | Português | Русский

NiceEval is an agent eval tool that helps teams measure, evaluate, and improve AI in production. With NiceEval, teams can compare models, iterate on agents, catch regressions, and keep improving their AI applications using real user data.

NiceEval is local-first at its core: your evals run in your own environment. When your team needs to share evals or track regressions, you can push a Report to platforms like BrainTrust, or export a custom report.

Why NiceEval when DeepEval, LangFuse, and BrainTrust already exist

NiceEval is an Agent-Native eval tool. The Dataset / golden pattern of building an Input and an Expected Output doesn't fit real agent evaluation. Agents today need to be evaluated at a finer grain — multi-turn conversations, multi-agent collaboration, tool calls, skill loading — and NiceEval does this better.

It also coexists with LangFuse and BrainTrust: use them for tracing, or upload eval results to both.

Architecture

NiceEval supports two integration modes, depending on whether the agent under test needs an isolated sandbox filesystem.

Mode 1: Sandbox (Docker, E2B) — run coding agents like Codex and Claude Code that need a sandbox

   evals/*.eval.ts
        │
        ▼
   ┌────────────┐
   │  NiceEval  │
   └────────────┘
        │
        │ Agent adapter (official)
        ▼
   ┌────────────────────────────────┐
   │         Docker Sandbox         │
   │    ┌────────────────────────┐  │
   │    │ Codex / Claude Code    │  │
   │    │ apps needing isolation │  │
   │    └────────────────────────┘  │
   └────────────────────────────────┘

Mode 2: Direct — connect straight to your own AI Agent

   evals/*.eval.ts
        │
        ▼
   ┌────────────┐
   │  NiceEval  │
   └────────────┘
        │
        │ Agent adapter (official, or your own implementation)
        ▼
   ┌──────────────────────────┐
   │    your own AI Agent     │
   │   (AI SDK·LangGraph·Pi)  │
   └──────────────────────────┘

• NiceEval core owns discovery, scheduling, scoring, reporting, and artifacts.
• Agent adapters are the open boundary: you decide how to call the system under test.
• Coding agents that need filesystem isolation run inside the Docker Sandbox; your own AI agent can connect directly, without Docker.

Core concepts at a glance

Concept	In one line
Eval	A test case: written in evals/*.eval.ts, describing what to check.
Experiment	A checked-in run configuration: which Adapter, which model, which flags.
Adapter	The layer that connects to the system under test: implement one send, get back a standard event stream.
Sandbox	Only needed for coding agents that require an isolated workspace; a direct web agent doesn't need one.
Tier	Three levels of Adapter integration effort: Tier 1 wires up send only, Tier 2 adds OTel for a call waterfall, Tier 3 makes invasive changes for feature A/B testing.

See the full glossary in the architecture overview.

Example

// evals/eval-tool-call.eval.ts
import { defineEval } from "niceeval";

export default defineEval({
  description: "Verify the agent calls the weather tool and answers from its result",

  async test(t) {
    const turn = await t.send("What's the weather in Beijing today?");
    t.succeeded();

    await t.group("calls get_weather with the right city", () => {
      t.calledTool("get_weather", { input: { city: "Beijing" } });
      t.messageIncludes(/°C|sunny|cloudy|rain/);
    });

    const second = await t.send("What about Shanghai tomorrow?");
    second.messageIncludes("Shanghai");

    t.judge.autoevals
      .closedQA("Does the reply use the tool's weather data instead of making up a temperature?")
      .atLeast(0.7);
  },
});

// experiments/local.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "./adapter"; // your agent adapter, pointed at the system under test

export default defineExperiment({
  agent: webAgent({ baseUrl: "http://127.0.0.1:5188" }),
  model: "gpt-5.5"
});

pnpm exec niceeval exp local eval-tool-call  // run only eval-tool-call under the local experiment
pnpm exec niceeval view // view eval results

Quick Start

READ https://niceeval.com/INIT.md and install niceeval for this repo.

Start from the scenario that matches what you need to evaluate:

• Claude Code / Codex plugin eval
• Claude Code / Codex skill eval
• AI Agent application eval

Roadmap

Official Adapters

• Agent Software

  • Claude Code
  • Codex
  • Bub
  • OpenClaw
  • Hermess Agent
  • Alma
  • ...

• Agent Frameworks

  • AI SDK
  • Claude SDK
  • Codex SDK
  • Pi Agent SDK
  • LangGraph
  • vm0
  • Cursor Agent SDK

Documentation

• Quickstart

Acknowledgements

This project was inspired by — or had its code learned by AI from — the projects below:

• eve: the main DX and API inspiration
• agent eval
• ponytail

Thanks to the following communities

• WIP