Reference

Agents

client.agents.verify_connection(...) -> VerifyConnectionResponse

📝 Description

Verify an agent's connection and persist the result when successful

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.verify_connection(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — The agent whose connection to verify

model: typing.Optional[str] — Model to verify. Omit for a basic connection check. Provide it for a model-specific check before benchmarking that model

messages: typing.Optional[typing.List[typing.Dict[str, str]]] — Sample chat messages to send during verification. Omit to use the default probe

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.resolve(...) -> ResolveAgentNamesResponse

📝 Description

Get the IDs for your agents by their names

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.resolve(
    names=[
        "my-agent",
        "support-bot"
    ],
)

⚙️ Parameters

names: typing.List[str] — Agent names to resolve to IDs

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.list(...) -> PaginatedResponseAgentSummary

📝 Description

Get the list of all your agents

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.list()

⚙️ Parameters

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.create(...) -> AgentCreateResponse

📝 Description

Create an agent to test inside Calibrate or connect your existing agent to Calibrate

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.create(
    name="name",
)

⚙️ Parameters

name: str — Agent name, unique within the workspace

type: typing.Optional[AgentCreateType]

agent: built inside Calibrate
connection: your existing agent connected to Calibrate

config: typing.Optional[typing.Dict[str, typing.Any]]

Agent behavioral config. The keys depend on type.

type=agent, built inside Calibrate:

system_prompt: the agent's instructions
llm.model: provider/model, e.g. openai/gpt-4.1 or google/gemini-2.5-flash
stt.provider: deepgram, openai, cartesia, elevenlabs, google, sarvam, or smallest
tts.provider: cartesia, openai, google, elevenlabs, sarvam, or smallest
settings.agent_speaks_first, settings.max_assistant_turns
system_tools.end_call: let the agent end the call
data_extraction_fields: [{name, type, description, required}]

{
  "system_prompt": "You are a helpful support agent.",
  "llm": {"model": "openai/gpt-4.1"},
  "stt": {"provider": "deepgram"},
  "tts": {"provider": "elevenlabs"},
  "settings": {"agent_speaks_first": true, "max_assistant_turns": 50}
}

type=connection, your own HTTP endpoint:

agent_url: public HTTP(S) endpoint your agent is called at
agent_headers: headers sent on each request, e.g. auth
benchmark_provider: openrouter by default. Other values: openai, google, anthropic, meta-llama, mistralai, deepseek, x-ai, cohere, qwen, or ai21

{
  "agent_url": "https://api.example.com/agent",
  "agent_headers": {"Authorization": "Bearer <token>"},
  "benchmark_provider": "openrouter"
}

For type=agent, omitted keys inherit managed defaults. Omit config entirely to use all defaults. For type=connection, config is stored as-is and must contain agent_url

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.get(...) -> RoutersAgentsAgentResponse

📝 Description

Get one agent by its ID

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.get(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — The agent to retrieve

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.update(...) -> RoutersAgentsAgentResponse

📝 Description

Update an agent's configuration

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.update(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — The agent to update

name: typing.Optional[str] — New agent name. Omit to leave the name unchanged

config: typing.Optional[typing.Dict[str, typing.Any]]

Agent behavioral config. The keys depend on type.

type=agent, built inside Calibrate:

system_prompt: the agent's instructions
llm.model: provider/model, e.g. openai/gpt-4.1 or google/gemini-2.5-flash
stt.provider: deepgram, openai, cartesia, elevenlabs, google, sarvam, or smallest
tts.provider: cartesia, openai, google, elevenlabs, sarvam, or smallest
settings.agent_speaks_first, settings.max_assistant_turns
system_tools.end_call: let the agent end the call
data_extraction_fields: [{name, type, description, required}]

{
  "system_prompt": "You are a helpful support agent.",
  "llm": {"model": "openai/gpt-4.1"},
  "stt": {"provider": "deepgram"},
  "tts": {"provider": "elevenlabs"},
  "settings": {"agent_speaks_first": true, "max_assistant_turns": 50}
}

type=connection, your own HTTP endpoint:

agent_url: public HTTP(S) endpoint your agent is called at
agent_headers: headers sent on each request, e.g. auth
benchmark_provider: openrouter by default. Other values: openai, google, anthropic, meta-llama, mistralai, deepseek, x-ai, cohere, qwen, or ai21

{
  "agent_url": "https://api.example.com/agent",
  "agent_headers": {"Authorization": "Bearer <token>"},
  "benchmark_provider": "openrouter"
}

Replaces the stored config. Omit to leave unchanged

For type=connection, changing agent_url or agent_headers resets the connection and benchmark verification flags

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

Tests

client.tests.bulk_create(...) -> BulkTestUploadResponse

📝 Description

Create many test cases at once and link them to your agents

🔌 Usage

from calibrate import Calibrate, BulkTestItem, ChatMessage
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.bulk_create(
    type="response",
    tests=[
        BulkTestItem(
            name="name",
            conversation_history=[
                ChatMessage(
                    role="user",
                )
            ],
        )
    ],
)

⚙️ Parameters

type: BulkTestUploadType

What the test judges:

response: judges the generated reply
tool_call: diffs the generated tool calls
conversation: judges the full conversation

Applied to every test in the batch

tests: typing.List[BulkTestItem] — Test items to create, at most 500 per request, with names unique within the batch

agent_uuids: typing.Optional[typing.List[str]] — IDs of agents to link every created test to. Omit to link none

language: typing.Optional[str] — Language written to each test's config.settings.language. Omit to leave unset

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.tests.list(...) -> PaginatedResponseTestListResponse

📝 Description

List all the test cases for your agents

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.list()

⚙️ Parameters

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.tests.create(...) -> TestCreateResponse

📝 Description

Create a test that runs your agent against a conversation and evaluates its answer quality or the tools it calls

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.create(
    name="name",
    type="response",
)

⚙️ Parameters

name: str — Name of the test, unique within the workspace

type: TestCreateType

What the test judges:

response: judges the generated reply
tool_call: diffs the generated tool calls
conversation: judges the full conversation

config: typing.Optional[typing.Dict[str, typing.Any]]

The calibrate test config. Three top-level keys.

history: the required conversation up to the agent's turn. Each item is {role, content} with role one of user, assistant, tool. A tool message also carries tool_call_id and name.
evaluation: the required {type, ...}, where type matches the test's type below.
settings: an optional object, e.g. {"language": "en"}.

evaluation by test type:

response: judge the agent's reply, graded by the linked evaluators. {"type": "response"}
conversation: append the reply and judge the whole conversation. {"type": "conversation"}
tool_call: diff the agent's tool calls against expected ones. Add tool_calls, a list of {tool, arguments, accept_any_arguments?}.

For tool_call, each expected argument value is one of:

{"match_type": "exact", "value": <any>}: must equal value
{"match_type": "llm_judge", "criteria": "..."}: judged against the criteria
{"match_type": "any"}: any value, only checks the argument was passed

response / conversation example:

{
  "history": [{"role": "user", "content": "What is your return policy?"}],
  "evaluation": {"type": "response"},
  "settings": {"language": "en"}
}

tool_call example:

{
  "history": [{"role": "user", "content": "Book room 101 for tomorrow"}],
  "evaluation": {
    "type": "tool_call",
    "tool_calls": [
      {
        "tool": "book_room",
        "arguments": {
          "room": {"match_type": "exact", "value": "101"},
          "date": {"match_type": "llm_judge", "criteria": "tomorrow's date"}
        },
        "accept_any_arguments": false
      }
    ]
  }
}

Evaluators are linked via the separate evaluators field, not inside config.

Omit to create the test with no config and fill it in later via update

evaluators: typing.Optional[typing.List[RoutersTestsEvaluatorRef]] — Evaluators to link. Used by response and conversation tests

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.tests.get(...) -> TestResponse

📝 Description

Get an agent test case by its ID

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.get(
    test_uuid="b1c2d3e4-f5a6-7890-bcde-f12345678901",
)

⚙️ Parameters

test_uuid: str — Test to retrieve

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.tests.update(...) -> TestResponse

📝 Description

Update an agent test case

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.update(
    test_uuid="b1c2d3e4-f5a6-7890-bcde-f12345678901",
)

⚙️ Parameters

test_uuid: str — Test to update

name: typing.Optional[str] — New test name. Omit to leave unchanged

type: typing.Optional[TestUpdateType]

What the test judges:

response: judges the generated reply
tool_call: diffs the generated tool calls
conversation: judges the full conversation

Immutable. Omit it, or send the current value

config: typing.Optional[typing.Dict[str, typing.Any]]

The calibrate test config. Three top-level keys.

history: the required conversation up to the agent's turn. Each item is {role, content} with role one of user, assistant, tool. A tool message also carries tool_call_id and name.
evaluation: the required {type, ...}, where type matches the test's type below.
settings: an optional object, e.g. {"language": "en"}.

evaluation by test type:

response: judge the agent's reply, graded by the linked evaluators. {"type": "response"}
conversation: append the reply and judge the whole conversation. {"type": "conversation"}
tool_call: diff the agent's tool calls against expected ones. Add tool_calls, a list of {tool, arguments, accept_any_arguments?}.

For tool_call, each expected argument value is one of:

{"match_type": "exact", "value": <any>}: must equal value
{"match_type": "llm_judge", "criteria": "..."}: judged against the criteria
{"match_type": "any"}: any value, only checks the argument was passed

response / conversation example:

{
  "history": [{"role": "user", "content": "What is your return policy?"}],
  "evaluation": {"type": "response"},
  "settings": {"language": "en"}
}

tool_call example:

{
  "history": [{"role": "user", "content": "Book room 101 for tomorrow"}],
  "evaluation": {
    "type": "tool_call",
    "tool_calls": [
      {
        "tool": "book_room",
        "arguments": {
          "room": {"match_type": "exact", "value": "101"},
          "date": {"match_type": "llm_judge", "criteria": "tomorrow's date"}
        },
        "accept_any_arguments": false
      }
    ]
  }
}

Evaluators are linked via the separate evaluators field, not inside config.

Replaces the stored config. Omit to leave unchanged

evaluators: typing.Optional[typing.List[RoutersTestsEvaluatorRef]] — New evaluator links for the test. Omit to leave unchanged. An empty list clears them, except on conversation tests, which must keep at least one

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

AgentTests

client.agent_tests.link(...) -> AgentTestsCreateResponse

📝 Description

Link one or more tests to an agent. Tests that are already linked are skipped.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.link(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    test_uuids=[
        "b1c2d3e4-f5a6-7890-bcde-f12345678901"
    ],
)

⚙️ Parameters

agent_uuid: str — Agent to link tests to

test_uuids: typing.List[str] — Tests to link. Any that are already linked are skipped

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.list_for_agent(...) -> PaginatedResponseTestListResponse

📝 Description

List the tests linked to an agent.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.list_for_agent(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — Agent whose linked tests to list

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.list_runs_for_agent(...) -> PaginatedResponseAgentTestRunListItem

📝 Description

List an agent's test runs with their results

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.list_runs_for_agent(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — Agent whose test runs to list

type: typing.Optional[ListRunsForAgentAgentTestsRequestType]

Filter by run type. Omit to return both:

llm-unit-test: single runs of an agent's tests
llm-benchmark: multi-model comparisons

status: typing.Optional[TaskStatus] — Filter by run status. Omit for all statuses

has_failures: typing.Optional[bool] — Filter by whether the run has any failing test case or model. true returns only runs with failures (or errors), false only clean runs. Omit for both

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.run(...) -> AgentTestRunCreateResponse

📝 Description

Run an agent's linked tests as a background job, returning a task ID to poll.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.run(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — Agent to test

test_uuids: typing.Optional[typing.List[str]] — Tests to run. Omit to run all tests linked to the agent

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.run_batch(...) -> BatchTestRunResponse

📝 Description

Run agent tests for every agent, or for a selected set.

🔌 Usage

from calibrate import Calibrate, BatchRunRequest
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.run_batch(
    request=BatchRunRequest(),
)

⚙️ Parameters

request: typing.Optional[BatchRunRequest]

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.get_run(...) -> TestRunStatusResponse

📝 Description

Poll a test run for its status and evaluation results.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.get_run(
    task_id="a3b2c1d0-e5f4-3210-abcd-ef1234567890",
)

⚙️ Parameters

task_id: str — Test run to poll for status and results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.benchmark(...) -> AgentTestRunCreateResponse

📝 Description

Run a multi-model benchmark on an agent's linked tests as a background job.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.benchmark(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    models=[
        "openai/gpt-4.1",
        "anthropic/claude-sonnet-4"
    ],
)

⚙️ Parameters

agent_uuid: str — Agent to benchmark

models: typing.List[str] — Model names to benchmark

test_uuids: typing.Optional[typing.List[str]] — A subset of the agent's linked tests to benchmark. Each ID must be linked to the agent. Omit to run all linked tests

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.get_benchmark(...) -> BenchmarkStatusResponse

📝 Description

Get the results of a benchmark run

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.get_benchmark(
    task_id="a3b2c1d0-e5f4-3210-abcd-ef1234567890",
)

⚙️ Parameters

task_id: str — Benchmark run to poll for status and results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

Evaluators

client.evaluators.list(...) -> PaginatedResponseEvaluatorResponse

📝 Description

List your evaluators

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.evaluators.list()

⚙️ Parameters

evaluator_type: typing.Optional[ListEvaluatorsRequestEvaluatorType] — Filter by what the evaluator judges. Omit for all types

data_type: typing.Optional[ListEvaluatorsRequestDataType] — Filter by modality. Omit for all

include_defaults: typing.Optional[bool] — When true, include the built-in default evaluators alongside the ones you created

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.evaluators.create(...) -> EvaluatorCreateResponse

📝 Description

Create an evaluator along with its first version, which is set live

🔌 Usage

from calibrate import Calibrate, EvaluatorVersionCreate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.evaluators.create(
    name="name",
    version=EvaluatorVersionCreate(
        judge_model="judge_model",
        system_prompt="system_prompt",
    ),
)

⚙️ Parameters

name: str — Evaluator name, unique within your workspace

version: EvaluatorVersionCreate — The evaluator's first version. Set as live when you create the evaluator

description: typing.Optional[str] — Description. Omit to leave blank

evaluator_type: typing.Optional[EvaluatorCreateEvaluatorType]

What the evaluator judges:

tts: TTS audio
stt: one transcript
llm: a reply with its conversation history
llm-general: a standalone input and output pair
conversation: a full conversation

data_type: typing.Optional[EvaluatorCreateDataType]

The modality the judge reads:

text
audio

output_type: typing.Optional[EvaluatorCreateOutputType]

How the evaluator scores:

binary: pass or fail
rating: a numeric score, using the scale in output_config

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.evaluators.get(...) -> EvaluatorDetailResponse

📝 Description

Get one evaluator with its full version history

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.evaluators.get(
    evaluator_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

evaluator_uuid: str — Evaluator to retrieve

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.evaluators.create_version(...) -> VersionCreateResponse

📝 Description

Add a new version to an evaluator you created

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.evaluators.create_version(
    evaluator_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    judge_model="judge_model",
    system_prompt="system_prompt",
)

⚙️ Parameters

evaluator_uuid: str — Evaluator to add a version to

judge_model: str — The model that runs the judge, named the way its provider does, for example openai/gpt-4.1 or anthropic/claude-sonnet-4

system_prompt: str — Judge system prompt. May contain {{variable}} placeholders

output_config: typing.Optional[OutputConfig] — The scale points and their labels. Required for a rating evaluator. A binary evaluator uses the default Correct/Wrong labels unless you set your own

variables: typing.Optional[typing.List[VariableSpec]] — Declared prompt variables. Omit if the prompt has none. After the first version the variable names are fixed. You can change a variable's description or default, but not add, remove, or rename one

make_live: typing.Optional[bool] — When true, immediately point the evaluator's live version at this new version

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

AnnotationTasks

client.annotation_tasks.list(...) -> PaginatedResponseAnnotationTaskResponse

📝 Description

List annotation tasks with linked evaluators

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.list()

⚙️ Parameters

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.create(...) -> AnnotationTaskCreateResponse

📝 Description

Create an annotation task for annotators to label items against evaluators

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.create(
    name="name",
    type="stt",
)

⚙️ Parameters

name: str — Task name, unique within your workspace

type: AnnotationTaskCreateType

Task type. Determines the shape of each item's payload.

stt: judge a transcript on its own
llm: judge one response with its conversation history
llm-general: judge a standalone input -> output pair
conversation: judge a full conversation

description: typing.Optional[str] — A description for the task. Omit for none

evaluator_ids: typing.Optional[typing.List[str]] — IDs of evaluators to link when the task is created, in order. Each must be one you created or a built-in default. Omit to create with no linked evaluators

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.get(...) -> AnnotationTaskResponse

📝 Description

Get one annotation task with linked evaluators, items, and labelling jobs

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.get(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

task_uuid: str — Task to retrieve

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.link_evaluator(...) -> EvaluatorLinkResponse

📝 Description

Link an evaluator to a task, appending it to the display order

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.link_evaluator(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    evaluator_id="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

evaluator_id: str — The evaluator to link. Must be one you created or a built-in default

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.add_items(...) -> BulkCreateItemsResponse

📝 Description

Bulk-create annotation items in a task, optionally seeding human annotations

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.add_items(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    items=[],
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

items: typing.List[AnnotationItemPayload] — Items to insert. Insertion order is preserved

annotator_id: typing.Optional[str] — Annotator these initial annotations belong to. Required when any item carries annotations

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.update_items(...) -> BulkUpdateItemsResponse

📝 Description

Bulk-update item payloads in a task

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.update_items(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    updates=[],
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

updates: typing.List[ItemUpdatePayload] — The new payload for each item you're updating. Entries not in this task, or referencing deleted items, are skipped

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.create_evaluator_run(...) -> EvaluatorRunLaunchResponse

📝 Description

Run evaluators on task items as a background job

🔌 Usage

from calibrate import Calibrate, EvaluatorRunRequestEntry
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.create_evaluator_run(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    evaluators=[
        EvaluatorRunRequestEntry(
            evaluator_id="f47ac10b-58cc-4372-a567-0e02b2c3d479",
        )
    ],
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

evaluators: typing.List[EvaluatorRunRequestEntry] — The evaluators to run. Each must be linked to the task

item_ids: typing.Optional[typing.List[str]] — Item IDs to run on. Required when select_all=false. Ignored when select_all=true

select_all: typing.Optional[bool] — When true, run on every item in the task. Set q to run only items whose name matches it

q: typing.Optional[str] — Case-insensitive substring filter on payload.name. Applies only when select_all=true

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.get_evaluator_run(...) -> EvaluatorRunResponse

📝 Description

Get one evaluator-run job with results and human-agreement summary

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.get_evaluator_run(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    job_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

job_uuid: str — The evaluator run to act on

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.get_agreement(...) -> TaskAgreementResponse

📝 Description

Get human-vs-human and human-vs-evaluator agreement metrics for a task

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.get_agreement(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

bucket: typing.Optional[GetAgreementAnnotationTasksRequestBucket] — How to bucket points in the trend series

days: typing.Optional[int] — Trailing window in days for the trend series

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.get_summary(...) -> TaskSummaryResponse

📝 Description

Get a paginated summary table of items, evaluator runs, and human annotations for a task

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.get_summary(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

item_id: typing.Optional[str] — Filter rows to a single item. The full task-wide annotator union is still returned in annotators

live_only: typing.Optional[bool] — When true, emit only one row for each (item, evaluator) pair using the evaluator's live version. Versions other than the live one that have runs are excluded

q: typing.Optional[str] — Case-insensitive substring search on payload.name. Blank is a no-op

sort_by: typing.Optional[str] — Sort key for the results

order: typing.Optional[GetSummaryAnnotationTasksRequestOrder] — Sort direction

limit: typing.Optional[int] — Maximum number of items to return

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

FilesExpand file tree

reference.md

Latest commit

History

reference.md

File metadata and controls

Reference

Agents

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

Tests

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

AgentTests

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

Evaluators

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters