Releases · braintrustdata/autoevals

What's Changed

ci(publish-js): align pnpm version with mise by Abhijeet Prasad (@AbhiPrasad) in #184
[OAI] Allow forcing Responses API for non-gpt-5 model names by Kenny Wong (@wong-codaio) in #190
feat: add [ LiteLLM AI Gateway ] for provider independence by Aarish Alam (@RheagalFire) in #186
ci: Improve autoevals CI and fix failing Python CI by Abhijeet Prasad (@AbhiPrasad) in #195
docs: document score result shape by Rul1an (@Rul1an) in #189
fix: Limit all extra to production dependencies by Abhijeet Prasad (@AbhiPrasad) in #196
Allow autoevals to support both zod 3 and zod 4 by Caitlin Pinn (@cpinn) in #155
Move from legacy proxy to gateway by Erin McNulty (@erin2722) in #191
chore(release): bump autoevals version to 0.3.0 by Abhijeet Prasad (@AbhiPrasad) in #197

New Contributors

Aarish Alam (@RheagalFire) made their first contribution in #186
Rul1an (@Rul1an) made their first contribution in #189
Erin McNulty (@erin2722) made their first contribution in #191

Full Changelog: py-0.2.0...py-0.3.0

What's Changed

Update README.md by davidatbraintrust in #1
Setup: Include template *.yaml files in output by Kenny Wong (@wong-codaio) in #2
Updates to Levenshtein + methods by Kenny Wong (@wong-codaio) in #3
Make the package isomorphic by Andy Brenneke (@abrenneke) in #5
Package uses cjs for commonjs, so .js needs to be module by Andy Brenneke (@abrenneke) in #6
Add file extensions for true ESM by Andy Brenneke (@abrenneke) in #7
Remove fancy regex from JS implementation by Ankur Goyal (@ankrgyl) in #8
Update link to Autoevals docs by Kenny (@dashk) in #13
Prettier: Fixes by Kenny (@dashk) in #11
Github Action: Trigger pre-commit hooks by Kenny (@dashk) in #12
Create JS workflow by Kenny (@dashk) in #15
Create Python workflow by Kenny (@dashk) in #14
Add numeric and json diff by Ankur Goyal (@ankrgyl) in #16
Use function calling to parse responses by Ankur Goyal (@ankrgyl) in #18
Add tracing support by Ankur Goyal (@ankrgyl) in #17
Remove pydantic as a dependency by Ankur Goyal (@ankrgyl) in #19
Enable azure openai engines by Edward Atkins (@ecatkins) in #20
Move score tracing into sdk framework. by Manu Goyal (@manugoyal) in #22
Update OpenAI version to v4 and add JS tracing by Ankur Goyal (@ankrgyl) in #21
Disable threading checks and synchronize access with sync/async locks by Ankur Goyal (@ankrgyl) in #23
Pin openai version to the one we use. by Manu Goyal (@manugoyal) in #24
Update pydoc-markdown requirement by Manu Goyal (@manugoyal) in #25
Improve mustache by Ankur Goyal (@ankrgyl) in #26
Support multiple openai versions in python by Ankur Goyal (@ankrgyl) in #27
Set the braintrust proxy as the api url to facilitate caching by Ankur Goyal (@ankrgyl) in #33
Remove validity score by Manu Goyal (@manugoyal) in #34
Reduce dependencies on SDK span logic. by Manu Goyal (@manugoyal) in #35
Support python3.8 by Ankur Goyal (@ankrgyl) in #36
Replace duplicated definitions with @braintrust/core. by Manu Goyal (@manugoyal) in #37
Update packaging script locations. by Manu Goyal (@manugoyal) in #38
Embedding distance by Ankur Goyal (@ankrgyl) in #39
Rename EmbeddingDistance to EmbeddingSimilarity by Ankur Goyal (@ankrgyl) in #41
Update bundling to resemble sdk and proxy by Ankur Goyal (@ankrgyl) in #40
Add Levenshtein by Ankur Goyal (@ankrgyl) in #42
Add a manifest of all autoevals in typescript and fix export error by Ankur Goyal (@ankrgyl) in #43
Add turbo.json by Ankur Goyal (@ankrgyl) in #45
Fix binding issue when tracing inside autoevals. by Manu Goyal (@manugoyal) in #46
Align with updates to JS tracing API. by Manu Goyal (@manugoyal) in #47
Updates to track the fact that Scores can be null by Ankur Goyal (@ankrgyl) in #48
Support OPENAI_BASE_URL in autoevals by Ankur Goyal (@ankrgyl) in #49
Fix tracing bug. by Manu Goyal (@manugoyal) in #50
Use openai wrapper by Ankur Goyal (@ankrgyl) in #52
Add Sql scorer to python by Austin Moehle (@aphinx) in #53
Merge node-specific env initialization into env.ts. by Manu Goyal (@manugoyal) in #54
README fix + use enums to improve classification accuracy by Ankur Goyal (@ankrgyl) in #55
Fall back to BRAINTRUST_API_KEY if OPENAI_API_KEY is not set. by Manu Goyal (@manugoyal) in #57
Don't swallow errors into scores. by Manu Goyal (@manugoyal) in #56
Bump core version by Ankur Goyal (@ankrgyl) in #58
fix: move tsx and @types/node to devDependencies by Travis Fischer (@transitive-bullshit) in #62
Add list comparison metric by Ankur Goyal (@ankrgyl) in #59
RAGAS retrieval metrics by Ankur Goyal (@ankrgyl) in #60
Added Moderation as a scorer by Kenny (@dashk) in #63
Add generation and end-to-end RAGAS metrics by Austin Moehle (@aphinx) in #64
fix matter => manner typo by bardia-pourvakil in #66
Add Valid JSON as a scorer by tara (@tara-nagar) in #67
Update autoevals README to remove usage of experiment.log by Austin Moehle (@aphinx) in #68
Update openai package by Ankur Goyal (@ankrgyl) in #69
Fix list contains DX by Ankur Goyal (@ankrgyl) in #71
Some housekeeping updates by Ankur Goyal (@ankrgyl) in #72
Add support for partial syntax by Ankur Goyal (@ankrgyl) in #73
Readme fixes by Dan Lee (@danielericlee) in #75
Add public evals by Ankur Goyal (@ankrgyl) in #74
Update links for npm package by tara (@tara-nagar) in #76
update pre-commit prettier to 3.3.2 by john (@j13huang) in #77
Add ExactMatch scorer by Ankur Goyal (@ankrgyl) in #79
Propagate args to embedding metrics by Ankur Goyal (@ankrgyl) in #83
Change proxy URL by Ankur Goyal (@ankrgyl) in #85
Fix rationale by Ankur Goyal (@ankrgyl) in #88
Add scorer purpose by Ankur Goyal (@ankrgyl) in #87
Propagate prompt template through manifest by Ankur Goyal (@ankrgyl) in #86
Fix a few typos found via doc tests by Ankur Goyal (@ankrgyl) in #92
Support AzureOpenAI service by Ben Perlmutter (@mongodben) in #91
Add customizable embedding model to AnswerRelevancy metric by Ben Perlmutter (@mongodben) in #94
Ensure EmbeddingSimilarity scores do not exceed 1 by Dan Lee (@danielericlee) in #95
Fix ExactMatch export by Ankur Goyal (@ankrgyl) in #97
Fix embedding similarity crash by Austin Moehle (@aphinx) in #90
Parse JSON in JSONDiff by Ankur Goyal (@ankrgyl) in #99
feature: configurable python (OpenAI) client by Olmo Maldonado (@ibolmo) in #102
Support non-strings in valid json by Ankur Goyal (@ankrgyl) in #103
bump by Sachin Padmanabhan (@sachinpad) in #106
update version by Olmo Maldonado (@ibolmo) in #107
bump autoevals version by Sachin Padmanabhan (@sachinpad) in #108
wrap openai moderations by Sachin Padmanabhan (@sachinpad) in #109
escape objects to stringified in autoevals by Sachin Padmanabhan (@sachinpad) in #110
Upgrade turborepo by Manu Goyal (@manugoyal) in #111
Update autoevals manifest by Ankur Goyal (@ankrgyl) in #112
Update docs to show usage with custom AI models through the AI proxy by Hurshal Patel (@choochootrain) in #113
Improve readme for AI proxy examples based on feedback by Hurshal Patel (@choochootrain) in #114
Pass through azure open ai params to cachedChatCompletion by Olmo Maldonado (@ibolmo) in #117
bump version by Olmo Maldonado (@ibolmo) in #119
Replace Levenshtein with polyeven by Zubin Pahuja (@zpahuja) in #122
Bump the npm_and_yarn group across 1 directory with 2 updates by Dependabot (@dependabot)[bot] in #123
Bump tsup from 8.3.5 to 8.4.0 in the npm_and_yarn group across 1 directory by Dependabot (@dependabot)[bot] in #124
Bump tsup by Manu Goyal (@manugoyal) in #125
Uniform improved client= (direct & global) option for Python and JS by Olmo Maldonado (@ibolmo) in #121
Revert "Unif...