Skip to content
@llmonade

LLMonade

README.md
LLMonade_tagline

Welcome to LLMonade

LLMonade is an open-source AI evaluation tool that helps small development teams systematically improve their LLM-integrated applications through guided, manual error analysis. The streamlined interface pulls real trace data, allows for quick annotation of outputs, automatically categorizes failure patterns, and identifies the highest ROI fixes. LLMonade’s step-by-step approach empowers teams new to AI evaluation to adopt the proven best practice of manual error analysis.

🍋 What is LLMonade?

LLMonade is an evaluation tool that provdes:

  • a guided error analysis workflow
  • a frictionless annotation interface
  • automated trace ingestion with an existing observability tool
  • automated deployment to AWS

For more information, please visit our case study.

☀️ Why Error Analysis Matters

Error analysis is one of the most frequently skipped or superficially performed steps in the AI development lifecycle, especially by teams that are new to building with LLMs. LLMonade provides a focused approach to manual error analysis that identifies the highest-ROI issues.

llmonade_process

🍃 Key Features

  • Root Spans Extract, Transform, Load Pipeline: LLMonade integrates seamlessly with the AI observability tool Phoenix.
  • Automated ETL: Once deployed, the ETL pipeline automatically ingests existing traces from Phoenix and any new traces that get generated.
  • Effective Human Review Interface: Purpose-built interface that removes all friction from data inspection: keyboard shortcuts keep the review process moving quickly, llm-structured input/output allows for an easy-on-the-eye data viewing experience, unified data viewer displays all information needed to evaluate an output without needed to switch between windows.

🪴 Getting Started

To get starting using LLMonade, visit our CLI deployer's installation guide.

👨‍🌾 Meet the Team

LLMonade is built by a passionate team of engineers dedicated to empowering development teams to make targeted improvements with confidence.

Alex Harnett | Software Engineer | San Francisco, CA
Josh Cutts | Software Engineer | Portland, OR
Justin Shaber | Software Engineer | San Francisco, CA
Noah Raynor | Software Engineer | San Luis Obispo, CA

🌳 LLMonade’s Architecture

_full-arch-detail-edit
🍋 🌳 ✅ 🍋

Popular repositories Loading

  1. server server Public

    AI evaluation tool that helps development teams systematically improve their LLM powered applications through guided, manual error analysis. (React, Express, AWS, SQL)

    TypeScript

  2. client client Public

    React frontend for Error Analysis App

    TypeScript

  3. etl-pipeline etl-pipeline Public

    Serverless ETL pipeline that extracts project and span data from the Phoenix API

    TypeScript

  4. CLI CLI Public

    CLI tool that uses AWS CDK-based deployment framework to provision and configure serverless infrastructure to self host LLMonade

    JavaScript

  5. llmonade.github.io llmonade.github.io Public

    In-depth case study for LLMonade, an open source evaluation tool for LLM applications.

    Astro

  6. .github .github Public

Repositories

Showing 6 of 6 repositories

Top languages

Loading…

Most used topics

Loading…