Abstractor.ai - LLM Summarization Benchmark

A comprehensive benchmarking pipeline for evaluating and comparing Large Language Model (LLM) performance on text summarization tasks across multiple providers and prompt styles.

Overview

Abstractor.ai provides a systematic way to test and compare different LLM models from OpenAI, Anthropic, and Google on their summarization capabilities. The pipeline measures three key dimensions:

• Speed: Response time for each API call
• Cost: Token usage and financial cost per request
• Quality: Semantic similarity between generated and reference summaries

This tool is useful for:

• Selecting the best LLM for your summarization use cases
• Understanding cost-performance trade-offs
• Evaluating quality across different prompt styles
• Benchmarking new models as they become available

Features

Multiple LLM Providers

• OpenAI: GPT-4o, GPT-4o mini, GPT-4 Turbo, GPT-3.5 Turbo
• Anthropic: Claude 3.5 Sonnet, Claude 3.5 Haiku, Claude 3 Opus
• Google: Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro

Flexible Prompt Styles

• Concise: 2-3 sentence summaries
• Detailed: Comprehensive summaries with all important points
• Bullet Points: Organized key takeaways in list format
• Executive: Decision-focused summaries with actionable insights

Comprehensive Metrics

• Response time tracking
• Token usage (input/output)
• Cost calculation based on current pricing
• Semantic similarity scoring against reference summaries
• Performance comparisons across models and prompt types

Detailed Reports

• Visualization dashboards showing performance comparisons
• Response time analysis by model and prompt type
• Cost-quality trade-off analysis
• Downloadable CSV and JSON reports with full results

Installation

Prerequisites

• Python 3.8+
• API keys for the LLM providers you want to use

Setup

1. Clone the repository:

git clone https://github.com/yourusername/abstractor.ai.git
cd abstractor.ai

2. Create a virtual environment:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

4. Configure API keys:

cp .env.example .env

Edit .env and add your API keys:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=...

Quick Start

1. Add Your Test Data

Place text files to summarize in data/texts/:

data/texts/
├── article1.txt
├── article2.txt
└── article3.txt

Optionally add reference summaries in data/references/ (for quality evaluation):

data/references/
├── article1.txt
├── article2.txt
└── article3.txt

2. Configure the Benchmark

Open llm_summarization_benchmark.ipynb and customize:

• Models to test: Comment/uncomment model selections
• Configuration: Adjust max_tokens, temperature, and output paths
• Rate limiting: Set delay_between_requests to avoid API rate limits

Example configuration:

MODELS_TO_TEST = {
    "openai": ["gpt-4o-mini"],
    "anthropic": ["claude-3-5-haiku-20241022"],
    "google": ["gemini-2.5-flash"]
}

3. Run the Benchmark

Open and run the Jupyter notebook:

jupyter notebook llm_summarization_benchmark.ipynb

The notebook will:

1. Load your text files
2. Generate summaries using each model with different prompts
3. Evaluate quality against references (if available)
4. Generate visualizations and reports

4. Review Results

Results are saved to results/ directory:

• benchmark_results_*.csv: Raw data for all API calls
• benchmark_report_*.png: Performance visualization dashboard
• full_report_*.json: Complete analysis including model and prompt breakdowns

Project Structure

abstractor.ai/
├── llm_summarization_benchmark.ipynb  # Main benchmark pipeline
├── requirements.txt                    # Python dependencies
├── .env.example                        # Example environment variables
├── data/
│   ├── texts/                         # Input text files to summarize
│   └── references/                    # Reference summaries for evaluation
└── results/                           # Generated reports and visualizations

Key Metrics Explained

Response Time

Total seconds taken for an API call to complete. Lower is faster.

Cost

Calculated based on token usage and current pricing per 1M tokens. Use this to understand API expenses.

Similarity Score

Semantic similarity between the generated summary and reference summary (0.0 to 1.0). Higher indicates better quality alignment with expected output.

Token Usage

Input tokens consumed by the prompt + text, and output tokens generated. Affects both speed and cost.

Pricing Reference

Prices are based on current rates (as of late 2024) but may change. Check individual provider documentation for the latest:

• OpenAI: GPT-4o ($2.50/$10 per 1M), GPT-4o mini ($0.15/$0.60 per 1M)
• Anthropic: Claude 3.5 Sonnet ($3/$15 per 1M), Claude 3.5 Haiku ($0.80/$4 per 1M)
• Google: Gemini Flash ($0.10/$0.40 per 1M), Gemini Pro ($1.25/$5 per 1M)

Update pricing in the notebook's PRICING dictionary to keep costs accurate.

Tips & Best Practices

Avoiding Rate Limits

• Use delay_between_requests parameter to add delays between API calls
• Start with fewer models and build up
• Consider testing during off-peak hours

Cost Optimization

• Test with cheaper models first (mini/Flash variants)
• Shorter maximum token limits reduce output costs
• Batch similar tests together

Quality Evaluation

• Provide reference summaries for meaningful quality scoring
• Test multiple prompt styles to find the best one for your use case
• Use diverse text samples for comprehensive benchmarking

Reproducibility

• Save your benchmark configurations in git
• Document any custom changes to prompts or settings
• Keep results from different runs for trend analysis

Troubleshooting

"API key not found" error

• Ensure .env file exists with correct API keys
• Verify environment variables are loaded (check output at startup)
• Try restarting the notebook kernel

Rate limit errors

• Increase delay_between_requests parameter
• Reduce number of models tested at once
• Spread benchmarks across multiple time periods

No quality scores

• Ensure reference summaries are in data/references/
• Verify filenames match between texts and references exactly
• Check that similarity_model loads correctly (requires internet for first download)

Response blocked errors (especially Google Gemini)

• Google's safety filters may block some content types
• This is reflected in the error logs for those specific requests
• Filter out blocked responses from final analysis

Requirements

See requirements.txt for full dependency list:

• LLM API clients (openai, anthropic, google-generativeai)
• Data processing (pandas, numpy)
• ML/evaluation (scikit-learn, sentence-transformers)
• Visualization (matplotlib, seaborn)
• Utilities (python-dotenv)
• Jupyter environment

License

MIT License - feel free to use and modify for your benchmarking needs.

Contributing

Contributions welcome! Areas for enhancement:

• Support for additional LLM providers
• Custom evaluation metrics
• Real-time result streaming
• Web dashboard for results visualization
• Scheduling and automated recurring benchmarks

Future Enhancements

• Multi-language support
• Streaming response support
• Real-time cost tracking dashboard
• Automated model comparison reports
• Integration with LLM leaderboards