This project implements a complete evaluation pipeline for Large Language Models running locally using Ollama. It provides a web interface where users can submit prompts, receive model responses, and evaluate output quality using multiple NLP metrics.
The system runs a local LLM instance, sends prompts through a Flask API, and evaluates the generated output using standard NLP metrics. It simulates a real-world AI evaluation workflow used in production environments.
Architecture flow:
Browser → Flask App → Ollama API → Local LLM → Metrics Engine → Results UI
- Local LLM execution (no external API required)
- Interactive web interface for prompt testing
- Automated evaluation metrics
- Fully containerized environment
- Reproducible setup using Docker
- Realistic AI evaluation pipeline
Measures exact token overlap between generated output and reference text. Best suited for strict textual similarity tasks such as translation.
Measures the longest common subsequence between texts. Useful for evaluating content similarity and summarization tasks.
Uses sentence embeddings to compare semantic meaning rather than exact wording. More suitable for modern LLM evaluation.
- Docker
- Docker Compose
Clone repository:
git clone https://github.com/eduardo-data/llm-evaluation-platform.git
cd llm-eval-docker
Build containers:
docker compose build
Start services:
docker compose up -d
Download model (first run only):
docker exec -it ollama ollama pull mistral
Open your browser:
http://localhost:5000
Input:
- Prompt (required)
- Reference answer (optional)
The system returns:
- Model response
- BLEU score
- ROUGE-L score
- Semantic similarity score
This project simulates real-world LLM validation pipelines used for:
- AI response quality analysis
- Prompt engineering testing
- Conversational AI evaluation
- Model benchmarking
- Retrieval-Augmented Generation validation
- Python
- Flask
- Ollama
- Sentence Transformers
- Scikit-learn
- Docker
Modern AI systems require automated evaluation pipelines to validate output quality and consistency. This project demonstrates practical experience with:
- running local LLMs
- integrating model APIs
- evaluating generated outputs
- building testing pipelines
- containerizing applications
- BERTScore metric
- Multi-model comparison
- Prompt A/B testing
- Experiment tracking dashboard
- Evaluation history logging
Luiz Eduardo
Data Scientist
LinkedIn: https://www.linkedin.com/in/luizeduardodatascientist/
GitHub: https://github.com/eduardo-data