VLM Playground

All-in-one desktop tool for evaluating and benchmarking Vision Language Models.

English | 中文

Core Features

Playground - Model Chat Testing

• Quick Mode: Single-model real-time chat with multi-turn conversation and image upload support (up to 20 images)
• Free Comparison Mode: Test 2-3 models simultaneously to compare performance across different configurations
• Flexible Configuration: Set System Prompt and User Prompt templates independently
• Real-time Feedback: Displays response time and model name

Datasets

• Multi-format Import:
  • Plain image folders
  • LLaVA format (JSON + images)
  • ShareGPT format
  • YOLO format (images + labels)
  • COCO format (annotations JSON)
• Smart Sampling: Random sampling by count or percentage
• Easy Management: View dataset details, statistics, and quick previews

Evaluators

• Reusable Templates: Create evaluation rules and prompt templates
• Version Control: Multiple versions per evaluator for easy iterative refinement
• Flexible Composition: Dual-layer configuration with System Prompt + User Prompt
• One-click Reuse: Evaluators can be shared across multiple experiments

Batch Experiments

• Automated Execution: Generate results in bulk based on datasets + evaluators + model configurations
• Real-time Progress: Seamless progress updates without page refresh
• Flexible Control: Cancel running experiments or re-run failed/completed ones
• Smart Recovery: Reuses existing configuration and image lists when re-running

Human Review

• Dual Mode Toggle:
  • Normal Mode: View experiment results and review with buttons
  • Annotation Mode: Quick annotation via dropdown (Unreviewed / Pass / Fail)
• Smart Filtering: Filter by review status (All / Unreviewed only / Reviewed only)
• Statistics Panel: Real-time display of totals, reviewed count, passed, failed, and pass rate

Multi-format Export

• CSV: Tabular data, ideal for viewing and analyzing in Excel
• JSON: Fully structured data with all fields preserved
• LLaVA: Fine-tuning format with image paths and conversation data
• ShareGPT: Conversation format supporting system/user/assistant roles
• Alpaca: Instruction format with instruction-output structure
• Flexible Filtering: Optionally export only reviewed results or only passed results

Model Configuration

• Multi-model Support: Configure multiple LLMs simultaneously (OpenAI, Azure, custom endpoints, etc.)
• Detailed Settings: API Key, Base URL, model name, temperature, max tokens
• Default Model: Set a frequently used model as default for automatic selection
• Secure Storage: API Keys are encrypted and stored in the local database

Typical Workflow

1. Configure models
   |
2. Import datasets
   |
3. Create evaluators (configure prompt templates)
   |
4. Create and run experiments
   |
5. Human review and annotation of results
   |
6. Export data for model fine-tuning

Tech Stack

Frontend

• Electron 28.x - Cross-platform desktop application framework
• React 18.x + TypeScript - Type-safe modern UI framework
• Ant Design 5.x - Enterprise-grade UI component library
• Vite 5.x - Fast development and build tool
• React Router 6.x - Client-side routing
• Redux Toolkit - State management

Backend

• Prisma 6.x - Modern TypeScript ORM
• SQLite - Local database managed by Prisma
• Node.js IPC - Communication between main and renderer processes
• axios - HTTP client for calling LLM APIs
• crypto-js - Automatic API Key encryption (via Prisma middleware)
• fs-extra - Enhanced file system operations

Visualization

• ECharts 5.x - Data statistics and charting
• echarts-for-react - React integration

Quick Start

Install Dependencies

npm install

Development Mode

npm run dev

This will start both:

1. The Vite dev server (React hot module replacement)
2. The Electron application window

Build for Production

# Build React and Electron code
npm run build

# Package into an executable
npm run dist

Packaged applications are output to the release/ directory:

• macOS: .dmg installer
• Windows: .exe installer
• Linux: .AppImage executable

Project Structure

vlm-annotation-platform/
├── electron/                # Electron main process
│   ├── main.ts             # Main process entry point
│   ├── preload.ts          # Secure IPC bridge
│   ├── db/
│   │   └── database.ts     # SQLite database management
│   ├── ipc/
│   │   ├── llmConfig.ts    # LLM configuration management
│   │   ├── dataset.ts      # Dataset import
│   │   ├── evaluator.ts    # Evaluator management
│   │   ├── experiment.ts   # Experiment execution
│   │   ├── playground.ts   # Playground chat
│   │   ├── export.ts       # Data export
│   │   └── batch.ts        # Batch processing tasks
│   └── services/
│       ├── llm.ts          # LLM API calls
│       └── imageProcessor.ts # Image processing
├── src/                    # React frontend
│   ├── main.tsx           # React entry point
│   ├── App.tsx            # Main application layout
│   ├── pages/
│   │   ├── WelcomePage.tsx        # Home page
│   │   ├── PlaygroundPage.tsx     # Playground
│   │   ├── DatasetViewer.tsx      # Dataset details
│   │   └── ExperimentViewer.tsx   # Experiment details
│   ├── components/
│   │   ├── DatasetListView.tsx    # Dataset list
│   │   ├── EvaluatorListView.tsx  # Evaluator list
│   │   ├── ExperimentListView.tsx # Experiment list
│   │   ├── PlaygroundChat.tsx     # Chat component
│   │   ├── LLMConfigDialog.tsx    # Model configuration
│   │   ├── WorkflowDiagram.tsx    # Workflow diagram
│   │   └── ...
│   └── types/
│       └── index.ts       # TypeScript type definitions
├── build/
│   └── icons/             # Application icons
└── package.json

Data Storage

Location: ~/.vlm-toolkit/

~/.vlm-toolkit/
├── app.db              # SQLite database
├── datasets/           # Dataset files
├── cache/             # Thumbnail cache
└── exports/           # Exported files

Database Schema

• llm_configs: LLM configurations (API Key, Base URL, model parameters, etc.)
• datasets: Dataset metadata (name, type, path, statistics)
• images: Image information (filename, path, parent dataset)
• evaluators: Evaluators (name, version, prompt templates)
• experiments: Experiments (linked dataset, evaluator, model, progress status)
• experiment_results: Experiment results (question, answer, human review status)
• batch_jobs: Batch processing jobs (experiment ID, image list, progress)

User Guide

1. First-time Setup

1. Configure Your First Model

   • Click "Model Configuration" in the left sidebar or the "Configure Your First Model" button on the home page
   • Select a provider (OpenAI, Azure OpenAI, Custom)
   • Enter the API Key and other settings
   • Click "Test Connection" to verify the configuration

2. Import a Dataset

   • Click "Datasets" then "Import Dataset"
   • Select the dataset type and file/folder
   • The system will automatically detect and import it

3. Create an Evaluator

   • Click "Evaluators" then "Create Evaluator"
   • Set the System Prompt (optional) and User Prompt template
   • Once saved, it can be reused across multiple experiments

2. Running Experiments

1. Create an Experiment

   • Click "Experiments" then "Create Experiment"
   • Select a dataset, evaluator, and model configuration
   • Optionally configure a sampling strategy (all / by count / by percentage)

2. Start the Run

   • Click "Start Experiment" in the experiment list
   • Monitor real-time progress updates (no page refresh needed)
   • Cancel at any time

3. View Results

   • Click "View" to open the experiment details
   • Review the question and answer for each image
   • Check statistics (total count, reviewed count, pass rate, etc.)

3. Human Review

1. Switch to Annotation Mode

   • On the experiment details page, toggle to "Annotation Mode"
   • The review column changes to dropdown selectors

2. Quick Annotation

   • Use the dropdown to mark items: Unreviewed / Pass / Fail
   • Use the filter to quickly locate unreviewed items

3. Export Data

   • Click "Export Results"
   • Choose the export format and filter criteria
   • Select a save location

4. Using the Playground

Quick Mode (single model):

1. Select a model configuration
2. Set System/User Prompt (optional)
3. Upload images or type text
4. Send and view results

Free Comparison Mode (2-3 models):

1. Choose different models or configure different prompts for each group
2. Enter the same question
3. Click "Run" to test all models simultaneously
4. Compare outputs and response times across models

Application Icons

The app supports custom icons. See build/icons/README.md for details.

Recommended:

• Purple gradient theme (#667eea to #764ba2)
• Data analysis / evaluation-themed icon design
• Clean, modern design style

Performance

• Local Execution: All data is stored locally to protect privacy
• Real-time Updates: Event-driven via IPC for seamless progress updates
• Batch Processing: Supports large-scale dataset evaluation
• Smart Caching: Image path caching for faster loading
• Concurrent Calls: Playground comparison mode supports concurrent API calls

Security

• API Key Encryption: Stored encrypted using crypto-js
• Local Storage: Data is never uploaded to the cloud
• Sandbox Isolation: Electron process isolation for protection
• Context Isolation: Secure communication between renderer and main processes

FAQ

Q: Dataset import failed?

A: Verify that the dataset format meets the requirements and check the console for error messages. Supported formats: plain image folders, LLaVA, ShareGPT, YOLO, COCO.

Q: Model API call failed?

A:

1. Verify the API Key is correct
2. Check the network connection
3. Confirm the Base URL is configured correctly
4. Review the error message in the response

Q: Experiment progress not updating?

A: Progress is now driven by real-time events. If the issue persists, check whether the experiment status is "Running".

Q: Exported data format is wrong?

A: Different formats serve different purposes:

• CSV: Manual review
• JSON: Programmatic processing
• LLaVA/ShareGPT/Alpaca: Model fine-tuning

Q: How do I re-run an experiment?

A: Experiments in any status (pending, cancelled, failed, completed) can be re-run. The system will reuse the existing configuration and image list.

Development

Prisma ORM Usage

// Import Prisma Client
import { getPrismaClient } from '../db/prisma';
const prisma = getPrismaClient();

// Query
const records = await prisma.table.findMany({
  where: { id },
  include: { relation: true }
});

// Create
const record = await prisma.table.create({
  data: { ...data }
});

// Update
await prisma.table.update({
  where: { id },
  data: { ...updates }
});

// Delete
await prisma.table.delete({
  where: { id }
});

// Transaction
await prisma.$transaction([
  prisma.table1.update({...}),
  prisma.table2.create({...})
]);

Prisma CLI Commands

# Generate Prisma Client
npm run prisma:generate

# Push schema to database (development)
npm run db:push

# Open Prisma Studio (database GUI)
npm run db:studio

# Create a migration (production)
npx prisma migrate dev --name migration_name

IPC Communication

// Frontend call
const result = await window.electronAPI.experiment.run(experimentId);

// Backend handler
ipcMain.handle('experiment:run', async (_, experimentId) => {
  // Processing logic
  return result;
});

// Event listener
const unsubscribe = window.electronAPI.batch.onProgress((data) => {
  console.log(data);
});

Contributing

Issues and Pull Requests are welcome!

License

MIT License

---

VLM Playground - Making Vision Language Model evaluation simpler and more efficient