A Comparative Benchmarking of Qdrant and Milvus Vector Databases for High-Dimensional Similarity Search
NTUA ECE Semester Project – DBLab, Winter 2025
VECTOR-ADIS is a comprehensive benchmarking framework for evaluating vector database performance in single-node deployments. We compare Qdrant (Rust-based, monolithic) and Milvus (C++/Go, microservices) under identical hardware conditions, measuring:
- Throughput (requests per second)
- Latency (mean, p95, p99)
- Accuracy (Precision@k)
- Resource consumption (CPU, memory, I/O)
The framework supports both in-memory and memory-constrained (on-disk) evaluation modes using a three-regime methodology:
| Regime | Memory Pressure | Dataset Example | RAM Limit |
|---|---|---|---|
| A (In-Memory) | Data fits comfortably | dbpedia-100K (614 MB) | 2 GB |
| B (Borderline) | Near memory limit | laion-small-clip (~2 GB) | 2 GB |
| C (Memory-Exceeding) | Data exceeds RAM | deep-image-96 (3.8 GB) | 4 GB |
Standard Benchmarks (In-Memory):
| Metric | Qdrant | Milvus | Δ |
|---|---|---|---|
| Mean RPS | 600.73 | 395.10 | +52% |
| Mean Latency | 12.3 ms | 34.1 ms | -64% |
| P95 Latency | 28.5 ms | 89.2 ms | -68% |
| Precision@10 | 96.67% | 96.67% | = |
Memory-Constrained Benchmarks:
| Regime | Winner | Key Result |
|---|---|---|
| A (In-Memory) | Qdrant | 1.5× higher throughput |
| B (Borderline) | Qdrant | 4.6× throughput, 14× lower p95 latency |
| C (Memory-Exceeding) | Milvus | 2.8× throughput (IVF_SQ8 vs mmap HNSW) |
Important: Regime C reveals a significant finding reversal—under extreme memory pressure, Milvus IVF_SQ8 (407.8 RPS) outperforms Qdrant mmap (144.1 RPS) by 2.8×, while Qdrant dominates in Regimes A–B.
Read the full thesis for detailed methodology, results, and analysis
VECTOR-ADIS/
├── benchmark/ # Core benchmark utilities
│ ├── config_read.py # Configuration loader
│ ├── convert.py # Result consolidation
│ └── dataset.py # Dataset download/management
├── dataset_reader/ # Format adapters (HDF5, JSON, sparse)
├── datasets/ # Dataset storage + registry (datasets.json)
├── engine/
│ ├── base_client/ # Abstract interfaces
│ ├── clients/ # Qdrant & Milvus implementations
│ └── servers/ # Docker Compose configurations
├── experiments/
│ └── configurations/ # Engine parameter manifests (JSON)
│ ├── qdrant-single-node.json
│ ├── milvus-single-node.json
│ ├── qdrant-on-disk.json
│ └── milvus-on-disk.json
├── scripts/ # Shell scripts & plotting utilities
├── results/ # Raw JSON benchmark outputs
├── consolidated/ # Merged results (results.json)
├── plots/ # Generated visualizations
├── thesis/ # IEEE conference paper (LaTeX)
├── docs/ # Documentation & reports
├── run.py # Main CLI entry point
├── Makefile # Build automation
└── pyproject.toml # Python dependencies (Poetry)
- Python 3.11+
- Poetry (dependency management)
- Docker (container runtime)
- LaTeX distribution (for thesis compilation, optional)
# Clone repository
git clone https://github.com/your-org/vector-adis.git
cd vector-adis
# Install Python dependencies
make install
# Start vector database servers
cd engine/servers/qdrant-single-node && docker compose up -d && cd -
cd engine/servers/milvus-single-node && docker compose up -d && cd -# Run individual engines
make qdrant # Full Qdrant benchmark suite
make milvus # Full Milvus benchmark suite
make run-all # Both engines sequentially
# Quick smoke tests (glove-100, 2 min timeout)
make benchmark-quick-qdrant
make benchmark-quick-milvusmake qdrant-on-disk # Qdrant with mmap storage
make milvus-on-disk # Milvus with disk indexing
make run-all-on-disk # Both engines on-diskNote on DISKANN: Milvus standalone mode does not support DISKANN (requires distributed deployment). Therefore, memory-constrained comparisons use Qdrant mmap-backed HNSW vs Milvus IVF_SQ8. See memory-constrained-evaluation.md for details.
# Custom dataset selection
python run.py --engines qdrant --datasets glove-100-angular,deep-image-96-angular
# Adjust timeout and host
python run.py --engines milvus --host 192.168.1.100 --timeout 7200
# Skip upload phase (reuse existing data)
python run.py --engines qdrant --skip-uploadmake convert # Consolidate results → consolidated/results.json
make plot # Generate benchmark plots → plots/
make plot-thesis # Generate thesis-safe plots (publication quality)
make monitor # Real-time Docker resource monitoringThe research paper is formatted as an IEEE conference paper using LaTeX.
make thesis # Build PDF → thesis/main.pdf
make thesis-watch # Auto-rebuild on file changes
make thesis-clean # Remove LaTeX build artifactsThe benchmark suite includes 11 datasets spanning various dimensions and characteristics:
| Dataset | Dimensions | Vectors | Metric |
|---|---|---|---|
| glove-100-angular | 100 | 1.2M | Cosine |
| deep-image-96-angular | 96 | 10M | Cosine |
| arxiv-titles-384-angular | 384 | 2.2M | Cosine |
| cohere-wiki-100k | 768 | 100K | Cosine |
| laion-small-clip | 512 | 100K | Cosine |
| dbpedia-openai-100K-1536 | 1536 | 100K | Cosine |
| dbpedia-openai-1M-1536 | 1536 | 1M | Cosine |
| h-and-m-2048-angular | 2048 | 105K | Cosine |
| msmarco-sparse | Sparse | 8.8M | Dot |
| random-100 | 100 | 100K | Cosine |
| random-match-keyword-100 | 100 | 100K | Cosine |
Datasets are automatically downloaded on first use. Registry: datasets/datasets.json
make clean # Stop and remove containers
make reset # Full environment reset
make status # Show container status
make logs # Tail Qdrant logs
make sync-results HOST=server # Rsync results from remote
make archive # Create results tarball
make help # Show all available targetsStudents:
- Stavriana Zannetou (el21608@mail.ntua.gr)
- Sevastianos Ioannidis (el19859@mail.ntua.gr)
- Marios Christos Syrmakezis (el15182@mail.ntua.gr)
Supervisor:
- Professor Dimitrios Tsoumakos, Director of DBLab
School of Electrical and Computer Engineering, National Technical University of Athens
This project is licensed under the MIT License. See LICENSE for details.