English | 中文

vizseek · Multimodal Visual Search Engine

Vision-language aligned vector retrieval — search images with a sentence or an image (ZH/EN queries)

Headline

SigLIP 2 maps ZH/EN/image into one shared vector space — one index serves bilingual retrieval. On a COCO 2017 val subset: EN caption→image R@1 = 0.761 / R@5 = 0.942 / R@10 = 0.979; image→image mean nearest-neighbor similarity 0.796; both Faiss flat and Qdrant hit R@10 = 1.0 (matching the exact baseline).

Retrieval axis	R@1	R@5	R@10	Notes
EN text→image	0.761	0.942	0.979	2502 captions, GT image ranks first
ZH text→image	0.353	0.824	0.941	17 queries (one-off early eval, not archived in reports/ — indicative only)
image→image	—	—	—	mean nearest-neighbor cosine 0.796 (vector quality signal)

Source: reports/metrics.json + reports/recall_by_k.csv (--quick 500-image subset, real SigLIP2 encodings). Reproduce with python run_all.py --quick (EN eval queries all ~2502 captions — EVAL_QUERY_SAMPLE=0 means no truncation). The Chinese sample is only 17, so it is indicative rather than statistically robust; those ZH numbers come from an early one-off manual eval and are not archived in reports/ (quick mode skips the ZH axis, so the audit script cannot verify them) — re-run a full python -m vizseek.evaluate with COCO-CN available to re-measure. Full 5K gallery numbers will replace these after full encoding.

---

Overview

vizseek is an end-to-end multimodal visual retrieval system. Enter a Chinese or English description and the system recalls the most semantically relevant images from a gallery; image-to-image search is also supported. The backbone is SigLIP 2, a multilingual vision-language encoder that maps images, English captions, and Chinese queries into a single shared vector space, so one index serves both languages.

This is a vision + multimodal retrieval project focused on cross-modal vector retrieval and vector-database engineering.

Highlights

• Vision-language retrieval: built on the SigLIP 2 multilingual encoder; English retrieval reaches R@1=0.76. Chinese queries reuse the same model and index (no separate Chinese-CLIP stack) — R@10=0.94 but R@1=0.35, so Chinese works but is markedly less precise than English.
• Dual vector-DB fair comparison: identical embeddings indexed in both Faiss (exact flat; IVF ANN pending full-scale data) and Qdrant (local HNSW), benchmarked across query latency (P50/P95/QPS) and retrieval quality (Recall@10 vs exact baseline).
• Standard COCO retrieval protocol: COCO 2017 val (5K images + 25K captions) + COCO-CN (Chinese axis), reporting Recall@1/5/10.
• Encode-time self-check: guards against transformers issue #43994 (the SigLIP similarity anomaly) — encode.py probes a known image-text pair before the full run and aborts if the normalized cosine falls below threshold, so broken encodings never ship silently.
• Production-grade engineering: CI (lint + test, CPU-torch compatible), dual entry points (Makefile + Windows-friendly run_all.py), --quick/--limit smoke modes, Streamlit UI.

Architecture

COCO 2017 val images (5K) + captions (25K)
        │
        ▼
   preprocess ──► image_meta.json + text_meta.json + (zh queries)
        │
        ▼
      encode ──► SigLIP 2 (google/siglip2-base-patch16-224, 768-d, fp16)
        │         ├── self-check: known image-text pair cosine > 0.10
        │         └── L2 normalize → image_embeddings.npy / text_embeddings.npy
        │
        ▼
   build_index ─► Faiss flat (exact baseline) + Faiss IVF (ANN) + Qdrant local (HNSW)
        │
        ▼
    evaluate  ─► Recall@1/5/10 (text→image, EN + ZH) + image coherence
        │
        ▼
   benchmark  ─► Faiss vs Qdrant: P50/P95 latency, QPS, disk, Recall@10
        │
        ▼
   dashboard ─► Streamlit: text-search / image-search / metrics / vector-DB comparison

Quick Start

# 1. Install (verified on RTX 4060 8GB; CPU also works but encodes slowly)
python -m venv .venv
# Linux / macOS: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install -r requirements-cuda.txt   # CUDA 12.1
# python -m pip install -r requirements-cpu.txt  # CPU-only
python -m pip install -e .

# 2. Full pipeline (download → preprocess → encode → index → evaluate → benchmark)
python run_all.py

# 3. Quick smoke run (500-image subset, end to end)
python run_all.py --quick

# 4. Launch the interactive dashboard
make dashboard

Step by step: make download / make encode / make index / make evaluate / make benchmark

Retrieval Results (COCO 2017 val · 500-image subset · SigLIP2-base)

Numbers below are from --quick mode (500-image subset) with real SigLIP2 encodings. Full 5K gallery numbers will replace these after full encoding. Reproduce: python run_all.py --quick (the EN column evaluates all ~2502 captions, no truncation; for the ZH column's provenance see "On Chinese retrieval" below).

Retrieval quality

Metric	English (2502 captions)	Chinese (17 queries)	Notes
Recall@1	0.761	0.353	fraction of queries whose GT image ranks first
Recall@5	0.942	0.824	top-5 hit
Recall@10	0.979	0.941	top-10 hit
image coherence	0.796	—	mean cosine to nearest neighbor (vector quality signal)

Vector-DB comparison (same 500 vectors)

Engine	P50 latency	QPS	Recall@10	Disk
Faiss flat	0.033 ms	27,747	1.0000	1.5 MB
Qdrant local	2.083 ms	453	1.0000	4.0 MB

Faiss flat is ~60× faster at this small scale (pure in-memory brute force); both report Recall@10 = 1.0 (identical to the exact baseline). Qdrant's value is metadata filtering, persistence, and horizontal scaling — features that matter at full scale and in production.

Note: Faiss IVF (approximate index) is implemented but needs ≥1024 images to be meaningful (auto-skipped on the 500-image subset, which uses the exact flat index). IVF enters the comparison once the full 5K gallery is encoded.

On Chinese retrieval

SigLIP 2's multilingual capability makes Chinese queries usable (R@10=0.94 means the relevant image is usually in the top-10), but exact ranking is clearly weaker than English (R@1=0.35 vs 0.76). This matches SigLIP2-base's profile — Chinese is supported but not its strong suit. The Chinese sample is only 17 (COCO-CN annotates 2014 images, which overlap little with the 2017 gallery), so these numbers are indicative, not statistically robust.

Provenance: these ZH numbers come from an early one-off manual eval (when the COCO-CN cache was available) and were never written to reports/metrics.json / recall_by_k.csv; quick-mode evaluation skips the ZH axis, so make audit currently verifies the EN column only (once an eval run records text_to_image_zh, the audit checks the ZH column too). To re-measure: run a full python -m vizseek.evaluate with the COCO-CN cache present (ZH queries are encoded on the fly and recorded in reports).

Key Design Decisions

Why SigLIP 2 over vanilla CLIP?

Vanilla CLIP has poor Chinese support, forcing a dual-model setup (Chinese-CLIP + OpenCLIP). SigLIP 2 is natively multilingual — one model aligns ZH/EN/image into one space, which is the technical basis for "one index serves bilingual retrieval."

The SigLIP similarity pitfall (issue #43994)

get_image_features / get_text_features return unnormalized vectors. Computing inner products directly or misusing softmax yields meaningless similarities (this is the community-reported #43994). This project:

1. Force L2-normalizes every vector after encoding (torch.nn.functional.normalize)
2. Inner product on normalized vectors == cosine similarity
3. Self-checks a known image-text pair before the full run: matched cosine must be > 0.10 and exceed unmatched pairs, else abort

Tech stack / data / metrics detail

Tech Stack

Layer	Tool	Notes
Encoder	SigLIP 2 (google/siglip2-base-patch16-224)	Multilingual, 768-d, encodes ZH/EN/image in one model
DL	PyTorch 2.5 + transformers ≥4.50	fp16, batched
Vector DB A	Faiss-cpu	IndexFlatIP (exact) + IndexIVFFlat (ANN); inner product on L2-normalized == cosine
Vector DB B	Qdrant (local mode)	HNSW + Distance.COSINE; metadata payloads
Data	COCO 2017 val + COCO-CN	5K images / 25K EN captions / ZH subset
UI	Streamlit + plotly	text/image search, metrics, benchmark viz

Data

Property	Value
Gallery	COCO 2017 val (5,000 images)
English captions	COCO Captions (~25,014, ~5 per image)
Chinese queries	COCO-CN (best-effort; raw text file from the official GitHub repo; degrades gracefully if absent)
Protocol	text-to-image Recall@1/5/10

Metrics

Task	Metric	Notes
text→image (EN)	Recall@1/5/10	EN caption retrieves its GT image
text→image (ZH)	Recall@1/5/10	ZH query retrieval (cross-lingual axis, depends on COCO-CN)
Vector-DB	P50/P95 latency, QPS, Recall@10 vs exact	fair comparison on identical vectors

Project structure / quality assurance

Project Structure

vizseek/
├── src/vizseek/
│   ├── config.py             # paths + model name + index params (central config)
│   ├── download_data.py      # COCO download (integrity check + --quick subsample)
│   ├── preprocess.py         # manifest build + query-set gen + COCO-CN defensive probe
│   ├── encode.py             # SigLIP 2 encoding (self-check + L2 normalize + --limit)
│   ├── build_index.py        # Faiss flat/IVF + Qdrant local dual index
│   ├── evaluate.py           # Recall@K (EN + ZH) + image coherence
│   ├── benchmark_vector_db.py# Faiss vs Qdrant four-axis comparison
│   └── audit_consistency.py  # README metrics vs reports/*.json audit
├── dashboard/app.py          # Streamlit interactive retrieval UI
├── tests/                    # config + logic tests (CI needs no model/data)
├── run_all.py                # Windows-compatible pipeline runner
└── Makefile                  # standard dev entry point

Quality Assurance

Level	Trigger	Command	Purpose
L1	each commit	pre-commit (ruff)	format, basic lint
L2	each push/PR	GitHub Actions CI	lint + test (CPU torch)
L3	manual	make verify	lint + format-check + typecheck + test + audit
L4	at encode time	encode.py self-check	model-load correctness (guards #43994)

License

MIT