UNION: A Lightweight Target Representation for Efficient Image-Guided Retrieval with Optional Textual Queries

Abstract

Image-Guided Retrieval with Optional Text (IGROT) is a general retrieval setting where a query consists of an anchor image, with or without accompanying text, aiming to retrieve semantically relevant target images. This formulation unifies two major tasks: Composed Image Retrieval (CIR) and Sketch-Based Image Retrieval (SBIR). In this work, we address IGROT under low-data supervision by introducing UNION, a lightweight and generalizable target representation that fuses the image embedding with a null-text prompt. Unlike traditional approaches that rely on frozen target features, UNION enhances semantic alignment with multimodal queries while requiring no architectural modifications to pretrained vision-language models. With only 5k training samples—from LlavaSCo for CIR and Training-Sketchy for SBIR—our method achieves competitive results across benchmarks, including CIRCO mAP@50 of 38.5 and Sketchy mAP@200 of 82.7, surpassing many heavily supervised baselines. This demonstrates the robustness and efficiency of UNION in bridging vision and language across diverse query types.

Setup

micromamba create -n union python=3.9
micromamba activate union
pip install -r requirements_blip.txt 
pip install -r requirements.txt

Datasets:

The dataset folder is here.

• LlavaSCo: Please refer to this link
  LaSCo dataset can be downloaded here.
• Training-Sketchy: Please refer to this link.
  Sketchy dataset can be downloaded at their website or Google Drive.

Model Zoo

Pretrained Model	Link
CLIP ViT-B/32	here
CLIP ViT-L/14	here
BLIP-B (COCO)	here

Inference

We prepare two different files for inference stage. You can train and inference, if you run this:

bash scripts/inference.sh

or only train the model:

bash scripts/train.sh

Note: If you want to change the target image feature type (original, sum, union), please change it in the script files, then one more in utils.py.

Due to the weight conversion, the performance may be slightly different:

In Zero-Shot Composed Image Retrieval: FashionIQ, CIRRand CIRCO

Method	Backbone	# Params	# Triplets	FashionIQ (R)		CIRR (R)		CIRCO (mAP)
				@10	@50	@10	@50	@10	@50
Pic2Word	CLIP-L	429M	3M	24.7	43.7	65.3	87.8	9.5	11.3
i-SEARLE	CLIP-L	442M	205K	29.2	49.5	66.7	88.8	13.6	16.3
CIReVL	CLIP-L	12.5B	-	28.6	48.6	64.9	86.3	19.1	20.9
MLLM-I2W	CLIP-L	-	3M	30.3	50.1	68.4	92.4	-	-
PLI	CLIP-L	428M	695K	35.4	57.4	69.3	89.6	14.2	16.4
LinCIR	CLIP-L	-	5.5M	26.4	46.6	66.9	88.2	13.9	16.2
MagicLens	CLIP-L	465M	36.5M	30.7	52.5	74.4	92.6	30.8	34.4
CoLLM	BLIP-B	-	3.4M	34.6	56.0	78.6	94.2	20.4	23.1
TransAgg	BLIP-B	235M	32K	34.4	55.1	77.9	93.4	32.2	36.2
TransAgg + UNION	BLIP-B	235M	5K	31.9	51.5	77.6	92.9	34.5	38.5

In Zero-Shot Sketch-Based Image Retrieval: Sketchy, TUBerlin and QuickDraw

Method	Backbone	# Pairs	Sketchy		TU-Berlin		QuickDraw
			mAP@200	Prec@200	mAP	Prec@100	mAP	Prec@200
DCDL	CLIP-B	57K/15K/236K	72.6	76.9	63.4	74.1	33.6	29.6
CAT	CLIP-B	57K/15K/236K	71.3	72.5	63.1	72.2	20.2	38.8
IVT	ViT-B	57K/15K/236K	61.5	69.4	55.7	62.9	32.4	16.2
ZSE-SBIR	ViT-L	57K/15K/236K	52.5	62.4	54.2	65.7	14.5	21.6
MagicLens	CLIP-L	36.7M	68.2	75.8	62.9	73.1	15.1	20.4
TransAgg	CLIP-L	5K	79.6	75.8	45.4	68.2	30.1	43.5
TransAgg + UNION	CLIP-L	5K	82.7	79.9	51.0	69.8	33.4	41.5

Citing this work

Add citation details here, usually a pastable BibTeX snippet:

@inproceedings{le2025union,
  title={UNION: A Lightweight Target Representation for Efficient Image-Guided Retrieval with Optional Textual Queries},
  author={Hoang-Bao, Le and Allie, Tran and Binh, T. Nguyen and Liting, Zhou and Cathal, Gurrin},
  booktitle={2025 IEEE International Conference on Data Mining Workshops (ICDMW)},
  pages={1471-1479},
  year={2025},
  organization={IEEE}
}

Acknowledgement

We extend our gratitude to the open-source efforts of TransAgg.