RIPRAG

RIPRAG is a robust defense framework against knowledge corruption attacks on Retrieval-Augmented Generation (RAG) systems. Built upon the PoisonedRAG attack framework, RIPRAG integrates multiple defense mechanisms including query rewriting, RobustRAG keyword-based defense, RAGuard perplexity-based filtering, and hybrid retrieval fusion to mitigate adversarial corpus poisoning.

🔍 Quick Usage

📃 Setup environment

conda create -n RIPRAG python=3.10

conda activate RIPRAG

git submodule init
git submodule update
pip install -r requirements.txt
python -m spacy download en_core_web_sm

💽 Dataset (optional, suggested)

When running our code, the datasets will be automatically downloaded and saved in datasets. You could also run this line to manually download datasets.

python prepare_dataset.py

🔑 Set API key

Enter your api key in the model_configs folder. For local models (VLLM provider), configure the appropriate URL and port.

🚀 Start Required Services

RIPRAG requires several backend services to be running:

1. vLLM servers for LLM inference and embeddings:

# LLM for attack generation (RIP/iter methods) — port 3000
vllm serve unsloth/Qwen3-8B-unsloth-bnb-4bit --enable-lora --lora-modules riprag=$RIPRAG_PATH --port 3000 

# LLM for RAG answering — port 2004 (or as configured in model_configs/)
vllm serve Qwen/Qwen3-8B --port 2004

# Embedding models
vllm serve Qwen/Qwen3-Embedding-0.6B --port 2000
vllm serve BAAI/bge-m3 --port 2001

# Rerank models
vllm serve --model-id BAAI/bge-reranker-v2-m3 --port 2002

3. Milvus vector database on http://localhost:19530:

# Using Docker
docker run -d --name milvus-standalone -p 19530:19530 -p 9091:9091 milvusdb/milvus:latest milvus run standalone

📝 Build Embeddings and Indexes

Before running experiments with the database-backed pipeline (main_db.py), generate embeddings for the corpus and queries, then create indexes:

# Generate Milvus embeddings for all datasets
python gen_emb.py --dataset nq
python gen_emb.py --dataset hotpotqa
python gen_emb.py --dataset msmarco

# Create IVF indexes
python create_idx.py

📝 Reproduce experiments

Attack Experiments (with RIP on-the-fly adversarial generation)

# Run RIP attack with Milvus-backed pipeline (attack_db.py)
python main_db.py --eval_dataset nq --top_k 10 --attack_method RIP --adv_per_query 5 --model_name glm_9b

Defense Experiments

RIPRAG supports multiple defense/query methods via --query_method:

Method	Description
direct	Standard retrieval with the original query (no defense)
rewrite	LLM rewrites the query before retrieval to mitigate adversarial influence
answer	LLM generates a detailed answer, used as the retrieval query
RobustRAG	Keyword-based defense: extracts keywords from per-context responses, filters by frequency, and generates answer from keyword hints
RAGuard	Perplexity-based filtering: removes retrieved passages with anomalous perplexity patterns

# Direct query (baseline attack)
python main_db.py --eval_dataset nq --top_k 10 --attack_method RIP --adv_per_query 5 --query_method direct

# Query rewriting defense
python main_db.py --eval_dataset nq --top_k 10 --attack_method RIP --adv_per_query 5 --query_method rewrite

# Answer-based retrieval defense
python main_db.py --eval_dataset nq --top_k 10 --attack_method RIP --adv_per_query 5 --query_method answer

# RobustRAG keyword-based defense
python main_db.py --eval_dataset nq --top_k 10 --attack_method RIP --adv_per_query 5 --query_method RobustRAG

# RAGuard perplexity-based defense
python main_db.py --eval_dataset nq --top_k 10 --attack_method RIP --adv_per_query 5 --query_method RAGuard

Using run_RIP.py (Contriever-based pipeline)

# RIPRAG with Contriever retrieval and RIP attack
python run_RIP.py --model_name glm_9b --top_k 10 --attack_method RIP --adv_per_query 5 --query_method direct

# RIPRAG with specific defense
python run_RIP.py --top_k 10 --attack_method RIP --adv_per_query 1 --RIPRAG=10-1-rrf --model_name deepseek_v3.2

The --RIPRAG argument expects the RIPRAG LoRA adapter name set in vllm.

Black-box Attack Baseline (pre-generated adversarial texts)

# LM_targeted attack (uses pre-generated adversarial texts)
python run.py --model_name glm_9b --top_k 10 --attack_method LM_targeted --adv_per_query 1

White-box Attack Baseline (HotFlip)

python run.py --model_name glm_9b --top_k 10 --attack_method hotflip --adv_per_query 1

🤖 RL Training for Adversarial Text Generation

RIPRAG uses trl to train a LoRA adapter on unsloth/Qwen3-8B that generates effective adversarial texts:

python rl.py

The RL pipeline uses two reward functions:

• Similarity reward (similarity_reward): BM25 similarity between the query and generated adversarial text, gated on containing the incorrect answer.
• Attack reward (attack_reward): Full attack evaluation as a reward — generates text, runs the full attack pipeline, and returns the ASR binary score.

🏗 Architecture

Defense Mechanisms

RIPRAG implements five query/defense methods against knowledge corruption attacks:

1. Direct: Baseline — no defense applied. The original query is used directly for retrieval.
2. Rewrite: The LLM rewrites the query into an alternative form that preserves semantics but may break adversarial alignment. Cached in cache/{model_name}_rewrite_cache.json.
3. Answer: Instead of the query, a detailed LLM-generated answer is used as the retrieval query. Cached in cache/{model_name}_answer_cache.json.
4. RobustRAG: A keyword-based defense that (a) generates per-context responses, (b) extracts keywords from each response, (c) filters keywords by frequency across responses, and (d) generates the final answer using only high-frequency keyword hints. Adapted from RobustRAG.
5. RAGuard: A perplexity-based defense that computes log-perplexity for the first and second halves of each retrieved passage, then filters out passages with anomalous perplexity patterns (both inter-half difference pd_r and max perplexity pm_r). Cached in cache/internlm2.5_7b_pd_r_cache.json and cache/internlm2.5_7b_pm_r_cache.json.

Retrieval Architecture

RIPRAG supports two retrieval backends:

Backend	Script	Retriever	Storage
Contriever (JSON)	attack.py	Single-encoder Contriever	results/beir_results/*.json
Milvus Hybrid	attack_db.py	Dual-encoder (Qwen3 + BGE-M3) + RRF fusion + Re-rank	Milvus vector DB

The Milvus pipeline provides stronger defense through:

• Dual embeddings: Qwen3-Embedding-0.6B and BGE-M3 embeddings for each passage/query.
• Hybrid search: Both embeddings are used for retrieval, combined via Reciprocal Rank Fusion (RRF).
• Re-ranking: BGE-Reranker-v2-M3 re-ranks the fused results for final top-k selection.

Model Configs

JSON configuration files in model_configs/ specify the LLM provider, model name, API keys, and inference parameters. Key configs:

Config	Provider	Model
qwen3_8b_config.json	vllm	Qwen/Qwen3-8B
deepseek_v3.2_config.json	deepseek	DeepSeek V3.2
glm_9b_config.json	vllm	GLM4-9B
internlm2.5_7b_config.json	vllm	internlm2.5-7B-chat
qwen3_0.6b_config.json	vllm	Qwen3-Embedding (for perplexity)

IMPORTANT: You can't access DeepSeek V3.2 via deepseek_v3.2_config.json, it has been deprecated now

Acknowledgement

• Our code is built upon the PoisonedRAG framework (USENIX Security 2025).
• The defense implementations are adapted from RobustRAG and RAGuard.
• RL training built with TRL and Unsloth.

Citation

If you use this code, please cite the original RIPRAG paper:

@inproceedings{xi2026riprag,
  title={RIPRAG: Hack a Black-box Retrieval-Augmented Generation Question-Answering System with Reinforcement Learning},
  author={Xi, Meng and Lv, Sihan and Jin, Yechen and Cheng, Guanjie and Wang, Naibo and Li, Ying and Yin, Jianwei},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2026},
  pages={16882--16902},
  year={2026}
}