Skip to content

OmniSimpleMem

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

OmniSimpleMem

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
assets
assets
 
 
benchmarks
benchmarks
 
 
configs
configs
 
 
examples
examples
 
 
omni_memory
omni_memory
 
 
tests
tests
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
requirements.txt
requirements.txt
 
 
setup.py
setup.py
 
 

README.md

Omni-SimpleMem: Unified Multimodal Memory for Lifelong AI Agents

Extending SimpleMem to multimodal — store, organize, and recall text, image, audio & video experiences with state-of-the-art accuracy.

Apache 2.0 License Python 3.9+ 126 Tests Passed 13 Packages SOTA AutoResearchClaw

Quick Start  ·  Architecture  ·  Results  ·  Benchmarks  ·  Config  ·  Citation

Omni-SimpleMem Framework Overview

🚀 Quick Start

Installation

git clone https://github.com/aiming-lab/SimpleMem.git
cd SimpleMem/OmniSimpleMem
pip install -e .
📦 Optional dependency groups 
pip install -e ".[all]"      # Everything
pip install -e ".[visual]"   # Image/video (torch, transformers, CLIP)
pip install -e ".[audio]"    # Audio (soundfile, librosa)
pip install -e ".[vector]"   # FAISS vector search
pip install -e ".[server]"   # FastAPI REST server
pip install -e ".[dev]"      # Development (pytest)

Basic Usage

from omni_memory import OmniMemoryOrchestrator, OmniMemoryConfig

config = OmniMemoryConfig()
config.embedding.model_name = "all-MiniLM-L6-v2"
config.embedding.embedding_dim = 384

orchestrator = OmniMemoryOrchestrator(config=config, data_dir="./my_memory")

# Store
orchestrator.add_text(
    "User loves hiking in the Rocky Mountains every summer.",
    tags=["session_id:D1", "timestamp:2024-06-15"],
)

# Query
result = orchestrator.query("What does the user enjoy?", top_k=5)
for item in result.items:
    print(item["summary"])

orchestrator.close()
🌐 REST API Server 
pip install -e ".[server]"
export OPENAI_API_KEY=your_key_here
python examples/api_server.py
# Visit http://localhost:8000/docs
📝 More examples

Highlights

📈 +411%
LoCoMo F1		 📈 +214%
Mem-Gallery F1		 ⚡ 5.81 q/s
3.5x faster		 🧠 4 modalities
Text · Image · Audio · Video		 🔬 ~50 experiments
Fully autonomous
Benchmark Previous SOTA Omni-SimpleMem (GPT-5.1) Improvement
LoCoMo (F1) 0.432 (SimpleMem) 0.613 +41.9%
Mem-Gallery (F1) 0.697 (MuRAG) 0.810 +16.2%

Omni-SimpleMem achieves SOTA across all five LLM backbones (GPT-4o, GPT-4o-mini, GPT-4.1-nano, GPT-5.1, GPT-5-nano), outperforming six baselines including MemVerse, Mem0, Claude-Mem, A-MEM, MemGPT, and SimpleMem. Architecture autonomously discovered by AutoResearchClaw (~50 experiments, ~72 hours, zero human intervention).

🏗️ Architecture

Omni-SimpleMem builds on three architectural principles:

1. Selective Ingestion

Modality Method Effect
🎞️ Visual CLIP cosine-similarity scene-change detection ~70% storage reduction
🎙️ Audio VAD silence/noise filtering ~40% reduction
📝 Text Jaccard overlap redundancy filter Dedup

All modalities unify into Multimodal Atomic Units (MAUs): <summary, embedding, cold_pointer, timestamp, modality, links>. Lightweight metadata lives in hot storage (RAM); raw media in cold storage (disk/S3), loaded on demand.

2. Progressive Retrieval with Hybrid Search

Hybrid search: FAISS (dense) + BM25 (sparse) merged via set-union (autonomously discovered). Results expand progressively under a token budget:

Level Content Cost
🔍 Preview Summaries (~10 tokens) Low
📄 Details Full text + metadata Medium
📦 Evidence Raw content from cold storage High

3. Knowledge Graph Augmentation

A graph with 7 entity types and 12 relation types enables cross-modal multi-hop reasoning via seed identification → h-hop expansion → distance-decayed scoring.

📐 Full system diagram 
Input (Text / Image / Audio / Video)
  │
  ▼
┌─────────────────────────────┐
│   Entropy-Driven Filtering  │  ← CLIP / VAD / Jaccard
└─────────────┬───────────────┘
              ▼
┌─────────────────────────────┐
│   Two-Tier Storage          │
│   Hot: summaries, embeddings│  ← FAISS + BM25 index
│   Cold: raw media (disk/S3) │
└─────────────┬───────────────┘
              ▼
┌─────────────────────────────┐
│   Hybrid Search             │
│   Dense (FAISS) ∪ Sparse    │  ← Set-union merging
└─────────────┬───────────────┘
              ▼
┌─────────────────────────────┐
│   Pyramid Retrieval         │  ← Token-budget aware
│   Preview → Details →       │
│   Evidence                  │
└─────────────┬───────────────┘
              ▼
┌─────────────────────────────┐
│   Knowledge Graph Traversal │  ← h-hop expansion
└─────────────┬───────────────┘
              ▼
         LLM Answer

📊 Results

LoCoMo — Overall F1
Method GPT-4o GPT-4o-mini GPT-4.1-nano GPT-5.1 GPT-5-nano
Mem0 0.397 0.364 0.310 0.390 0.352
A-MEM 0.394 0.357 0.216 0.385 0.348
MemGPT 0.404 0.364 0.316 0.385 0.355
SimpleMem 0.432 0.404 0.342 0.418 0.388
Omni-SimpleMem 0.598 0.519 0.492 0.613 0.522
Mem-Gallery — Overall F1
Method GPT-4o GPT-4o-mini GPT-4.1-nano GPT-5.1 GPT-5-nano
Mem0 0.298 0.291 0.268 0.270 0.283
A-MEM 0.370 0.330 0.365 0.408 0.505
MemGPT 0.435 0.398 0.360 0.425 0.388
SimpleMem 0.535 0.498 0.518 0.538 0.522
Omni-SimpleMem 0.797 0.749 0.780 0.810 0.787

Efficiency

Efficiency Comparison

🔬 Benchmarks

Omni-SimpleMem is evaluated on two benchmarks:

  • LoCoMo — 1,986 QA pairs across 10 conversations and 5 categories (Multi-hop, Single-hop, Temporal, Open-domain, Adversarial)
  • Mem-Gallery — 1,711 QA pairs from 240 dialogues across 9 multimodal categories

Reproducing LoCoMo Results

# 1. Download dataset
git clone https://github.com/snap-research/locomo.git

# 2. Set API key
export OPENAI_API_KEY="your-openai-api-key"

# 3. Run benchmark
python benchmarks/locomo/run_locomo.py \
    --data-path /path/to/locomo/data/locomo10.json \
    --model gpt-4o -o ./locomo_results

# Quick test (1 conversation, 20 QA pairs)
python benchmarks/locomo/run_locomo.py \
    --data-path /path/to/locomo/data/locomo10.json \
    --max-conversations 1 --max-qa 20

⚙️ Configuration

Full configuration reference 
from omni_memory import OmniMemoryConfig

config = OmniMemoryConfig()

# Embedding
config.embedding.model_name = "all-MiniLM-L6-v2"  # Local (384d)
config.embedding.embedding_dim = 384

# LLM
config.llm.summary_model = "gpt-4o-mini"
config.llm.query_model = "gpt-4o"
config.llm.temperature = 0.0

# Retrieval
config.retrieval.default_top_k = 20
config.retrieval.enable_hybrid_search = True
config.retrieval.enable_graph_traversal = True

# Storage
config.storage.base_dir = "./my_memory_data"

# Unified model shortcut
config.set_unified_model("gpt-4o")

See configs/ for benchmark-specific YAML configurations.

🧪 Testing

pip install -e ".[dev]"
pytest tests/ -v

📁 Package Structure

Click to expand 
OmniSimpleMem/
├── omni_memory/               # Core package
│   ├── orchestrator.py        # Central coordinator
│   ├── app.py                 # FastAPI REST server
│   ├── core/                  # MAU, config, events
│   ├── storage/               # FAISS vector store, cold storage, dedup
│   ├── retrieval/             # Pyramid retriever, BM25, query processor
│   ├── processors/            # Text, image, audio, video processors
│   ├── triggers/              # CLIP visual & VAD audio triggers
│   ├── knowledge/             # Knowledge graph & entity extraction
│   ├── graph/                 # Event management
│   ├── parametric/            # Memory consolidation
│   ├── routing/               # Query routing
│   ├── evolution/             # Self-evolution
│   └── utils/                 # Embedding, model utilities, logging
├── configs/                   # Benchmark YAML configs
├── benchmarks/                # LoCoMo & Mem-Gallery adapters
├── tests/                     # 126 unit tests
├── examples/                  # Usage examples
├── setup.py
├── requirements.txt
└── LICENSE                    # Apache 2.0

🔧 Implementation Details

Component Details
🔎 Dense retrieval FAISS — all-MiniLM-L6-v2 (384d) or text-embedding-3-large (3072d)
🔤 Sparse retrieval BM25 via rank_bm25
🖼️ Visual embeddings OpenVision CLIP ViT-L/14 (768d)
🕸️ Knowledge graph In-memory, 7 entity types, 12 relation types
⚡ Concurrency Thread-safe (RLock), parallel workers
💾 Storage JSONL MAU persistence, FAISS index serialization

🙏 Acknowledgments

Omni-SimpleMem's architecture was discovered through AutoResearchClaw, an autonomous 23-stage research pipeline. The most impactful discoveries were bug fixes (+175%), prompt engineering (+188%), and architectural changes (+44%) — each exceeding all hyperparameter tuning combined.

📄 License

Apache 2.0

📌 Citation

@article{omnisimplemem2026,
  title   = {{Omni-SimpleMem}: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory},
  author  = {Liu, Jiaqi and Ling, Zipeng and Qiu, Shi and Liu, Yanqing and Han, Siwei and Xia, Peng and Tu, Haoqin and Zheng, Zeyu and Xie, Cihang and Fleming, Charles and Ding, Mingyu and Yao, Huaxiu},
  journal = {arXiv preprint arXiv:2604.01007},
  year    = {2026},
}

Part of the SimpleMem family  ·  Architecture discovered by AutoResearchClaw