From 4ff8fa981c8c99598b168b3cd435439737117c32 Mon Sep 17 00:00:00 2001 From: Kristiyan Ivanov Date: Thu, 23 Apr 2026 19:36:27 +0300 Subject: [PATCH] added new cookbooks for the new versions --- .../01-getting-started.md | 160 ++++++++ .../02-llm-and-tool-cache.md | 211 ++++++++++ .../03-session-store.md | 143 +++++++ .../04-provider-adapters.md | 263 +++++++++++++ .../betterdb-agent-cache-py/05-langgraph.md | 199 ++++++++++ content/betterdb-agent-cache-py/meta.json | 97 +++++ .../01-getting-started.md | 7 +- .../02-llm-and-tool-cache.md | 68 +++- .../05-provider-adapters.md | 297 ++++++++++++++ content/betterdb-agent-cache/meta.json | 20 + .../01-getting-started.html | 189 +++++++++ .../02-llm-and-tool-cache.html | 267 +++++++++++++ .../03-session-store.html | 182 +++++++++ .../04-provider-adapters.html | 314 +++++++++++++++ .../betterdb-agent-cache-py/05-langgraph.html | 256 ++++++++++++ .../01-getting-started.html | 7 +- .../02-llm-and-tool-cache.html | 49 ++- .../betterdb-agent-cache/04-langgraph.html | 2 +- .../05-provider-adapters.html | 365 ++++++++++++++++++ cookbooks/betterdb/index.html | 73 +++- index.html | 2 +- 21 files changed, 3151 insertions(+), 20 deletions(-) create mode 100644 content/betterdb-agent-cache-py/01-getting-started.md create mode 100644 content/betterdb-agent-cache-py/02-llm-and-tool-cache.md create mode 100644 content/betterdb-agent-cache-py/03-session-store.md create mode 100644 content/betterdb-agent-cache-py/04-provider-adapters.md create mode 100644 content/betterdb-agent-cache-py/05-langgraph.md create mode 100644 content/betterdb-agent-cache-py/meta.json create mode 100644 content/betterdb-agent-cache/05-provider-adapters.md create mode 100644 cookbooks/betterdb-agent-cache-py/01-getting-started.html create mode 100644 cookbooks/betterdb-agent-cache-py/02-llm-and-tool-cache.html create mode 100644 cookbooks/betterdb-agent-cache-py/03-session-store.html create mode 100644 cookbooks/betterdb-agent-cache-py/04-provider-adapters.html create mode 100644 cookbooks/betterdb-agent-cache-py/05-langgraph.html create mode 100644 cookbooks/betterdb-agent-cache/05-provider-adapters.html diff --git a/content/betterdb-agent-cache-py/01-getting-started.md b/content/betterdb-agent-cache-py/01-getting-started.md new file mode 100644 index 0000000..d983eae --- /dev/null +++ b/content/betterdb-agent-cache-py/01-getting-started.md @@ -0,0 +1,160 @@ +`betterdb-agent-cache` is a multi-tier exact-match cache for AI agent workloads backed by Valkey. It combines three cache tiers behind one connection: + +- **LLM tier** — caches full LLM responses by exact match on model + messages + params +- **Tool tier** — caches tool/function call results by tool name and argument hash +- **Session tier** — key-value store for agent state with per-field TTL + +Works on vanilla Valkey 7+, Amazon ElastiCache, Google Cloud Memorystore, and Amazon MemoryDB with no modules required. + +## Prerequisites + +- **Valkey 7+** (no modules needed) +- Python 3.11+ + +## Step 1: Start Valkey + +```bash +docker run -d --name valkey -p 6379:6379 valkey/valkey:latest +``` + +## Step 2: Install + +```bash +pip install betterdb-agent-cache +``` + +For provider-specific adapters, install the relevant extra: + +```bash +pip install "betterdb-agent-cache[openai]" # OpenAI +pip install "betterdb-agent-cache[anthropic]" # Anthropic +pip install "betterdb-agent-cache[langchain]" # LangChain +pip install "betterdb-agent-cache[langgraph]" # LangGraph +pip install "betterdb-agent-cache[llamaindex]" # LlamaIndex +``` + +## Step 3: Create an AgentCache + +```python +import asyncio +import valkey.asyncio as valkey_client +from betterdb_agent_cache import AgentCache, TierDefaults +from betterdb_agent_cache.types import AgentCacheOptions + +async def main(): + client = valkey_client.Valkey(host="localhost", port=6379) + + cache = AgentCache(AgentCacheOptions( + client=client, + tier_defaults={ + "llm": TierDefaults(ttl=3600), # LLM responses: 1 hour + "tool": TierDefaults(ttl=300), # Tool results: 5 minutes + "session": TierDefaults(ttl=1800), # Session state: 30 min (sliding) + }, + # cost_table is optional — 1,900+ models covered by default + )) +``` + +No `initialize()` call needed — all tiers use plain Valkey commands (SET, GET, HINCRBY) with no index creation. + +## Step 4: LLM Tier + +```python +params = { + "model": "gpt-4o-mini", + "messages": [{"role": "user", "content": "What is Valkey?"}], + "temperature": 0, +} + +# Check for a cached response +llm_result = await cache.llm.check(params) + +if llm_result.hit: + print("LLM cache HIT:", llm_result.response) +else: + print("LLM cache MISS → calling LLM...") + response = await call_llm(params) # your LLM call + + from betterdb_agent_cache.types import LlmStoreOptions + await cache.llm.store(params, response, LlmStoreOptions( + tokens={"input": 14, "output": 62}, # enables cost tracking + )) +``` + +The cache key is a SHA-256 hash of the canonicalized params. `{"messages": [...], "model": "gpt-4o"}` and `{"model": "gpt-4o", "messages": [...]}` produce the same key. + +## Step 5: Tool Tier + +```python +tool_name = "get_weather" +tool_args = {"city": "Sofia", "units": "metric"} + +tool_result = await cache.tool.check(tool_name, tool_args) + +if tool_result.hit: + print("Tool cache HIT:", tool_result.response) +else: + print("Tool cache MISS → calling API...") + import json + data = await get_weather(tool_args) # your tool call + + from betterdb_agent_cache.types import ToolStoreOptions + await cache.tool.store(tool_name, tool_args, json.dumps(data), ToolStoreOptions( + cost=0.001, # API call cost in dollars, for tracking + )) +``` + +Tool args are serialized with recursively sorted keys before hashing, so `{"city": "Sofia", "units": "metric"}` and `{"units": "metric", "city": "Sofia"}` hit the same cache entry. + +## Step 6: Session Tier + +```python +thread_id = "user-42-session-1" + +# Store agent state fields +await cache.session.set(thread_id, "last_intent", "book_flight") +await cache.session.set(thread_id, "destination", "Sofia") +await cache.session.set(thread_id, "departure_date", "2026-06-01") + +# Retrieve a field (also refreshes its TTL) +intent = await cache.session.get(thread_id, "last_intent") +# "book_flight" + +# Get all fields for a thread +state = await cache.session.get_all(thread_id) +# {"last_intent": "book_flight", "destination": "Sofia", "departure_date": "2026-06-01"} +``` + +Each `get()` refreshes the TTL on that field (sliding window). The session stays alive as long as the agent is actively using it. + +## Key Formats + +All keys are prefixed with the `name` option (default: `betterdb_ac`): + +| Tier | Key Pattern | +|------|-------------| +| LLM cache | `betterdb_ac:llm:{sha256hash}` | +| Tool cache | `betterdb_ac:tool:{toolName}:{sha256hash}` | +| Session field | `betterdb_ac:session:{threadId}:{field}` | +| Stats hash | `betterdb_ac:__stats` | + +```python +cache = AgentCache(AgentCacheOptions( + client=client, + name="myapp_prod", # all keys prefixed with 'myapp_prod:' +)) +``` + +## Cleanup + +```python +# Destroy all state for a thread +await cache.session.destroy_thread(thread_id) + +# Invalidate all LLM cache entries for a model +await cache.llm.invalidate_by_model("gpt-4o-mini") + +# Close the client when done +await cache.shutdown() +await client.aclose() +``` diff --git a/content/betterdb-agent-cache-py/02-llm-and-tool-cache.md b/content/betterdb-agent-cache-py/02-llm-and-tool-cache.md new file mode 100644 index 0000000..a8e3fb2 --- /dev/null +++ b/content/betterdb-agent-cache-py/02-llm-and-tool-cache.md @@ -0,0 +1,211 @@ +## LLM Cache Tier + +The LLM cache stores full LLM responses by exact match on all parameters that affect the output. + +### What Gets Hashed + +The cache key is a SHA-256 hash of these fields (with recursively sorted dict keys for determinism): + +| Field | Notes | +|-------|-------| +| `model` | `'gpt-4o'`, `'claude-opus-4-6'`, etc. | +| `messages` | Full message list including roles and content | +| `temperature` | `None` is treated as unset | +| `top_p` | `None` is treated as unset | +| `max_tokens` | `None` is treated as unset | +| `tools` | Tool definitions if present | +| `tool_choice` | `None` is treated as unset | +| `seed` | `None` is treated as unset | +| `stop` | `None` is treated as unset | +| `response_format` | `None` is treated as unset | +| `reasoning_effort` | For models supporting extended thinking | +| `prompt_cache_key` | Pass-through for provider-level prompt caching | + +### Check and Store + +```python +import valkey.asyncio as valkey_client +from openai import AsyncOpenAI +from betterdb_agent_cache import AgentCache, TierDefaults +from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions + +client = valkey_client.Valkey(host="localhost", port=6379) +openai = AsyncOpenAI() + +# cost_table is optional — 1,900+ models covered by default (see "Default Cost Table" below) +cache = AgentCache(AgentCacheOptions( + client=client, + tier_defaults={"llm": TierDefaults(ttl=3600)}, +)) + +async def cached_completion(params: dict) -> str: + cached = await cache.llm.check(params) + if cached.hit: + return cached.response or "" + + response = await openai.chat.completions.create(**params) + text = response.choices[0].message.content or "" + usage = response.usage + + await cache.llm.store(params, text, LlmStoreOptions( + tokens={ + "input": usage.prompt_tokens if usage else 0, + "output": usage.completion_tokens if usage else 0, + }, + )) + return text +``` + +Token counts in `store()` enable cost savings tracking via the cost table. Without `tokens`, cost tracking is skipped for that entry. + +### Invalidating by Model + +```python +deleted = await cache.llm.invalidate_by_model("gpt-4o-mini") +print(f"Deleted {deleted} entries") +``` + +--- + +## Tool Cache Tier + +The tool cache stores tool/function call results. Valuable for expensive external API calls — weather, geocoding, database queries, web search — that return stable results for the same inputs. + +### Check and Store + +```python +import json +from betterdb_agent_cache.types import ToolStoreOptions + +async def cached_tool(name: str, args: dict) -> dict: + cached = await cache.tool.check(name, args) + if cached.hit: + return json.loads(cached.response) + + result = await call_tool(name, args) # your tool implementation + + await cache.tool.store(name, args, json.dumps(result), ToolStoreOptions( + ttl=300, # per-call TTL override + cost=0.005, # API call cost in dollars + )) + return result +``` + +### Per-Tool TTL Policies + +```python +from betterdb_agent_cache.types import ToolPolicy + +# Weather data: short TTL, changes frequently +await cache.tool.set_policy("get_weather", ToolPolicy(ttl=300)) # 5 min + +# Stock prices: very short TTL +await cache.tool.set_policy("get_stock_price", ToolPolicy(ttl=60)) # 1 min + +# Geocoding: long TTL, addresses don't change +await cache.tool.set_policy("geocode_address", ToolPolicy(ttl=86400)) # 24h + +# Web search: medium TTL +await cache.tool.set_policy("web_search", ToolPolicy(ttl=3600)) # 1h +``` + +TTL precedence: per-call `ttl` > tool policy > `tier_defaults["tool"].ttl` > `default_ttl`. + +### Invalidation + +```python +# Invalidate all results for a specific tool +deleted = await cache.tool.invalidate_by_tool("get_weather") +print(f"Deleted {deleted} weather cache entries") + +# Invalidate one specific call +existed = await cache.tool.invalidate("get_weather", {"city": "Sofia"}) +``` + +--- + +## Cost Tracking and Stats + +### Aggregate Stats + +```python +stats = await cache.stats() +print(stats.llm.hits, stats.llm.misses, f"{stats.llm.hit_rate:.0%}") +# 150 50 75% + +print(f"Cost saved: ${stats.cost_saved_micros / 1_000_000:.4f}") +# Cost saved: $12.5000 + +# Per-tool breakdown +for tool_name, tool_stats in stats.per_tool.items(): + print(f"{tool_name}: {tool_stats.hit_rate:.0%} hit rate") +``` + +Cost savings are stored as microdollars (integers) to avoid floating-point precision issues. Divide by `1_000_000` to get dollars. + +### Tool Effectiveness Recommendations + +```python +effectiveness = await cache.tool_effectiveness() +for entry in effectiveness: + print(f"{entry.tool}: {entry.hit_rate:.0%} hit rate — {entry.recommendation}") +# get_weather: 85% hit rate — increase_ttl +# web_search: 62% hit rate — optimal +# rare_api_call: 8% hit rate — decrease_ttl_or_disable +``` + +| Recommendation | Condition | Action | +|---|---|---| +| `increase_ttl` | Hit rate > 80% and TTL < 1 hour | Results are stable and reused frequently — extend TTL | +| `optimal` | Hit rate 40–80% | No change needed | +| `decrease_ttl_or_disable` | Hit rate < 40% | Results change too fast or rarely repeated | + +--- + +## Default Cost Table + +`betterdb-agent-cache` ships a bundled cost table sourced from [LiteLLM's model pricing data](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) and refreshed on every release. Cost tracking works out of the box for 1,900+ models — no `cost_table` configuration required. + +```python +# No cost_table needed — GPT-4o, Claude, Gemini, and 1,900+ others are covered +cache = AgentCache(AgentCacheOptions(client=client)) +``` + +### Overriding Specific Models + +User-supplied `cost_table` entries merge on top of the defaults: + +```python +from betterdb_agent_cache import ModelCost + +cache = AgentCache(AgentCacheOptions( + client=client, + cost_table={ + "my-fine-tuned-gpt4o": ModelCost(input_per_1k=0.005, output_per_1k=0.015), + }, +)) +``` + +### Inspecting the Bundled Table + +```python +from betterdb_agent_cache import DEFAULT_COST_TABLE + +print(DEFAULT_COST_TABLE.get("gpt-4o-mini")) +# ModelCost(input_per_1k=0.00015, output_per_1k=0.0006) + +print(len(DEFAULT_COST_TABLE)) +# 1900+ +``` + +### Disabling the Default Table + +```python +cache = AgentCache(AgentCacheOptions( + client=client, + use_default_cost_table=False, + cost_table={ + "gpt-4o": ModelCost(input_per_1k=0.0025, output_per_1k=0.010), + }, +)) +``` diff --git a/content/betterdb-agent-cache-py/03-session-store.md b/content/betterdb-agent-cache-py/03-session-store.md new file mode 100644 index 0000000..7a42de5 --- /dev/null +++ b/content/betterdb-agent-cache-py/03-session-store.md @@ -0,0 +1,143 @@ +The session tier is a per-thread key-value store for agent state. Designed for the patterns that come up in multi-step, multi-turn agents: storing user intent, intermediate reasoning, extracted entities, and tool call context between invocations. + +## How It Works + +Each session field is stored as an individual Valkey key: + +``` +betterdb_ac:session:{thread_id}:{field} +``` + +Individual keys allow **per-field TTL** — different fields can have different expiry times — and `get()` automatically refreshes the TTL on each read (sliding window). The trade-off is that `get_all()` and `destroy_thread()` require a SCAN + pipeline rather than a single `HGETALL` or `DEL`. + +## Basic Operations + +```python +import valkey.asyncio as valkey_client +from betterdb_agent_cache import AgentCache, TierDefaults +from betterdb_agent_cache.types import AgentCacheOptions + +client = valkey_client.Valkey(host="localhost", port=6379) +cache = AgentCache(AgentCacheOptions( + client=client, + tier_defaults={"session": TierDefaults(ttl=1800)}, # 30 min sliding window +)) + +thread_id = "user-42-thread-001" + +# Write state fields +await cache.session.set(thread_id, "intent", "book_flight") +await cache.session.set(thread_id, "destination", "Sofia") +await cache.session.set(thread_id, "departure", "2026-06-01") + +# Read a field (also refreshes its TTL) +intent = await cache.session.get(thread_id, "intent") +# "book_flight" + +# Returns None for fields that don't exist or have expired +missing = await cache.session.get(thread_id, "return_date") +# None +``` + +## Getting All Fields + +```python +state = await cache.session.get_all(thread_id) +print(state) +# {"intent": "book_flight", "destination": "Sofia", "departure": "2026-06-01"} +``` + +`get_all()` uses SCAN to find all keys for the thread and fetches them in a pipeline. Designed for sessions with dozens of fields — not thousands. + +## Refreshing TTLs + +```python +# On each user interaction, refresh all session fields to keep the session alive +await cache.session.touch(thread_id) +``` + +`touch()` extends the TTL on every field for the thread. Call it at the start of each agent invocation to prevent partial expiry where some fields outlive others. + +## Deleting State + +```python +# Delete a single field +await cache.session.delete(thread_id, "departure") + +# Destroy the entire thread +deleted = await cache.session.destroy_thread(thread_id) +print(f"Deleted {deleted} keys") +``` + +## Multi-Step Agent Example + +A booking agent that accumulates state across multiple tool calls: + +```python +import json + +async def run_booking_agent(thread_id: str, user_message: str) -> dict: + # Load existing state + state = await cache.session.get_all(thread_id) + + # Extract intent + if "book" in user_message and "flight" in user_message: + await cache.session.set(thread_id, "intent", "book_flight") + + # Extract destination + import re + dest_match = re.search(r"to (\w+)", user_message, re.IGNORECASE) + if dest_match: + await cache.session.set(thread_id, "destination", dest_match.group(1)) + + # Extract departure date + date_match = re.search(r"on (\d{4}-\d{2}-\d{2})", user_message) + if date_match: + await cache.session.set(thread_id, "departure", date_match.group(1)) + + # Refresh all TTLs so the session doesn't partially expire + await cache.session.touch(thread_id) + + # Read current state + current_state = await cache.session.get_all(thread_id) + + # Search for flights only when we have enough context + if (current_state.get("destination") and current_state.get("departure") + and not current_state.get("search_results")): + args = { + "destination": current_state["destination"], + "date": current_state["departure"], + } + cached = await cache.tool.check("search_flights", args) + if not cached.hit: + flight_data = await search_flights(current_state) + await cache.tool.store("search_flights", args, json.dumps(flight_data)) + await cache.session.set(thread_id, "search_results", json.dumps(flight_data)) + else: + await cache.session.set(thread_id, "search_results", cached.response) + + return await cache.session.get_all(thread_id) +``` + +## Session State Patterns + +| Pattern | Implementation | +|---------|---------------| +| User intent tracking | `set(thread, "intent", value)` on each turn | +| Entity accumulation | One field per entity: `set(thread, "city", "Sofia")` | +| Conversation stage | `set(thread, "stage", "collecting_dates")` | +| Intermediate results | Store JSON: `set(thread, "search_results", json.dumps(data))` | +| Error recovery | `set(thread, "last_failed_step", "payment")` | +| Session keepalive | `touch(thread)` at start of each invocation | + +## Per-Field TTL Override + +```python +# Short-lived field: search results that go stale quickly +await cache.session.set(thread_id, "flight_prices", json.dumps(prices), ttl=300) + +# Long-lived field: user preferences that persist across sessions +await cache.session.set(thread_id, "preferred_seat", "aisle", ttl=86400 * 30) +``` + +The per-call `ttl` overrides `tier_defaults["session"].ttl` for that specific field. diff --git a/content/betterdb-agent-cache-py/04-provider-adapters.md b/content/betterdb-agent-cache-py/04-provider-adapters.md new file mode 100644 index 0000000..db367ae --- /dev/null +++ b/content/betterdb-agent-cache-py/04-provider-adapters.md @@ -0,0 +1,263 @@ +Provider adapters translate the native params of each SDK into the canonical `LlmCacheParams` format before hashing. The pattern is the same across all four adapters: + +1. Call `prepare_params()` on the native SDK params dict +2. Pass the result to `cache.llm.check()` +3. On a miss, call the provider and build a list of content block dicts from the response +4. Store with `cache.llm.store_multipart()` — handles text, tool calls, and reasoning blocks together + +## Binary Normalizer + +The binary normalizer controls how binary content — images, audio, documents — is reduced to a stable string before being included in the cache key hash. + +The **default** is `passthrough`: the full base64 data string is included literally. Zero-latency and correct, but a re-encoded image that is byte-for-byte identical but with different base64 padding will produce a different hash. + +For production use, switch to `hash_base64`: + +```python +from betterdb_agent_cache import compose_normalizer, hash_base64 + +# Hash base64 image/audio/document bytes — O(n) in content size, no network calls +normalizer = compose_normalizer({"base64": hash_base64}) +``` + +Other built-in helpers: + +| Helper | Behavior | +|--------|----------| +| `hash_base64(data)` | SHA-256 of decoded bytes — stable across re-encodings | +| `hash_bytes(data)` | SHA-256 of raw `bytes` | +| `hash_url(url)` | Lowercases scheme+host, sorts query params, returns `"url:"` | +| `fetch_and_hash(url)` | Fetches the URL (requires `aiohttp`) and returns SHA-256 of the body | +| `passthrough(ref)` | Returns the raw data as-is (default) | + +--- + +## OpenAI Chat Completions + +```bash +pip install "betterdb-agent-cache[openai]" +``` + +```python +import json +import valkey.asyncio as valkey_client +from openai import AsyncOpenAI +from betterdb_agent_cache import AgentCache, TierDefaults, compose_normalizer, hash_base64 +from betterdb_agent_cache.adapters.openai import OpenAIPrepareOptions, prepare_params +from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions + +client = valkey_client.Valkey(host="localhost", port=6379) +cache = AgentCache(AgentCacheOptions( + client=client, + tier_defaults={"llm": TierDefaults(ttl=3600)}, +)) +normalizer = compose_normalizer({"base64": hash_base64}) +opts = OpenAIPrepareOptions(normalizer=normalizer) +openai = AsyncOpenAI() + +async def chat(params: dict) -> str: + # Translate OpenAI params → canonical LlmCacheParams + cache_params = await prepare_params(params, opts) + + cached = await cache.llm.check(cache_params) + if cached.hit: + return cached.response or "" + + response = await openai.chat.completions.create(**params, stream=False) + choice = response.choices[0] + + blocks = [] + if choice.message.content: + blocks.append({"type": "text", "text": choice.message.content}) + for tc in choice.message.tool_calls or []: + try: + args = json.loads(tc.function.arguments or "{}") + except json.JSONDecodeError: + args = {"__raw": tc.function.arguments} + blocks.append({"type": "tool_call", "id": tc.id, "name": tc.function.name, "args": args}) + + await cache.llm.store_multipart(cache_params, blocks, LlmStoreOptions(tokens={ + "input": response.usage.prompt_tokens if response.usage else 0, + "output": response.usage.completion_tokens if response.usage else 0, + })) + + return " ".join(b["text"] for b in blocks if b.get("type") == "text") +``` + +`prepare_params` handles all message roles: `system`, `developer`, `user`, `assistant`, `tool`, and legacy `function`. Multi-modal user messages (images, audio, file uploads) are normalized through the binary normalizer before hashing. + +--- + +## OpenAI Responses API + +```bash +pip install "betterdb-agent-cache[openai]" +``` + +```python +import valkey.asyncio as valkey_client +from openai import AsyncOpenAI +from betterdb_agent_cache import AgentCache, TierDefaults, compose_normalizer, hash_base64 +from betterdb_agent_cache.adapters.openai_responses import OpenAIResponsesPrepareOptions, prepare_params +from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions + +client = valkey_client.Valkey(host="localhost", port=6379) +cache = AgentCache(AgentCacheOptions( + client=client, + tier_defaults={"llm": TierDefaults(ttl=3600)}, +)) +normalizer = compose_normalizer({"base64": hash_base64}) +opts = OpenAIResponsesPrepareOptions(normalizer=normalizer) +openai = AsyncOpenAI() + +async def respond(params: dict) -> str: + cache_params = await prepare_params(params, opts) + + cached = await cache.llm.check(cache_params) + if cached.hit: + return cached.response or "" + + response = await openai.responses.create(**params) + + blocks = [] + for item in response.output or []: + if item.type == "message": + for part in item.content or []: + if part.type == "output_text": + blocks.append({"type": "text", "text": part.text}) + elif item.type == "reasoning": + text = " ".join(s.text for s in (item.summary or []) if s.type == "reasoning_text") + blocks.append({"type": "reasoning", "text": text}) + elif item.type == "function_call": + import json + try: + args = json.loads(item.arguments or "{}") + except json.JSONDecodeError: + args = {"__raw": item.arguments} + blocks.append({"type": "tool_call", "id": item.call_id, "name": item.name, "args": args}) + + await cache.llm.store_multipart(cache_params, blocks, LlmStoreOptions(tokens={ + "input": response.usage.input_tokens if response.usage else 0, + "output": response.usage.output_tokens if response.usage else 0, + })) + + return " ".join(b["text"] for b in blocks if b.get("type") == "text") +``` + +`instructions` is prepended as a `system` message. The adapter handles `reasoning` items, `function_call`/`function_call_output` item types, and multi-modal input parts. `reasoning.effort` maps to `reasoning_effort` in the cache key. + +--- + +## Anthropic Messages + +```bash +pip install "betterdb-agent-cache[anthropic]" +``` + +```python +import valkey.asyncio as valkey_client +import anthropic as sdk +from betterdb_agent_cache import AgentCache, TierDefaults, compose_normalizer, hash_base64 +from betterdb_agent_cache.adapters.anthropic import AnthropicPrepareOptions, prepare_params +from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions + +client = valkey_client.Valkey(host="localhost", port=6379) +cache = AgentCache(AgentCacheOptions( + client=client, + tier_defaults={"llm": TierDefaults(ttl=3600)}, +)) +normalizer = compose_normalizer({"base64": hash_base64}) +opts = AnthropicPrepareOptions(normalizer=normalizer) +anthropic = sdk.AsyncAnthropic() + +async def chat(params: dict) -> str: + cache_params = await prepare_params(params, opts) + + cached = await cache.llm.check(cache_params) + if cached.hit: + return cached.response or "" + + response = await anthropic.messages.create(**params) + + blocks = [] + for block in response.content: + if block.type == "text": + blocks.append({"type": "text", "text": block.text}) + elif block.type == "tool_use": + blocks.append({"type": "tool_call", "id": block.id, "name": block.name, "args": block.input}) + elif block.type == "thinking": + blocks.append({"type": "reasoning", "text": block.thinking, + "opaqueSignature": getattr(block, "signature", None)}) + + await cache.llm.store_multipart(cache_params, blocks, LlmStoreOptions(tokens={ + "input": response.usage.input_tokens, + "output": response.usage.output_tokens, + })) + + return " ".join(b["text"] for b in blocks if b.get("type") == "text") +``` + +`params["system"]` is prepended as a `system` message. `tool_result` blocks in user messages are split into separate `tool` messages. `thinking` and `redacted_thinking` blocks map to `reasoning` blocks. `stop_sequences` maps to `stop` in the cache key. + +--- + +## LlamaIndex + +```bash +pip install "betterdb-agent-cache[llamaindex]" +``` + +```python +import valkey.asyncio as valkey_client +from llama_index.llms.openai import OpenAI +from betterdb_agent_cache import AgentCache, TierDefaults, compose_normalizer, hash_base64 +from betterdb_agent_cache.adapters.llamaindex import LlamaIndexPrepareOptions, prepare_params +from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions + +client = valkey_client.Valkey(host="localhost", port=6379) +cache = AgentCache(AgentCacheOptions( + client=client, + tier_defaults={"llm": TierDefaults(ttl=3600)}, +)) +normalizer = compose_normalizer({"base64": hash_base64}) +llm = OpenAI(model="gpt-4o-mini") + +async def chat(messages: list) -> str: + # model must be supplied explicitly — LlamaIndex messages don't carry it + opts = LlamaIndexPrepareOptions( + model="gpt-4o-mini", + normalizer=normalizer, + temperature=0, + ) + cache_params = await prepare_params(messages, opts) + + cached = await cache.llm.check(cache_params) + if cached.hit: + return cached.response or "" + + response = await llm.achat(messages) + text = str(response.message.content) + blocks = [{"type": "text", "text": text}] + + usage = getattr(response.raw, "usage", None) + await cache.llm.store_multipart(cache_params, blocks, LlmStoreOptions(tokens={ + "input": getattr(usage, "prompt_tokens", 0) if usage else 0, + "output": getattr(usage, "completion_tokens", 0) if usage else 0, + })) + return text +``` + +The model name is not available from LlamaIndex message objects — supply it via `LlamaIndexPrepareOptions.model`. The `memory` and `developer` roles are mapped to `system`. Tool calls are read from `message.options["toolCall"]` and tool results from `message.options["toolResult"]`. + +--- + +## Adapter Import Paths + +| Provider | Import path | +|---|---| +| OpenAI Chat Completions | `betterdb_agent_cache.adapters.openai` | +| OpenAI Responses API | `betterdb_agent_cache.adapters.openai_responses` | +| Anthropic Messages | `betterdb_agent_cache.adapters.anthropic` | +| LlamaIndex | `betterdb_agent_cache.adapters.llamaindex` | +| LangChain | `betterdb_agent_cache.adapters.langchain` | +| LangGraph | `betterdb_agent_cache.adapters.langgraph` | diff --git a/content/betterdb-agent-cache-py/05-langgraph.md b/content/betterdb-agent-cache-py/05-langgraph.md new file mode 100644 index 0000000..cb36dbc --- /dev/null +++ b/content/betterdb-agent-cache-py/05-langgraph.md @@ -0,0 +1,199 @@ +`BetterDBSaver` is a LangGraph checkpoint saver backed by `betterdb-agent-cache`. It stores graph state on **vanilla Valkey 7+** with no modules required. + +## Why BetterDBSaver vs langgraph-checkpoint-redis + +| | `BetterDBSaver` | `langgraph-checkpoint-redis` | +|---|---|---| +| Valkey support | ✅ Valkey 7+ | ❌ Redis 8+ only | +| Module requirements | ✅ None | ❌ RedisJSON + RediSearch | +| Works on ElastiCache | ✅ Any tier | ❌ Requires Redis 8 + modules | +| Works on Memorystore | ✅ Any tier | ❌ Requires Redis 8 + modules | +| Checkpoint storage | Plain JSON strings | RedisJSON path operations | +| Filtered listing | SCAN + parse | O(1) RediSearch index | +| Async | ✅ Fully async | ✅ | +| Best for | Any Valkey/Redis deployment | Redis 8+ with modules, millions of checkpoints | + +The trade-off: `alist()` with filtering uses SCAN. For typical deployments with hundreds of checkpoints per thread this is fast. `BetterDBSaver` is async-only — the sync `get_tuple()`, `list()`, and `put()` methods raise `RuntimeError`. + +## Prerequisites + +```bash +pip install "betterdb-agent-cache[langgraph]" langchain-openai +``` + +## Step 1: Create the Checkpointer + +```python +import valkey.asyncio as valkey_client +from betterdb_agent_cache import AgentCache, TierDefaults +from betterdb_agent_cache.adapters.langgraph import BetterDBSaver +from betterdb_agent_cache.types import AgentCacheOptions + +client = valkey_client.Valkey(host="localhost", port=6379) + +cache = AgentCache(AgentCacheOptions( + client=client, + tier_defaults={ + "session": TierDefaults(ttl=86400), # 24h session state + }, +)) + +checkpointer = BetterDBSaver(cache=cache) +``` + +## Step 2: Build a Graph with Checkpointing + +```python +import json +import random +from typing import Annotated +from typing_extensions import TypedDict + +from langchain_openai import ChatOpenAI +from langchain_core.messages import HumanMessage, AIMessage, ToolMessage +from langgraph.graph import StateGraph, END +from langgraph.graph.message import add_messages + +from betterdb_agent_cache.adapters.langchain import BetterDBLlmCache + + +class State(TypedDict): + messages: Annotated[list, add_messages] + + +# Pass BetterDBLlmCache to LangChain so LLM responses are cached automatically +model = ChatOpenAI( + model="gpt-4o-mini", + temperature=0, + cache=BetterDBLlmCache(cache=cache), +) + +_TOOLS = [{"type": "function", "function": { + "name": "get_weather", + "description": "Get current weather for a city", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, +}}] +model_with_tools = model.bind_tools(_TOOLS) + + +async def get_weather(city: str) -> str: + """Tool with its own cache tier.""" + cached = await cache.tool.check("get_weather", {"city": city}) + if cached.hit: + return cached.response or "" + result = json.dumps({ + "city": city, + "temperature": round(15 + random.random() * 15), + "condition": random.choice(["sunny", "cloudy", "rainy"]), + }) + await cache.tool.store("get_weather", {"city": city}, result) + return result + + +async def call_model(state: State) -> dict: + response = await model_with_tools.ainvoke(state["messages"]) + return {"messages": [response]} + + +async def call_tools(state: State) -> dict: + last: AIMessage = state["messages"][-1] + results = [] + for tc in getattr(last, "tool_calls", []) or []: + if tc["name"] == "get_weather": + result = await get_weather(tc["args"].get("city", "")) + results.append(ToolMessage(content=result, tool_call_id=tc["id"])) + return {"messages": results} + + +def should_continue(state: State) -> str: + last: AIMessage = state["messages"][-1] + return "tools" if getattr(last, "tool_calls", None) else END + + +graph = ( + StateGraph(State) + .add_node("agent", call_model) + .add_node("tools", call_tools) + .add_edge("__start__", "agent") + .add_conditional_edges("agent", should_continue) + .add_edge("tools", "agent") + .compile(checkpointer=checkpointer) # attach BetterDBSaver here +) +``` + +## Step 3: Run the Graph Across Multiple Turns + +```python +async def run_turn(thread_id: str, message: str) -> str: + result = await graph.ainvoke( + {"messages": [HumanMessage(message)]}, + config={"configurable": {"thread_id": thread_id}}, + ) + return result["messages"][-1].content + + +# First message — LangGraph stores the checkpoint in Valkey +r1 = await run_turn("user-42-thread-001", "What is the weather in Sofia?") +print(r1) +# "The weather in Sofia is 18°C and sunny." + +# Second message — graph resumes from the Valkey checkpoint +r2 = await run_turn("user-42-thread-001", "And in Berlin?") +print(r2) +# "The weather in Berlin is 14°C and cloudy. Earlier you asked about Sofia (18°C, sunny)." +``` + +The graph has full conversation history because `BetterDBSaver` loaded the checkpoint from Valkey before the second invocation. + +## Checkpoint Key Format + +Checkpoints are stored as plain JSON strings in the session namespace: + +``` +betterdb_ac:session:{thread_id}:checkpoint:{checkpoint_id} +betterdb_ac:session:{thread_id}:__checkpoint_latest +betterdb_ac:session:{thread_id}:writes:{checkpoint_id}|{task_id}|{channel}|{idx} +``` + +They live under the session namespace so `destroy_thread(thread_id)` cleans up both session state and checkpoints in one call: + +```python +await cache.session.destroy_thread("user-42-thread-001") +``` + +## Combined: LLM + Tool + Session + Checkpointing + +All four capabilities from a single `AgentCache` instance: + +```python +cache = AgentCache(AgentCacheOptions( + client=client, + tier_defaults={ + "llm": TierDefaults(ttl=3600), + "tool": TierDefaults(ttl=300), + "session": TierDefaults(ttl=86400), + }, +)) + +# LangGraph checkpointing +checkpointer = BetterDBSaver(cache=cache) + +# LangChain LLM caching +model = ChatOpenAI(model="gpt-4o-mini", cache=BetterDBLlmCache(cache=cache)) + +# Tool caching: use cache.tool.check() / cache.tool.store() in tool nodes +# Session state: use cache.session.set() / get() for per-thread context + +# When a conversation ends: +await cache.session.destroy_thread(thread_id) + +# Monitor cost savings: +stats = await cache.stats() +print(f"Saved ${stats.cost_saved_micros / 1_000_000:.4f} so far") +``` + +> **Note:** `active_sessions` in Prometheus metrics is approximate — it uses an in-memory counter that resets on process restart. diff --git a/content/betterdb-agent-cache-py/meta.json b/content/betterdb-agent-cache-py/meta.json new file mode 100644 index 0000000..096ee8a --- /dev/null +++ b/content/betterdb-agent-cache-py/meta.json @@ -0,0 +1,97 @@ +{ + "trackName": "betterdb-agent-cache (Python)", + "cookbooks": [ + { + "num": "01", + "source": "01-getting-started.md", + "output": "01-getting-started.html", + "title": "Getting Started with betterdb-agent-cache", + "h1": "Getting Started with betterdb-agent-cache", + "lead": "Three cache tiers - LLM responses, tool results, and session state - behind one connection. Works on vanilla Valkey 7+ with no modules.", + "breadcrumb": "Getting Started", + "difficulty": "Beginner", + "time": "15 min", + "language": "Python", + "next": { + "file": "02-llm-and-tool-cache.html", + "title": "02 - LLM & Tool Cache" + } + }, + { + "num": "02", + "source": "02-llm-and-tool-cache.md", + "output": "02-llm-and-tool-cache.html", + "title": "LLM & Tool Cache", + "h1": "LLM & Tool Cache", + "lead": "Cache exact LLM responses and tool call results. Track cost savings per model. Use tool_effectiveness() to tune per-tool TTLs automatically.", + "breadcrumb": "LLM & Tool Cache", + "difficulty": "Intermediate", + "time": "20 min", + "language": "Python", + "prev": { + "file": "01-getting-started.html", + "title": "01 - Getting Started" + }, + "next": { + "file": "03-session-store.html", + "title": "03 - Session Store" + } + }, + { + "num": "03", + "source": "03-session-store.md", + "output": "03-session-store.html", + "title": "Agent Session Store", + "h1": "Agent Session Store", + "lead": "Persist per-agent state with per-field TTL and sliding-window refresh. Store intent, intermediate results, and reasoning chains across invocations.", + "breadcrumb": "Session Store", + "difficulty": "Intermediate", + "time": "20 min", + "language": "Python", + "prev": { + "file": "02-llm-and-tool-cache.html", + "title": "02 - LLM & Tool Cache" + }, + "next": { + "file": "04-provider-adapters.html", + "title": "04 - Provider Adapters" + } + }, + { + "num": "04", + "source": "04-provider-adapters.md", + "output": "04-provider-adapters.html", + "title": "Provider Adapters", + "h1": "Provider Adapters", + "lead": "Drop-in caching for OpenAI Chat Completions, Responses API, Anthropic Messages, and LlamaIndex. One prepare_params() call translates provider params into the canonical cache key.", + "breadcrumb": "Provider Adapters", + "difficulty": "Intermediate", + "time": "25 min", + "language": "Python", + "prev": { + "file": "03-session-store.html", + "title": "03 - Session Store" + }, + "next": { + "file": "05-langgraph.html", + "title": "05 - LangGraph Checkpointing" + } + }, + { + "num": "05", + "source": "05-langgraph.md", + "output": "05-langgraph.html", + "title": "LangGraph Checkpointing", + "h1": "LangGraph Checkpointing", + "lead": "Persist LangGraph agent state on vanilla Valkey 7+ with BetterDBSaver - no Redis 8, no RedisJSON, no RediSearch modules required.", + "breadcrumb": "LangGraph Checkpointing", + "difficulty": "Advanced", + "time": "25 min", + "language": "Python", + "prev": { + "file": "04-provider-adapters.html", + "title": "04 - Provider Adapters" + } + } + ] +} diff --git a/content/betterdb-agent-cache/01-getting-started.md b/content/betterdb-agent-cache/01-getting-started.md index 892e4d9..32d9a50 100644 --- a/content/betterdb-agent-cache/01-getting-started.md +++ b/content/betterdb-agent-cache/01-getting-started.md @@ -46,10 +46,7 @@ const cache = new AgentCache({ tool: { ttl: 300 }, // Tool results: 5 minutes session: { ttl: 1800 }, // Session state: 30 minutes (sliding) }, - costTable: { - 'gpt-4o': { inputPer1k: 0.0025, outputPer1k: 0.010 }, - 'gpt-4o-mini': { inputPer1k: 0.00015, outputPer1k: 0.0006 }, - }, + // costTable is optional — 100+ models are covered by default }); ``` @@ -133,7 +130,7 @@ All keys are prefixed with the `name` option (default: `betterdb_ac`): | LLM cache | `betterdb_ac:llm:{sha256hash}` | | Tool cache | `betterdb_ac:tool:{toolName}:{sha256hash}` | | Session field | `betterdb_ac:session:{threadId}:{field}` | -| Stats hash | `betterdb_ac:stats` | +| Stats hash | `betterdb_ac:__stats` | To use a separate namespace per environment or application: diff --git a/content/betterdb-agent-cache/02-llm-and-tool-cache.md b/content/betterdb-agent-cache/02-llm-and-tool-cache.md index b25e57d..1d03f9d 100644 --- a/content/betterdb-agent-cache/02-llm-and-tool-cache.md +++ b/content/betterdb-agent-cache/02-llm-and-tool-cache.md @@ -27,12 +27,10 @@ import OpenAI from 'openai'; const client = new Valkey({ host: 'localhost', port: 6379 }); const openai = new OpenAI(); +// costTable is optional — 1,900+ models covered by default (see "Default Cost Table" below) const cache = new AgentCache({ client, tierDefaults: { llm: { ttl: 3600 } }, - costTable: { - 'gpt-4o-mini': { inputPer1k: 0.00015, outputPer1k: 0.0006 }, - }, }); async function cachedCompletion(params: { @@ -178,3 +176,67 @@ console.log(effectiveness); | `increase_ttl` | Hit rate > 80% and TTL < 1 hour | Extend TTL - results are stable and reused frequently | | `optimal` | Hit rate 40–80% | No change needed | | `decrease_ttl_or_disable` | Hit rate < 40% | Results change too fast or are rarely repeated - consider disabling cache for this tool | + +--- + +## Default Cost Table + +Starting with v0.4.0, `@betterdb/agent-cache` ships a bundled cost table sourced from [LiteLLM's model pricing data](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) and refreshed on every release. Cost tracking works out of the box for 1,900+ models — no `costTable` configuration required. + +```typescript +// No costTable needed — GPT-4o, Claude, Gemini, and many others are covered +const cache = new AgentCache({ + client, + tierDefaults: { llm: { ttl: 3600 } }, +}); +``` + +Pass `tokens` when storing and cost savings are tracked automatically: + +```typescript +await cache.llm.store(params, response, { + tokens: { input: usage.prompt_tokens, output: usage.completion_tokens }, +}); + +const stats = await cache.stats(); +console.log(`Cost saved: $${(stats.costSavedMicros / 1_000_000).toFixed(4)}`); +``` + +### Overriding Specific Models + +User-supplied `costTable` entries merge on top of the defaults. You only need to supply entries for models not in the bundled table or when you want to override a price: + +```typescript +const cache = new AgentCache({ + client, + costTable: { + 'my-fine-tuned-gpt4o': { inputPer1k: 0.005, outputPer1k: 0.015 }, + }, +}); +``` + +### Inspecting the Bundled Table + +```typescript +import { DEFAULT_COST_TABLE } from '@betterdb/agent-cache'; + +console.log(DEFAULT_COST_TABLE['gpt-4o-mini']); +// { inputPer1k: 0.00015, outputPer1k: 0.0006 } + +console.log(Object.keys(DEFAULT_COST_TABLE).length); +// 1900+ +``` + +### Disabling the Default Table + +Set `useDefaultCostTable: false` to opt out entirely and supply your own table: + +```typescript +const cache = new AgentCache({ + client, + useDefaultCostTable: false, + costTable: { + 'gpt-4o': { inputPer1k: 0.0025, outputPer1k: 0.010 }, + }, +}); +``` diff --git a/content/betterdb-agent-cache/05-provider-adapters.md b/content/betterdb-agent-cache/05-provider-adapters.md new file mode 100644 index 0000000..22a90aa --- /dev/null +++ b/content/betterdb-agent-cache/05-provider-adapters.md @@ -0,0 +1,297 @@ +Provider adapters translate the native params of each SDK into the canonical `LlmCacheParams` format before hashing. The pattern is the same across all four adapters: + +1. Call `prepareParams()` on the native SDK params +2. Pass the result to `cache.llm.check()` +3. On a miss, call the provider and build `ContentBlock[]` from the response +4. Store with `cache.llm.storeMultipart()` — this handles text, tool calls, and reasoning blocks together + +All four adapters support multi-modal content (images, audio, documents) and use the same pluggable binary normalizer for stable hashing. + +## Binary Normalizer + +The binary normalizer controls how binary content — images, audio, documents — is reduced to a stable string before being included in the cache key hash. + +The **default** is `passthrough`: the full base64 data string is included literally. This is zero-latency and correct, but means a re-encoded image byte-for-byte identical in content but with different base64 padding will produce a different hash. + +For production use, switch to `hashBase64` to hash the decoded bytes instead: + +```typescript +import { composeNormalizer, hashBase64 } from '@betterdb/agent-cache'; + +// Hash base64 image/audio/document bytes — O(n) in content size, no network calls +const normalizer = composeNormalizer({ base64: hashBase64 }); +``` + +Other built-in helpers: + +| Helper | Behavior | +|--------|----------| +| `hashBase64(data)` | SHA-256 of decoded bytes — stable across re-encodings | +| `hashBytes(buf)` | SHA-256 of a `Buffer` or `Uint8Array` | +| `hashUrl(url)` | Lowercases scheme+host, sorts query params, returns `"url:"` | +| `fetchAndHash(url)` | Fetches the URL and returns SHA-256 of the body — most stable, adds latency | +| `passthrough(ref)` | Returns the raw data as-is (default) | + +For per-kind control: + +```typescript +import { composeNormalizer, hashBase64, fetchAndHash } from '@betterdb/agent-cache'; + +const normalizer = composeNormalizer({ + base64: hashBase64, // Hash inline images/audio + byKind: { + document: async (ref) => { // Fetch and hash remote documents + if (ref.source.type === 'url') return fetchAndHash(ref.source.url); + return hashBase64((ref.source as { data: string }).data); + }, + }, +}); +``` + +--- + +## OpenAI Chat Completions + +```bash +npm install @betterdb/agent-cache openai iovalkey +``` + +```typescript +import Valkey from 'iovalkey'; +import OpenAI from 'openai'; +import { AgentCache, composeNormalizer, hashBase64 } from '@betterdb/agent-cache'; +import { prepareParams } from '@betterdb/agent-cache/openai'; +import type { ContentBlock, TextBlock, ToolCallBlock } from '@betterdb/agent-cache'; +import type { ChatCompletionCreateParamsNonStreaming } from 'openai/resources/chat/completions'; + +const valkey = new Valkey({ host: 'localhost', port: 6379 }); +const cache = new AgentCache({ client: valkey, tierDefaults: { llm: { ttl: 3600 } } }); +const normalizer = composeNormalizer({ base64: hashBase64 }); +const openai = new OpenAI(); + +async function chat(params: ChatCompletionCreateParamsNonStreaming): Promise { + // Translate OpenAI params → canonical LlmCacheParams + const cacheParams = await prepareParams(params, { normalizer }); + + const cached = await cache.llm.check(cacheParams); + if (cached.hit) return cached.response ?? ''; + + const response = await openai.chat.completions.create({ ...params, stream: false }); + const choice = response.choices[0]; + + // Build content blocks — text and tool calls both need to be stored + const blocks: ContentBlock[] = []; + if (choice.message.content) { + blocks.push({ type: 'text', text: choice.message.content } as TextBlock); + } + if (choice.message.tool_calls) { + for (const tc of choice.message.tool_calls) { + let args: unknown; + try { args = JSON.parse(tc.function.arguments || '{}'); } catch { args = { __raw: tc.function.arguments }; } + blocks.push({ type: 'tool_call', id: tc.id, name: tc.function.name, args } as ToolCallBlock); + } + } + + await cache.llm.storeMultipart(cacheParams, blocks, { + tokens: { + input: response.usage?.prompt_tokens ?? 0, + output: response.usage?.completion_tokens ?? 0, + }, + }); + + const textBlocks = blocks.filter((b): b is TextBlock => b.type === 'text'); + return textBlocks.map(b => b.text).join(''); +} +``` + +`prepareParams` handles all message roles: `system`, `developer`, `user`, `assistant`, `tool`, and legacy `function`. Multi-modal user messages (images, audio, file uploads) are normalized through the binary normalizer before hashing. + +### What Gets Hashed (OpenAI Chat) + +In addition to the base fields, `prepareParams` also maps these OpenAI params into the cache key: + +| OpenAI param | `LlmCacheParams` field | +|---|---| +| `tool_choice` | `toolChoice` | +| `seed` | `seed` | +| `stop` | `stop` | +| `response_format` | `responseFormat` | +| `prompt_cache_key` | `promptCacheKey` | + +--- + +## OpenAI Responses API + +```bash +npm install @betterdb/agent-cache openai iovalkey +``` + +```typescript +import Valkey from 'iovalkey'; +import OpenAI from 'openai'; +import { AgentCache, composeNormalizer, hashBase64 } from '@betterdb/agent-cache'; +import { prepareParams } from '@betterdb/agent-cache/openai-responses'; +import type { ContentBlock, TextBlock, ToolCallBlock, ReasoningBlock } from '@betterdb/agent-cache'; +import type { ResponseCreateParams } from 'openai/resources/responses/responses'; + +const valkey = new Valkey({ host: 'localhost', port: 6379 }); +const cache = new AgentCache({ client: valkey, tierDefaults: { llm: { ttl: 3600 } } }); +const normalizer = composeNormalizer({ base64: hashBase64 }); +const openai = new OpenAI(); + +async function respond(params: ResponseCreateParams): Promise { + const cacheParams = await prepareParams(params, { normalizer }); + + const cached = await cache.llm.check(cacheParams); + if (cached.hit) return cached.response ?? ''; + + const response = await openai.responses.create(params); + + const blocks: ContentBlock[] = []; + for (const item of response.output ?? []) { + if (item.type === 'message') { + for (const part of item.content ?? []) { + if (part.type === 'output_text') blocks.push({ type: 'text', text: part.text } as TextBlock); + } + } else if (item.type === 'reasoning') { + const text = (item.summary ?? []).filter((s: { type?: string }) => s.type === 'reasoning_text').map((s: { text: string }) => s.text).join(''); + blocks.push({ type: 'reasoning', text } as ReasoningBlock); + } else if (item.type === 'function_call') { + let args: unknown; + try { args = JSON.parse(item.arguments || '{}'); } catch { args = { __raw: item.arguments }; } + blocks.push({ type: 'tool_call', id: item.call_id, name: item.name, args } as ToolCallBlock); + } + } + + await cache.llm.storeMultipart(cacheParams, blocks, { + tokens: { + input: response.usage?.input_tokens ?? 0, + output: response.usage?.output_tokens ?? 0, + }, + }); + + const textBlocks = blocks.filter((b): b is TextBlock => b.type === 'text'); + return textBlocks.map(b => b.text).join(''); +} +``` + +`instructions` is prepended as a `system` message in the canonical format. The adapter handles `reasoning` items (extended thinking), `function_call` / `function_call_output` item types, and multi-modal `input_image` / `input_file` parts. `reasoning.effort` maps to `reasoningEffort` in the cache key. + +--- + +## Anthropic Messages + +```bash +npm install @betterdb/agent-cache @anthropic-ai/sdk iovalkey +``` + +```typescript +import Valkey from 'iovalkey'; +import Anthropic from '@anthropic-ai/sdk'; +import { AgentCache, composeNormalizer, hashBase64 } from '@betterdb/agent-cache'; +import { prepareParams } from '@betterdb/agent-cache/anthropic'; +import type { ContentBlock, TextBlock, ToolCallBlock, ReasoningBlock } from '@betterdb/agent-cache'; +import type { MessageCreateParamsNonStreaming } from '@anthropic-ai/sdk/resources'; + +const valkey = new Valkey({ host: 'localhost', port: 6379 }); +const cache = new AgentCache({ client: valkey, tierDefaults: { llm: { ttl: 3600 } } }); +const normalizer = composeNormalizer({ base64: hashBase64 }); +const anthropic = new Anthropic(); + +async function chat(params: MessageCreateParamsNonStreaming): Promise { + const cacheParams = await prepareParams(params, { normalizer }); + + const cached = await cache.llm.check(cacheParams); + if (cached.hit) return cached.response ?? ''; + + const response = await anthropic.messages.create(params); + + const blocks: ContentBlock[] = []; + for (const block of response.content) { + if (block.type === 'text') { + blocks.push({ type: 'text', text: block.text } as TextBlock); + } else if (block.type === 'tool_use') { + blocks.push({ type: 'tool_call', id: block.id, name: block.name, args: block.input } as ToolCallBlock); + } else if (block.type === 'thinking') { + blocks.push({ type: 'reasoning', text: block.thinking } as ReasoningBlock); + } + } + + await cache.llm.storeMultipart(cacheParams, blocks, { + tokens: { + input: response.usage.input_tokens, + output: response.usage.output_tokens, + }, + }); + + const textBlocks = blocks.filter((b): b is TextBlock => b.type === 'text'); + return textBlocks.map(b => b.text).join(''); +} +``` + +`params.system` is prepended as a `system` message. `tool_result` blocks in user messages are split into separate `tool` messages in the canonical format. `thinking` and `redacted_thinking` blocks map to `ReasoningBlock`. `stop_sequences` maps to `stop` in the cache key. + +--- + +## LlamaIndex + +```bash +npm install @betterdb/agent-cache @llamaindex/core @llamaindex/openai iovalkey +``` + +```typescript +import Valkey from 'iovalkey'; +import { OpenAI } from '@llamaindex/openai'; +import { AgentCache, composeNormalizer, hashBase64 } from '@betterdb/agent-cache'; +import { prepareParams } from '@betterdb/agent-cache/llamaindex'; +import type { ContentBlock, TextBlock } from '@betterdb/agent-cache'; +import type { ChatMessage } from '@llamaindex/core/llms'; + +const valkey = new Valkey({ host: 'localhost', port: 6379 }); +const cache = new AgentCache({ client: valkey, tierDefaults: { llm: { ttl: 3600 } } }); +const normalizer = composeNormalizer({ base64: hashBase64 }); +const llm = new OpenAI({ model: 'gpt-4o-mini' }); + +async function chat(messages: ChatMessage[]): Promise { + // model must be supplied explicitly — LlamaIndex messages don't carry it + const cacheParams = await prepareParams(messages, { + model: 'gpt-4o-mini', + normalizer, + temperature: 0, + }); + + const cached = await cache.llm.check(cacheParams); + if (cached.hit) return cached.response ?? ''; + + const response = await llm.chat({ messages }); + const text = response.message.content as string; + + const blocks: ContentBlock[] = [{ type: 'text', text } as TextBlock]; + const usage = (response.raw as { usage?: { prompt_tokens?: number; completion_tokens?: number } } | null)?.usage; + + await cache.llm.storeMultipart(cacheParams, blocks, { + tokens: { + input: usage?.prompt_tokens ?? 0, + output: usage?.completion_tokens ?? 0, + }, + }); + + return text; +} +``` + +The LlamaIndex adapter differs from the others in that the model name is not available from `ChatMessage` objects — you supply it via `opts.model`. The `memory` and `developer` roles are mapped to `system`. Tool calls are read from `message.options.toolCall` and tool results from `message.options.toolResult`. + +--- + +## Adapter Import Paths + +| Provider | Import path | +|---|---| +| OpenAI Chat Completions | `@betterdb/agent-cache/openai` | +| OpenAI Responses API | `@betterdb/agent-cache/openai-responses` | +| Anthropic Messages | `@betterdb/agent-cache/anthropic` | +| LlamaIndex | `@betterdb/agent-cache/llamaindex` | +| LangChain | `@betterdb/agent-cache/langchain` | +| Vercel AI SDK | `@betterdb/agent-cache/ai` | +| LangGraph | `@betterdb/agent-cache/langgraph` | diff --git a/content/betterdb-agent-cache/meta.json b/content/betterdb-agent-cache/meta.json index 1320963..fde59e0 100644 --- a/content/betterdb-agent-cache/meta.json +++ b/content/betterdb-agent-cache/meta.json @@ -71,6 +71,26 @@ "prev": { "file": "03-session-store.html", "title": "03 - Session Store" + }, + "next": { + "file": "05-provider-adapters.html", + "title": "05 - Provider Adapters" + } + }, + { + "num": "05", + "source": "05-provider-adapters.md", + "output": "05-provider-adapters.html", + "title": "Provider Adapters", + "h1": "Provider Adapters", + "lead": "Drop-in caching for OpenAI Chat Completions, Responses API, Anthropic Messages, and LlamaIndex. One prepareParams() call translates provider params into the canonical cache key.", + "breadcrumb": "Provider Adapters", + "difficulty": "Intermediate", + "time": "25 min", + "language": "TypeScript", + "prev": { + "file": "04-langgraph.html", + "title": "04 - LangGraph Checkpointing" } } ] diff --git a/cookbooks/betterdb-agent-cache-py/01-getting-started.html b/cookbooks/betterdb-agent-cache-py/01-getting-started.html new file mode 100644 index 0000000..fd888a0 --- /dev/null +++ b/cookbooks/betterdb-agent-cache-py/01-getting-started.html @@ -0,0 +1,189 @@ + + + + + +01 - Getting Started with betterdb-agent-cache - Valkey for AI + + + + + + + +
+
Home → betterdb-agent-cache (Python) → 01 Getting Started
+
BeginnerPython~15 min
+

Getting Started with betterdb-agent-cache

+

Three cache tiers - LLM responses, tool results, and session state - behind one connection. Works on vanilla Valkey 7+ with no modules.

+ +

betterdb-agent-cache is a multi-tier exact-match cache for AI agent workloads backed by Valkey. It combines three cache tiers behind one connection:

+
    +
  • LLM tier — caches full LLM responses by exact match on model + messages + params
    • +
  • Tool tier — caches tool/function call results by tool name and argument hash
    • +
  • Session tier — key-value store for agent state with per-field TTL
    • +
+

Works on vanilla Valkey 7+, Amazon ElastiCache, Google Cloud Memorystore, and Amazon MemoryDB with no modules required.

+

Prerequisites

+
    +
  • Valkey 7+ (no modules needed)
    • +
  • Python 3.11+
    • +
+

Step 1: Start Valkey

+
docker run -d --name valkey -p 6379:6379 valkey/valkey:latest
+
+

Step 2: Install

+
pip install betterdb-agent-cache
+
+

For provider-specific adapters, install the relevant extra:

+
pip install "betterdb-agent-cache[openai]"       # OpenAI
+pip install "betterdb-agent-cache[anthropic]"    # Anthropic
+pip install "betterdb-agent-cache[langchain]"    # LangChain
+pip install "betterdb-agent-cache[langgraph]"    # LangGraph
+pip install "betterdb-agent-cache[llamaindex]"   # LlamaIndex
+
+

Step 3: Create an AgentCache

+
import asyncio
+import valkey.asyncio as valkey_client
+from betterdb_agent_cache import AgentCache, TierDefaults
+from betterdb_agent_cache.types import AgentCacheOptions
+
+async def main():
+    client = valkey_client.Valkey(host="localhost", port=6379)
+
+    cache = AgentCache(AgentCacheOptions(
+        client=client,
+        tier_defaults={
+            "llm":     TierDefaults(ttl=3600),   # LLM responses: 1 hour
+            "tool":    TierDefaults(ttl=300),     # Tool results: 5 minutes
+            "session": TierDefaults(ttl=1800),    # Session state: 30 min (sliding)
+        },
+        # cost_table is optional — 1,900+ models covered by default
+    ))
+
+

No initialize() call needed — all tiers use plain Valkey commands (SET, GET, HINCRBY) with no index creation.

+

Step 4: LLM Tier

+
params = {
+    "model": "gpt-4o-mini",
+    "messages": [{"role": "user", "content": "What is Valkey?"}],
+    "temperature": 0,
+}
+
+# Check for a cached response
+llm_result = await cache.llm.check(params)
+
+if llm_result.hit:
+    print("LLM cache HIT:", llm_result.response)
+else:
+    print("LLM cache MISS → calling LLM...")
+    response = await call_llm(params)  # your LLM call
+
+    from betterdb_agent_cache.types import LlmStoreOptions
+    await cache.llm.store(params, response, LlmStoreOptions(
+        tokens={"input": 14, "output": 62},  # enables cost tracking
+    ))
+
+

The cache key is a SHA-256 hash of the canonicalized params. {"messages": [...], "model": "gpt-4o"} and {"model": "gpt-4o", "messages": [...]} produce the same key.

+

Step 5: Tool Tier

+
tool_name = "get_weather"
+tool_args = {"city": "Sofia", "units": "metric"}
+
+tool_result = await cache.tool.check(tool_name, tool_args)
+
+if tool_result.hit:
+    print("Tool cache HIT:", tool_result.response)
+else:
+    print("Tool cache MISS → calling API...")
+    import json
+    data = await get_weather(tool_args)  # your tool call
+
+    from betterdb_agent_cache.types import ToolStoreOptions
+    await cache.tool.store(tool_name, tool_args, json.dumps(data), ToolStoreOptions(
+        cost=0.001,  # API call cost in dollars, for tracking
+    ))
+
+

Tool args are serialized with recursively sorted keys before hashing, so {"city": "Sofia", "units": "metric"} and {"units": "metric", "city": "Sofia"} hit the same cache entry.

+

Step 6: Session Tier

+
thread_id = "user-42-session-1"
+
+# Store agent state fields
+await cache.session.set(thread_id, "last_intent", "book_flight")
+await cache.session.set(thread_id, "destination", "Sofia")
+await cache.session.set(thread_id, "departure_date", "2026-06-01")
+
+# Retrieve a field (also refreshes its TTL)
+intent = await cache.session.get(thread_id, "last_intent")
+# "book_flight"
+
+# Get all fields for a thread
+state = await cache.session.get_all(thread_id)
+# {"last_intent": "book_flight", "destination": "Sofia", "departure_date": "2026-06-01"}
+
+

Each get() refreshes the TTL on that field (sliding window). The session stays alive as long as the agent is actively using it.

+

Key Formats

+

All keys are prefixed with the name option (default: betterdb_ac):

+ + + + + + + + + + + + + + + + + + + + + + + +
TierKey Pattern
LLM cachebetterdb_ac:llm:{sha256hash}
Tool cachebetterdb_ac:tool:{toolName}:{sha256hash}
Session fieldbetterdb_ac:session:{threadId}:{field}
Stats hashbetterdb_ac:__stats
+
cache = AgentCache(AgentCacheOptions(
+    client=client,
+    name="myapp_prod",  # all keys prefixed with 'myapp_prod:'
+))
+
+

Cleanup

+
# Destroy all state for a thread
+await cache.session.destroy_thread(thread_id)
+
+# Invalidate all LLM cache entries for a model
+await cache.llm.invalidate_by_model("gpt-4o-mini")
+
+# Close the client when done
+await cache.shutdown()
+await client.aclose()
+
+ + +
+
+ +
+
+ + + + + + \ No newline at end of file diff --git a/cookbooks/betterdb-agent-cache-py/02-llm-and-tool-cache.html b/cookbooks/betterdb-agent-cache-py/02-llm-and-tool-cache.html new file mode 100644 index 0000000..b8c8859 --- /dev/null +++ b/cookbooks/betterdb-agent-cache-py/02-llm-and-tool-cache.html @@ -0,0 +1,267 @@ + + + + + +02 - LLM & Tool Cache - Valkey for AI + + + + + + + +
+
Home → betterdb-agent-cache (Python) → 02 LLM & Tool Cache
+
IntermediatePython~20 min
+

LLM & Tool Cache

+

Cache exact LLM responses and tool call results. Track cost savings per model. Use tool_effectiveness() to tune per-tool TTLs automatically.

+ +

LLM Cache Tier

+

The LLM cache stores full LLM responses by exact match on all parameters that affect the output.

+

What Gets Hashed

+

The cache key is a SHA-256 hash of these fields (with recursively sorted dict keys for determinism):

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldNotes
model'gpt-4o', 'claude-opus-4-6', etc.
messagesFull message list including roles and content
temperatureNone is treated as unset
top_pNone is treated as unset
max_tokensNone is treated as unset
toolsTool definitions if present
tool_choiceNone is treated as unset
seedNone is treated as unset
stopNone is treated as unset
response_formatNone is treated as unset
reasoning_effortFor models supporting extended thinking
prompt_cache_keyPass-through for provider-level prompt caching
+

Check and Store

+
import valkey.asyncio as valkey_client
+from openai import AsyncOpenAI
+from betterdb_agent_cache import AgentCache, TierDefaults
+from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions
+
+client = valkey_client.Valkey(host="localhost", port=6379)
+openai = AsyncOpenAI()
+
+# cost_table is optional — 1,900+ models covered by default (see "Default Cost Table" below)
+cache = AgentCache(AgentCacheOptions(
+    client=client,
+    tier_defaults={"llm": TierDefaults(ttl=3600)},
+))
+
+async def cached_completion(params: dict) -> str:
+    cached = await cache.llm.check(params)
+    if cached.hit:
+        return cached.response or ""
+
+    response = await openai.chat.completions.create(**params)
+    text = response.choices[0].message.content or ""
+    usage = response.usage
+
+    await cache.llm.store(params, text, LlmStoreOptions(
+        tokens={
+            "input":  usage.prompt_tokens     if usage else 0,
+            "output": usage.completion_tokens if usage else 0,
+        },
+    ))
+    return text
+
+

Token counts in store() enable cost savings tracking via the cost table. Without tokens, cost tracking is skipped for that entry.

+

Invalidating by Model

+
deleted = await cache.llm.invalidate_by_model("gpt-4o-mini")
+print(f"Deleted {deleted} entries")
+
+
+

Tool Cache Tier

+

The tool cache stores tool/function call results. Valuable for expensive external API calls — weather, geocoding, database queries, web search — that return stable results for the same inputs.

+

Check and Store

+
import json
+from betterdb_agent_cache.types import ToolStoreOptions
+
+async def cached_tool(name: str, args: dict) -> dict:
+    cached = await cache.tool.check(name, args)
+    if cached.hit:
+        return json.loads(cached.response)
+
+    result = await call_tool(name, args)  # your tool implementation
+
+    await cache.tool.store(name, args, json.dumps(result), ToolStoreOptions(
+        ttl=300,    # per-call TTL override
+        cost=0.005, # API call cost in dollars
+    ))
+    return result
+
+

Per-Tool TTL Policies

+
from betterdb_agent_cache.types import ToolPolicy
+
+# Weather data: short TTL, changes frequently
+await cache.tool.set_policy("get_weather",    ToolPolicy(ttl=300))    # 5 min
+
+# Stock prices: very short TTL
+await cache.tool.set_policy("get_stock_price", ToolPolicy(ttl=60))    # 1 min
+
+# Geocoding: long TTL, addresses don't change
+await cache.tool.set_policy("geocode_address", ToolPolicy(ttl=86400)) # 24h
+
+# Web search: medium TTL
+await cache.tool.set_policy("web_search",     ToolPolicy(ttl=3600))   # 1h
+
+

TTL precedence: per-call ttl > tool policy > tier_defaults["tool"].ttl > default_ttl.

+

Invalidation

+
# Invalidate all results for a specific tool
+deleted = await cache.tool.invalidate_by_tool("get_weather")
+print(f"Deleted {deleted} weather cache entries")
+
+# Invalidate one specific call
+existed = await cache.tool.invalidate("get_weather", {"city": "Sofia"})
+
+
+

Cost Tracking and Stats

+

Aggregate Stats

+
stats = await cache.stats()
+print(stats.llm.hits, stats.llm.misses, f"{stats.llm.hit_rate:.0%}")
+# 150 50 75%
+
+print(f"Cost saved: ${stats.cost_saved_micros / 1_000_000:.4f}")
+# Cost saved: $12.5000
+
+# Per-tool breakdown
+for tool_name, tool_stats in stats.per_tool.items():
+    print(f"{tool_name}: {tool_stats.hit_rate:.0%} hit rate")
+
+

Cost savings are stored as microdollars (integers) to avoid floating-point precision issues. Divide by 1_000_000 to get dollars.

+

Tool Effectiveness Recommendations

+
effectiveness = await cache.tool_effectiveness()
+for entry in effectiveness:
+    print(f"{entry.tool}: {entry.hit_rate:.0%} hit rate — {entry.recommendation}")
+# get_weather:   85% hit rate — increase_ttl
+# web_search:    62% hit rate — optimal
+# rare_api_call:  8% hit rate — decrease_ttl_or_disable
+
+ + + + + + + + + + + + + + + + + + + + + + + +
RecommendationConditionAction
increase_ttlHit rate > 80% and TTL < 1 hourResults are stable and reused frequently — extend TTL
optimalHit rate 40–80%No change needed
decrease_ttl_or_disableHit rate < 40%Results change too fast or rarely repeated
+
+

Default Cost Table

+

betterdb-agent-cache ships a bundled cost table sourced from LiteLLM's model pricing data and refreshed on every release. Cost tracking works out of the box for 1,900+ models — no cost_table configuration required.

+
# No cost_table needed — GPT-4o, Claude, Gemini, and 1,900+ others are covered
+cache = AgentCache(AgentCacheOptions(client=client))
+
+

Overriding Specific Models

+

User-supplied cost_table entries merge on top of the defaults:

+
from betterdb_agent_cache import ModelCost
+
+cache = AgentCache(AgentCacheOptions(
+    client=client,
+    cost_table={
+        "my-fine-tuned-gpt4o": ModelCost(input_per_1k=0.005, output_per_1k=0.015),
+    },
+))
+
+

Inspecting the Bundled Table

+
from betterdb_agent_cache import DEFAULT_COST_TABLE
+
+print(DEFAULT_COST_TABLE.get("gpt-4o-mini"))
+# ModelCost(input_per_1k=0.00015, output_per_1k=0.0006)
+
+print(len(DEFAULT_COST_TABLE))
+# 1900+
+
+

Disabling the Default Table

+
cache = AgentCache(AgentCacheOptions(
+    client=client,
+    use_default_cost_table=False,
+    cost_table={
+        "gpt-4o": ModelCost(input_per_1k=0.0025, output_per_1k=0.010),
+    },
+))
+
+ + +
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/cookbooks/betterdb-agent-cache-py/03-session-store.html b/cookbooks/betterdb-agent-cache-py/03-session-store.html new file mode 100644 index 0000000..b6c6e5c --- /dev/null +++ b/cookbooks/betterdb-agent-cache-py/03-session-store.html @@ -0,0 +1,182 @@ + + + + + +03 - Agent Session Store - Valkey for AI + + + + + + + +
+
Home → betterdb-agent-cache (Python) → 03 Session Store
+
IntermediatePython~20 min
+

Agent Session Store

+

Persist per-agent state with per-field TTL and sliding-window refresh. Store intent, intermediate results, and reasoning chains across invocations.

+ +

The session tier is a per-thread key-value store for agent state. Designed for the patterns that come up in multi-step, multi-turn agents: storing user intent, intermediate reasoning, extracted entities, and tool call context between invocations.

+

How It Works

+

Each session field is stored as an individual Valkey key:

+
betterdb_ac:session:{thread_id}:{field}
+
+

Individual keys allow per-field TTL — different fields can have different expiry times — and get() automatically refreshes the TTL on each read (sliding window). The trade-off is that get_all() and destroy_thread() require a SCAN + pipeline rather than a single HGETALL or DEL.

+

Basic Operations

+
import valkey.asyncio as valkey_client
+from betterdb_agent_cache import AgentCache, TierDefaults
+from betterdb_agent_cache.types import AgentCacheOptions
+
+client = valkey_client.Valkey(host="localhost", port=6379)
+cache = AgentCache(AgentCacheOptions(
+    client=client,
+    tier_defaults={"session": TierDefaults(ttl=1800)},  # 30 min sliding window
+))
+
+thread_id = "user-42-thread-001"
+
+# Write state fields
+await cache.session.set(thread_id, "intent", "book_flight")
+await cache.session.set(thread_id, "destination", "Sofia")
+await cache.session.set(thread_id, "departure", "2026-06-01")
+
+# Read a field (also refreshes its TTL)
+intent = await cache.session.get(thread_id, "intent")
+# "book_flight"
+
+# Returns None for fields that don't exist or have expired
+missing = await cache.session.get(thread_id, "return_date")
+# None
+
+

Getting All Fields

+
state = await cache.session.get_all(thread_id)
+print(state)
+# {"intent": "book_flight", "destination": "Sofia", "departure": "2026-06-01"}
+
+

get_all() uses SCAN to find all keys for the thread and fetches them in a pipeline. Designed for sessions with dozens of fields — not thousands.

+

Refreshing TTLs

+
# On each user interaction, refresh all session fields to keep the session alive
+await cache.session.touch(thread_id)
+
+

touch() extends the TTL on every field for the thread. Call it at the start of each agent invocation to prevent partial expiry where some fields outlive others.

+

Deleting State

+
# Delete a single field
+await cache.session.delete(thread_id, "departure")
+
+# Destroy the entire thread
+deleted = await cache.session.destroy_thread(thread_id)
+print(f"Deleted {deleted} keys")
+
+

Multi-Step Agent Example

+

A booking agent that accumulates state across multiple tool calls:

+
import json
+
+async def run_booking_agent(thread_id: str, user_message: str) -> dict:
+    # Load existing state
+    state = await cache.session.get_all(thread_id)
+
+    # Extract intent
+    if "book" in user_message and "flight" in user_message:
+        await cache.session.set(thread_id, "intent", "book_flight")
+
+    # Extract destination
+    import re
+    dest_match = re.search(r"to (\w+)", user_message, re.IGNORECASE)
+    if dest_match:
+        await cache.session.set(thread_id, "destination", dest_match.group(1))
+
+    # Extract departure date
+    date_match = re.search(r"on (\d{4}-\d{2}-\d{2})", user_message)
+    if date_match:
+        await cache.session.set(thread_id, "departure", date_match.group(1))
+
+    # Refresh all TTLs so the session doesn't partially expire
+    await cache.session.touch(thread_id)
+
+    # Read current state
+    current_state = await cache.session.get_all(thread_id)
+
+    # Search for flights only when we have enough context
+    if (current_state.get("destination") and current_state.get("departure")
+            and not current_state.get("search_results")):
+        args = {
+            "destination": current_state["destination"],
+            "date": current_state["departure"],
+        }
+        cached = await cache.tool.check("search_flights", args)
+        if not cached.hit:
+            flight_data = await search_flights(current_state)
+            await cache.tool.store("search_flights", args, json.dumps(flight_data))
+            await cache.session.set(thread_id, "search_results", json.dumps(flight_data))
+        else:
+            await cache.session.set(thread_id, "search_results", cached.response)
+
+    return await cache.session.get_all(thread_id)
+
+

Session State Patterns

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PatternImplementation
User intent trackingset(thread, "intent", value) on each turn
Entity accumulationOne field per entity: set(thread, "city", "Sofia")
Conversation stageset(thread, "stage", "collecting_dates")
Intermediate resultsStore JSON: set(thread, "search_results", json.dumps(data))
Error recoveryset(thread, "last_failed_step", "payment")
Session keepalivetouch(thread) at start of each invocation
+

Per-Field TTL Override

+
# Short-lived field: search results that go stale quickly
+await cache.session.set(thread_id, "flight_prices", json.dumps(prices), ttl=300)
+
+# Long-lived field: user preferences that persist across sessions
+await cache.session.set(thread_id, "preferred_seat", "aisle", ttl=86400 * 30)
+
+

The per-call ttl overrides tier_defaults["session"].ttl for that specific field.

+ + +
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/cookbooks/betterdb-agent-cache-py/04-provider-adapters.html b/cookbooks/betterdb-agent-cache-py/04-provider-adapters.html new file mode 100644 index 0000000..2c44d4b --- /dev/null +++ b/cookbooks/betterdb-agent-cache-py/04-provider-adapters.html @@ -0,0 +1,314 @@ + + + + + +04 - Provider Adapters - Valkey for AI + + + + + + + +
+
Home → betterdb-agent-cache (Python) → 04 Provider Adapters
+
IntermediatePython~25 min
+

Provider Adapters

+

Drop-in caching for OpenAI Chat Completions, Responses API, Anthropic Messages, and LlamaIndex. One prepare_params() call translates provider params into the canonical cache key.

+ +

Provider adapters translate the native params of each SDK into the canonical LlmCacheParams format before hashing. The pattern is the same across all four adapters:

+
    +
  1. Call prepare_params() on the native SDK params dict
    2. +
  2. Pass the result to cache.llm.check()
    3. +
  3. On a miss, call the provider and build a list of content block dicts from the response
    4. +
  4. Store with cache.llm.store_multipart() — handles text, tool calls, and reasoning blocks together
    5. +
+

Binary Normalizer

+

The binary normalizer controls how binary content — images, audio, documents — is reduced to a stable string before being included in the cache key hash.

+

The default is passthrough: the full base64 data string is included literally. Zero-latency and correct, but a re-encoded image that is byte-for-byte identical but with different base64 padding will produce a different hash.

+

For production use, switch to hash_base64:

+
from betterdb_agent_cache import compose_normalizer, hash_base64
+
+# Hash base64 image/audio/document bytes — O(n) in content size, no network calls
+normalizer = compose_normalizer({"base64": hash_base64})
+
+

Other built-in helpers:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
HelperBehavior
hash_base64(data)SHA-256 of decoded bytes — stable across re-encodings
hash_bytes(data)SHA-256 of raw bytes
hash_url(url)Lowercases scheme+host, sorts query params, returns "url:<normalised>"
fetch_and_hash(url)Fetches the URL (requires aiohttp) and returns SHA-256 of the body
passthrough(ref)Returns the raw data as-is (default)
+
+

OpenAI Chat Completions

+
pip install "betterdb-agent-cache[openai]"
+
+
import json
+import valkey.asyncio as valkey_client
+from openai import AsyncOpenAI
+from betterdb_agent_cache import AgentCache, TierDefaults, compose_normalizer, hash_base64
+from betterdb_agent_cache.adapters.openai import OpenAIPrepareOptions, prepare_params
+from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions
+
+client = valkey_client.Valkey(host="localhost", port=6379)
+cache = AgentCache(AgentCacheOptions(
+    client=client,
+    tier_defaults={"llm": TierDefaults(ttl=3600)},
+))
+normalizer = compose_normalizer({"base64": hash_base64})
+opts = OpenAIPrepareOptions(normalizer=normalizer)
+openai = AsyncOpenAI()
+
+async def chat(params: dict) -> str:
+    # Translate OpenAI params → canonical LlmCacheParams
+    cache_params = await prepare_params(params, opts)
+
+    cached = await cache.llm.check(cache_params)
+    if cached.hit:
+        return cached.response or ""
+
+    response = await openai.chat.completions.create(**params, stream=False)
+    choice = response.choices[0]
+
+    blocks = []
+    if choice.message.content:
+        blocks.append({"type": "text", "text": choice.message.content})
+    for tc in choice.message.tool_calls or []:
+        try:
+            args = json.loads(tc.function.arguments or "{}")
+        except json.JSONDecodeError:
+            args = {"__raw": tc.function.arguments}
+        blocks.append({"type": "tool_call", "id": tc.id, "name": tc.function.name, "args": args})
+
+    await cache.llm.store_multipart(cache_params, blocks, LlmStoreOptions(tokens={
+        "input":  response.usage.prompt_tokens     if response.usage else 0,
+        "output": response.usage.completion_tokens if response.usage else 0,
+    }))
+
+    return " ".join(b["text"] for b in blocks if b.get("type") == "text")
+
+

prepare_params handles all message roles: system, developer, user, assistant, tool, and legacy function. Multi-modal user messages (images, audio, file uploads) are normalized through the binary normalizer before hashing.

+
+

OpenAI Responses API

+
pip install "betterdb-agent-cache[openai]"
+
+
import valkey.asyncio as valkey_client
+from openai import AsyncOpenAI
+from betterdb_agent_cache import AgentCache, TierDefaults, compose_normalizer, hash_base64
+from betterdb_agent_cache.adapters.openai_responses import OpenAIResponsesPrepareOptions, prepare_params
+from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions
+
+client = valkey_client.Valkey(host="localhost", port=6379)
+cache = AgentCache(AgentCacheOptions(
+    client=client,
+    tier_defaults={"llm": TierDefaults(ttl=3600)},
+))
+normalizer = compose_normalizer({"base64": hash_base64})
+opts = OpenAIResponsesPrepareOptions(normalizer=normalizer)
+openai = AsyncOpenAI()
+
+async def respond(params: dict) -> str:
+    cache_params = await prepare_params(params, opts)
+
+    cached = await cache.llm.check(cache_params)
+    if cached.hit:
+        return cached.response or ""
+
+    response = await openai.responses.create(**params)
+
+    blocks = []
+    for item in response.output or []:
+        if item.type == "message":
+            for part in item.content or []:
+                if part.type == "output_text":
+                    blocks.append({"type": "text", "text": part.text})
+        elif item.type == "reasoning":
+            text = " ".join(s.text for s in (item.summary or []) if s.type == "reasoning_text")
+            blocks.append({"type": "reasoning", "text": text})
+        elif item.type == "function_call":
+            import json
+            try:
+                args = json.loads(item.arguments or "{}")
+            except json.JSONDecodeError:
+                args = {"__raw": item.arguments}
+            blocks.append({"type": "tool_call", "id": item.call_id, "name": item.name, "args": args})
+
+    await cache.llm.store_multipart(cache_params, blocks, LlmStoreOptions(tokens={
+        "input":  response.usage.input_tokens  if response.usage else 0,
+        "output": response.usage.output_tokens if response.usage else 0,
+    }))
+
+    return " ".join(b["text"] for b in blocks if b.get("type") == "text")
+
+

instructions is prepended as a system message. The adapter handles reasoning items, function_call/function_call_output item types, and multi-modal input parts. reasoning.effort maps to reasoning_effort in the cache key.

+
+

Anthropic Messages

+
pip install "betterdb-agent-cache[anthropic]"
+
+
import valkey.asyncio as valkey_client
+import anthropic as sdk
+from betterdb_agent_cache import AgentCache, TierDefaults, compose_normalizer, hash_base64
+from betterdb_agent_cache.adapters.anthropic import AnthropicPrepareOptions, prepare_params
+from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions
+
+client = valkey_client.Valkey(host="localhost", port=6379)
+cache = AgentCache(AgentCacheOptions(
+    client=client,
+    tier_defaults={"llm": TierDefaults(ttl=3600)},
+))
+normalizer = compose_normalizer({"base64": hash_base64})
+opts = AnthropicPrepareOptions(normalizer=normalizer)
+anthropic = sdk.AsyncAnthropic()
+
+async def chat(params: dict) -> str:
+    cache_params = await prepare_params(params, opts)
+
+    cached = await cache.llm.check(cache_params)
+    if cached.hit:
+        return cached.response or ""
+
+    response = await anthropic.messages.create(**params)
+
+    blocks = []
+    for block in response.content:
+        if block.type == "text":
+            blocks.append({"type": "text", "text": block.text})
+        elif block.type == "tool_use":
+            blocks.append({"type": "tool_call", "id": block.id, "name": block.name, "args": block.input})
+        elif block.type == "thinking":
+            blocks.append({"type": "reasoning", "text": block.thinking,
+                           "opaqueSignature": getattr(block, "signature", None)})
+
+    await cache.llm.store_multipart(cache_params, blocks, LlmStoreOptions(tokens={
+        "input":  response.usage.input_tokens,
+        "output": response.usage.output_tokens,
+    }))
+
+    return " ".join(b["text"] for b in blocks if b.get("type") == "text")
+
+

params["system"] is prepended as a system message. tool_result blocks in user messages are split into separate tool messages. thinking and redacted_thinking blocks map to reasoning blocks. stop_sequences maps to stop in the cache key.

+
+

LlamaIndex

+
pip install "betterdb-agent-cache[llamaindex]"
+
+
import valkey.asyncio as valkey_client
+from llama_index.llms.openai import OpenAI
+from betterdb_agent_cache import AgentCache, TierDefaults, compose_normalizer, hash_base64
+from betterdb_agent_cache.adapters.llamaindex import LlamaIndexPrepareOptions, prepare_params
+from betterdb_agent_cache.types import AgentCacheOptions, LlmStoreOptions
+
+client = valkey_client.Valkey(host="localhost", port=6379)
+cache = AgentCache(AgentCacheOptions(
+    client=client,
+    tier_defaults={"llm": TierDefaults(ttl=3600)},
+))
+normalizer = compose_normalizer({"base64": hash_base64})
+llm = OpenAI(model="gpt-4o-mini")
+
+async def chat(messages: list) -> str:
+    # model must be supplied explicitly — LlamaIndex messages don't carry it
+    opts = LlamaIndexPrepareOptions(
+        model="gpt-4o-mini",
+        normalizer=normalizer,
+        temperature=0,
+    )
+    cache_params = await prepare_params(messages, opts)
+
+    cached = await cache.llm.check(cache_params)
+    if cached.hit:
+        return cached.response or ""
+
+    response = await llm.achat(messages)
+    text = str(response.message.content)
+    blocks = [{"type": "text", "text": text}]
+
+    usage = getattr(response.raw, "usage", None)
+    await cache.llm.store_multipart(cache_params, blocks, LlmStoreOptions(tokens={
+        "input":  getattr(usage, "prompt_tokens",     0) if usage else 0,
+        "output": getattr(usage, "completion_tokens", 0) if usage else 0,
+    }))
+    return text
+
+

The model name is not available from LlamaIndex message objects — supply it via LlamaIndexPrepareOptions.model. The memory and developer roles are mapped to system. Tool calls are read from message.options["toolCall"] and tool results from message.options["toolResult"].

+
+

Adapter Import Paths

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProviderImport path
OpenAI Chat Completionsbetterdb_agent_cache.adapters.openai
OpenAI Responses APIbetterdb_agent_cache.adapters.openai_responses
Anthropic Messagesbetterdb_agent_cache.adapters.anthropic
LlamaIndexbetterdb_agent_cache.adapters.llamaindex
LangChainbetterdb_agent_cache.adapters.langchain
LangGraphbetterdb_agent_cache.adapters.langgraph
+ + +
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/cookbooks/betterdb-agent-cache-py/05-langgraph.html b/cookbooks/betterdb-agent-cache-py/05-langgraph.html new file mode 100644 index 0000000..8df18e7 --- /dev/null +++ b/cookbooks/betterdb-agent-cache-py/05-langgraph.html @@ -0,0 +1,256 @@ + + + + + +05 - LangGraph Checkpointing - Valkey for AI + + + + + + + +
+
Home → betterdb-agent-cache (Python) → 05 LangGraph Checkpointing
+
AdvancedPython~25 min
+

LangGraph Checkpointing

+

Persist LangGraph agent state on vanilla Valkey 7+ with BetterDBSaver - no Redis 8, no RedisJSON, no RediSearch modules required.

+ +

BetterDBSaver is a LangGraph checkpoint saver backed by betterdb-agent-cache. It stores graph state on vanilla Valkey 7+ with no modules required.

+

Why BetterDBSaver vs langgraph-checkpoint-redis

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
BetterDBSaverlanggraph-checkpoint-redis
Valkey support✅ Valkey 7+❌ Redis 8+ only
Module requirements✅ None❌ RedisJSON + RediSearch
Works on ElastiCache✅ Any tier❌ Requires Redis 8 + modules
Works on Memorystore✅ Any tier❌ Requires Redis 8 + modules
Checkpoint storagePlain JSON stringsRedisJSON path operations
Filtered listingSCAN + parseO(1) RediSearch index
Async✅ Fully async✅
Best forAny Valkey/Redis deploymentRedis 8+ with modules, millions of checkpoints
+

The trade-off: alist() with filtering uses SCAN. For typical deployments with hundreds of checkpoints per thread this is fast. BetterDBSaver is async-only — the sync get_tuple(), list(), and put() methods raise RuntimeError.

+

Prerequisites

+
pip install "betterdb-agent-cache[langgraph]" langchain-openai
+
+

Step 1: Create the Checkpointer

+
import valkey.asyncio as valkey_client
+from betterdb_agent_cache import AgentCache, TierDefaults
+from betterdb_agent_cache.adapters.langgraph import BetterDBSaver
+from betterdb_agent_cache.types import AgentCacheOptions
+
+client = valkey_client.Valkey(host="localhost", port=6379)
+
+cache = AgentCache(AgentCacheOptions(
+    client=client,
+    tier_defaults={
+        "session": TierDefaults(ttl=86400),  # 24h session state
+    },
+))
+
+checkpointer = BetterDBSaver(cache=cache)
+
+

Step 2: Build a Graph with Checkpointing

+
import json
+import random
+from typing import Annotated
+from typing_extensions import TypedDict
+
+from langchain_openai import ChatOpenAI
+from langchain_core.messages import HumanMessage, AIMessage, ToolMessage
+from langgraph.graph import StateGraph, END
+from langgraph.graph.message import add_messages
+
+from betterdb_agent_cache.adapters.langchain import BetterDBLlmCache
+
+
+class State(TypedDict):
+    messages: Annotated[list, add_messages]
+
+
+# Pass BetterDBLlmCache to LangChain so LLM responses are cached automatically
+model = ChatOpenAI(
+    model="gpt-4o-mini",
+    temperature=0,
+    cache=BetterDBLlmCache(cache=cache),
+)
+
+_TOOLS = [{"type": "function", "function": {
+    "name": "get_weather",
+    "description": "Get current weather for a city",
+    "parameters": {
+        "type": "object",
+        "properties": {"city": {"type": "string"}},
+        "required": ["city"],
+    },
+}}]
+model_with_tools = model.bind_tools(_TOOLS)
+
+
+async def get_weather(city: str) -> str:
+    """Tool with its own cache tier."""
+    cached = await cache.tool.check("get_weather", {"city": city})
+    if cached.hit:
+        return cached.response or ""
+    result = json.dumps({
+        "city": city,
+        "temperature": round(15 + random.random() * 15),
+        "condition": random.choice(["sunny", "cloudy", "rainy"]),
+    })
+    await cache.tool.store("get_weather", {"city": city}, result)
+    return result
+
+
+async def call_model(state: State) -> dict:
+    response = await model_with_tools.ainvoke(state["messages"])
+    return {"messages": [response]}
+
+
+async def call_tools(state: State) -> dict:
+    last: AIMessage = state["messages"][-1]
+    results = []
+    for tc in getattr(last, "tool_calls", []) or []:
+        if tc["name"] == "get_weather":
+            result = await get_weather(tc["args"].get("city", ""))
+            results.append(ToolMessage(content=result, tool_call_id=tc["id"]))
+    return {"messages": results}
+
+
+def should_continue(state: State) -> str:
+    last: AIMessage = state["messages"][-1]
+    return "tools" if getattr(last, "tool_calls", None) else END
+
+
+graph = (
+    StateGraph(State)
+    .add_node("agent", call_model)
+    .add_node("tools", call_tools)
+    .add_edge("__start__", "agent")
+    .add_conditional_edges("agent", should_continue)
+    .add_edge("tools", "agent")
+    .compile(checkpointer=checkpointer)  # attach BetterDBSaver here
+)
+
+

Step 3: Run the Graph Across Multiple Turns

+
async def run_turn(thread_id: str, message: str) -> str:
+    result = await graph.ainvoke(
+        {"messages": [HumanMessage(message)]},
+        config={"configurable": {"thread_id": thread_id}},
+    )
+    return result["messages"][-1].content
+
+
+# First message — LangGraph stores the checkpoint in Valkey
+r1 = await run_turn("user-42-thread-001", "What is the weather in Sofia?")
+print(r1)
+# "The weather in Sofia is 18°C and sunny."
+
+# Second message — graph resumes from the Valkey checkpoint
+r2 = await run_turn("user-42-thread-001", "And in Berlin?")
+print(r2)
+# "The weather in Berlin is 14°C and cloudy. Earlier you asked about Sofia (18°C, sunny)."
+
+

The graph has full conversation history because BetterDBSaver loaded the checkpoint from Valkey before the second invocation.

+

Checkpoint Key Format

+

Checkpoints are stored as plain JSON strings in the session namespace:

+
betterdb_ac:session:{thread_id}:checkpoint:{checkpoint_id}
+betterdb_ac:session:{thread_id}:__checkpoint_latest
+betterdb_ac:session:{thread_id}:writes:{checkpoint_id}|{task_id}|{channel}|{idx}
+
+

They live under the session namespace so destroy_thread(thread_id) cleans up both session state and checkpoints in one call:

+
await cache.session.destroy_thread("user-42-thread-001")
+
+

Combined: LLM + Tool + Session + Checkpointing

+

All four capabilities from a single AgentCache instance:

+
cache = AgentCache(AgentCacheOptions(
+    client=client,
+    tier_defaults={
+        "llm":     TierDefaults(ttl=3600),
+        "tool":    TierDefaults(ttl=300),
+        "session": TierDefaults(ttl=86400),
+    },
+))
+
+# LangGraph checkpointing
+checkpointer = BetterDBSaver(cache=cache)
+
+# LangChain LLM caching
+model = ChatOpenAI(model="gpt-4o-mini", cache=BetterDBLlmCache(cache=cache))
+
+# Tool caching: use cache.tool.check() / cache.tool.store() in tool nodes
+# Session state: use cache.session.set() / get() for per-thread context
+
+# When a conversation ends:
+await cache.session.destroy_thread(thread_id)
+
+# Monitor cost savings:
+stats = await cache.stats()
+print(f"Saved ${stats.cost_saved_micros / 1_000_000:.4f} so far")
+
+
+

Note: active_sessions in Prometheus metrics is approximate — it uses an in-memory counter that resets on process restart.

+
+ + +
+ +
+
+
+ + + + + + \ No newline at end of file diff --git a/cookbooks/betterdb-agent-cache/01-getting-started.html b/cookbooks/betterdb-agent-cache/01-getting-started.html index 235f06e..6fa3340 100644 --- a/cookbooks/betterdb-agent-cache/01-getting-started.html +++ b/cookbooks/betterdb-agent-cache/01-getting-started.html @@ -65,10 +65,7 @@

Step 3: Create an AgentCache

tool: { ttl: 300 }, // Tool results: 5 minutes session: { ttl: 1800 }, // Session state: 30 minutes (sliding) }, - costTable: { - 'gpt-4o': { inputPer1k: 0.0025, outputPer1k: 0.010 }, - 'gpt-4o-mini': { inputPer1k: 0.00015, outputPer1k: 0.0006 }, - }, + // costTable is optional — 100+ models are covered by default });

No initialize() call needed - all three tiers use plain Valkey commands (SET, GET, HINCRBY) with no index creation.

@@ -152,7 +149,7 @@

Key Formats

Stats hash -betterdb_ac:stats +betterdb_ac:__stats

To use a separate namespace per environment or application:

diff --git a/cookbooks/betterdb-agent-cache/02-llm-and-tool-cache.html b/cookbooks/betterdb-agent-cache/02-llm-and-tool-cache.html index 131d1b0..a4def7b 100644 --- a/cookbooks/betterdb-agent-cache/02-llm-and-tool-cache.html +++ b/cookbooks/betterdb-agent-cache/02-llm-and-tool-cache.html @@ -74,12 +74,10 @@

Check and Store

const client = new Valkey({ host: 'localhost', port: 6379 }); const openai = new OpenAI(); +// costTable is optional — 1,900+ models covered by default (see "Default Cost Table" below) const cache = new AgentCache({ client, tierDefaults: { llm: { ttl: 3600 } }, - costTable: { - 'gpt-4o-mini': { inputPer1k: 0.00015, outputPer1k: 0.0006 }, - }, }); async function cachedCompletion(params: { @@ -214,6 +212,51 @@

Tool Effectiveness Recommendations

Results change too fast or are rarely repeated - consider disabling cache for this tool +
+

Default Cost Table

+

Starting with v0.4.0, @betterdb/agent-cache ships a bundled cost table sourced from LiteLLM's model pricing data and refreshed on every release. Cost tracking works out of the box for 1,900+ models — no costTable configuration required.

+
// No costTable needed — GPT-4o, Claude, Gemini, and many others are covered
+const cache = new AgentCache({
+  client,
+  tierDefaults: { llm: { ttl: 3600 } },
+});
+
+

Pass tokens when storing and cost savings are tracked automatically:

+
await cache.llm.store(params, response, {
+  tokens: { input: usage.prompt_tokens, output: usage.completion_tokens },
+});
+
+const stats = await cache.stats();
+console.log(`Cost saved: $${(stats.costSavedMicros / 1_000_000).toFixed(4)}`);
+
+

Overriding Specific Models

+

User-supplied costTable entries merge on top of the defaults. You only need to supply entries for models not in the bundled table or when you want to override a price:

+
const cache = new AgentCache({
+  client,
+  costTable: {
+    'my-fine-tuned-gpt4o': { inputPer1k: 0.005, outputPer1k: 0.015 },
+  },
+});
+
+

Inspecting the Bundled Table

+
import { DEFAULT_COST_TABLE } from '@betterdb/agent-cache';
+
+console.log(DEFAULT_COST_TABLE['gpt-4o-mini']);
+// { inputPer1k: 0.00015, outputPer1k: 0.0006 }
+
+console.log(Object.keys(DEFAULT_COST_TABLE).length);
+// 1900+
+
+

Disabling the Default Table

+

Set useDefaultCostTable: false to opt out entirely and supply your own table:

+
const cache = new AgentCache({
+  client,
+  useDefaultCostTable: false,
+  costTable: {
+    'gpt-4o': { inputPer1k: 0.0025, outputPer1k: 0.010 },
+  },
+});
+
diff --git a/cookbooks/betterdb-agent-cache/04-langgraph.html b/cookbooks/betterdb-agent-cache/04-langgraph.html index da73ccf..2b9f961 100644 --- a/cookbooks/betterdb-agent-cache/04-langgraph.html +++ b/cookbooks/betterdb-agent-cache/04-langgraph.html @@ -215,7 +215,7 @@

Combined: LLM + Tool + Session + Checkpointing

-
+
diff --git a/cookbooks/betterdb-agent-cache/05-provider-adapters.html b/cookbooks/betterdb-agent-cache/05-provider-adapters.html new file mode 100644 index 0000000..0b96e22 --- /dev/null +++ b/cookbooks/betterdb-agent-cache/05-provider-adapters.html @@ -0,0 +1,365 @@ + + + + + +05 - Provider Adapters - Valkey for AI + + + + + + + +
+
Home → @betterdb/agent-cache → 05 Provider Adapters
+
IntermediateTypeScript~25 min
+

Provider Adapters

+

Drop-in caching for OpenAI Chat Completions, Responses API, Anthropic Messages, and LlamaIndex. One prepareParams() call translates provider params into the canonical cache key.

+ +

Provider adapters translate the native params of each SDK into the canonical LlmCacheParams format before hashing. The pattern is the same across all four adapters:

+
    +
  1. Call prepareParams() on the native SDK params
    2. +
  2. Pass the result to cache.llm.check()
    3. +
  3. On a miss, call the provider and build ContentBlock[] from the response
    4. +
  4. Store with cache.llm.storeMultipart() — this handles text, tool calls, and reasoning blocks together
    5. +
+

All four adapters support multi-modal content (images, audio, documents) and use the same pluggable binary normalizer for stable hashing.

+

Binary Normalizer

+

The binary normalizer controls how binary content — images, audio, documents — is reduced to a stable string before being included in the cache key hash.

+

The default is passthrough: the full base64 data string is included literally. This is zero-latency and correct, but means a re-encoded image byte-for-byte identical in content but with different base64 padding will produce a different hash.

+

For production use, switch to hashBase64 to hash the decoded bytes instead:

+
import { composeNormalizer, hashBase64 } from '@betterdb/agent-cache';
+
+// Hash base64 image/audio/document bytes — O(n) in content size, no network calls
+const normalizer = composeNormalizer({ base64: hashBase64 });
+
+

Other built-in helpers:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
HelperBehavior
hashBase64(data)SHA-256 of decoded bytes — stable across re-encodings
hashBytes(buf)SHA-256 of a Buffer or Uint8Array
hashUrl(url)Lowercases scheme+host, sorts query params, returns "url:<normalized>"
fetchAndHash(url)Fetches the URL and returns SHA-256 of the body — most stable, adds latency
passthrough(ref)Returns the raw data as-is (default)
+

For per-kind control:

+
import { composeNormalizer, hashBase64, fetchAndHash } from '@betterdb/agent-cache';
+
+const normalizer = composeNormalizer({
+  base64: hashBase64,           // Hash inline images/audio
+  byKind: {
+    document: async (ref) => {  // Fetch and hash remote documents
+      if (ref.source.type === 'url') return fetchAndHash(ref.source.url);
+      return hashBase64((ref.source as { data: string }).data);
+    },
+  },
+});
+
+
+

OpenAI Chat Completions

+
npm install @betterdb/agent-cache openai iovalkey
+
+
import Valkey from 'iovalkey';
+import OpenAI from 'openai';
+import { AgentCache, composeNormalizer, hashBase64 } from '@betterdb/agent-cache';
+import { prepareParams } from '@betterdb/agent-cache/openai';
+import type { ContentBlock, TextBlock, ToolCallBlock } from '@betterdb/agent-cache';
+import type { ChatCompletionCreateParamsNonStreaming } from 'openai/resources/chat/completions';
+
+const valkey = new Valkey({ host: 'localhost', port: 6379 });
+const cache = new AgentCache({ client: valkey, tierDefaults: { llm: { ttl: 3600 } } });
+const normalizer = composeNormalizer({ base64: hashBase64 });
+const openai = new OpenAI();
+
+async function chat(params: ChatCompletionCreateParamsNonStreaming): Promise<string> {
+  // Translate OpenAI params → canonical LlmCacheParams
+  const cacheParams = await prepareParams(params, { normalizer });
+
+  const cached = await cache.llm.check(cacheParams);
+  if (cached.hit) return cached.response ?? '';
+
+  const response = await openai.chat.completions.create({ ...params, stream: false });
+  const choice = response.choices[0];
+
+  // Build content blocks — text and tool calls both need to be stored
+  const blocks: ContentBlock[] = [];
+  if (choice.message.content) {
+    blocks.push({ type: 'text', text: choice.message.content } as TextBlock);
+  }
+  if (choice.message.tool_calls) {
+    for (const tc of choice.message.tool_calls) {
+      let args: unknown;
+      try { args = JSON.parse(tc.function.arguments || '{}'); } catch { args = { __raw: tc.function.arguments }; }
+      blocks.push({ type: 'tool_call', id: tc.id, name: tc.function.name, args } as ToolCallBlock);
+    }
+  }
+
+  await cache.llm.storeMultipart(cacheParams, blocks, {
+    tokens: {
+      input: response.usage?.prompt_tokens ?? 0,
+      output: response.usage?.completion_tokens ?? 0,
+    },
+  });
+
+  const textBlocks = blocks.filter((b): b is TextBlock => b.type === 'text');
+  return textBlocks.map(b => b.text).join('');
+}
+
+

prepareParams handles all message roles: system, developer, user, assistant, tool, and legacy function. Multi-modal user messages (images, audio, file uploads) are normalized through the binary normalizer before hashing.

+

What Gets Hashed (OpenAI Chat)

+

In addition to the base fields, prepareParams also maps these OpenAI params into the cache key:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
OpenAI paramLlmCacheParams field
tool_choicetoolChoice
seedseed
stopstop
response_formatresponseFormat
prompt_cache_keypromptCacheKey
+
+

OpenAI Responses API

+
npm install @betterdb/agent-cache openai iovalkey
+
+
import Valkey from 'iovalkey';
+import OpenAI from 'openai';
+import { AgentCache, composeNormalizer, hashBase64 } from '@betterdb/agent-cache';
+import { prepareParams } from '@betterdb/agent-cache/openai-responses';
+import type { ContentBlock, TextBlock, ToolCallBlock, ReasoningBlock } from '@betterdb/agent-cache';
+import type { ResponseCreateParams } from 'openai/resources/responses/responses';
+
+const valkey = new Valkey({ host: 'localhost', port: 6379 });
+const cache = new AgentCache({ client: valkey, tierDefaults: { llm: { ttl: 3600 } } });
+const normalizer = composeNormalizer({ base64: hashBase64 });
+const openai = new OpenAI();
+
+async function respond(params: ResponseCreateParams): Promise<string> {
+  const cacheParams = await prepareParams(params, { normalizer });
+
+  const cached = await cache.llm.check(cacheParams);
+  if (cached.hit) return cached.response ?? '';
+
+  const response = await openai.responses.create(params);
+
+  const blocks: ContentBlock[] = [];
+  for (const item of response.output ?? []) {
+    if (item.type === 'message') {
+      for (const part of item.content ?? []) {
+        if (part.type === 'output_text') blocks.push({ type: 'text', text: part.text } as TextBlock);
+      }
+    } else if (item.type === 'reasoning') {
+      const text = (item.summary ?? []).filter((s: { type?: string }) => s.type === 'reasoning_text').map((s: { text: string }) => s.text).join('');
+      blocks.push({ type: 'reasoning', text } as ReasoningBlock);
+    } else if (item.type === 'function_call') {
+      let args: unknown;
+      try { args = JSON.parse(item.arguments || '{}'); } catch { args = { __raw: item.arguments }; }
+      blocks.push({ type: 'tool_call', id: item.call_id, name: item.name, args } as ToolCallBlock);
+    }
+  }
+
+  await cache.llm.storeMultipart(cacheParams, blocks, {
+    tokens: {
+      input: response.usage?.input_tokens ?? 0,
+      output: response.usage?.output_tokens ?? 0,
+    },
+  });
+
+  const textBlocks = blocks.filter((b): b is TextBlock => b.type === 'text');
+  return textBlocks.map(b => b.text).join('');
+}
+
+

instructions is prepended as a system message in the canonical format. The adapter handles reasoning items (extended thinking), function_call / function_call_output item types, and multi-modal input_image / input_file parts. reasoning.effort maps to reasoningEffort in the cache key.

+
+

Anthropic Messages

+
npm install @betterdb/agent-cache @anthropic-ai/sdk iovalkey
+
+
import Valkey from 'iovalkey';
+import Anthropic from '@anthropic-ai/sdk';
+import { AgentCache, composeNormalizer, hashBase64 } from '@betterdb/agent-cache';
+import { prepareParams } from '@betterdb/agent-cache/anthropic';
+import type { ContentBlock, TextBlock, ToolCallBlock, ReasoningBlock } from '@betterdb/agent-cache';
+import type { MessageCreateParamsNonStreaming } from '@anthropic-ai/sdk/resources';
+
+const valkey = new Valkey({ host: 'localhost', port: 6379 });
+const cache = new AgentCache({ client: valkey, tierDefaults: { llm: { ttl: 3600 } } });
+const normalizer = composeNormalizer({ base64: hashBase64 });
+const anthropic = new Anthropic();
+
+async function chat(params: MessageCreateParamsNonStreaming): Promise<string> {
+  const cacheParams = await prepareParams(params, { normalizer });
+
+  const cached = await cache.llm.check(cacheParams);
+  if (cached.hit) return cached.response ?? '';
+
+  const response = await anthropic.messages.create(params);
+
+  const blocks: ContentBlock[] = [];
+  for (const block of response.content) {
+    if (block.type === 'text') {
+      blocks.push({ type: 'text', text: block.text } as TextBlock);
+    } else if (block.type === 'tool_use') {
+      blocks.push({ type: 'tool_call', id: block.id, name: block.name, args: block.input } as ToolCallBlock);
+    } else if (block.type === 'thinking') {
+      blocks.push({ type: 'reasoning', text: block.thinking } as ReasoningBlock);
+    }
+  }
+
+  await cache.llm.storeMultipart(cacheParams, blocks, {
+    tokens: {
+      input: response.usage.input_tokens,
+      output: response.usage.output_tokens,
+    },
+  });
+
+  const textBlocks = blocks.filter((b): b is TextBlock => b.type === 'text');
+  return textBlocks.map(b => b.text).join('');
+}
+
+

params.system is prepended as a system message. tool_result blocks in user messages are split into separate tool messages in the canonical format. thinking and redacted_thinking blocks map to ReasoningBlock. stop_sequences maps to stop in the cache key.

+
+

LlamaIndex

+
npm install @betterdb/agent-cache @llamaindex/core @llamaindex/openai iovalkey
+
+
import Valkey from 'iovalkey';
+import { OpenAI } from '@llamaindex/openai';
+import { AgentCache, composeNormalizer, hashBase64 } from '@betterdb/agent-cache';
+import { prepareParams } from '@betterdb/agent-cache/llamaindex';
+import type { ContentBlock, TextBlock } from '@betterdb/agent-cache';
+import type { ChatMessage } from '@llamaindex/core/llms';
+
+const valkey = new Valkey({ host: 'localhost', port: 6379 });
+const cache = new AgentCache({ client: valkey, tierDefaults: { llm: { ttl: 3600 } } });
+const normalizer = composeNormalizer({ base64: hashBase64 });
+const llm = new OpenAI({ model: 'gpt-4o-mini' });
+
+async function chat(messages: ChatMessage[]): Promise<string> {
+  // model must be supplied explicitly — LlamaIndex messages don't carry it
+  const cacheParams = await prepareParams(messages, {
+    model: 'gpt-4o-mini',
+    normalizer,
+    temperature: 0,
+  });
+
+  const cached = await cache.llm.check(cacheParams);
+  if (cached.hit) return cached.response ?? '';
+
+  const response = await llm.chat({ messages });
+  const text = response.message.content as string;
+
+  const blocks: ContentBlock[] = [{ type: 'text', text } as TextBlock];
+  const usage = (response.raw as { usage?: { prompt_tokens?: number; completion_tokens?: number } } | null)?.usage;
+
+  await cache.llm.storeMultipart(cacheParams, blocks, {
+    tokens: {
+      input: usage?.prompt_tokens ?? 0,
+      output: usage?.completion_tokens ?? 0,
+    },
+  });
+
+  return text;
+}
+
+

The LlamaIndex adapter differs from the others in that the model name is not available from ChatMessage objects — you supply it via opts.model. The memory and developer roles are mapped to system. Tool calls are read from message.options.toolCall and tool results from message.options.toolResult.

+
+

Adapter Import Paths

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProviderImport path
OpenAI Chat Completions@betterdb/agent-cache/openai
OpenAI Responses API@betterdb/agent-cache/openai-responses
Anthropic Messages@betterdb/agent-cache/anthropic
LlamaIndex@betterdb/agent-cache/llamaindex
LangChain@betterdb/agent-cache/langchain
Vercel AI SDK@betterdb/agent-cache/ai
LangGraph@betterdb/agent-cache/langgraph
+ + +
+ +
+
+
+ + + + + + \ No newline at end of file diff --git a/cookbooks/betterdb/index.html b/cookbooks/betterdb/index.html index 360cd3f..238861b 100644 --- a/cookbooks/betterdb/index.html +++ b/cookbooks/betterdb/index.html @@ -322,8 +322,8 @@
Home → Frameworks → BetterDB

BetterDB + Valkey

-

Two npm packages for Valkey-backed LLM caching. Semantic caching with vector search, and - multi-tier agent caching with LangGraph support. TypeScript, built-in OTel + Prometheus.

+

TypeScript and Python packages for Valkey-backed LLM caching. Semantic caching with vector search, and + multi-tier agent caching with LangGraph support. Built-in OTel + Prometheus.

@@ -428,6 +428,75 @@

LangGraph Checkpointing

class="tag-cyan">TypeScript
+ +
05
+
+

Provider Adapters

+

Drop-in caching for OpenAI Chat Completions, Responses API, Anthropic Messages, and LlamaIndex — plus the pluggable binary normalizer for multi-modal content.

+
Intermediate~25 minTypeScript
+
+ +
+ + + +
+
+
+ + + + + +
+
+

betterdb-agent-cache (Python)

+

Python port · feature-identical to the TypeScript package · async/await · 1,900+ model cost table

+
+
+
+ +
01
+
+

Getting Started

+

LLM cache, tool cache, and session store on vanilla Valkey 7+ — three tiers, one connection.

+
Beginner~15 minPython
+
+ + +
02
+
+

LLM & Tool Cache

+

Exact-match LLM caching with cost tracking. Per-tool TTL policies and tool_effectiveness() recommendations.

+
Intermediate~20 minPython
+
+ + +
03
+
+

Agent Session Store

+

Per-field TTL, sliding-window refresh, and multi-step agent state patterns.

+
Intermediate~20 minPython
+
+ + +
04
+
+

Provider Adapters

+

Drop-in caching for OpenAI Chat, Responses API, Anthropic, and LlamaIndex with the pluggable binary normalizer.

+
Intermediate~25 minPython
+
+ + +
05
+
+

LangGraph Checkpointing

+

BetterDBSaver on vanilla Valkey 7+ - no Redis 8, no RedisJSON, no RediSearch required.

+
Advanced~25 minPython
+
+
diff --git a/index.html b/index.html index bc5ef28..bd0ed4d 100644 --- a/index.html +++ b/index.html @@ -196,7 +196,7 @@

Haystack

BetterDB

Two packages for Valkey-backed LLM caching. @betterdb/semantic-cache — Valkey-native vector similarity cache. @betterdb/agent-cache — multi-tier LLM, tool, and session cache with LangGraph support.

semantic-cacheagent-cacheLangGraph
-
7 CookbooksTypeScript
+
13 CookbooksTypeScriptPython