Problem
Self-hosted users on Anthropic's API are getting hit pretty hard on input tokens. CORE sends a literal F ton of context on every single request: system prompt, memory graph, conversation history. And for those of us on lower API tiers, that burns through the per-minute token budget fast and causes the chat to freeze mid-response.
The proposed fix:
Please implement Anthropic prompt caching
(https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching)
The repeated stuff like system prompt, memory context, persona could be cached between requests. That repeated content costs 90% less when cached and doesn't count against the rate limit. New memories and fresh conversation turns would stay uncached as normal.
It's a win-win: users aren't getting rate limited or paying through the nose, and output quality doesn't change at all since only the static boilerplate gets cached.
References
Problem
Self-hosted users on Anthropic's API are getting hit pretty hard on input tokens. CORE sends a literal F ton of context on every single request: system prompt, memory graph, conversation history. And for those of us on lower API tiers, that burns through the per-minute token budget fast and causes the chat to freeze mid-response.
The proposed fix:
Please implement Anthropic prompt caching
(https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching)
The repeated stuff like system prompt, memory context, persona could be cached between requests. That repeated content costs 90% less when cached and doesn't count against the rate limit. New memories and fresh conversation turns would stay uncached as normal.
It's a win-win: users aren't getting rate limited or paying through the nose, and output quality doesn't change at all since only the static boilerplate gets cached.
References