Summary
The gateway's cache key currently does not account for all generation parameters that influence an AI model's output.
As a result, two requests with different inference settings may generate the same cache key and incorrectly return a cached response produced using different parameters.
Current Behavior
The cache key generation includes only a subset of the request information.
There is also an existing TODO in the implementation:
// TODO: Consider accepting a struct with all OpenRouter parameters
This indicates that support for all generation parameters has not yet been implemented.
Why this is a problem
Many LLM generation parameters directly affect the output, including:
- temperature
- top_p
- max_tokens
- frequency_penalty
- presence_penalty
- stop sequences
- seed (if supported)
If these values are omitted from the cache key, requests such as
Request A
{
"prompt": "Explain transformers",
"temperature": 0.0
}and
Request B
{
"prompt": "Explain transformers",
"temperature": 1.0
}may resolve to the same cache entry even though they are expected to produce different responses.
This can lead to:
- incorrect cached responses
- confusing user behavior
- difficult-to-debug cache inconsistencies
- reduced cache correctness
Expected Behavior
The cache key should uniquely represent every request attribute that can change the generated response.
Possible approaches include:
- hashing the complete normalized request payload
- using a request struct containing every generation parameter
- canonical JSON serialization before hashing
Possible Solution
Refactor cache key generation to use a canonical representation of the complete inference request instead of only selected fields.
This ensures that semantically different requests always generate different cache keys while identical requests continue to benefit from cache reuse.
Summary
The gateway's cache key currently does not account for all generation parameters that influence an AI model's output.
As a result, two requests with different inference settings may generate the same cache key and incorrectly return a cached response produced using different parameters.
Current Behavior
The cache key generation includes only a subset of the request information.
There is also an existing TODO in the implementation:
// TODO: Consider accepting a struct with all OpenRouter parametersThis indicates that support for all generation parameters has not yet been implemented.
Why this is a problem
Many LLM generation parameters directly affect the output, including:
If these values are omitted from the cache key, requests such as
Request A
{ "prompt": "Explain transformers", "temperature": 0.0 }and
Request B
{ "prompt": "Explain transformers", "temperature": 1.0 }may resolve to the same cache entry even though they are expected to produce different responses.
This can lead to:
Expected Behavior
The cache key should uniquely represent every request attribute that can change the generated response.
Possible approaches include:
Possible Solution
Refactor cache key generation to use a canonical representation of the complete inference request instead of only selected fields.
This ensures that semantically different requests always generate different cache keys while identical requests continue to benefit from cache reuse.