Problem
When the LLM is served through a gateway or a model nao has no pricing data for (e.g. LiteLLM, self-hosted models), nao test reports costs as $0. Token counts are captured correctly, but with no per-token price nao cannot compute a cost, so every run shows zero.
This makes the cost dimension of the eval framework unusable. Reported by a user benchmarking models on quality, latency and cost via nao test, who is also demoing the output to clients, where $0 reads as broken.
Expected behavior
Users can supply their own input/output token prices per model in configuration. When nao has no built-in pricing for a model, it uses the user-supplied values to compute cost from the token counts it already collects, so nao test and cost reporting show real numbers instead of zero.
Customer ID
0f2a033d-54ef-4a64-be25-365f43eb0529
Problem
When the LLM is served through a gateway or a model nao has no pricing data for (e.g. LiteLLM, self-hosted models),
nao testreports costs as $0. Token counts are captured correctly, but with no per-token price nao cannot compute a cost, so every run shows zero.This makes the cost dimension of the eval framework unusable. Reported by a user benchmarking models on quality, latency and cost via
nao test, who is also demoing the output to clients, where $0 reads as broken.Expected behavior
Users can supply their own input/output token prices per model in configuration. When nao has no built-in pricing for a model, it uses the user-supplied values to compute cost from the token counts it already collects, so
nao testand cost reporting show real numbers instead of zero.Customer ID
0f2a033d-54ef-4a64-be25-365f43eb0529