Add update_genai_prices flag to logfire.configure()#1960
Open
JonathanTsen wants to merge 5 commits into
Open
Conversation
Adds a new opt-in flag (default False) that, when enabled, starts a daemon thread refreshing the genai-prices model pricing snapshot from upstream every hour. This keeps operation.cost attributes correct for newly released models (e.g. gemini-3.5-flash) without requiring a package upgrade. The thread is gracefully a no-op (with a one-shot warning) when genai-prices is not installed, and is properly torn down on reconfigure or interpreter exit. Also exposed via the LOGFIRE_UPDATE_GENAI_PRICES=1 environment variable.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
The `genai-prices` package depends on `pydantic.Tag`, which was introduced in
pydantic 2.5. On the `pydantic 2.4` CI matrix entry, importing
`genai_prices.update_prices` raises `AttributeError: module 'pydantic._migration'
has no attribute 'Tag'` — not `ImportError`. Three fixes:
- `config.py`: broaden the lazy-import guard from `except ImportError` to also
catch generic import-time exceptions, emitting an informative warning instead
of crashing `logfire.configure()` over a non-critical feature.
- `tests/test_configure.py`: skip the 4 `patch('genai_prices.update_prices...')`
tests via `pytest.importorskip` when the module cannot be imported in the
current environment (e.g. the pydantic 2.4 matrix). The graceful-when-not-
installed test stays unconditional because it exercises exactly that path.
- `docs/integrations/llms/pydanticai.md`: mark the new example block as
`skip-run="true" skip-reason="external-connection"`, matching the other LLM
integration docs which also need a real network connection at runtime.
`pytest.importorskip()` only catches `ImportError`. When importing `genai_prices.update_prices` on the pydantic 2.4 CI matrix, the failure mode is `AttributeError: module 'pydantic._migration' has no attribute 'Tag'`, which escapes the guard and surfaces as a test failure. Replace the four `pytest.importorskip` calls with a local helper that calls `importlib.import_module` inside a broad `try/except` and `pytest.skip`s on any exception during import. The helper sits near the affected tests so the intent stays local.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The `if not emscripten` guard around `_maybe_start_genai_prices_updater()` had no test exercising the emscripten=True path on non-pyodide CI, so the branch coverage check failed by exactly one branch (1430->exit, 99.99% total). Add `test_update_genai_prices_skipped_under_emscripten` which monkeypatches `platform_is_emscripten` to return True and verifies the updater is not instantiated even when the flag is enabled.
Reviewers shouldn't need to dig through the `genai-prices` repo to confirm where the model costs come from. Reference the two upstream sources of truth directly in the "Keep model pricing up to date" section.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Pydantic AI uses
genai-pricesto populate theoperation.costattribute on LLM spans. The catalogue mirrors the upstream provider pricing pages — for example Google Gemini API pricing and OpenAI API pricing — and is shipped as a snapshot inside the package. So when a brand-new model is released (e.g.gemini-3.5-flash)calc_priceraisesLookupError, which pydantic-ai then swallows silently inpydantic_ai/_instrumentation.py. The result is that newly released models show up in Logfire with correct token counts but nooperation.cost, until a newgenai-pricesrelease is published, pulled in transitively, and reinstalled.genai-pricesitself already exposes an opt-inUpdatePricesbackground updater that refreshes the catalogue from the upstream repo. This PR makes that updater available through a single Logfire configure flag.Change
A new opt-in kwarg
update_genai_prices: bool | None = Noneonlogfire.configure().False(no behaviour change for existing users).True, Logfire instantiatesgenai_prices.update_prices.UpdatePricesand starts the daemon thread that polls upstream every hour.atexit.genai-pricesis not installed (or fails to import — e.g. on pydantic < 2.5 environments missingpydantic.Tag), the flag emits a one-shotUserWarningand is a no-op.LOGFIRE_UPDATE_GENAI_PRICES=1environment variable, following the existingConfigParampattern.Usage
Or via env var:
Pricing source of truth
The model prices used to compute
operation.costare maintained inpydantic/genai-pricesand mirror the official provider catalogues:This PR doesn't change how the prices are derived — it only lets Logfire users pick up upstream updates without waiting for a
genai-pricesrelease.Files changed
logfire/_internal/config_params.py— newUPDATE_GENAI_PRICESConfigParamand registration inCONFIG_PARAMS.logfire/_internal/config.py— kwarg threaded throughconfigure(),_LogfireConfigData,_load_configuration,LogfireConfig.__init__/configure; lifecycle in_initializevia two helpers (_maybe_start_genai_prices_updater/_stop_genai_prices_updater); new field surfaced inemit_configuration_span.logfire-api/logfire_api/_internal/config.pyiandconfig_params.pyi— regenerated stubs.docs/integrations/llms/pydanticai.md— new "Keep model pricing up to date" section.tests/test_configure.py— 6 new tests covering default-off, enabled-starts, env var, gracefulImportError, reconfigure-stops-old, and emscripten-skip; existingtest_configuration_span_emitted_when_opted_insnapshot updated to include the new field.Tests
All new tests mock
genai_prices.update_prices.UpdatePrices, so no network or real threads are needed in CI.