feat: PostHog LLM observability ($ai_generation traces) - #320
Open
farhat-is-coding wants to merge 1 commit into
Open
feat: PostHog LLM observability ($ai_generation traces)#320farhat-is-coding wants to merge 1 commit into
farhat-is-coding wants to merge 1 commit into
Conversation
Emit one $ai_generation event per LLM provider call at the single LLMService._execute_llm_completion choke point, covering all providers (OpenAI, Claude, Gemini, Llama, custom/LiteLLM), streaming and structured paths, and post-tool follow-up calls. - LLMObservabilityService: process-wide PostHog client, strict no-op when POSTHOG_API_KEY is unset, fire-and-forget (never raises into chat path) - GenerationTracker: per-call accumulator for chunks, usage, first-token time; finish() runs from finally so aborted streams still capture - LLMGenerationContext / LLMGenerationRecord DTOs per rules.md - Trace identity: message id -> $ai_trace_id, conversation id -> $ai_session_id, user id -> distinct_id - Detects both raised exceptions and provider 'Error:' sentinel chunks - POSTHOG_LLM_CAPTURE_CONTENT=False keeps telemetry but omits content - docs/integration/posthog-llm-observability.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Integrates PostHog AI observability via manual capture: one
$ai_generationevent per LLM provider call, giving per-turn trace waterfalls (prompt + output + tool rounds), token/cost dashboards, latency breakdowns (incl. time-to-first-token), and error-rate tracking across all providers.How
Capture happens at a single choke point —
LLMService._execute_llm_completion— so chat, Socratic, workflow steps, and post-tool follow-up calls are all covered with zero per-caller instrumentation:core/services/llm_observability_service.py—LLMObservabilityService(process-wide singleton, PostHog client) +GenerationTracker(per-call accumulator;finish()runs fromfinally, so client disconnects still capture).core/services/dtos/llm_generation_dto.py— frozenLLMGenerationContext/LLMGenerationRecordDTOs (rules.md §1).$ai_trace_id(one chat turn incl. tool follow-ups), conversation id →$ai_session_id, user id →distinct_id(anonymousfor public bots)."Error: ..."sentinel chunks (which never raise) — so silent provider failures become visible.Config
POSTHOG_API_KEYPOSTHOG_HOSThttps://us.i.posthog.comPOSTHOG_LLM_CAPTURE_CONTENTTrueFalse= telemetry only, no prompt/response contentSafety
Verification
python manage.py checkclean; black + isort (black profile) clean on all touched Python files.POSTHOG_API_KEY) — integration doc atdocs/integration/posthog-llm-observability.md.