feat(telemetry): add OpenTelemetry tracing, metrics, logs and cross-service e2e test - #383
Open
prvnsmpth wants to merge 14 commits into
Open
feat(telemetry): add OpenTelemetry tracing, metrics, logs and cross-service e2e test#383prvnsmpth wants to merge 14 commits into
prvnsmpth wants to merge 14 commits into
Conversation
prvnsmpth
force-pushed
the
fix/open-telemetry
branch
11 times, most recently
from
July 29, 2026 04:21
18b34d2 to
5bb6302
Compare
…ervice e2e test - Configure W3C TraceContext propagation with CLIENT/SERVER spans across web, AI, searcher, indexer, and connector-manager - Add OTLP metrics providers with HTTP RED, queue, searcher, connector sync, and AI embedding metrics - Add OTLP log export with trace/span correlation and PII-safe attribute allowlist - Persist traceparent/tracestate in queue rows for async producer→consumer trace links - Fix Rust BatchSpanProcessor Tokio runtime panic that silently dropped all trace exports - Add e2e cross-service tracing test that validates parent/child span continuity across Node.js and Rust services - Remove bundled local OTel Collector, smoke/validate scripts, and observability profile from compose
- Delete migration 106 that added traceparent/tracestate columns - Remove traceparent/tracestate fields from Rust queue event structs and SQL queries - Remove PRODUCER span wrapping and W3C carrier injection from enqueue operations - Remove consumer trace link creation from indexer queue processor - Remove trace context extraction and consumer link building from AI batch processor - Delete test files for removed queue tracing functionality - Remove the queue module from telemetry.rs Async queue consumers now create fresh new-root traces without explicit OTel links, which is the standard pattern for async boundaries. Business correlation IDs in the events provide traceability when needed.
Exception messages (e.g. 'Connection refused', 'Timeout') are not sensitive and are critical for debugging production failures. Restore them across all AI Python provider, tool, streaming, and email files.
- Restore exception args in 'Failed to initialize services' error - Restore exception args in 'Failed to fetch Gemini model metadata' debug - Add exc_info=True to 'Online processing loop error' for traceback - Restore error object in TypeScript title generation catch handler - Restore errorMessage in action failure context object
Internal IDs like chatId, userId, documentId, sourceId are not PII. They are critical for debugging production issues. Restore them in error/warn log messages and context objects across all web routes.
- Restore chat_id in streaming/generate.py and streaming/run.py
- Restore exception details ({e}) in connector_handler.py and search_handler.py warnings
- Restore event_count in anthropic.py message completed log
- Restore exception details in main.py memory initialization warning
- Restore chatId in 7 TypeScript stream/+server.ts error/info logs
- Restore error context in oauth/callback/+server.ts (3 sites)
- Restore user IDs in accountLinking.ts (6 sites)
- Restore messageId in email/types.ts
- Restore aiServiceUrl in uploads/+server.ts
- Restore modelId in chat/+server.ts
- Restore url in config.ts
- Restore userId in hooks.server.ts Request completed
Restore sync_run_id, source_id, url, provider, addr, and other
internal Omni identifiers that were over-scrubbed from debug!/info!
logs across searcher, indexer, connector-manager, and shared.
Restore exception messages ({e}) in error!/warn! logs that are
critical for debugging production failures.
Remove unused traceparent/tracestate fields from ConnectorEventQueueItem
model (leftover from reverted queue trace context feature).
Correctly kept stripped: search queries, RAG prompts, connector API
response bodies, and raw HTTP request bodies.
…oisy stream logs - Add otel_skip filter across Python/Rust/TypeScript to exclude high-cardinality info logs (model params, stream summary) from OTel - Restore tool names and event keys in provider log statements - Drop noisy stream-level logs (content block start/stop, stream created)
- Restore full request params, messages, tool input, text/JSON deltas, citations, adapted messages, and LLM response body debug logs - Tag sensitive-content logs with otel_skip so they appear in local stdout but are excluded from OTel export - Restore exception details and call_id in stripped error/warning logs - Bounded event metadata (type, stop_reason, index) restored without otel_skip
…email, embeddings - Restore full event content, text deltas, citations, search queries, response bodies, and exception details in non-provider modules - Split bounded metadata (index, status) from sensitive content in same log call — bounded part goes to OTel, sensitive part uses otel_skip - Restore tool name in tool use block start log (bounded, no otel_skip)
- Restore search query in agent search request and search error logs with otel_skip - Restore userId in add-message debug log (internal ID) - Restore error object in instrumentation shutdown and OAuth unlink console.error calls
- Restore search queries in fulltext/semantic/hybrid search logs with otel_skip - Restore query embedding, RAG prompt, AI answer cache info with otel_skip - Restore connector response bodies in error logs with otel_skip - Restore document IDs in indexer create/update/delete logs
prvnsmpth
force-pushed
the
fix/open-telemetry
branch
from
July 29, 2026 09:39
35a2e7d to
704e080
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.