feat(observability): OpenTelemetry instrumentation (spans + metrics + traceparent)#17
Merged
Conversation
Re-instrument D:\moss-drobotics with OpenTelemetry, correcting the tracing/metrics mixed-mode tech debt from the from-remote fork: tracing moves onto the official SDK so trace and metric share one Resource, errors go through recordException, and spans are batched. Local file trace is preserved as a FileSpanProcessor alongside the OTLP exporter. Co-Authored-By: Claude <noreply@anthropic.com>
13-task TDD plan implementing the 2026-07-16 observability design: deps, metrics handle, SDK-backed withSpan, FileSpanProcessor, NodeSDK assembly, trace-context injection, and the 6 call-site wirings (session/turn/llm/tool spans + metrics, web traceparent), ending with end-to-end verification against the local OTLP receiver. Co-Authored-By: Claude <noreply@anthropic.com>
12-task TDD plan wiring OTel SDK (trace+metric+local file trace) into moss-agent: dependencies, metrics handles, FileSpanProcessor, SDK-backed withSpan rewrite, sdk/index entrypoint, web traceparent injection, CLI init, and the four instrumentation sites (llm/tool/turn/session spans + metrics), ending with an end-to-end integration spec. Co-Authored-By: Claude <noreply@anthropic.com>
Adds 8 OpenTelemetry packages to @rdk-moss/agent: - stable: api@1.9, resources@2.9, sdk-metrics@2.9, sdk-trace-base@2.9, semantic-conventions@1.43 - experimental (aligned at 0.220): sdk-node, exporter-trace-otlp-http, exporter-metrics-otlp-http Note: sdk-trace-base has graduated to stable (2.9.0), not experimental — the plan's ^0.200 assumption was corrected to the real published versions. No code imports the deps yet; build + 80 spec files still pass. Co-Authored-By: Claude <noreply@anthropic.com>
metrics.ts exposes a noop-by-default instrument handle (llmTokens, llmDuration, toolInvocations, toolDuration, sessionCount, sessionDuration, sessionToolCount). Before setGlobalMeterProvider, metrics.getMeter() returns a noop meter, so business call-sites invoke .add()/.record() unconditionally at zero cost. sessionToolCount corrects the from-remote misnaming of per-turn tool count as "session.turns". Re-exported from observability/index.ts. Co-Authored-By: Claude <noreply@anthropic.com>
FileSpanProcessor implements the OTel SpanProcessor interface, buffering
ended ReadableSpans and flushing them to
{workspaceDir}/.moss/analytics/traces.jsonl every 30s and on shutdown.
serializeSpan maps SDK HrTime ([sec, ns]) to epoch-ms and SpanStatusCode
to ok/error. readTraceStats aggregates the JSONL for CLI reporting.
Verified against the installed sdk-trace-base@2.9.0 (ReadableSpan,
SpanProcessor, TimedEvent shapes) — the spec's BigInt HrTime was
corrected to the real [number, number] shape.
Co-Authored-By: Claude <noreply@anthropic.com>
withSpan now uses trace.getTracer() + context propagation; on error it records the exception as a span event and sets ERROR with a redacted message (String-coerced because redactSensitiveData returns unknown). The fn callback receives the real OTel Span. Legacy setTracer/setTraceRedactor/TraceRegistry/getTracer are retained as noop shims so cli-main.ts, moss-agent.ts, and agent-loop-llm-call.ts imports keep resolving until callers migrate to initObservability. Added sessionAttributes builder for the session-span task. Co-Authored-By: Claude <noreply@anthropic.com>
sdk.ts assembles a single NodeSDK: one shared Resource (service.name + service.version from package.json), BatchSpanProcessor→OTLP trace exporter, FileSpanProcessor for local JSONL, and a PeriodicExportingMetricReader→ OTLP metric exporter. Uses metricReaders (plural) — metricReader is deprecated. Best-effort init: never crashes the agent. index.ts now exports initObservability (env-gated noop when disabled), shutdownObservability, propagateHeaders (W3C traceparent via the SDK propagation API + defaultTextMapSetter). tracing-on ⇒ metrics-on by default; local file trace on by default. Co-Authored-By: Claude <noreply@anthropic.com>
Both outbound fetches now route headers through propagateHeaders, which adds a W3C traceparent for the active span (or passes through unchanged when no span is active). Lets the receiver correlate upstream HTTP calls with the agent's trace tree. Co-Authored-By: Claude <noreply@anthropic.com>
initObservability runs before MossAgent construction (no-op unless MOSS_OTEL_ENABLED / MOSS_OTEL_URL set). shutdownObservability runs in the main finally block and on beforeExit to flush in-flight traces/metrics. Co-Authored-By: Claude <noreply@anthropic.com>
Span renamed agent.llm_turn → moss.llm.request for naming consistency. Records moss.llm.tokens (input/output, with direction + model) and moss.llm.request.duration on success; records duration with status=error on failure. Noop when metrics disabled. Co-Authored-By: Claude <noreply@anthropic.com>
… metrics Splits executeOneToolCall into a thin outer wrapper (span + metrics) and the unchanged inner logic. Each tool call becomes one moss.tool.invoke span carrying is_error/outcome attributes, plus moss.tool.invocations and moss.tool.invoke.duration metrics. The inner function's control flow and multiple early-return paths are untouched. Co-Authored-By: Claude <noreply@anthropic.com>
Adds an imperative startSpan() / ActiveSpan API to tracing.ts for call-sites that cannot use the callback-based withSpan (loop bodies with continue/break/throw). Each agent-loop iteration opens a moss.agent.turn span; executeLlmTurn and processLlmResponse run inside runInSpanContext so child spans (moss.llm.request, moss.tool.invoke) nest under the turn. The span is ended in the existing finally block (runs on continue/break/throw), marking ERROR when the turn ended with a consecutive-turn error. Co-Authored-By: Claude <noreply@anthropic.com>
…etrics chat() now runs its body inside a moss.session span (sessionKey + model attributes). Session metrics — count, duration, per-turn tool count — are recorded in a finally block so they fire on both success and error outcomes (ok/incomplete/error). The session span is the root of the session → turn → llm/tool trace tree. Co-Authored-By: Claude <noreply@anthropic.com>
Enables the SDK (MOSS_OTEL_ENABLED=1), runs a nested withSpan chain (session → turn → llm), shuts down, and asserts the FileSpanProcessor landed all three spans in traces.jsonl. Does not require a live receiver — OTLP sends are fire-and-forget, the file trace writes regardless. Proves the full trace+file stack works, not just compiles. Co-Authored-By: Claude <noreply@anthropic.com>
mossMetrics previously created its instruments at module load via metrics.getMeter() — which returns a NOOP meter until setGlobalMeterProvider runs. The cached noop instruments never switched to real ones after the SDK started, so metrics were silently dropped (verified: receiver got 0 metrics even with MOSS_OTEL_ENABLED=1). Instruments now resolve lazily on first use (cached after first resolution), so they bind to the real MeterProvider once the SDK starts. The e2e spec now asserts getMeterProvider() is the noop before init and a real provider after — a regression guard for this exact failure. Found via manual receiver verification (D:\otel), not the spec suite. Co-Authored-By: Claude <noreply@anthropic.com>
The moss.session root span and session metrics were in chat(), but the CLI (oneshot / piped / TUI) all drive the agent via streamChat() → streamChatViaAgentLoop, which never calls chat(). So the session span and session metrics were unreachable from any CLI entry — verified: real LLM oneshot produced only turn+llm spans, no moss.session, no session metrics. Moved the session span (startSpan) and session metrics into streamChatViaAgentLoop's finally, covering every entry. chat() is restored to its original event-loop logic (the mesh/channel path now also gets a single session span via streamChat). Verified end-to-end with a real LLM: moss.session → moss.agent.turn → moss.llm.request tree, real token metrics, and session metrics all land in the receiver. Co-Authored-By: Claude <noreply@anthropic.com>
…ropagation) The session span and the turn/llm child spans ended up in separate traces: moss.session alone in one trace, moss.agent.turn + moss.llm.request in another. Verified by tracing the real LLM run — turn.parent was NULL, not the session span. Two root causes, both fixed: 1. The session span was opened but runAgentLoop ran outside its context. Added runInSpanContextGen to ActiveSpan — it drives an async generator while re-activating the span context on each .next() resume (generators suspend at yield and lose the stack-local active context). 2. runAgentLoop spawns a background async IIFE that captures the context active at call time. It was called before the session context was activated. Moved the miniStream creation INSIDE the session-context generator so the background loop inherits the session context. Verified end-to-end with a real LLM: the newest trace now has all three spans in one traceId with session → turn → llm parent linkage correct. Co-Authored-By: Claude <noreply@anthropic.com>
Two minor metric-semantic issues found during runtime verification: 1. session outcome: a hard LLM failure (HTTP 401) produced stopReason='error' but was labeled 'incomplete' (only 'end_turn' was 'ok', everything else 'incomplete'). Now 'error' stopReason → outcome='error', so the session error-rate reflects real failures. Verified: 401 run now records session.count outcome=error (was incomplete). 2. tool span/metrics: denied / pre-blocked / hook-blocked / unknown-tool outcomes were all marked is_error and counted as status='error', inflating the tool error-rate. Now only a completed-and-failed execution is status='error'; the pre-execution outcomes count as 'blocked' (user refusal / guardrail / unknown tool are not execution errors). Added outcome_kind span attribute. Verified: successful exec → status='ok'. Co-Authored-By: Claude <noreply@anthropic.com>
MOSS_TRACE=console (documented in cli help) was a dead switch — the old
setTracer('console') became a noop shim after the SDK rewrite. Restore it
as a real ConsoleSpanProcessor that streams span open/close to stderr with
parent-depth indentation, duration, status, and key attributes — useful for
watching the trace tree run in a terminal without a receiver.
Also fixes two lazy-resolution bugs exposed while wiring this up:
- tracing.ts bound `tracer` eagerly at module load (noop before SDK start),
same failure mode as the metrics instruments. Now resolved lazily via
resolveTracer() so spans bind to the real TracerProvider after SDK start.
- sdk.ts returned early when cfg.enabled=false, blocking the standalone
console-only path. Now starts the SDK if either OTel or console is on.
Verified: real LLM oneshot with a tool call streams the full
session → turn → llm/tool tree to stderr with correct nesting.
Co-Authored-By: Claude <noreply@anthropic.com>
Resolve conflicts across 6 files where observability instrumentation (HEAD) overlaps with main's tool-nudge + run-lifecycle changes: - cli-main.ts, web-fetch.ts, web-search.ts: keep both sides' imports (propagateHeaders + ensureKeepAliveDispatcherInstalled / logLLMUsage). - execute-tool-call.ts: keep isMossError (main) + withSpan/toolAttributes/ mossMetrics (HEAD). - agent-loop.ts: adopt main's shouldBufferAssistantOutput (do not suppress streaming for an always-installed completionGate) + pass pendingToolAborts; keep HEAD's turnSpan wrapping so calls close with '}))'. - moss-agent.ts: keep HEAD's session-span metrics + sessionSpan.end(), and restore main's run-lifecycle calls (noteRunStarted before the loop, noteRunFinished/retireSteeringMessages/clearPendingStructuredValidation in finally). logLLMUsage import retained. Verified: tsc --noEmit clean; observability/web/tool/loop specs pass. Co-Authored-By: Claude <noreply@anthropic.com>
Bring in upstream's 27 commits since afb478a: exec_wait, /rewind, sessions search/list/export, ACP stdio, prompt-cache observability, config-profile safety fixes, redact refactor, 0.6.0 release. Resolve 2 dependency-file conflicts (cli-main.ts / moss-agent.ts auto-merged cleanly this round): - packages/moss-agent/package.json: keep the 8 @opentelemetry/* deps (our instrumentation) and adopt upstream's @rdk-moss/core ^0.6.0. - package-lock.json: same resolution, then npm install --package-lock-only to reconcile the 0.6.0 core/agent entries. Verified: tsc --noEmit clean; observability (5) + web/tool + upstream new-feature specs (acp, exec-wait, rewind, sessions-search, todo) all pass. Co-Authored-By: Claude <noreply@anthropic.com>
… links The link extractor matched TS computed-key syntax inside example code (e.g. `[semconv.ATTR_SERVICE_NAME]: cfg.serviceName,`) as a markdown reference definition (`[name]: url`), flagging `cfg.serviceName,` etc. as broken links. Strip fenced (```/~~~) and indented code blocks before extracting links so code in examples is not scanned. Fixes the workspace-hygiene failures that blocked `npm run verify` for the observability instrumentation docs. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds end-to-end OpenTelemetry observability to the agent runtime. The existing
observability/module ships a lightweight tracing API (withSpan,setTracer('console')) but only one span is actually emitted (moss.llm.turninagent-loop-llm-call.ts), with no metrics, no trace propagation, and no file export.This PR completes the instrumentation:
moss.session→moss.agent.turn→moss.llm.request/moss.tool.invoke. Child spans nest under the session span viaAsyncLocalStoragecontext, sharing onetraceId.mossMetrics): session count/duration/tool-count, tool invocations + duration by status, LLM request timing.web-fetch/web-searchoutbound requests so downstream services correlate.FileSpanProcessor(local JSONL trace export) +ConsoleSpanProcessor(MOSS_TRACE=console), assembled via a newsdk.tsentrypoint.beforeExit.The
observability/tracing.tsis rewritten onto the OTel SDKwithSpan, keeping the existingsetTracer/setTraceRedactor/TraceRegistryshims for compatibility.Type of change
Why the
@opentelemetry/*dependenciesThe upstream light tracer is a hand-rolled
TraceSpan/Tracerinterface with anoopTracerdefault. To emit real spans/metrics to OTLP collectors (and to get context propagation, exporters, and semantic conventions for free), this switches the agent to the standard OpenTelemetry SDK. Added deps:@opentelemetry/api,sdk-node,sdk-trace-base,sdk-metrics,resources,semantic-conventions,exporter-trace-otlp-http,exporter-metrics-otlp-http. Default behavior is still a no-op unlessMOSS_OTEL_ENABLED/MOSS_OTEL_URL/MOSS_TRACEis set — zero overhead when off.Checklist
npm run verifygreen (boundaries + hygiene + build + typecheck + lint + test). Note: one test (git-status-snapshot.spec.mjs) fails on Windows only withEBUSYwhile tearing down a%TEMP%dir locked by antivirus/indexing; it passes with an alternateTMPDIRand on Linux/CI. Not a code defect.initObservability/shutdownObservability/propagateHeadersexported fromobservability/index.ts.Also included
fix(hygiene): the workspace-hygiene link extractor matched TS computed-key syntax inside example code blocks ([ATTR]: value,) as markdown reference definitions, flaggingcfg.serviceName,etc. as broken links and blockingnpm run verify. Now strips fenced/indented code blocks before scanning. Fixes the hygiene failures the new observability docs surfaced.🤖 Generated with Claude Code