diff --git a/docs/superpowers/plans/2026-07-16-observability-instrumentation.md b/docs/superpowers/plans/2026-07-16-observability-instrumentation.md new file mode 100644 index 00000000..caef8ff9 --- /dev/null +++ b/docs/superpowers/plans/2026-07-16-observability-instrumentation.md @@ -0,0 +1,1431 @@ +# Moss-Drobotics 可观测性（埋点）实现计划 + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 为 `D:\moss-drobotics` 重新埋入 OpenTelemetry trace+metrics，纠正 from-remote fork 的 tracing/metrics 混搭债，并保留本地文件 trace。 + +**Architecture:** tracing 与 metrics 同走官方 OTel SDK（`@opentelemetry/sdk-node`），共享一个 Resource；业务代码通过 `withSpan` / `mossMetrics.*` 无条件调用，SDK 在未启用时返回 noop（零开销）；本地文件 trace 实现为 `FileSpanProcessor` 挂到 tracer，与 OTLP exporter 并存。 + +**Tech Stack:** TypeScript（ESM）、Node.js、OpenTelemetry JS SDK、moss-agent monorepo（包名 `@rdk-moss/agent`）、测试用 plain Node `*.spec.mjs` + `node:assert/strict`（经 `scripts/run-package-tests.mjs` 跑）。 + +## Global Constraints + +- 工作目录：`D:\moss-drobotics`，仓库根。所有 git 命令用 `git -C /d/moss-drobotics` 或先 `cd /d/moss-drobotics`。 +- 包：所有代码改在 `packages/moss-agent/`，npm 包名 `@rdk-moss/agent`，workspace 命令形如 `npm run build -w @rdk-moss/agent`。 +- ESM：`"type": "module"`，import 路径必须带 `.js` 后缀（如 `'./tracing.js'`）。 +- 构建：`npm run build -w @rdk-moss/agent`（产 `dist/`）；测试先 build 再 import `dist/`。 +- 测试：`npm run test -w @rdk-moss/agent`（内部 `npm run build && node ../../scripts/run-package-tests.mjs`，遍历 `packages/moss-agent/test/*.spec.mjs`）。spec 用 `node:assert/strict`，从 `dist/` import。 +- 关闭即零开销：未启用时 `withSpan` 走 noop、`mossMetrics.*` 是 noop instrument，业务代码不写 `if` 分支。 +- 不改变现有控制流语义：埋点只包 span / 记 metric，不吞错误、不改返回值。 +- 版本对齐：tracing experimental 包统一 `^0.200`；metrics 侧沿用 from-remote 的 `api@^1.9` / `sdk-metrics@^2.9` / `exporter-metrics-otlp-http@^0.220` / `resources@^2.9` / `semantic-conventions@^1.43`。若 install 时 peer 冲突，把 tracing experimental 包升到 `^0.220` 与 metrics exporter 对齐。 + +--- + +## File Structure + +新建 / 修改（均在 `packages/moss-agent/`）： + +- **Create** `src/observability/metrics.ts` — `mossMetrics` 仪器句柄（counter/histogram）。 +- **Create** `src/observability/file-trace.ts` — `FileSpanProcessor`：消费 SDK `ReadableSpan`，缓冲后 flush 到 `.moss/analytics/traces.jsonl`，含 `getStats` 聚合。 +- **Create** `src/observability/sdk.ts` — `initObservabilitySdk` / `shutdownObservabilitySdk`：装配 NodeSDK、Resource、spanProcessors、metricReader。 +- **Rewrite** `src/observability/tracing.ts` — 从旧手写 tracer 改为 SDK-backed `withSpan` + attributes 构造器；保留 `setTracer`/`getTracer`/`TraceRegistry` 等**导出壳**（noop）以免破坏 `cli-main.ts:65,188` 与 `moss-agent.ts`、`agent-loop-llm-call.ts:22` 的 import。 +- **Rewrite** `src/observability/index.ts` — 公共入口：`initObservability` / `shutdownObservability` + 重导出 withSpan/mossMetrics/attributes，保留 redact/llm-usage 导出。 +- **Modify** `packages/moss-agent/package.json` — 加 8 个 `@opentelemetry/*` 依赖。 +- **Modify** `src/cli-main.ts` — agent 创建前调 `initObservability`；退出调 `shutdownObservability`；保留 `setTracer('console')` 调用（noop shim）。 +- **Modify** `src/core/agent/moss-agent.ts` — `chat()` 开 `moss.session` 根 span + session metrics。 +- **Modify** `src/core/loop/agent-loop.ts` — 主循环每轮外包 `moss.agent.turn` span。 +- **Modify** `src/core/loop/agent-loop-llm-call.ts` — span 名改 `moss.llm.request`，加 llm metrics。 +- **Modify** `src/core/tools/execute-tool-call.ts` — `executeOneToolCall` 内包 `moss.tool.invoke` span + tool metrics。 +- **Modify** `src/tools/web-fetch.ts` + `src/tools/web-search.ts` — 出站 fetch headers 用 `propagation.inject` 注入 traceparent。 +- **Test** `test/observability-tracing.spec.mjs`、`test/observability-file-trace.spec.mjs`、`test/observability-metrics.spec.mjs`、`test/observability-noop.spec.mjs`、`test/observability-integration.spec.mjs`。 + +--- + +## Task 1: 加 OTel 依赖并安装 + +**Files:** +- Modify: `packages/moss-agent/package.json`（dependencies 块） + +**Interfaces:** +- Consumes: 无 +- Produces: 可 import 的 `@opentelemetry/*` 模块（后续 task 依赖） + +- [ ] **Step 1: 在 `dependencies` 块加入 8 个包** + +打开 `packages/moss-agent/package.json`，在 `"dependencies"` 对象内（按字母序插入）加入： + +```json +"@opentelemetry/api": "^1.9.0", +"@opentelemetry/exporter-metrics-otlp-http": "^0.220.0", +"@opentelemetry/exporter-trace-otlp-http": "^0.200.0", +"@opentelemetry/resources": "^2.9.0", +"@opentelemetry/sdk-metrics": "^2.9.0", +"@opentelemetry/sdk-node": "^0.200.0", +"@opentelemetry/sdk-trace-base": "^0.200.0", +"@opentelemetry/semantic-conventions": "^1.43.0", +``` + +- [ ] **Step 2: 安装** + +Run: `cd /d/moss-drobotics && npm install -w @rdk-moss/agent` +Expected: 安装成功。若出现 `ERESOLVE` peer 冲突，把 4 个 tracing experimental 包（`sdk-node` / `sdk-trace-base` / `exporter-trace-otlp-http`，以及视情况 `exporter-metrics-otlp-http`）统一升到 `^0.220.0` 后重试。 + +- [ ] **Step 3: 验证可 import 且 build 不破** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent` +Expected: build 成功（此步尚未引用新包，仅确认依赖到位、不破坏现有构建）。 + +- [ ] **Step 4: 跑现有测试确认无回归** + +Run: `cd /d/moss-drobotics && npm run test -w @rdk-moss/agent` +Expected: 全部 spec 通过（应与改动前一致）。 + +- [ ] **Step 5: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/package.json package-lock.json +git commit -m "chore(agent): add @opentelemetry/* dependencies for instrumentation" +``` + +--- + +## Task 2: metrics.ts — mossMetrics 仪器句柄 + +**Files:** +- Create: `packages/moss-agent/src/observability/metrics.ts` +- Test: `packages/moss-agent/test/observability-metrics.spec.mjs` + +**Interfaces:** +- Consumes: `@opentelemetry/api` 的 `metrics` +- Produces: `mossMetrics`（对象，含 counter/histogram；未注册 provider 时为 noop） + +- [ ] **Step 1: 写 spec（验证导出 + noop 行为）** + +Create `packages/moss-agent/test/observability-metrics.spec.mjs`: + +```javascript +#!/usr/bin/env node +// mossMetrics instruments export + noop behavior (no provider registered). +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import path from 'node:path'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'metrics.js')).href); +const { mossMetrics } = mod; + +// 未 setGlobalMeterProvider 时返回 noop meter，instruments 仍可调用不抛错。 +assert.ok(mossMetrics, 'mossMetrics should be exported'); +assert.equal(typeof mossMetrics.llmTokens.add, 'function', 'llmTokens.add is a function'); +assert.equal(typeof mossMetrics.llmDuration.record, 'function', 'llmDuration.record is a function'); +assert.equal(typeof mossMetrics.toolInvocations.add, 'function', 'toolInvocations.add is a function'); +assert.equal(typeof mossMetrics.toolDuration.record, 'function', 'toolDuration.record is a function'); +assert.equal(typeof mossMetrics.sessionCount.add, 'function', 'sessionCount.add is a function'); +assert.equal(typeof mossMetrics.sessionDuration.record, 'function', 'sessionDuration.record is a function'); +assert.equal(typeof mossMetrics.sessionToolCount.record, 'function', 'sessionToolCount.record is a function'); + +// noop 调用零成本、不抛 +assert.doesNotThrow(() => { + mossMetrics.llmTokens.add(10, { direction: 'input', model: 'm' }); + mossMetrics.llmDuration.record(123, { model: 'm' }); + mossMetrics.toolInvocations.add(1, { tool: 't', status: 'ok' }); + mossMetrics.sessionCount.add(1, { outcome: 'ok' }); + mossMetrics.sessionToolCount.record(3, { outcome: 'ok' }); +}); + +console.error('[spec] observability-metrics OK'); +``` + +- [ ] **Step 2: 运行确认失败（模块不存在）** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent && node packages/moss-agent/test/observability-metrics.spec.mjs` +Expected: FAIL（`dist/observability/metrics.js` 不存在，import 报 `ERR_MODULE_NOT_FOUND`）。 + +- [ ] **Step 3: 实现 metrics.ts** + +Create `packages/moss-agent/src/observability/metrics.ts`: + +```typescript +/** + * OpenTelemetry metrics for Moss — instrument handles. + * + * Uses the global MeterProvider. When none is registered (observability + * disabled), metrics.getMeter() returns a noop meter whose instruments are + * no-ops, so business code calls .add()/.record() unconditionally at zero cost. + * + * Usage: + * import { mossMetrics } from './observability/index.js'; + * mossMetrics.llmTokens.add(inputTokens, { direction: 'input', model }); + */ +import { metrics } from '@opentelemetry/api'; + +const meter = metrics.getMeter('moss-agent'); + +export const mossMetrics = { + // LLM + llmTokens: meter.createCounter('moss.llm.tokens', { unit: '{token}' }), + llmDuration: meter.createHistogram('moss.llm.request.duration', { unit: 'ms' }), + // tool + toolInvocations: meter.createCounter('moss.tool.invocations'), + toolDuration: meter.createHistogram('moss.tool.invoke.duration', { unit: 'ms' }), + // session + sessionCount: meter.createCounter('moss.session.count'), + sessionDuration: meter.createHistogram('moss.session.duration', { unit: 'ms' }), + // 每轮工具数（纠正 from-remote 把它误命名为 session.turns 的错位） + sessionToolCount: meter.createHistogram('moss.session.tool_count'), +}; +``` + +- [ ] **Step 4: build + 跑 spec 确认通过** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent && node packages/moss-agent/test/observability-metrics.spec.mjs` +Expected: PASS，输出 `[spec] observability-metrics OK`。 + +- [ ] **Step 5: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/observability/metrics.ts packages/moss-agent/test/observability-metrics.spec.mjs +git commit -m "feat(agent): add mossMetrics OTel instrument handles" +``` + +--- + +## Task 3: file-trace.ts — FileSpanProcessor 本地落盘 + +**Files:** +- Create: `packages/moss-agent/src/observability/file-trace.ts` +- Test: `packages/moss-agent/test/observability-file-trace.spec.mjs` + +**Interfaces:** +- Consumes: `@opentelemetry/sdk-trace-base` 的 `SpanProcessor`、`ReadableSpan` +- Produces: `FileSpanProcessor`（class）、`serializeSpan`（函数，被 getStats 复用）、`readTraceStats`（从 jsonl 聚合） + +- [ ] **Step 1: 写 spec（落盘 + 聚合）** + +Create `packages/moss-agent/test/observability-file-trace.spec.mjs`: + +```javascript +#!/usr/bin/env node +// FileSpanProcessor buffers spans, flushes to traces.jsonl, and readTraceStats aggregates. +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import os from 'node:os'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'file-trace.js')).href); +const { FileSpanProcessor, readTraceStats } = mod; + +// 构造一个最小 ReadableSpan 形状 +function fakeSpan(name, attrs, isError) { + const now = Date.now(); + return { + name, + kind: 0, + spanContext: () => ({ traceId: 't'.repeat(32), spanId: 's'.repeat(16) }), + startTime: [[now - 50, 0]], + endTime: [[now, 0]], + attributes: attrs ?? {}, + status: isError ? { code: 2, message: 'boom' } : { code: 1 }, + events: [], + resource: { attributes: [] }, + instrumentationScope: { name: 'moss-agent' }, + }; +} + +const tmp = await fs.mkdtemp(path.join(os.tmpdir(), 'moss-trace-')); +const proc = new FileSpanProcessor(tmp); +proc.onStart({}, fakeSpan('noop')); +proc.onEnd(fakeSpan('moss.tool.invoke', { toolName: 'read_file' }, false)); +proc.onEnd(fakeSpan('moss.tool.invoke', { toolName: 'read_file' }, true)); +proc.onEnd(fakeSpan('moss.llm.request', { model: 'm' }, false)); +await proc.forceFlush(); + +const file = path.join(tmp, '.moss', 'analytics', 'traces.jsonl'); +const content = await fs.readFile(file, 'utf8'); +const lines = content.split('\n').filter(Boolean); +assert.equal(lines.length, 3, 'should write 3 span lines'); + +const stats = await readTraceStats(file); +assert.equal(stats.totalSpans, 3, 'totalSpans'); +assert.equal(stats.totalErrors, 1, 'totalErrors'); +assert.ok(stats.byName['moss.tool.invoke'], 'tool span aggregated by name'); +assert.equal(stats.byName['moss.tool.invoke'].count, 2, '2 tool spans'); +assert.equal(stats.byName['moss.tool.invoke'].errors, 1, '1 tool error'); +assert.equal(stats.toolSpans[0].toolName, 'read_file', 'tool breakdown by toolName'); + +await proc.shutdown(); +await fs.rm(tmp, { recursive: true, force: true }); +console.error('[spec] observability-file-trace OK'); +``` + +- [ ] **Step 2: 运行确认失败** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent && node packages/moss-agent/test/observability-file-trace.spec.mjs` +Expected: FAIL（`dist/observability/file-trace.js` 不存在）。 + +- [ ] **Step 3: 实现 file-trace.ts** + +Create `packages/moss-agent/src/observability/file-trace.ts`: + +```typescript +/** + * FileSpanProcessor — serializes SDK ReadableSpans to a local JSONL file. + * + * Attached to the TracerProvider alongside the OTLP exporter, so the same + * spans ship to the receiver AND land on disk. Best-effort: flush failures + * never block the agent. + * + * Path: {workspaceDir}/.moss/analytics/traces.jsonl + */ +import type { SpanProcessor, ReadableSpan } from '@opentelemetry/sdk-trace-base'; +import { SpanStatusCode } from '@opentelemetry/api'; +import fs from 'node:fs/promises'; +import path from 'node:path'; + +const FLUSH_INTERVAL_MS = 30_000; + +export interface SerializedSpan { + name: string; + startTime: number; + endTime: number; + attributes: Record; + events: Array<{ name: string; time: number; attrs?: Record }>; + status: 'ok' | 'error'; + statusMessage?: string; +} + +export interface TraceStats { + totalSpans: number; + totalErrors: number; + errorRate: number; + byName: Record; + toolSpans: Array<{ toolName: string; count: number; errors: number; avgDurationMs: number }>; +} + +/** Convert an SDK ReadableSpan into the JSONL-friendly SerializedSpan shape. */ +export function serializeSpan(span: ReadableSpan): SerializedSpan { + const attrs: Record = {}; + for (const [k, v] of Object.entries(span.attributes ?? {})) { + if (typeof v === 'string' || typeof v === 'number' || typeof v === 'boolean') attrs[k] = v; + } + return { + name: span.name, + startTime: Number(span.startTime[0]), + endTime: Number(span.endTime[0]), + attributes: attrs, + events: (span.events ?? []).map((e) => ({ + name: e.name, + time: Number(e.time[0]), + ...(e.attributes ? { attrs: e.attributes as Record } : {}), + })), + status: span.status.code === SpanStatusCode.ERROR ? 'error' : 'ok', + ...(span.status.code === SpanStatusCode.ERROR && span.status.message + ? { statusMessage: span.status.message } + : {}), + }; +} + +export class FileSpanProcessor implements SpanProcessor { + private buffer: ReadableSpan[] = []; + private timer: ReturnType | null = null; + private readonly file: string; + + constructor(workspaceDir: string) { + this.file = path.join(workspaceDir, '.moss', 'analytics', 'traces.jsonl'); + this.timer = setInterval(() => { void this.flush(); }, FLUSH_INTERVAL_MS); + } + + onStart(_span: ReadableSpan, _parentContext: unknown): void {} + + onEnd(span: ReadableSpan): void { + this.buffer.push(span); + } + + async flush(): Promise { + if (this.buffer.length === 0) return; + const snapshot = this.buffer.splice(0); + const lines = snapshot.map(serializeSpan).map((s) => JSON.stringify(s)).join('\n') + '\n'; + try { + await fs.mkdir(path.dirname(this.file), { recursive: true }); + await fs.appendFile(this.file, lines, 'utf-8'); + } catch { + // Silently ignore — never block the agent. + } + } + + async forceFlush(): Promise { + await this.flush(); + } + + async shutdown(): Promise { + if (this.timer) clearInterval(this.timer); + this.timer = null; + await this.flush(); + } +} + +function emptyStats(): TraceStats { + return { totalSpans: 0, totalErrors: 0, errorRate: 0, byName: {}, toolSpans: [] }; +} + +/** Read aggregated stats from a traces.jsonl file (for CLI reporting). */ +export async function readTraceStats(file: string): Promise { + let lines: string[]; + try { + lines = (await fs.readFile(file, 'utf-8')).split('\n').filter(Boolean); + } catch { + return emptyStats(); + } + const stats = emptyStats(); + for (const line of lines) { + let span: SerializedSpan; + try { span = JSON.parse(line); } catch { continue; } + stats.totalSpans++; + if (span.status === 'error') stats.totalErrors++; + const entry = stats.byName[span.name] ??= { count: 0, errors: 0, avgDurationMs: 0 }; + entry.count++; + if (span.status === 'error') entry.errors++; + const duration = span.endTime - span.startTime; + entry.avgDurationMs = (entry.avgDurationMs * (entry.count - 1) + duration) / entry.count; + if (span.name === 'moss.tool.invoke') { + const toolName = String(span.attributes.toolName ?? 'unknown'); + let tool = stats.toolSpans.find((t) => t.toolName === toolName); + if (!tool) { tool = { toolName, count: 0, errors: 0, avgDurationMs: 0 }; stats.toolSpans.push(tool); } + tool.count++; + if (span.status === 'error') tool.errors++; + tool.avgDurationMs = (tool.avgDurationMs * (tool.count - 1) + duration) / tool.count; + } + } + stats.errorRate = stats.totalSpans > 0 ? stats.totalErrors / stats.totalSpans : 0; + stats.toolSpans.sort((a, b) => b.count - a.count); + return stats; +} +``` + +- [ ] **Step 4: build + 跑 spec 确认通过** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent && node packages/moss-agent/test/observability-file-trace.spec.mjs` +Expected: PASS，输出 `[spec] observability-file-trace OK`。 + +- [ ] **Step 5: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/observability/file-trace.ts packages/moss-agent/test/observability-file-trace.spec.mjs +git commit -m "feat(agent): add FileSpanProcessor for local JSONL trace export" +``` + +--- + +## Task 4: 重写 tracing.ts（SDK-backed withSpan + 保留导出壳） + +**Files:** +- Rewrite: `packages/moss-agent/src/observability/tracing.ts` +- Test: `packages/moss-agent/test/observability-tracing.spec.mjs` + +**Interfaces:** +- Consumes: `@opentelemetry/api`（`trace`、`context`、`SpanStatusCode`） +- Produces: `withSpan(name, attrs, fn)`；attributes 构造器 `turnAttributes` / `toolAttributes` / `llmRequestAttributes` / `sessionAttributes`；**保留** `setTracer` / `getTracer` / `TraceRegistry` / `Tracer` / `TraceSpan` 导出（noop shim，供现有 import 不破）。 + +**注意：** 现有 `cli-main.ts:65` `import { setTracer } from './observability/tracing.js'` 与 `:188` `setTracer('console')`；`moss-agent.ts` 与 `agent-loop-llm-call.ts:22` 也从 tracing.js import。本 task 把 withSpan 改 SDK 实现，同时保留 `setTracer`/`TraceRegistry` 等为 noop shim，使现有 import 不需在本 task 内改动（cli-main 的真正初始化在 Task 7）。 + +- [ ] **Step 1: 写 spec（withSpan 成功/异常路径 + attributes 形状 + noop shim 存在）** + +Create `packages/moss-agent/test/observability-tracing.spec.mjs`: + +```javascript +#!/usr/bin/env node +// withSpan: success path sets OK, error path records exception + rethrows. +// attributes constructors produce expected shape. setTracer shim is a noop. +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import path from 'node:path'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'tracing.js')).href); +const { withSpan, turnAttributes, toolAttributes, llmRequestAttributes, sessionAttributes, setTracer, getTracer, TraceRegistry } = mod; + +// 未注册 tracer provider 时为 noop tracer，withSpan 仍正常执行 fn 并返回结果。 +const result = await withSpan('test.span', { a: 1 }, async (span) => { + assert.equal(typeof span.setAttribute, 'function'); + assert.equal(typeof span.addEvent, 'function'); + assert.equal(typeof span.setStatus, 'function'); + assert.equal(typeof span.end, 'function'); + span.setAttribute('k', 'v'); + span.addEvent('ev', { x: 1 }); + return 42; +}); +assert.equal(result, 42, 'withSpan returns fn result'); + +// 异常路径：rethrow（不吞错） +await assert.rejects( + withSpan('test.err', {}, async () => { throw new Error('boom'); }), + /boom/, +); + +// attributes 构造器形状 +assert.deepEqual(turnAttributes('r1', 3, 'm'), { runId: 'r1', turn: 3, model: 'm' }); +assert.deepEqual(toolAttributes('r1', 'read_file', 'tc1'), { runId: 'r1', toolName: 'read_file', toolCallId: 'tc1' }); +assert.deepEqual(llmRequestAttributes('r1', 'm', 100), { runId: 'r1', model: 'm', inputTokens: 100 }); +assert.deepEqual(sessionAttributes('r1', 'm', 'sk'), { runId: 'r1', model: 'm', sessionKey: 'sk' }); + +// 旧 API 保留为 noop shim，不抛 +assert.doesNotThrow(() => setTracer('console')); +assert.ok(getTracer()); +assert.ok(new TraceRegistry()); +console.error('[spec] observability-tracing OK'); +``` + +- [ ] **Step 2: 运行确认失败** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent && node packages/moss-agent/test/observability-tracing.spec.mjs` +Expected: FAIL（旧 tracing.ts 的 `withSpan` 第三参是 `(span: TraceSpan) => Promise` 但签名带 `parent?`，且 attributes 构造器缺 `sessionAttributes`；旧实现 `withSpan` 在异常时虽 rethrow 但不 `recordException`——本 spec 主要因缺 `sessionAttributes` 导出而失败）。 + +- [ ] **Step 3: 重写 tracing.ts** + +Replace **entire** `packages/moss-agent/src/observability/tracing.ts` with: + +```typescript +/** + * Tracing — SDK-backed withSpan + attributes. + * + * Uses the global TracerProvider. When none is registered (observability + * disabled), trace.getTracer() returns a noop tracer and withSpan runs fn + * directly with zero overhead. + * + * Legacy setTracer/TraceRegistry/Tracer/TraceSpan are kept as noop shims so + * existing imports (cli-main.ts, moss-agent.ts, agent-loop-llm-call.ts) do + * not break until callers are migrated to initObservability. + */ +import { trace, context, SpanStatusCode } from '@opentelemetry/api'; +import type { Span } from '@opentelemetry/api'; +import { errorMessage } from '../errors.js'; +import { redactSensitiveData } from './redact.js'; + +const tracer = trace.getTracer('moss-agent'); + +/** Public span handle passed to withSpan's fn. */ +export interface TraceSpan { + setAttribute(key: string, value: string | number | boolean): void; + addEvent(name: string, attributes?: Record): void; + setStatus(ok: boolean, message?: string): void; + end(): void; +} + +/** Legacy Tracer interface — kept as noop shim. */ +export interface Tracer { + startSpan( + name: string, + attributes?: Record, + parent?: TraceSpan, + ): TraceSpan; +} + +const noopSpan: TraceSpan = { + setAttribute() {}, + addEvent() {}, + setStatus() {}, + end() {}, +}; + +const noopTracer: Tracer = { + startSpan() { return noopSpan; }, +}; + +/** + * Run fn inside a span. On success sets OK; on throw records the exception + * (type/message/stack as a span event), sets ERROR with a redacted message, + * and rethrows. Never swallows errors. + */ +export async function withSpan( + name: string, + attributes: Record | undefined, + fn: (span: Span & TraceSpan) => Promise, +): Promise { + const span = tracer.startSpan(name, { attributes }) as Span & TraceSpan; + return context.with(trace.setSpan(context.active(), span), async () => { + try { + const result = await fn(span); + span.setStatus({ code: SpanStatusCode.OK }); + return result; + } catch (err) { + span.recordException(err as Error); + span.setStatus({ + code: SpanStatusCode.ERROR, + message: redactSensitiveData(errorMessage(err)), + }); + throw err; + } finally { + span.end(); + } + }); +} + +// ── Attributes constructors (span dimensions, centralized) ────────────── + +export function turnAttributes( + runId: string, + turn: number, + model: string, +): Record { + return { runId, turn, model }; +} + +export function toolAttributes( + runId: string, + toolName: string, + toolCallId: string, +): Record { + return { runId, toolName, toolCallId }; +} + +export function llmRequestAttributes( + runId: string, + model: string, + inputTokens: number, +): Record { + return { runId, model, inputTokens }; +} + +export function sessionAttributes( + runId: string, + model: string, + sessionKey: string, +): Record { + return { runId, model, sessionKey }; +} + +// ── Legacy noop shims (do not remove — existing imports depend on them) ── + +export class TraceRegistry { + setTracer(_tracer: Tracer | 'console'): void {} + setTraceRedactor(_fn: (text: string) => string): void {} + getTracer(): Tracer { return noopTracer; } + redactMessage(text: string): string { return text; } +} + +export function setTracer(_tracer: Tracer | 'console'): void { + // No-op under the SDK model. Tracer/MeterProvider is configured via + // initObservability() in observability/index.ts. +} + +export function getTracer(): Tracer { + return noopTracer; +} +``` + +- [ ] **Step 4: build + 跑 spec 确认通过** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent && node packages/moss-agent/test/observability-tracing.spec.mjs` +Expected: PASS，输出 `[spec] observability-tracing OK`。 + +- [ ] **Step 5: 确认现有 import 未破（build + 全量测试）** + +Run: `cd /d/moss-drobotics && npm run test -w @rdk-moss/agent` +Expected: 全部 spec 通过（cli-main/moss-agent/llm-call 的 `setTracer`/`withSpan` import 仍有效，因 shim 保留）。 + +- [ ] **Step 6: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/observability/tracing.ts packages/moss-agent/test/observability-tracing.spec.mjs +git commit -m "feat(agent): rewrite tracing.ts onto OTel SDK withSpan (legacy shims retained)" +``` + +--- + +## Task 5: sdk.ts + index.ts — SDK 装配与公共入口 + +**Files:** +- Create: `packages/moss-agent/src/observability/sdk.ts` +- Rewrite: `packages/moss-agent/src/observability/index.ts` +- Test: `packages/moss-agent/test/observability-noop.spec.mjs` + +**Interfaces:** +- Consumes: Task 2 `mossMetrics`、Task 3 `FileSpanProcessor`、`@opentelemetry/sdk-node` 等 +- Produces: `initObservability(opts)`、`shutdownObservability()`、`propagateHeaders(headers)`（web 工具用） + +- [ ] **Step 1: 写 spec（默认 noop：不启用时 initObservability 是 noop，且出站 headers 注入是 passthrough）** + +Create `packages/moss-agent/test/observability-noop.spec.mjs`: + +```javascript +#!/usr/bin/env node +// initObservability is a noop when disabled; propagateHeaders passes through when no active span. +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import path from 'node:path'; +import os from 'node:os'; +import fs from 'node:fs/promises'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'index.js')).href); +const { initObservability, shutdownObservability, propagateHeaders } = mod; + +// 不设任何 env，initObservability 应 noop（不创建文件、不抛） +const tmp = await fs.mkdtemp(path.join(os.tmpdir(), 'moss-noop-')); +delete process.env.MOSS_OTEL_ENABLED; +delete process.env.MOSS_OTEL_URL; +assert.doesNotThrow(() => initObservability({ workspaceDir: tmp })); +await shutdownObservability(); + +// propagateHeaders 无 active span 时原样返回（补一个已存在的 header） +const out = propagateHeaders({ 'x-custom': '1' }); +assert.equal(out['x-custom'], '1', 'passes existing headers through'); +assert.ok(out, 'returns a headers object'); + +await fs.rm(tmp, { recursive: true, force: true }); +console.error('[spec] observability-noop OK'); +``` + +- [ ] **Step 2: 运行确认失败** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent && node packages/moss-agent/test/observability-noop.spec.mjs` +Expected: FAIL（`index.js` 未导出 `initObservability` / `shutdownObservability` / `propagateHeaders`）。 + +- [ ] **Step 3: 实现 sdk.ts** + +Create `packages/moss-agent/src/observability/sdk.ts`: + +```typescript +/** + * OpenTelemetry SDK assembly — single NodeSDK for trace + metric, shared Resource. + * Local file trace attaches as a FileSpanProcessor alongside the OTLP exporter. + */ +import { NodeSDK } from '@opentelemetry/sdk-node'; +import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'; +import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; +import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; +import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; +import { resourceFromAttributes } from '@opentelemetry/resources'; +import { semconv } from '@opentelemetry/semantic-conventions'; +import fs from 'node:fs'; +import path from 'node:path'; +import { FileSpanProcessor } from './file-trace.js'; +import type { SpanProcessor } from '@opentelemetry/sdk-trace-base'; + +export interface ObservabilityConfig { + serviceName: string; + otlpUrl: string; + enabled: boolean; + metricsEnabled: boolean; + fileTraceEnabled: boolean; + workspaceDir: string; +} + +let sdk: NodeSDK | null = null; + +/** Read package version from packages/moss-agent/package.json (not hardcoded). */ +function readPackageVersion(): string { + try { + const pkgPath = path.join(import.meta.dirname, '..', '..', 'package.json'); + return JSON.parse(fs.readFileSync(pkgPath, 'utf-8')).version ?? '0.0.0'; + } catch { + return '0.0.0'; + } +} + +export function initObservabilitySdk(cfg: ObservabilityConfig): void { + if (!cfg.enabled || sdk) return; + try { + const resource = resourceFromAttributes({ + [semconv.ATTR_SERVICE_NAME]: cfg.serviceName, + [semconv.ATTR_SERVICE_VERSION]: readPackageVersion(), + }); + + const spanProcessors: SpanProcessor[] = [ + new BatchSpanProcessor(new OTLPTraceExporter({ url: `${cfg.otlpUrl}/v1/traces` })), + ]; + if (cfg.fileTraceEnabled) { + spanProcessors.push(new FileSpanProcessor(cfg.workspaceDir)); + } + + const metricReader = cfg.metricsEnabled + ? new PeriodicExportingMetricReader({ + exporter: new OTLPMetricExporter({ url: `${cfg.otlpUrl}/v1/metrics` }), + exportIntervalMillis: 10_000, + }) + : undefined; + + sdk = new NodeSDK({ + resource, + spanProcessors, + ...(metricReader ? { metricReader } : {}), + }); + sdk.start(); + } catch { + // Best-effort — never block the agent. Failure means no telemetry, not a crash. + } +} + +export async function shutdownObservabilitySdk(): Promise { + if (!sdk) return; + try { await sdk.shutdown(); } catch { /* ignore */ } + sdk = null; +} +``` + +- [ ] **Step 4: 重写 index.ts** + +Replace **entire** `packages/moss-agent/src/observability/index.ts` with: + +```typescript +/** + * Observability public entrypoint. + * + * initObservability() wires up OTel SDK (trace + metric + local file trace) + * based on env vars; when disabled it is a no-op and withSpan / mossMetrics.* + * run as no-ops. Call once at CLI startup (before agent creation), call + * shutdownObservability() on exit to flush. + */ +import { initObservabilitySdk, shutdownObservabilitySdk } from './sdk.js'; + +export interface InitOptions { + workspaceDir: string; + serviceName?: string; + otlpUrl?: string; +} + +export function initObservability(opts: InitOptions): void { + const enabled = process.env.MOSS_OTEL_ENABLED === '1' || !!process.env.MOSS_OTEL_URL; + if (!enabled) return; + initObservabilitySdk({ + serviceName: process.env.MOSS_OTEL_SERVICE_NAME ?? opts.serviceName ?? 'moss', + otlpUrl: process.env.MOSS_OTEL_URL ?? opts.otlpUrl ?? 'http://localhost:4318', + enabled: true, + // tracing 开则 metrics 默认开（纠正 from-remote 两开关易漏配） + metricsEnabled: process.env.MOSS_METRICS_ENABLED !== '0', + // 本地文件 trace 默认开，MOSS_FILE_TRACE=0 关 + fileTraceEnabled: process.env.MOSS_FILE_TRACE !== '0', + workspaceDir: opts.workspaceDir, + }); +} + +export { shutdownObservabilitySdk as shutdownObservability } from './sdk.js'; + +export { + withSpan, + turnAttributes, + toolAttributes, + llmRequestAttributes, + sessionAttributes, +} from './tracing.js'; +export { mossMetrics } from './metrics.js'; +export { FileSpanProcessor, readTraceStats } from './file-trace.js'; +export type { SerializedSpan, TraceStats } from './file-trace.js'; + +// Re-export unchanged modules (drobotics already has these, identical to from-remote) +export { redactSensitiveData, parseTelemetryAllow } from './redact.js'; +export type { RedactOptions } from './redact.js'; +export { + logLLMUsage, + readUsageLog, + summarizeUsage, + formatUsageSummary, + estimateLLMCost, + registerModelPricing, +} from './llm-usage.js'; +export type { LLMUsageRecord, LLMUsageSummary } from './llm-usage.js'; + +// Legacy re-exports kept so cli-main's `import { setTracer } from './observability/tracing.js'` +// style imports continue to resolve through index too. +export { setTracer, getTracer, TraceRegistry } from './tracing.js'; +export type { Tracer, TraceSpan } from './tracing.js'; + +/** + * Inject W3C traceparent into outbound fetch headers for the current span. + * Returns headers unchanged when no active span (graceful degradation). + */ +import { propagation } from '@opentelemetry/api'; +export function propagateHeaders( + headers: Record = {}, +): Record { + try { + const injected: Record = { ...headers }; + propagation.inject(injected, { + set: (carrier, key, value) => { carrier[key] = String(value); }, + get: (carrier, key) => carrier[key], + }); + return injected; + } catch { + return headers; + } +} +``` + +- [ ] **Step 5: build + 跑 spec 确认通过** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent && node packages/moss-agent/test/observability-noop.spec.mjs` +Expected: PASS，输出 `[spec] observability-noop OK`。 + +- [ ] **Step 6: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/observability/sdk.ts packages/moss-agent/src/observability/index.ts packages/moss-agent/test/observability-noop.spec.mjs +git commit -m "feat(agent): add sdk.ts assembly + observability public entrypoint" +``` + +--- + +## Task 6: web-fetch / web-search 注入 traceparent + +**Files:** +- Modify: `packages/moss-agent/src/tools/web-fetch.ts`（line 541 附近 fetch headers） +- Modify: `packages/moss-agent/src/tools/web-search.ts`（line 233 附近 fetch headers） + +**Interfaces:** +- Consumes: Task 5 `propagateHeaders` +- Produces: 出站 HTTP 请求带 W3C traceparent（无 active span 时 passthrough） + +- [ ] **Step 1: web-fetch.ts — import propagateHeaders** + +在 `packages/moss-agent/src/tools/web-fetch.ts` 现有 import 区（line 16-22 之后）加： + +```typescript +import { propagateHeaders } from '../observability/index.js'; +``` + +- [ ] **Step 2: web-fetch.ts — 包裹 fetch headers** + +定位 line 541 附近的 `headers: {` 块（`fetchInit` 内）。改为在发起 fetch 前注入。找到 `res = await fetch(fetchUrl.toString(), fetchInit);`（line 552 附近），在其前面加： + +```typescript + if (fetchInit.headers && typeof fetchInit.headers === 'object' && !Array.isArray(fetchInit.headers)) { + fetchInit.headers = propagateHeaders(fetchInit.headers as Record); + } +``` + +（放在 `fetchInit` 已构造完成、`await fetch` 之前。若 `fetchInit.headers` 不存在则跳过，不影响原逻辑。） + +- [ ] **Step 3: web-search.ts — import propagateHeaders** + +在 `packages/moss-agent/src/tools/web-search.ts` 现有 import 区（line 32-35 之后）加： + +```typescript +import { propagateHeaders } from '../observability/index.js'; +``` + +- [ ] **Step 4: web-search.ts — 包裹主 fetch headers** + +定位 line 233 附近 `const res = await fetch(url, { ...init, signal: controller.signal });`。在它前面加： + +```typescript + const headersWithTrace = propagateHeaders((init?.headers ?? {}) as Record); + const res = await fetch(url, { ...init, headers: headersWithTrace, signal: controller.signal }); +``` + +（替换原 line 233 那一行。其余 line 272/370/477/591/675/735/798 的 `headers:` 块是 RSS/feed 子请求，本 task 不动——主搜索请求注入即可，避免过度改动。） + +- [ ] **Step 5: build 确认无类型错误** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent` +Expected: build 成功。 + +- [ ] **Step 6: 跑全量测试确认 web 工具无回归** + +Run: `cd /d/moss-drobotics && npm run test -w @rdk-moss/agent` +Expected: 全部 spec 通过。 + +- [ ] **Step 7: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/tools/web-fetch.ts packages/moss-agent/src/tools/web-search.ts +git commit -m "feat(agent): inject W3C traceparent into web-fetch / web-search outbound requests" +``` + +--- + +## Task 7: cli-main.ts — initObservability 初始化 + 退出 flush + +**Files:** +- Modify: `packages/moss-agent/src/cli-main.ts`（line 65 附近 import；line 611 `new MossAgent` 之前；main finally / 退出） + +**Interfaces:** +- Consumes: Task 5 `initObservability` / `shutdownObservability` +- Produces: CLI 启动时按 env 装配 SDK；退出时 flush 残留 span/metric + +- [ ] **Step 1: 加 import** + +在 `cli-main.ts` line 65 `import { setTracer } from './observability/tracing.js';` 之后加： + +```typescript +import { initObservability, shutdownObservability } from './observability/index.js'; +``` + +- [ ] **Step 2: agent 创建前调 initObservability** + +定位 line 611 附近 `const agent = new MossAgent({`。在它**之前**（line 608-610 区域，`// Enable OTel ...` 注释块如有也一并替换）插入： + +```typescript + // Enable observability (OTel tracing + metrics + local file trace) based on env. + // No-op when MOSS_OTEL_ENABLED is unset and MOSS_OTEL_URL absent. + initObservability({ workspaceDir: workspace }); +``` + +（`workspace` 变量在 line 465 已定义，此处可见。） + +- [ ] **Step 3: 注册退出 flush** + +`main()` 已有 `try { ... } finally { await closeMcpConnections(mcpConnections); }`（line 759-1016 区域）。在 finally 块内 `closeMcpConnections` 之后加 shutdown： + +定位 `} finally {` 接 `await closeMcpConnections(mcpConnections);`（line 1014-1015 附近），改为： + +```typescript + } finally { + await closeMcpConnections(mcpConnections); + await shutdownObservability(); + } +``` + +- [ ] **Step 4: 兜底 beforeExit flush（异常退出路径）** + +在 `cli-main.ts` 末尾 `main().catch(...)`（line 1019）**之前**加： + +```typescript +process.on('beforeExit', () => { void shutdownObservability(); }); +``` + +- [ ] **Step 5: build 确认无类型错误** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent` +Expected: build 成功。 + +- [ ] **Step 6: 跑全量测试确认无回归** + +Run: `cd /d/moss-drobotics && npm run test -w @rdk-moss/agent` +Expected: 全部 spec 通过。 + +- [ ] **Step 7: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/cli-main.ts +git commit -m "feat(agent): wire initObservability at CLI startup + flush on exit" +``` + +--- + +## Task 8: agent-loop-llm-call.ts — span 改名 + llm metrics + +**Files:** +- Modify: `packages/moss-agent/src/core/loop/agent-loop-llm-call.ts`（line 22 import；line 133 withSpan；success/catch 处加 metrics） + +**Interfaces:** +- Consumes: Task 4 `withSpan`/`turnAttributes`、Task 2 `mossMetrics` +- Produces: `moss.llm.request` span + `moss.llm.tokens` / `moss.llm.request.duration` metric + +- [ ] **Step 1: import mossMetrics** + +在 line 22 `import { withSpan, turnAttributes } from '../../observability/tracing.js';` 之后加： + +```typescript +import { mossMetrics } from '../../observability/index.js'; +``` + +- [ ] **Step 2: span 名改 moss.llm.request** + +定位 line 133-135： + +```typescript + const llmTurn = await withSpan( + 'agent.llm_turn', + turnAttributes(runId, state.turns, String(modelDef.id)), +``` + +把 `'agent.llm_turn'` 改为 `'moss.llm.request'`： + +```typescript + const llmTurn = await withSpan( + 'moss.llm.request', + turnAttributes(runId, state.turns, String(modelDef.id)), +``` + +- [ ] **Step 3: success 路径记 metrics** + +定位 line 177-191 区域 `if (llmTurn.usage) {` 块末尾、`return { control: 'continue', ...` 之前（line 191 `});` 之后），在 `recordLlmUsage(...)` 调用之后插入 metrics 记录： + +```typescript + // Metrics (noop when metrics disabled) + const llmModel = String(modelDef.id); + const llmDuration = Date.now() - llmTurnStartedAt; + mossMetrics.llmDuration.record(llmDuration, { model: llmModel }); + mossMetrics.llmTokens.add(llmTurn.usage.inputTokens, { direction: 'input', model: llmModel }); + mossMetrics.llmTokens.add(llmTurn.usage.outputTokens, { direction: 'output', model: llmModel }); +``` + +- [ ] **Step 4: catch 路径记 metrics** + +定位 line 213-223 区域 `} catch (llmError) {` 内的 `await recordLlmUsage({... success: false ...});` 之后（line 223 `});` 之后），插入： + +```typescript + // Metrics: record failed LLM call + mossMetrics.llmDuration.record(Date.now() - llmTurnStartedAt, { model: String(modelDef.id) }); +``` + +- [ ] **Step 5: build 确认无类型错误** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent` +Expected: build 成功。 + +- [ ] **Step 6: 跑全量测试** + +Run: `cd /d/moss-drobotics && npm run test -w @rdk-moss/agent` +Expected: 全部 spec 通过。 + +- [ ] **Step 7: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/core/loop/agent-loop-llm-call.ts +git commit -m "feat(agent): rename llm span to moss.llm.request + emit llm metrics" +``` + +--- + +## Task 9: execute-tool-call.ts — tool span + tool metrics + +**Files:** +- Modify: `packages/moss-agent/src/core/tools/execute-tool-call.ts`（line 30 附近 import；line 341 `startMs` 之后包 span；outcome 处记 metrics） + +**Interfaces:** +- Consumes: Task 4 `withSpan`/`toolAttributes`、Task 2 `mossMetrics` +- Produces: `moss.tool.invoke` span + `moss.tool.invocations` / `moss.tool.invoke.duration` metric + +- [ ] **Step 1: import** + +在 line 30 `import { runPreToolHookChain, validateToolInputObject } from './tool-pipeline.js';` 之后加： + +```typescript +import { withSpan, toolAttributes } from '../../observability/tracing.js'; +import { mossMetrics } from '../../observability/index.js'; +``` + +- [ ] **Step 2: 在 executeOneToolCall 内包 span + 记 metrics** + +定位 line 341 `const startMs = Date.now();`。它位于 `executeOneToolCall`（line 242 起）函数体内、`return { kind: 'completed', ... }`（line 540 附近）之前。 + +把 line 341 到函数 return 之间的主执行体用 `withSpan` 包裹。由于该函数体较长，采用**最小侵入**改法：在 `const startMs = Date.now();` 之后、原执行体最外层逻辑之外，用 withSpan 包住整个 try/catch。 + +实际改法——在 line 341 `const startMs = Date.now();` 之后插入： + +```typescript + return withSpan( + 'moss.tool.invoke', + toolAttributes(deps.sessionKey, call.name, call.id), + async (span) => { + try { + const outcome = await executeOneToolCallInner(call, deps); + const isErr = outcome.kind === 'completed' ? Boolean(outcome.isError) : true; + const dur = outcome.kind === 'completed' ? outcome.durationMs : Date.now() - startMs; + span.setAttribute('is_error', isErr); + if (outcome.kind === 'completed' && outcome.outcome) span.setAttribute('outcome', outcome.outcome); + mossMetrics.toolInvocations.add(1, { tool: call.name, status: isErr ? 'error' : 'ok' }); + mossMetrics.toolDuration.record(dur, { tool: call.name }); + return outcome; + } catch (err) { + mossMetrics.toolInvocations.add(1, { tool: call.name, status: 'error' }); + mossMetrics.toolDuration.record(Date.now() - startMs, { tool: call.name }); + throw err; + } + }, + ); +``` + +然后把**原** line 342 到 return 的函数体重命名为内部函数 `executeOneToolCallInner`： + +- 在 `export async function executeOneToolCall(...)` 签名之后、`const startMs = Date.now();` **之前**，原函数体不变； +- 将原顶层 `export async function executeOneToolCall` 改为 `async function executeOneToolCallInner`（去掉 export）； +- 在其**之后**新增对外导出的 `executeOneToolCall`，它只做「调 inner + 包 span/metrics」： + +```typescript +export async function executeOneToolCall( + call: ExecuteToolCallRef, + deps: ExecuteToolCallDeps, +): Promise { + const startMs = Date.now(); + return withSpan( + 'moss.tool.invoke', + toolAttributes(deps.sessionKey, call.name, call.id), + async (span) => { + try { + const outcome = await executeOneToolCallInner(call, deps); + const isErr = outcome.kind === 'completed' ? Boolean(outcome.isError) : true; + const dur = outcome.kind === 'completed' ? outcome.durationMs : Date.now() - startMs; + span.setAttribute('is_error', isErr); + if (outcome.kind === 'completed' && outcome.outcome) span.setAttribute('outcome', outcome.outcome); + mossMetrics.toolInvocations.add(1, { tool: call.name, status: isErr ? 'error' : 'ok' }); + mossMetrics.toolDuration.record(dur, { tool: call.name }); + return outcome; + } catch (err) { + mossMetrics.toolInvocations.add(1, { tool: call.name, status: 'error' }); + mossMetrics.toolDuration.record(Date.now() - startMs, { tool: call.name }); + throw err; + } + }, + ); +} +``` + +> 实现者注意：`ExecuteToolCallRef` / `ExecuteToolCallDeps` / `ExecuteToolCallOutcome` 是原函数已有的类型名（见文件内 `import type` 与函数签名），原样沿用；`outcome.outcome` 是 `ToolResultOutcome` 字符串字段（见 line 219-221）。若 TS 报 outcome 类型不匹配，按文件内实际类型调整 attribute 取值，不改控制流。 + +- [ ] **Step 3: build 确认无类型错误** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent` +Expected: build 成功（如有类型错误，按文件内实际类型修正 attribute 取值，不改 span 包裹与 metrics 逻辑）。 + +- [ ] **Step 4: 跑全量测试** + +Run: `cd /d/moss-drobotics && npm run test -w @rdk-moss/agent` +Expected: 全部 spec 通过（tool 执行相关 spec 不受影响——内层逻辑未变）。 + +- [ ] **Step 5: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/core/tools/execute-tool-call.ts +git commit -m "feat(agent): wrap tool execution in moss.tool.invoke span + emit tool metrics" +``` + +--- + +## Task 10: agent-loop.ts — moss.agent.turn span + +**Files:** +- Modify: `packages/moss-agent/src/core/loop/agent-loop.ts`（line 36 附近 import；主循环 line 330 `outerLoop: while (true)` 内每轮迭代外包 span） + +**Interfaces:** +- Consumes: Task 4 `withSpan`/`turnAttributes`、`runId`（循环内已有变量） +- Produces: `moss.agent.turn` span（每轮一个） + +- [ ] **Step 1: import** + +定位 line 36 `import { executeLlmTurn } from './agent-loop-llm-call.js';` 附近，在循环文件 import 区加： + +```typescript +import { withSpan, turnAttributes } from '../../observability/tracing.js'; +``` + +- [ ] **Step 2: 每轮迭代外包 span** + +定位 line 330 `outerLoop: while (true) {` 与 line 337 内层 `while (state.hasMoreToolCalls || state.pendingMessages.length > 0) {`。 + +在 line 356 `state.turns++;` 之后、该轮迭代主体（`executeLlmTurn` 等，line 426）之前，用 `withSpan` 包住单轮的 LLM+工具处理。由于 `outerLoop` / 内层 while 含 `break`/`continue`/`state.turns--` 等控制流，**最小侵入**做法是：在内层 while 循环体的开头（line 337 `while (...) {` 之后第一行）开 span，但要避免 break/continue 跳过 span.end。 + +**采用安全包裹法**——把 line 356 `state.turns++;` 之后到该轮结束的语句提取到 `await withSpan('moss.agent.turn', ..., async (span) => { ... })` 内，并让内层 while 改为： + +在 line 356 `state.turns++;` 之后插入（替换其后到内层 while 结束的整段为 span 包体）： + +```typescript + await withSpan('moss.agent.turn', turnAttributes(runId, state.turns, String(modelDef.id)), async () => { + // ... 原 line 358 起到内层 while 结束的全部语句（processLlmResponse / continue / break 等）原样搬入 ... + }); +``` + +> 实现者注意：`runId`、`modelDef` 在该作用域已定义（见 line 147 `runId`、循环参数）。内层 while 里的 `continue`/`break` 在 async 闭包内会作用于该闭包——若 `break` 需跳出 `outerLoop`，改用标志位（`let breakOuter = false;` 在闭包内设、闭包外判断）以替代原 `break outerLoop`。逐一核对 line 337-540 区域每个 `break`/`continue` 的跳转目标，确保 span 总能 end（withSpan 的 finally 保证）。不改任何条件判断与状态赋值的值。 + +- [ ] **Step 3: build 确认无类型错误** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent` +Expected: build 成功。若 TS 报闭包内 break/continue 作用域问题，按 Step 2 的标志位法修正，不改控制流语义。 + +- [ ] **Step 4: 跑全量测试（含 agent-loop 相关 spec）** + +Run: `cd /d/moss-drobotics && npm run test -w @rdk-moss/agent` +Expected: 全部 spec 通过。重点关注 `agent-loop-push-guard-isolation.spec.mjs`、`autonomous-loop.spec.mjs`。 + +- [ ] **Step 5: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/core/loop/agent-loop.ts +git commit -m "feat(agent): wrap each loop turn in moss.agent.turn span" +``` + +--- + +## Task 11: moss-agent.ts — moss.session 根 span + session metrics + +**Files:** +- Modify: `packages/moss-agent/src/core/agent/moss-agent.ts`（import 区；line 412 `chat()` 内） + +**Interfaces:** +- Consumes: Task 4 `withSpan`/`sessionAttributes`、Task 2 `mossMetrics` +- Produces: `moss.session` 根 span + `moss.session.count` / `moss.session.duration` / `moss.session.tool_count` metric + +- [ ] **Step 1: import** + +在 `moss-agent.ts` import 区（文件头部，`from '../loop/agent-loop.js'` 等附近）加： + +```typescript +import { withSpan, sessionAttributes } from '../../observability/tracing.js'; +import { mossMetrics } from '../../observability/index.js'; +``` + +- [ ] **Step 2: chat() 内包 session span + 记 metrics** + +定位 line 412 `async chat(sessionKey: string, userMessage: string, options?: ChatOptions): Promise {` 与 line 413 `let finalResult: ChatResult | undefined;`。 + +在 `chat()` 函数体最外层用 `withSpan('moss.session', sessionAttributes(runId, model, sessionKey), ...)` 包裹主体，并在结束记 metrics。最小侵入：在 line 412 函数体开头加 `const sessionStart = Date.now();`，把原返回路径包进 withSpan。 + +具体——找到 `chat()` 的最终 `return`（返回 `ChatResult`，通常在函数末尾，`finalResult` 被赋值处）。在 line 412 `let finalResult: ChatResult | undefined;` 之后插入： + +```typescript + const sessionStart = Date.now(); + return withSpan( + 'moss.session', + sessionAttributes(/* runId */ thisRunId, /* model */ String(model), sessionKey), + async () => { + // ... 原 chat() 主体（line 413 起到原 return）原样搬入，return 改为 return 到闭包 ... + }, + ).finally(() => { + const outcome = finalResult?.stopReason === 'end_turn' ? 'ok' : (finalResult ? 'incomplete' : 'error'); + mossMetrics.sessionCount.add(1, { outcome }); + mossMetrics.sessionDuration.record(Date.now() - sessionStart, { outcome }); + mossMetrics.sessionToolCount.record(finalResult?.toolCalls?.length ?? 0, { outcome }); + }); +``` + +> 实现者注意： +> - `runId` / `model` 在 `chat()` 作用域内的实际变量名按文件内为准（chat 可能在内部调用 `this.run(...)` 产生 runId；若 runId 在 withSpan 之前不可用，改用 `sessionKey` + `model` 作 attributes，`runId` 在闭包内拿到后 `span.setAttribute('runId', runId)`）。先读 line 412-570 区段确认变量可见性再填 attributes。 +> - `finalResult.toolCalls` 字段名按 `ChatResult` 类型（见 line 141 `SharedChatResult`）确认；若为 `toolsUsed` 或其他名，按实际字段。 +> - `.finally` 在 `finalResult` 赋值后执行（withSpan 的 fn return 时 finalResult 已被赋值），故 metric 能读到结果。 + +- [ ] **Step 3: build 确认无类型错误** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent` +Expected: build 成功。若 attributes 字段名/变量名不符，按文件实际修正，不改 span/metrics 逻辑与返回值。 + +- [ ] **Step 4: 跑全量测试** + +Run: `cd /d/moss-drobotics && npm run test -w @rdk-moss/agent` +Expected: 全部 spec 通过。 + +- [ ] **Step 5: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/src/core/agent/moss-agent.ts +git commit -m "feat(agent): wrap chat() in moss.session root span + emit session metrics" +``` + +--- + +## Task 12: 端到端集成验证 + +**Files:** +- Test: `packages/moss-agent/test/observability-integration.spec.mjs` + +**Interfaces:** +- Consumes: Task 5 `initObservability` / `shutdownObservability`、`withSpan`、`mossMetrics`、`readTraceStats` +- Produces: 验证三层 span + metrics + 本地文件 trace 在启用后真正产生数据 + +- [ ] **Step 1: 写集成 spec（启用 → 跑 withSpan → 文件落盘 → shutdown flush）** + +Create `packages/moss-agent/test/observability-integration.spec.mjs`: + +```javascript +#!/usr/bin/env node +// Integration: enable SDK, run nested withSpan (session→turn→llm), verify file trace lands. +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import os from 'node:os'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'index.js')).href); +const { initObservability, shutdownObservability, withSpan, mossMetrics } = mod; +const traceMod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'file-trace.js')).href); +const { readTraceStats } = traceMod; + +const tmp = await fs.mkdtemp(path.join(os.tmpdir(), 'moss-int-')); +process.env.MOSS_OTEL_ENABLED = '1'; +process.env.MOSS_OTEL_URL = 'http://localhost:4318'; // receiver 可能没起，fire-and-forget 不抛 +process.env.MOSS_FILE_TRACE = '1'; + +initObservability({ workspaceDir: tmp }); + +// 三层 span：session → turn → llm（模拟 agent 调用栈） +await withSpan('moss.session', { runId: 'r1', model: 'm', sessionKey: 'sk' }, async () => { + return withSpan('moss.agent.turn', { runId: 'r1', turn: 1, model: 'm' }, async () => { + return withSpan('moss.llm.request', { runId: 'r1', model: 'm', inputTokens: 100 }, async (span) => { + span.setAttribute('outputTokens', 50); + mossMetrics.llmTokens.add(100, { direction: 'input', model: 'm' }); + mossMetrics.llmTokens.add(50, { direction: 'output', model: 'm' }); + mossMetrics.llmDuration.record(42, { model: 'm' }); + return 'done'; + }); + }); +}); + +await shutdownObservability(); + +const file = path.join(tmp, '.moss', 'analytics', 'traces.jsonl'); +const stats = await readTraceStats(file); +assert.equal(stats.totalSpans, 3, 'three nested spans landed in file'); +assert.ok(stats.byName['moss.session'], 'session span present'); +assert.ok(stats.byName['moss.agent.turn'], 'turn span present'); +assert.ok(stats.byName['moss.llm.request'], 'llm span present'); + +await fs.rm(tmp, { recursive: true, force: true }); +delete process.env.MOSS_OTEL_ENABLED; +delete process.env.MOSS_OTEL_URL; +delete process.env.MOSS_FILE_TRACE; +console.error('[spec] observability-integration OK'); +``` + +- [ ] **Step 2: 运行确认（receiver 未起也应 PASS——文件 trace 不依赖网络）** + +Run: `cd /d/moss-drobotics && npm run build -w @rdk-moss/agent && node packages/moss-agent/test/observability-integration.spec.mjs` +Expected: PASS，输出 `[spec] observability-integration OK`。若因 OTLP 发送超时拖慢但不抛错属正常；若卡住，确认 OTLP exporter 是 fire-and-forget（`BatchSpanProcessor` 异步发送，shutdown 时 flush 有超时上限）。 + +- [ ] **Step 3: 跑全量测试确认整体无回归** + +Run: `cd /d/moss-drobotics && npm run test -w @rdk-moss/agent` +Expected: 全部 spec 通过。 + +- [ ] **Step 4: 手动 smoke（可选，需 receiver）** + +若本地 `D:\otel` receiver 可起： + +Run（在新窗口）：`D:\otel\start-receiver.cmd` +Run：`cd /d/moss-drobotics && MOSS_OTEL_ENABLED=1 MOSS_OTEL_SERVICE_NAME=moss node packages/moss-agent/dist/cli.js "hello"` +Expected: 面板 `http://localhost:3000` 看到 `moss.session` → `moss.agent.turn` → `moss.llm.request` span 树 + token/duration 指标；`.moss/analytics/traces.jsonl` 有内容。 + +- [ ] **Step 5: Commit** + +```bash +cd /d/moss-drobotics +git add packages/moss-agent/test/observability-integration.spec.mjs +git commit -m "test(agent): add observability end-to-end integration spec" +``` + +--- + +## Self-Review 结果 + +**1. Spec 覆盖：** +- 统一 SDK（一处 NodeSDK）→ Task 5 sdk.ts ✅ +- 三层 span（session→turn→llm/tool）→ Task 11 / 10 / 8 / 9 ✅ +- 共享 Resource → Task 5（resourceFromAttributes 一处）✅ +- recordException → Task 4 withSpan catch ✅ +- 批处理 → Task 5 BatchSpanProcessor ✅ +- 本地文件 trace 保留 → Task 3 FileSpanProcessor + Task 5 挂载 ✅ +- tracing 开则 metrics 默认开 → Task 5 index.ts ✅ +- 优雅 flush → Task 7 shutdown + beforeExit ✅ +- 关闭即零开销 → Task 4/2/5 noop + Task 5 spec 验证 ✅ +- web 注入 traceparent → Task 6 ✅ +- metrics 命名纠正（tool_count 非 turns）→ Task 2 ✅ +- 砍 ab-testing / 不搬 trace-exporter 旧实现 → 文件结构未含二者 ✅ +- 依赖 → Task 1 ✅ +- 验证 → Task 12 ✅ + +**2. Placeholder 扫描：** Task 9/10/11 含「按文件实际类型/变量名修正」的指引——这是因 drobotics 代码具体类型（`ExecuteToolCallOutcome.outcome`、`ChatResult.toolCalls`、`chat()` 内 runId 可见性）需在实现时核对，已给出核对位置与不改控制流的约束，非空 TODO。其余步骤均有完整代码。 + +**3. Type consistency：** `withSpan` 签名（Task 4）`withSpan(name, attrs, fn: (span: Span & TraceSpan) => Promise)` 被 Task 8/9/10/11 一致使用；`mossMetrics` 字段名（Task 2：`llmTokens`/`llmDuration`/`toolInvocations`/`toolDuration`/`sessionCount`/`sessionDuration`/`sessionToolCount`）与 Task 8/9/11 调用一致；`propagateHeaders`（Task 5 导出）与 Task 6 调用一致；`readTraceStats`（Task 3 导出）与 Task 12 调用一致。 diff --git a/docs/superpowers/specs/2026-07-16-observability-instrumentation-design.md b/docs/superpowers/specs/2026-07-16-observability-instrumentation-design.md new file mode 100644 index 00000000..e92a21df --- /dev/null +++ b/docs/superpowers/specs/2026-07-16-observability-instrumentation-design.md @@ -0,0 +1,361 @@ +--- +title: Moss-Drobotics 可观测性（埋点）重新设计 +status: draft +date: 2026-07-16 +owner: tongchun.zhao +--- + +# 重新埋点：D:\moss-drobotics 可观测性设计 + +## 1. 背景与目标 + +`D:\moss-drobotics` 是 `D-Robotics/moss` 最新代码的克隆，目前**没有任何运行时可观测性**：LLM 调用、工具执行、session 这些核心环节既不发 trace 也不发 metric，出问题只能靠日志盲猜。 + +参考实现 `D:\moss-from-remote`（用户的 fork）已经加了完整的 OTel 埋点，能发 trace+metric 到本地 receiver（端口 4318）并在 3000 面板展示。但它有一处技术债：**tracing 手写 OTLP/JSON（`otel-bridge.ts`，刻意零 SDK 依赖），metrics 却用了官方 `@opentelemetry/sdk-metrics`**——注释自己立的「不引入 SDK」原则被 metrics 打破，导致 trace 与 metric 各造一套 Resource、两套发送逻辑。 + +本设计目标：**从头为 moss-drobotics 埋点，纠正混搭债，同时保留本地文件 trace**。不是把 from-remote 原样搬过来。 + +### 设计原则 + +1. **一处 SDK**：tracing 与 metrics 同源，共享一个 Resource、一个 NodeSDK 实例。消除 from-remote 的混搭。 +2. **本地文件 trace 保留**：作为独立 SpanProcessor 落盘，与 OTLP exporter 并存（不是 from-remote 那套手写 tracer + 单独 exportSpan）。 +3. **YAGNI**：不搬 `ab-testing.ts`（A/B 测试不属可观测性）、不搬 from-remote 的 `trace-exporter.ts` 老实现（它假设了手写 tracer 的 SerializedSpan 形状，SDK 模式下重写为 SpanProcessor）。 +4. **error 走 recordException**：异常自动记成 span event，redact 仍套在 message 上防敏感信息泄露。 +5. **优雅 flush**：进程退出前 flush 残留 span/metric，不丢末批数据。 +6. **零改动业务语义**：埋点不改变任何现有控制流，关闭时零开销（noop 桩）。 + +## 2. 架构 + +### 2.1 三层 span 结构 + +贴合 agent 真实调用栈（drobotics 实际代码：`moss-agent.chat()` → `agent-loop` 主循环 → `executeLlmTurn` / `processLlmResponse` → `executeOneToolCall`）： + +``` +moss.session ← moss-agent.chat() 整个用户回合（根 span，1 个/用户消息） +├─ moss.agent.turn ← agent-loop 主循环每轮迭代（N 个/session） +│ ├─ moss.llm.request ← executeLlmTurn 里单次 LLM 调用 +│ └─ moss.tool.invoke ← executeOneToolCall 里单次工具执行（可并行） +``` + +上下文传播用 OTel SDK 自带的 `context` + `AsyncLocalStorage`（SDK 内部已实现），**业务代码不需要手动透传 `parentSpan` 参数**。这是对 from-remote 那套「到处传 parentSpan」的简化——手写桥接才需要手动透传，SDK 模式下子 span 自动继承当前 context 的父 span。 + +### 2.2 数据流 + +``` +业务调用点 (withSpan / mossMetrics.xxx) + │ + ▼ +TracerProvider / MeterProvider (NodeSDK 装配) + ├── BatchSpanProcessor ──► OTLPTraceExporter ──► http://localhost:4318/v1/traces + ├── FileSpanProcessor ──► traces.jsonl 落盘 (本地文件 trace，新增) + └── PeriodicMetricReader ─► OTLPMetricExporter ──► http://localhost:4318/v1/metrics + │ + ▼ + 本地 receiver (D:\otel) + 面板 localhost:3000 +``` + +三个 exporter 共享同一个 Resource（`service.name` + `service.version` + `process.*`），后端看到的 trace、metric、本地文件 resource 属性完全一致。 + +## 3. 文件组织 + +``` +packages/moss-agent/src/observability/ + index.ts ← 公共入口：initObservability / shutdownObservability / withSpan / mossMetrics + sdk.ts ← NodeSDK 装配、Resource、三个 exporter/processor 集中一处 + tracing.ts ← withSpan 包装、attributes 构造器（turn/tool/llm/session）、Tracer handle + metrics.ts ← mossMetrics 句柄 + instruments 定义 + file-trace.ts ← FileSpanProcessor + JSONL 落盘 + getStats 聚合（重写自 trace-exporter） + redact.ts ← 保留（drobotics 已有，与 from-remote 一致，不动） + llm-usage.ts ← 保留（drobotics 已有，本地 usage 日志，与埋点无关，不动） +``` + +对比 from-remote：砍掉 `otel-bridge.ts`（手写 OTLP/JSON，被 SDK 取代）、`ab-testing.ts`（非可观测性）；`trace-exporter.ts` 重写为 `file-trace.ts`（从「手写 tracer 的 SerializedSpan 落盘」改为「SDK SpanProcessor 落盘」，形状对齐 SDK 的 `ReadableSpan`）。 + +## 4. 组件设计 + +### 4.1 `sdk.ts` —— SDK 装配 + +单一初始化点。tracing 与 metrics 同一个 NodeSDK，共享 Resource。 + +```typescript +import { NodeSDK } from '@opentelemetry/sdk-node'; +import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'; +import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; +import { BatchSpanProcessor, SpanExporter } from '@opentelemetry/sdk-trace-base'; +import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; +import { resourceFromAttributes } from '@opentelemetry/resources'; +import { semconv } from '@opentelemetry/semantic-conventions'; +import { FileSpanProcessor } from './file-trace.js'; + +export interface ObservabilityConfig { + serviceName: string; + otlpUrl: string; // receiver 根地址，如 http://localhost:4318 + enabled: boolean; // tracing 开关 + metricsEnabled: boolean; + fileTraceEnabled: boolean; + workspaceDir: string; // 落盘到 {workspaceDir}/.moss/analytics/traces.jsonl +} + +let sdk: NodeSDK | null = null; + +export function initObservabilitySdk(cfg: ObservabilityConfig): void { + if (!cfg.enabled) return; + const pkg = readPackageVersion(); // 读 packages/moss-agent/package.json 的 version + const resource = resourceFromAttributes({ + [semconv.ATTR_SERVICE_NAME]: cfg.serviceName, + [semconv.ATTR_SERVICE_VERSION]: pkg, // 当前 0.5.3，但不硬编码 + }); + + const spanProcessors: SpanProcessor[] = [new BatchSpanProcessor( + new OTLPTraceExporter({ url: `${cfg.otlpUrl}/v1/traces` }) + )]; + if (cfg.fileTraceEnabled) { + spanProcessors.push(new FileSpanProcessor(cfg.workspaceDir)); + } + + const readers = cfg.metricsEnabled ? [new PeriodicExportingMetricReader({ + exporter: new OTLPMetricExporter({ url: `${cfg.otlpUrl}/v1/metrics` }), + exportIntervalMillis: 10_000, + })] : []; + + // NodeSDK 的 metricReader 选项接单个 MetricReader；若需要多 reader， + // 用底层 MeterProvider({ resource, readers }) 自行装配（from-remote 即如此）。 + // 本设计单 metric reader，走 NodeSDK 即可： + sdk = new NodeSDK({ + resource, + spanProcessors, + metricReader: readers[0], // 单数；readers 为空时 undefined，metrics 关闭 + }); + sdk.start(); +} + +export async function shutdownObservabilitySdk(): Promise { + if (!sdk) return; + await sdk.shutdown(); // flush 残留 span/metric + sdk = null; +} +``` + +### 4.2 `tracing.ts` —— withSpan 与 attributes + +```typescript +import { trace, context, SpanStatusCode } from '@opentelemetry/api'; +import { errorMessage } from '../errors.js'; +import { redactSensitiveData } from './redact.js'; + +const tracer = trace.getTracer('moss-agent'); + +export async function withSpan( + name: string, + attributes: Record | undefined, + fn: (span: Span) => Promise, +): Promise { + const span = tracer.startSpan(name, { attributes }); + return context.with(trace.setSpan(context.active(), span), async () => { + try { + const result = await fn(span); + span.setStatus({ code: SpanStatusCode.OK }); + return result; + } catch (err) { + span.recordException(err); // 异常自动记成 event：type/message/stack + span.setStatus({ + code: SpanStatusCode.ERROR, + message: redactSensitiveData(errorMessage(err)), // message 上 redact + }); + throw err; + } finally { + span.end(); + } + }); +} + +// attributes 构造器：集中定义 span 维度，避免散落 +export const sessionAttributes = (runId: string, model: string, sessionKey: string) => + ({ runId, model, sessionKey }); +export const turnAttributes = (runId: string, turn: number, model: string) => + ({ runId, turn, model }); +export const llmAttributes = (runId: string, model: string, inputTokens: number) => + ({ runId, model, inputTokens }); +export const toolAttributes = (runId: string, toolName: string, toolCallId: string) => + ({ runId, toolName, toolCallId }); +``` + +注意：drobotics 现有 `agent-loop-llm-call.ts:133` 已经在用 `withSpan('agent.llm_turn', turnAttributes(...))`。迁移时 span 名从 `agent.llm_turn` 改为 `moss.llm.request`（命名规范统一），attributes 构造器签名保持不变以最小化调用点改动。 + +### 4.3 `metrics.ts` —— instruments + +```typescript +import { metrics } from '@opentelemetry/api'; + +const meter = metrics.getMeter('moss-agent'); + +export const mossMetrics = { + // LLM + llmTokens: meter.createCounter('moss.llm.tokens', { unit: '{token}' }), + llmDuration: meter.createHistogram('moss.llm.request.duration', { unit: 'ms' }), + // tool + toolInvocations: meter.createCounter('moss.tool.invocations'), + toolDuration: meter.createHistogram('moss.tool.invoke.duration', { unit: 'ms' }), + // session + sessionCount: meter.createCounter('moss.session.count'), + sessionDuration: meter.createHistogram('moss.session.duration', { unit: 'ms' }), + sessionToolCount: meter.createHistogram('moss.session.tool_count'), // 纠正错位：每轮工具数，不是 turns +}; +``` + +SDK 的 `meter` 在未 `setGlobalMeterProvider` 时返回 noop meter，instruments 都是 noop——关闭即零开销，业务代码无条件调用 `.add()`/`.record()`，与 from-remote 的 noop 模式一致。 + +### 4.4 `file-trace.ts` —— 本地文件 trace（SpanProcessor 重写） + +from-remote 的 `TraceFileExporter` 假设手写 tracer 产出的 `SerializedSpan`。SDK 模式下重写为 `SpanProcessor`，消费 SDK 的 `ReadableSpan`： + +```typescript +import { SpanProcessor, ReadableSpan } from '@opentelemetry/sdk-trace-base'; +import fs from 'node:fs/promises'; +import path from 'node:path'; + +const FLUSH_INTERVAL_MS = 30_000; + +export class FileSpanProcessor implements SpanProcessor { + private buffer: ReadableSpan[] = []; + private timer: ReturnType | null = null; + private file: string; + + constructor(workspaceDir: string) { + this.file = path.join(workspaceDir, '.moss', 'analytics', 'traces.jsonl'); + this.timer = setInterval(() => this.flush(), FLUSH_INTERVAL_MS); + } + + onStart(): void {} // 无需处理 + onEnd(span: ReadableSpan): void { this.buffer.push(span); } + + async flush(): Promise { + if (this.buffer.length === 0) return; + const snapshot = this.buffer.splice(0); + const lines = snapshot.map(serializeSpan).join('\n') + '\n'; + try { + await fs.mkdir(path.dirname(this.file), { recursive: true }); + await fs.appendFile(this.file, lines, 'utf-8'); + } catch { /* never block agent */ } + } + + async shutdown(): Promise { + if (this.timer) clearInterval(this.timer); + await this.flush(); + } + async forceFlush(): Promise { await this.flush(); } +} +``` + +`serializeSpan` 把 `ReadableSpan` 拍平成 JSONL 一行（name / startTimeUnixNano / endTimeUnixNano / attributes / events / status），形状对齐 from-remote 的 `SerializedSpan` 便于复用其 `getStats` 聚合逻辑（保留 stats 面板能力）。 + +### 4.5 `index.ts` —— 公共入口与初始化 + +```typescript +import { initObservabilitySdk, shutdownObservabilitySdk } from './sdk.js'; + +export interface InitOptions { + workspaceDir: string; + serviceName?: string; + otlpUrl?: string; +} + +export function initObservability(opts: InitOptions): void { + const enabled = process.env.MOSS_OTEL_ENABLED === '1' || !!process.env.MOSS_OTEL_URL; + if (!enabled) return; // noop + initObservabilitySdk({ + serviceName: process.env.MOSS_OTEL_SERVICE_NAME ?? opts.serviceName ?? 'moss', + otlpUrl: process.env.MOSS_OTEL_URL ?? opts.otlpUrl ?? 'http://localhost:4318', + enabled: true, + // tracing 开则 metrics 默认开（纠正 from-remote 两个开关易漏配） + metricsEnabled: process.env.MOSS_METRICS_ENABLED !== '0', + // 本地文件 trace：默认开（用户要求保留），MOSS_FILE_TRACE=0 关 + fileTraceEnabled: process.env.MOSS_FILE_TRACE !== '0', + workspaceDir: opts.workspaceDir, + }); +} + +export { shutdownObservabilitySdk as shutdownObservability } from './sdk.js'; +export { withSpan, sessionAttributes, turnAttributes, llmAttributes, toolAttributes } from './tracing.js'; +export { mossMetrics } from './metrics.js'; +export { FileSpanProcessor } from './file-trace.js'; +``` + +`redact.ts`、`llm-usage.ts` 的导出从 drobotics 现有 `index.ts` 保留。 + +## 5. 调用点（5 处业务 + 1 处工具函数） + +| # | 文件 | 埋什么 | drobotics 现状 | +|---|------|--------|----------------| +| 1 | `cli-main.ts` | `initObservability()` 一次（agent 创建前）；退出时 `shutdownObservability()` | 仅有 `setTracer('console')` | +| 2 | `core/agent/moss-agent.ts` | `chat()` 开 `moss.session` 根 span + session metrics（count/duration/tool_count） | 无 | +| 3 | `core/loop/agent-loop.ts` | 主循环每轮迭代外包 `moss.agent.turn` span（新增这层） | 无 | +| 4 | `core/loop/agent-loop-llm-call.ts` | `withSpan('moss.llm.request')` + llm metrics | 已有 `withSpan('agent.llm_turn')`，需改名+加 metrics | +| 5 | `core/tools/execute-tool-call.ts` | `withSpan('moss.tool.invoke')` + tool metrics | 无 | +| 6 | `tools/web-fetch.ts` + `web-search.ts` | `propagation.inject(headers)` 注入 traceparent（收敛到一个工具函数） | 无 | + +对比 from-remote：少一处「手动透传 parentSpan」（SDK 自动传播）；web-fetch/web-search 的 `injectTraceparent` 改用 SDK `propagation.inject`。 + +## 6. 环境变量 + +| 变量 | 默认 | 作用 | +|------|------|------| +| `MOSS_OTEL_ENABLED` | unset=关 | 总开关（设 `1` 或配 `MOSS_OTEL_URL` 即开） | +| `MOSS_OTEL_URL` | `http://localhost:4318` | OTLP 根地址 | +| `MOSS_OTEL_SERVICE_NAME` | `moss` | service.name | +| `MOSS_METRICS_ENABLED` | tracing 开时默认开 | 设 `0` 单独关 metrics | +| `MOSS_FILE_TRACE` | 开 | 设 `0` 关本地文件 trace | +| `MOSS_TRACE_SAMPLE_RATIO` | 1.0 | 采样率（SDK TailSampling 或 ParentBased，初版可用 alwaysOn） | + +对比 from-remote：去掉了需要同时设 `MOSS_OTEL_ENABLED` + `MOSS_METRICS_ENABLED` 的冗余（tracing 开则 metrics 默认开）。 + +## 7. 依赖 + +`packages/moss-agent/package.json` 新增（drobotics 当前一个 OTel 包都没有）： + +```json +"@opentelemetry/api": "^1.9.0", +"@opentelemetry/sdk-node": "^0.200.0", +"@opentelemetry/sdk-trace-base": "^0.200.0", +"@opentelemetry/sdk-metrics": "^2.9.0", +"@opentelemetry/exporter-trace-otlp-http": "^0.200.0", +"@opentelemetry/exporter-metrics-otlp-http": "^0.220.0", +"@opentelemetry/resources": "^2.9.0", +"@opentelemetry/semantic-conventions": "^1.43.0" +``` + +**版本对齐策略（关键，避免歧义）**：OTel JS 分两条版本线——experimental `0.x`（`sdk-node` / `sdk-trace-*` / `exporter-trace-*`，仍带 `-experimental` tag）与 stable `1.x`/`2.x`（`api` / `sdk-metrics` / `resources` / `semantic-conventions`）。两者可共存——from-remote 已用 `api@1.9 + sdk-metrics@2.9 + exporter-metrics-otlp-http@0.220` 验证可跑。本设计补充 tracing 的 experimental 包，与 metrics 现有版本线各自向前对齐：tracing 侧统一 `^0.200`，metrics 侧沿用 from-remote 的 `^2.9` / `^0.220`。 + +实现时以 `npm install` 实际解析的版本为准，不锁死补丁号；若 `sdk-node@0.200` 与 `sdk-metrics@2.9` 出现 peer 冲突，优先把 tracing experimental 包升到与 `exporter-metrics-otlp-http@0.220` 同一 experimental 线（`^0.220`），保持 experimental 包彼此对齐。 + +## 8. 关闭即零开销 + +- 未 `setGlobalMeterProvider` 时，`metrics.getMeter()` 返回 noop meter，所有 instrument 是 noop——`.add()`/`.record()` 无条件调用零成本。 +- `trace.getTracer()` 返回的 tracer 在未注册 provider 时是 noop tracer，`startSpan` 返回 noop span。 +- `initObservability()` 在 `enabled=false` 时直接 return，不启动任何 timer/SDK。 +- 业务代码不写任何 `if (tracingEnabled)` 分支——全靠 noop 桩。 + +## 9. 错误处理 + +- `initObservability` 失败（receiver 连不上、SDK 构造异常）只记一行 log，不抛——监控是 best-effort。 +- `withSpan` 里业务抛错时 `recordException` + `setStatus(ERROR)`，再原样 rethrow——埋点不吞错误。 +- `FileSpanProcessor.flush` 落盘失败静默——不阻塞 agent。 +- 进程退出：`process.on('beforeExit', shutdownObservability)` 兜底 flush；CLI 正常退出路径也显式调用。 + +## 10. 验证 + +1. `npm install` 装齐依赖后 `npm run -w @rdk-moss/agent build` 通过（TS 类型对齐）。 +2. `MOSS_OTEL_ENABLED=1 node packages/moss-agent/dist/cli.js` 跑一次对话，确认： + - receiver（4318）收到 trace（session→turn→llm/tool 三层 span 在面板能展开成树）。 + - 4318 收到 metrics（token/duration/invocation 计数）。 + - `.moss/analytics/traces.jsonl` 有落盘内容，行数 = span 数。 +3. 不设任何 env 跑一次，确认无报错、无额外开销、无文件产生。 +4. 关掉 receiver 跑一次，确认 agent 正常工作（fire-and-forget）。 + +## 11. 不做（YAGNI） + +- 不搬 `ab-testing.ts`——A/B 测试是实验框架，与可观测性无关。 +- 不做分布式 context 跨进程传播（moss 是单进程 CLI，W3C traceparent 注入出站 HTTP 已够）。 +- 不做 TailSampling——本地单机，alwaysOn 采样即可，`MOSS_TRACE_SAMPLE_RATIO` 预留但初版不接。 +- 不做 metrics 语义到 OTel GenAI semconv 的完整对齐（那是另一项工作），只保证命名清晰、单位正确。 diff --git a/package-lock.json b/package-lock.json index 2e7495bd..9ddf6181 100644 --- a/package-lock.json +++ b/package-lock.json @@ -171,6 +171,118 @@ "node": "^12.22.0 || ^14.17.0 || >=16.0.0" } }, + "node_modules/@grpc/grpc-js": { + "version": "1.14.4", + "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.14.4.tgz", + "integrity": "sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ==", + "license": "Apache-2.0", + "dependencies": { + "@grpc/proto-loader": "^0.8.0", + "@js-sdsl/ordered-map": "^4.4.2" + }, + "engines": { + "node": ">=12.10.0" + } + }, + "node_modules/@grpc/proto-loader": { + "version": "0.8.1", + "resolved": "https://registry.npmjs.org/@grpc/proto-loader/-/proto-loader-0.8.1.tgz", + "integrity": "sha512-wtF6h+DY6M3YaDBPAmvuuA6jV8Sif9MjtOI5euKFWRgCDl5PeDpPsHR9u2l6St5ceY8AZgoNDww5+HvEsXFsGg==", + "license": "Apache-2.0", + "dependencies": { + "lodash.camelcase": "^4.3.0", + "long": "^5.0.0", + "protobufjs": "^7.5.5", + "yargs": "^17.7.2" + }, + "bin": { + "proto-loader-gen-types": "build/bin/proto-loader-gen-types.js" + }, + "engines": { + "node": ">=6" + } + }, + "node_modules/@grpc/proto-loader/node_modules/cliui": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", + "integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==", + "license": "ISC", + "dependencies": { + "string-width": "^4.2.0", + "strip-ansi": "^6.0.1", + "wrap-ansi": "^7.0.0" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/@grpc/proto-loader/node_modules/is-fullwidth-code-point": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", + "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/@grpc/proto-loader/node_modules/string-width": { + "version": "4.2.3", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", + "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", + "license": "MIT", + "dependencies": { + "emoji-regex": "^8.0.0", + "is-fullwidth-code-point": "^3.0.0", + "strip-ansi": "^6.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/@grpc/proto-loader/node_modules/wrap-ansi": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz", + "integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==", + "license": "MIT", + "dependencies": { + "ansi-styles": "^4.0.0", + "string-width": "^4.1.0", + "strip-ansi": "^6.0.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/wrap-ansi?sponsor=1" + } + }, + "node_modules/@grpc/proto-loader/node_modules/yargs": { + "version": "17.7.3", + "resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.3.tgz", + "integrity": "sha512-GZtjxm/J/4TSxuL3FNYjCmLktBTnIw/rVmKSIyKeYAZpmJB2ig9VauCC5xsa82GNKVKDAqpOn3KVzNt0zmrU0g==", + "license": "MIT", + "dependencies": { + "cliui": "^8.0.1", + "escalade": "^3.1.1", + "get-caller-file": "^2.0.5", + "require-directory": "^2.1.1", + "string-width": "^4.2.3", + "y18n": "^5.0.5", + "yargs-parser": "^21.1.1" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/@grpc/proto-loader/node_modules/yargs-parser": { + "version": "21.1.1", + "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz", + "integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, "node_modules/@humanwhocodes/config-array": { "version": "0.13.0", "resolved": "https://registry.npmjs.org/@humanwhocodes/config-array/-/config-array-0.13.0.tgz", @@ -233,6 +345,16 @@ "dev": true, "license": "BSD-3-Clause" }, + "node_modules/@js-sdsl/ordered-map": { + "version": "4.4.2", + "resolved": "https://registry.npmjs.org/@js-sdsl/ordered-map/-/ordered-map-4.4.2.tgz", + "integrity": "sha512-iUKgm52T8HOE/makSxjqoWhe95ZJA1/G1sYsGev2JDKUSS14KAgg1LHb+Ba+IPow0xflbnSkOsZcO08C7w1gYw==", + "license": "MIT", + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/js-sdsl" + } + }, "node_modules/@mixmark-io/domino": { "version": "2.2.0", "resolved": "https://registry.npmjs.org/@mixmark-io/domino/-/domino-2.2.0.tgz", @@ -277,6 +399,590 @@ "node": ">= 8" } }, + "node_modules/@opentelemetry/api": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/@opentelemetry/api/-/api-1.9.1.tgz", + "integrity": "sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==", + "license": "Apache-2.0", + "engines": { + "node": ">=8.0.0" + } + }, + "node_modules/@opentelemetry/api-logs": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/api-logs/-/api-logs-0.220.0.tgz", + "integrity": "sha512-CmVa4ImJ+ynfrPMNaAXHET6Bhb44SwzmfyVJFq9ni2jgXJR/l7C6gfVFddNmHP+ZOkP9cf4f9DBe68qVLTHc9w==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/api": "^1.3.0" + }, + "engines": { + "node": ">=8.0.0" + } + }, + "node_modules/@opentelemetry/configuration": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/configuration/-/configuration-0.220.0.tgz", + "integrity": "sha512-glfIVKnZevRin8fY/9uES/mhRtMT1lGINLHc9MIo5fTQZXswEEHamJtgjv4MTtzgnhHGC92mIS/0lzAUZMyE0w==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "yaml": "^2.8.3" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.9.0" + } + }, + "node_modules/@opentelemetry/context-async-hooks": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/context-async-hooks/-/context-async-hooks-2.9.0.tgz", + "integrity": "sha512-OQ0vzvbZBiUhjqLnUaoNfYmP8553Crr3aggB4y0ZUi815mZ7idpdJXQmoKdeBKJelYttoBlLSSHubmyw3wvX4w==", + "license": "Apache-2.0", + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.0.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/core": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/core/-/core-2.9.0.tgz", + "integrity": "sha512-m2nckMT80NnmjTYSPjJQObBJ+8dgkoajEOUbznL8AHZ3T3yHRk2P7gI1PhEBc1+lOnrYE9UWrWHqJDsmqjmNbw==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/semantic-conventions": "^1.29.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.0.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/exporter-logs-otlp-grpc": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-logs-otlp-grpc/-/exporter-logs-otlp-grpc-0.220.0.tgz", + "integrity": "sha512-s0sRPCSlXYqlgObOpCftomJllp3LfUL9FobQ5csg2172ydVhSEnu1ptpsVBJadazs5nUNp7vDuLE03FAFWTLOQ==", + "license": "Apache-2.0", + "dependencies": { + "@grpc/grpc-js": "^1.14.3", + "@opentelemetry/core": "2.9.0", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-grpc-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0", + "@opentelemetry/sdk-logs": "0.220.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-logs-otlp-http": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-logs-otlp-http/-/exporter-logs-otlp-http-0.220.0.tgz", + "integrity": "sha512-8186thl+pTw64iz/qEEen5oJZoZ/gO73XruChdaGlYdWOdBIQ42r+vHLf6a7vIDqTD4b8ZOoMlyxptanECaI9A==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/api-logs": "0.220.0", + "@opentelemetry/core": "2.9.0", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0", + "@opentelemetry/sdk-logs": "0.220.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-logs-otlp-proto": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-logs-otlp-proto/-/exporter-logs-otlp-proto-0.220.0.tgz", + "integrity": "sha512-8LZAxdJ0ENDAFwr4j0oY35mHBltiSzvlhdQAPGiC7p9VnxtuSq4SW1gfBAdW6t6hiQG6OwUl8w7KHaOdJPKHWg==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0", + "@opentelemetry/sdk-logs": "0.220.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-metrics-otlp-grpc": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-metrics-otlp-grpc/-/exporter-metrics-otlp-grpc-0.220.0.tgz", + "integrity": "sha512-U128izvJfX/dW9jRGP0gIfadR1Hg7ft3UEGIeRxLFK70m2BWw6AtNCOnsUygpw2zCgR/ygdWbGpcL6TmhW0ZGw==", + "license": "Apache-2.0", + "dependencies": { + "@grpc/grpc-js": "^1.14.3", + "@opentelemetry/core": "2.9.0", + "@opentelemetry/exporter-metrics-otlp-http": "0.220.0", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-grpc-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-metrics": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-metrics-otlp-http": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-metrics-otlp-http/-/exporter-metrics-otlp-http-0.220.0.tgz", + "integrity": "sha512-Yqt3RBw/bRVncaE9qIIhk4WfjbAQqXuP9FgAaU+IKPndnLEp/cUqZlSC324+bpmduRz7DoTjig8Ub0PeILWXUA==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-metrics": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-metrics-otlp-proto": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-metrics-otlp-proto/-/exporter-metrics-otlp-proto-0.220.0.tgz", + "integrity": "sha512-lyO+IQBdSvqHN/ZOW/OzrSWemtfD+HgWngn+HBNLhjy0YrCQQTz0OE/kSekH2Pl340dn9DWzhqHdz5Eftr+HLA==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/exporter-metrics-otlp-http": "0.220.0", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-metrics": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-prometheus": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-prometheus/-/exporter-prometheus-0.220.0.tgz", + "integrity": "sha512-JZD5DL/NBpVd2BHefvYosm3G40UZ/KzExLv5tc0eZe0CtrsHHtcOk3YPUxR2EINmUeBf8+w5UReTV8fFPn95lA==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-metrics": "2.9.0", + "@opentelemetry/semantic-conventions": "^1.29.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-trace-otlp-grpc": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-trace-otlp-grpc/-/exporter-trace-otlp-grpc-0.220.0.tgz", + "integrity": "sha512-bv1xmNhmNwIM6MdUBw4yYuJeVcEViVLk3uD69vOQMwueHBnfyl/u0HnBlB1FNY/Te0UOzJzvcbyR8wN6b+iGbA==", + "license": "Apache-2.0", + "dependencies": { + "@grpc/grpc-js": "^1.14.3", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-grpc-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0", + "@opentelemetry/sdk-trace": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-trace-otlp-http": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-trace-otlp-http/-/exporter-trace-otlp-http-0.220.0.tgz", + "integrity": "sha512-/+ExB3lRkf+erv4PnoywyL7RHKITidxtUpUTS55k7OQ0dB42S7gEF1gry7swb9MSm1hYLUhJg4QQh9W8SpwwqA==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-trace": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-trace-otlp-proto": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-trace-otlp-proto/-/exporter-trace-otlp-proto-0.220.0.tgz", + "integrity": "sha512-voTAD8XgJxlK7zLkXh8EzMB09zrQr3tyY/BsnDTlDiQU/UdK58MZ63A3mUjdEDrxMjCVmBHU3WQJhRmQe+Dvzg==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-trace": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/exporter-zipkin": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/exporter-zipkin/-/exporter-zipkin-2.9.0.tgz", + "integrity": "sha512-RwINoce2BH8T4obT5pMcAla2sWma1YZvYuaktWmTluQ0PkQdvv5D060rWI1+kawX+J2qBRcMbwrZJJNcMJUauQ==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-trace": "2.9.0", + "@opentelemetry/semantic-conventions": "^1.29.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.0.0" + } + }, + "node_modules/@opentelemetry/instrumentation": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/instrumentation/-/instrumentation-0.220.0.tgz", + "integrity": "sha512-xQx3E2WxP1mDvKzxLxX+CTCtNLa560YJZ3087qYHerl2YmiKpv7AH+dAy7vmx+eVrZ5BwhfWUAVoKOoxCNHcpw==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/api-logs": "0.220.0", + "import-in-the-middle": "^3.0.0", + "require-in-the-middle": "^8.0.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/otlp-exporter-base": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/otlp-exporter-base/-/otlp-exporter-base-0.220.0.tgz", + "integrity": "sha512-CXYo8UD5Mn9YbgebO2EL4wejtA+gxLmLiu6HCk2KH2BR7XhFN6/6p1UlCb23DYCjeYkndevLHuejCCN1yx4+OQ==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/otlp-transformer": "0.220.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/otlp-grpc-exporter-base": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/otlp-grpc-exporter-base/-/otlp-grpc-exporter-base-0.220.0.tgz", + "integrity": "sha512-/eIkBPMBTIvM3x/0mDX4aJeSkYifYClnBPr68PL1h5LV4VQv4+SV6CGrpiZ4fIWDnobVmhTWCm1J/QRdAWUfvA==", + "license": "Apache-2.0", + "dependencies": { + "@grpc/grpc-js": "^1.14.3", + "@opentelemetry/core": "2.9.0", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-transformer": "0.220.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/otlp-transformer": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/otlp-transformer/-/otlp-transformer-0.220.0.tgz", + "integrity": "sha512-lXGrv7KXZ0gNH9SVNUaa6vv6phVYGvJxfXAlMbzbakiXru75f5MZl8Z7oqiMMQD77riVHJCFlQvbZs/VVN2/4A==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/api-logs": "0.220.0", + "@opentelemetry/core": "2.9.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-logs": "0.220.0", + "@opentelemetry/sdk-metrics": "2.9.0", + "@opentelemetry/sdk-trace": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": "^1.3.0" + } + }, + "node_modules/@opentelemetry/propagator-b3": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/propagator-b3/-/propagator-b3-2.9.0.tgz", + "integrity": "sha512-WrOT1WsOUG+B7hstD2RYoMPIOK76G8E9AQHhMjUvrQaGx/oA7rPWQvvr1Rqv7+yy4R0ZMVwWLC4vW2xnkgWPAQ==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.0.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/propagator-jaeger": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/propagator-jaeger/-/propagator-jaeger-2.9.0.tgz", + "integrity": "sha512-4mYGty27rYvSM0jtp1ZUOqd3LfVRCYg9H5G9OFzSx5HViYToU21MFhWfco7x1HwXr7ER8yGOiCIHZUwjPksc0Q==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.0.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/resources": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/resources/-/resources-2.9.0.tgz", + "integrity": "sha512-jyA5MBLQ+Dkl3+JsZkUoUvL7yHvU64kLsvpXKarWm6347Sl1t1bXFTFykUePNpT5WH5pm9a2Qtt03iIYQhZ1Fg==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/semantic-conventions": "^1.29.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.3.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/sdk-logs": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/sdk-logs/-/sdk-logs-0.220.0.tgz", + "integrity": "sha512-WywcTkQtv2iNmt+6y5Kcd4rzvx9bLVsBa2Nwcmg01IUaBTkTow3W4d9KE5vNBpEDtb9tp21WcRBY/lANRrApYA==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/api-logs": "0.220.0", + "@opentelemetry/core": "2.9.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/semantic-conventions": "^1.29.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.4.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/sdk-metrics": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/sdk-metrics/-/sdk-metrics-2.9.0.tgz", + "integrity": "sha512-Xx8RGS4H5XEBl01WuCreMIpiah9cCXMbSkeuIePPdD2cUpq/vUzYmj8E/MK1OsbOc93FuAD4jfn2WOacKwLn7Q==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/resources": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.9.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/sdk-node": { + "version": "0.220.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/sdk-node/-/sdk-node-0.220.0.tgz", + "integrity": "sha512-wHtGyHhSKHNH3fym33xRu4Ef/HXTFvX8eQ42xdQdEO9LYx9Y2qNyBDJytyqVlvmo6abWZlNYTUthuAGUMYqYnQ==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/api-logs": "0.220.0", + "@opentelemetry/configuration": "0.220.0", + "@opentelemetry/context-async-hooks": "2.9.0", + "@opentelemetry/core": "2.9.0", + "@opentelemetry/exporter-logs-otlp-grpc": "0.220.0", + "@opentelemetry/exporter-logs-otlp-http": "0.220.0", + "@opentelemetry/exporter-logs-otlp-proto": "0.220.0", + "@opentelemetry/exporter-metrics-otlp-grpc": "0.220.0", + "@opentelemetry/exporter-metrics-otlp-http": "0.220.0", + "@opentelemetry/exporter-metrics-otlp-proto": "0.220.0", + "@opentelemetry/exporter-prometheus": "0.220.0", + "@opentelemetry/exporter-trace-otlp-grpc": "0.220.0", + "@opentelemetry/exporter-trace-otlp-http": "0.220.0", + "@opentelemetry/exporter-trace-otlp-proto": "0.220.0", + "@opentelemetry/exporter-zipkin": "2.9.0", + "@opentelemetry/instrumentation": "0.220.0", + "@opentelemetry/otlp-exporter-base": "0.220.0", + "@opentelemetry/otlp-grpc-exporter-base": "0.220.0", + "@opentelemetry/propagator-b3": "2.9.0", + "@opentelemetry/propagator-jaeger": "2.9.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-logs": "0.220.0", + "@opentelemetry/sdk-metrics": "2.9.0", + "@opentelemetry/sdk-trace": "2.9.0", + "@opentelemetry/sdk-trace-base": "2.9.0", + "@opentelemetry/sdk-trace-node": "2.9.0", + "@opentelemetry/semantic-conventions": "^1.29.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.3.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/sdk-trace": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/sdk-trace/-/sdk-trace-2.9.0.tgz", + "integrity": "sha512-sGA19HvtrrSKYsseHphluH6j3p6Xa3fqc7c7y8f/7mYWejc1lyDFcpSdD1kYa50HCLUeEo4zA5bW0pniaPszuw==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/semantic-conventions": "^1.29.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.3.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/sdk-trace-base": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/sdk-trace-base/-/sdk-trace-base-2.9.0.tgz", + "integrity": "sha512-cp9zmTl62R8PJrpvFcmc8N2JQU/xfa0S+61q511Nji+QxCfZ8Ifvg7H27G8cANe4crg4RTrWsVvanHiXjSp6ag==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/core": "2.9.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-trace": "2.9.0", + "@opentelemetry/semantic-conventions": "^1.29.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.3.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/sdk-trace-node": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/sdk-trace-node/-/sdk-trace-node-2.9.0.tgz", + "integrity": "sha512-ec9a7ps37huy5itYk0MalaZdSLlM6AXWp/FhtEjgMpp5leEGojBDvAl/UWttQnkMZOvFHKzRESn8TD3yKTF5nQ==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/context-async-hooks": "2.9.0", + "@opentelemetry/core": "2.9.0", + "@opentelemetry/sdk-trace-base": "2.9.0" + }, + "engines": { + "node": "^18.19.0 || >=20.6.0" + }, + "peerDependencies": { + "@opentelemetry/api": ">=1.0.0 <1.10.0" + } + }, + "node_modules/@opentelemetry/semantic-conventions": { + "version": "1.43.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/semantic-conventions/-/semantic-conventions-1.43.0.tgz", + "integrity": "sha512-eSYWTm620tTk45EKSedaUL8MFYI8hW164hIXsgIHyxu3VobUB3fFCu5t0hQby6OoWRPsG1KkKUG2M5UadiLiVg==", + "license": "Apache-2.0", + "engines": { + "node": ">=14" + } + }, + "node_modules/@protobufjs/aspromise": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", + "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/base64": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", + "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/codegen": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.5.tgz", + "integrity": "sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/eventemitter": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.1.tgz", + "integrity": "sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/fetch": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.1.tgz", + "integrity": "sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==", + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.1" + } + }, + "node_modules/@protobufjs/float": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", + "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/path": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", + "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/pool": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", + "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/utf8": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.2.tgz", + "integrity": "sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug==", + "license": "BSD-3-Clause" + }, "node_modules/@rdk-moss/agent": { "resolved": "packages/moss-agent", "link": true @@ -338,7 +1044,6 @@ "version": "22.19.19", "resolved": "https://registry.npmjs.org/@types/node/-/node-22.19.19.tgz", "integrity": "sha512-dyh/xO2Fh5bYrfWaaqGrRQQGkNdmYw6AmaAUvYeUMNTWQtvb796ikLdmTchRmOlOiIJ1TDXfWgVx1QkUlQ6Hew==", - "dev": true, "license": "MIT", "dependencies": { "undici-types": "~6.21.0" @@ -743,6 +1448,12 @@ "url": "https://github.com/sponsors/colinhacks" } }, + "node_modules/cjs-module-lexer": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/cjs-module-lexer/-/cjs-module-lexer-2.2.0.tgz", + "integrity": "sha512-4bHTS2YuzUvtoLjdy+98ykbNB5jS0+07EvFNXerqZQJ89F7DI6ET7OQo/HJuW6K0aVsKA9hj9/RVb2kQVOrPDQ==", + "license": "MIT" + }, "node_modules/cli-boxes": { "version": "4.0.1", "resolved": "https://registry.npmjs.org/cli-boxes/-/cli-boxes-4.0.1.tgz", @@ -988,7 +1699,6 @@ "version": "4.4.3", "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "dev": true, "license": "MIT", "dependencies": { "ms": "^2.1.3" @@ -1065,6 +1775,12 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/es-module-lexer": { + "version": "2.3.1", + "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.3.1.tgz", + "integrity": "sha512-shc1dbU90Yl/xq1QrC7QRtfcwURZuVRfPhZbDoldJ1cn1gzDvBaBWlv0eFolj5+0znnPJz5TXLxsN77X/12KTA==", + "license": "MIT" + }, "node_modules/es-toolkit": { "version": "1.47.0", "resolved": "https://registry.npmjs.org/es-toolkit/-/es-toolkit-1.47.0.tgz", @@ -1614,6 +2330,20 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/import-in-the-middle": { + "version": "3.3.1", + "resolved": "https://registry.npmjs.org/import-in-the-middle/-/import-in-the-middle-3.3.1.tgz", + "integrity": "sha512-0rymlHSFLwZ0ixx8DaQkoIyZojJPY2a0K2nEYslhKJ6jIYO/m0IcCb7iQsFPmS7WmKwISZiIrv5Icstrw/CmqA==", + "license": "Apache-2.0", + "dependencies": { + "cjs-module-lexer": "^2.2.0", + "es-module-lexer": "^2.2.0", + "module-details-from-path": "^1.0.4" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/imurmurhash": { "version": "0.1.4", "resolved": "https://registry.npmjs.org/imurmurhash/-/imurmurhash-0.1.4.tgz", @@ -1895,6 +2625,12 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/lodash.camelcase": { + "version": "4.3.0", + "resolved": "https://registry.npmjs.org/lodash.camelcase/-/lodash.camelcase-4.3.0.tgz", + "integrity": "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA==", + "license": "MIT" + }, "node_modules/lodash.merge": { "version": "4.6.2", "resolved": "https://registry.npmjs.org/lodash.merge/-/lodash.merge-4.6.2.tgz", @@ -1902,6 +2638,12 @@ "dev": true, "license": "MIT" }, + "node_modules/long": { + "version": "5.3.2", + "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", + "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", + "license": "Apache-2.0" + }, "node_modules/marked": { "version": "9.1.6", "resolved": "https://registry.npmjs.org/marked/-/marked-9.1.6.tgz", @@ -2010,11 +2752,16 @@ "node": ">=18.0.0" } }, + "node_modules/module-details-from-path": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/module-details-from-path/-/module-details-from-path-1.0.4.tgz", + "integrity": "sha512-EGWKgxALGMgzvxYF1UyGTy0HXX/2vHLkw6+NvDKW2jypWbHpjQuj4UMcqQWXHERJhVGKikolT06G3bcKe4fi7w==", + "license": "MIT" + }, "node_modules/ms": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "dev": true, "license": "MIT" }, "node_modules/mz": { @@ -2261,6 +3008,29 @@ "url": "https://github.com/prettier/prettier?sponsor=1" } }, + "node_modules/protobufjs": { + "version": "7.6.5", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.6.5.tgz", + "integrity": "sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw==", + "hasInstallScript": true, + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.2", + "@protobufjs/base64": "^1.1.2", + "@protobufjs/codegen": "^2.0.5", + "@protobufjs/eventemitter": "^1.1.1", + "@protobufjs/fetch": "^1.1.1", + "@protobufjs/float": "^1.0.2", + "@protobufjs/path": "^1.1.2", + "@protobufjs/pool": "^1.1.0", + "@protobufjs/utf8": "^1.1.1", + "@types/node": ">=13.7.0", + "long": "^5.3.2" + }, + "engines": { + "node": ">=12.0.0" + } + }, "node_modules/punycode": { "version": "2.3.1", "resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz", @@ -2447,6 +3217,19 @@ "node": ">=0.10.0" } }, + "node_modules/require-in-the-middle": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/require-in-the-middle/-/require-in-the-middle-8.0.1.tgz", + "integrity": "sha512-QT7FVMXfWOYFbeRBF6nu+I6tr2Tf3u0q8RIEjNob/heKY/nh7drD/k7eeMFmSQgnTtCzLDcCu/XEnpW2wk4xCQ==", + "license": "MIT", + "dependencies": { + "debug": "^4.3.5", + "module-details-from-path": "^1.0.3" + }, + "engines": { + "node": ">=9.3.0 || >=8.10.0 <9.0.0" + } + }, "node_modules/resolve-from": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-4.0.0.tgz", @@ -2889,7 +3672,6 @@ "version": "6.21.0", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", - "dev": true, "license": "MIT" }, "node_modules/unicode-emoji-modifier-base": { @@ -3051,6 +3833,21 @@ "node": ">=10" } }, + "node_modules/yaml": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.0.tgz", + "integrity": "sha512-2AvhNX3mb8zd6Zy7INTtSpl1F15HW6Wnqj0srWlkKLcpYl/gMIMJiyuGq2KeI2YFxUPjdlB+3Lc10seMLtL4cA==", + "license": "ISC", + "bin": { + "yaml": "bin.mjs" + }, + "engines": { + "node": ">= 14.6" + }, + "funding": { + "url": "https://github.com/sponsors/eemeli" + } + }, "node_modules/yargs": { "version": "16.2.0", "resolved": "https://registry.npmjs.org/yargs/-/yargs-16.2.0.tgz", @@ -3260,6 +4057,14 @@ "hasInstallScript": true, "license": "MIT", "dependencies": { + "@opentelemetry/api": "^1.9.0", + "@opentelemetry/exporter-metrics-otlp-http": "^0.220.0", + "@opentelemetry/exporter-trace-otlp-http": "^0.220.0", + "@opentelemetry/resources": "^2.9.0", + "@opentelemetry/sdk-metrics": "^2.9.0", + "@opentelemetry/sdk-node": "^0.220.0", + "@opentelemetry/sdk-trace-base": "^2.9.0", + "@opentelemetry/semantic-conventions": "^1.43.0", "@rdk-moss/core": "^0.6.0", "highlight.js": "^10.7.3", "ink": "^7.0.5", diff --git a/packages/moss-agent/package.json b/packages/moss-agent/package.json index ebeeabce..f3bc8f47 100644 --- a/packages/moss-agent/package.json +++ b/packages/moss-agent/package.json @@ -172,6 +172,14 @@ "docs": "typedoc --out docs-api src/index.ts src/core/index.ts src/knowledge/index.ts --tsconfig tsconfig.build.json" }, "dependencies": { + "@opentelemetry/api": "^1.9.0", + "@opentelemetry/exporter-metrics-otlp-http": "^0.220.0", + "@opentelemetry/exporter-trace-otlp-http": "^0.220.0", + "@opentelemetry/resources": "^2.9.0", + "@opentelemetry/sdk-metrics": "^2.9.0", + "@opentelemetry/sdk-node": "^0.220.0", + "@opentelemetry/sdk-trace-base": "^2.9.0", + "@opentelemetry/semantic-conventions": "^1.43.0", "@rdk-moss/core": "^0.6.0", "ink": "^7.0.5", "marked": "^5.1.0", diff --git a/packages/moss-agent/src/cli-main.ts b/packages/moss-agent/src/cli-main.ts index 4aed1772..72e79bc1 100644 --- a/packages/moss-agent/src/cli-main.ts +++ b/packages/moss-agent/src/cli-main.ts @@ -67,6 +67,7 @@ import { AgentMesh, createMeshTools, isMeshVerboseEnabled } from './mesh/agent-m import { MeshEventBus } from './mesh/index.js'; import { LanDiscovery } from './mesh/lan-discovery.js'; import { setTracer } from './observability/tracing.js'; +import { initObservability, shutdownObservability } from './observability/index.js'; import { resolveLLMUsageLogPath } from './observability/llm-usage.js'; import { redactSensitiveData } from './observability/redact.js'; import { resolveCliDetailMode } from './cli/output.js'; @@ -621,6 +622,11 @@ async function main() { }); const cliLlmProvider = parsedArgs.mock ? createMockLLMProvider() : createCliProvider(providerConfig); + + // Initialize observability (OTel tracing + metrics + local file trace) based on env. + // No-op when MOSS_OTEL_ENABLED is unset and MOSS_OTEL_URL absent. + initObservability({ workspaceDir: workspace }); + const agent = new MossAgent({ llmProvider: cliLlmProvider, sessionStore, model, workspaceDir: workspace, @@ -1041,9 +1047,13 @@ async function main() { } finally { await agent.close(); await closeMcpConnections(mcpConnections); + await shutdownObservability(); } } +// Flush any in-flight traces/metrics on unexpected exit paths. +process.on('beforeExit', () => { void shutdownObservability(); }); + main().catch((err) => { // Config file errors already carry a clean, actionable one-liner — show it // alone instead of a raw Node stack. A malformed/hand-edited config.json diff --git a/packages/moss-agent/src/core/agent/moss-agent.ts b/packages/moss-agent/src/core/agent/moss-agent.ts index 01ac5afa..8b588d49 100644 --- a/packages/moss-agent/src/core/agent/moss-agent.ts +++ b/packages/moss-agent/src/core/agent/moss-agent.ts @@ -37,7 +37,8 @@ import { } from '@rdk-moss/core/contracts/async-task'; import { compactHistoryIfNeeded, type SummarizeFn } from '../../context/compaction.js'; import { createRemoteCompactProviderFromEnv } from '../../context/remote-compaction.js'; -import { setTraceRedactor } from '../../observability/tracing.js'; +import { setTraceRedactor, startSpan, sessionAttributes } from '../../observability/tracing.js'; +import { mossMetrics } from '../../observability/index.js'; import { logLLMUsage } from '../../observability/llm-usage.js'; import { PlatformExtensionRegistry, @@ -1897,28 +1898,53 @@ export class MossAgent { userMessage: string, options?: ChatOptions ): AsyncGenerator { + const model = String(this.config.model); + const sessionStart = Date.now(); + // Session root span covers every CLI entry (oneshot / piped / TUI all + // funnel through streamChat). turn → llm/tool child spans nest under it + // via the active context. + const sessionSpan = startSpan('moss.session', sessionAttributes(sessionKey, model, sessionKey)); + let sessionResult: { toolCalls?: unknown[]; stopReason?: string } | undefined; const run = await this.createAgentLoopRun(sessionKey, userMessage, options); this.noteRunStarted(sessionKey, run.params.runId); - const miniStream = runAgentLoop(run.params); let done: Extract | undefined; + // Run the agent loop inside the session span's context so child spans + // (moss.agent.turn, moss.llm.request, moss.tool.invoke) nest under + // moss.session — share its traceId with the session span as parent. + // runInSpanContextGen keeps the span's context active across the + // generator's yields/awaits (AsyncLocalStorage propagation). The mini + // stream is created INSIDE the generator so runAgentLoop's background + // async task inherits the session context at startup. try { - for await (const event of this.adaptMiniStreamEvents(miniStream, run)) { + for await (const event of sessionSpan.runInSpanContextGen( + async function* (self: MossAgent) { + const miniStream = runAgentLoop(run.params); + for await (const ev of self.adaptMiniStreamEvents(miniStream, run)) { + yield ev; + } + const miniResult = await miniStream.result(); + done = run.adapter.getDoneEvent(miniResult); + sessionResult = done?.result; + }.call(this, this), + )) { yield event; } - - const miniResult = await miniStream.result(); - done = run.adapter.getDoneEvent(miniResult); - this.noteRunFinished(sessionKey, run.params.runId); - - - - - - - - } finally { + // Session metrics on every exit path (success or failure). + // end_turn → ok; explicit error stopReason → error; otherwise incomplete. + const outcome = sessionResult + ? (sessionResult.stopReason === 'end_turn' + ? 'ok' + : sessionResult.stopReason === 'error' ? 'error' : 'incomplete') + : 'error'; + mossMetrics.sessionCount.add(1, { outcome }); + mossMetrics.sessionDuration.record(Date.now() - sessionStart, { outcome }); + mossMetrics.sessionToolCount.record( + Array.isArray(sessionResult?.toolCalls) ? sessionResult!.toolCalls!.length : 0, + { outcome }, + ); + sessionSpan.end(outcome !== 'error'); this.noteRunFinished(sessionKey, run.params.runId); this.retireSteeringMessages(sessionKey, run.params.runId); clearPendingStructuredValidation(sessionKey, run.params.runId); @@ -1927,7 +1953,7 @@ export class MossAgent { } } - yield done; + yield done!; } async *streamChat( diff --git a/packages/moss-agent/src/core/loop/agent-loop-llm-call.ts b/packages/moss-agent/src/core/loop/agent-loop-llm-call.ts index 302ae628..57f6da91 100644 --- a/packages/moss-agent/src/core/loop/agent-loop-llm-call.ts +++ b/packages/moss-agent/src/core/loop/agent-loop-llm-call.ts @@ -20,6 +20,7 @@ import type { AgentLoopMutableState } from './agent-loop-state.js'; import type { CompactHookRegistry } from './compact-hooks.js'; import { isContextOverflowError, describeError } from '../../provider/errors.js'; import { withSpan, turnAttributes } from '../../observability/tracing.js'; +import { mossMetrics } from '../../observability/index.js'; import { redactSensitiveData } from '../../observability/redact.js'; import { totalPromptTokens } from '../llm/usage.js'; import { runAgentLoopLlmTurn } from './agent-loop-stream-helpers.js'; @@ -138,7 +139,7 @@ export async function executeLlmTurn(params: ExecuteLlmTurnParams): Promise { span.addEvent('prompt_window', { @@ -199,6 +200,12 @@ export async function executeLlmTurn(params: ExecuteLlmTurnParams): Promise }[] = []; + // Open a turn span covering this iteration's LLM + tool work. + // runInSpanContext() activates it around executeLlmTurn/processLlmResponse + // so child spans (moss.llm.request, moss.tool.invoke) nest under it. + // The finally ends the span on every exit: continue, break, throw. + const turnSpan = startSpan('moss.agent.turn', turnAttributes(runId, state.turns, String(modelDef.id))); try { const ctxResult = await prepareTurnContext({ @@ -1112,7 +1118,7 @@ export function runAgentLoop( } - const llmResult = await executeLlmTurn({ + const llmResult = await turnSpan.runInSpanContext(() => executeLlmTurn({ state, modelDef, piContext: ctxResult.piContext, @@ -1142,7 +1148,7 @@ export function runAgentLoop( suppressVisibleDeltas: Boolean( params.guardAssistantOutput || params.shouldBufferAssistantOutput?.() ), - }); + })); if (llmResult.control === 'retry') { state.turns--; @@ -1153,7 +1159,7 @@ export function runAgentLoop( turnToolCalls = llmResult.toolCalls; - const responseResult = await processLlmResponse({ + const responseResult = await turnSpan.runInSpanContext(() => processLlmResponse({ state, runId, assistantContent: llmResult.assistantContent, @@ -1191,7 +1197,7 @@ export function runAgentLoop( push: (e) => stream.push(e), buildCorrectionMessage, pendingToolAborts, - }); + })); state.consecutiveTurnErrors = 0; @@ -1315,9 +1321,12 @@ export function runAgentLoop( remainingBuffer: turnAssistantBuffer.length, sessionKey, }); - - + + + } + // Ends the turn span on every exit path: continue, break, throw. + turnSpan.end(state.consecutiveTurnErrors === 0); } } diff --git a/packages/moss-agent/src/core/tools/execute-tool-call.ts b/packages/moss-agent/src/core/tools/execute-tool-call.ts index f5e397d9..ba357b27 100644 --- a/packages/moss-agent/src/core/tools/execute-tool-call.ts +++ b/packages/moss-agent/src/core/tools/execute-tool-call.ts @@ -29,6 +29,8 @@ import { describeError, isTimeoutError, isTransientError } from '../../provider/ import { getRootLogger } from '../../logger.js'; import { runPreToolHookChain, validateToolInputObject } from './tool-pipeline.js'; import { MossError, ErrorCode, errorMessage, isMossError } from '../../errors.js'; +import { withSpan, toolAttributes } from '../../observability/tracing.js'; +import { mossMetrics } from '../../observability/index.js'; const logger = getRootLogger(); @@ -288,6 +290,42 @@ export type ExecuteToolCallOutcome = export async function executeOneToolCall( call: { id: string; name: string; input: Record }, deps: ExecuteToolCallDeps +): Promise { + const startMs = Date.now(); + return withSpan( + 'moss.tool.invoke', + toolAttributes(deps.sessionKey, call.name, call.id), + async (span) => { + try { + const outcome = await executeOneToolCallInner(call, deps); + // Only a completed-and-failed execution is a real tool error. + // denied (user refusal) / pre-blocked / hook-blocked / unknown-tool are + // not execution errors — classify as 'blocked' so the tool error-rate + // metric doesn't over-count them. + const isExecError = outcome.kind === 'completed' && Boolean(outcome.isError); + const metricStatus: string = + outcome.kind === 'completed' ? (isExecError ? 'error' : 'ok') : 'blocked'; + const durationMs = outcome.kind === 'completed' && typeof outcome.durationMs === 'number' + ? outcome.durationMs + : Date.now() - startMs; + span.setAttribute('is_error', isExecError); + span.setAttribute('outcome_kind', outcome.kind); + if (outcome.kind === 'completed' && outcome.outcome) span.setAttribute('outcome', outcome.outcome); + mossMetrics.toolInvocations.add(1, { tool: call.name, status: metricStatus }); + mossMetrics.toolDuration.record(durationMs, { tool: call.name }); + return outcome; + } catch (err) { + mossMetrics.toolInvocations.add(1, { tool: call.name, status: 'error' }); + mossMetrics.toolDuration.record(Date.now() - startMs, { tool: call.name }); + throw err; + } + }, + ); +} + +async function executeOneToolCallInner( + call: { id: string; name: string; input: Record }, + deps: ExecuteToolCallDeps ): Promise { try { diff --git a/packages/moss-agent/src/observability/console-trace.ts b/packages/moss-agent/src/observability/console-trace.ts new file mode 100644 index 00000000..8306f4b6 --- /dev/null +++ b/packages/moss-agent/src/observability/console-trace.ts @@ -0,0 +1,68 @@ +/** + * ConsoleSpanProcessor — emits a human-readable span lifecycle to stderr. + * + * For local debugging: MOSS_TRACE=console attaches this processor so you can + * watch the trace tree stream by as the agent runs (start/end per span, + * indented by parent depth, with duration + status). Zero network, zero disk. + * + * Uses SDK span lifecycle: onStart opens the line, onEnd completes it with + * duration + status. Because spans end out of order (children end before + * parents), we print on END (when duration/status are known) with a depth + * derived from the span's nesting via a parent stack — approximated by + * tracking open spans and their parent ids. + */ +import type { SpanProcessor, ReadableSpan } from '@opentelemetry/sdk-trace-base'; +import type { Context } from '@opentelemetry/api'; +import { SpanStatusCode } from '@opentelemetry/api'; + +interface OpenSpan { + name: string; + spanId: string; + parentSpanId?: string; + depth: number; +} + +export class ConsoleSpanProcessor implements SpanProcessor { + private open = new Map(); + + onStart(span: ReadableSpan, _parentContext: Context): void { + const spanId = span.spanContext().spanId; + const parentSpanId = span.parentSpanContext?.spanId; + // Depth: parent's depth + 1, or 0 for a root. + const parentDepth = parentSpanId ? this.open.get(parentSpanId)?.depth ?? 0 : -1; + const depth = parentDepth + 1; + this.open.set(spanId, { name: span.name, spanId, parentSpanId, depth }); + const indent = ' '.repeat(depth); + process.stderr.write(`[trace] ${indent}▶ ${span.name}\n`); + } + + onEnd(span: ReadableSpan): void { + const spanId = span.spanContext().spanId; + const info = this.open.get(spanId); + const depth = info?.depth ?? 0; + const indent = ' '.repeat(depth); + const durMs = Number(span.duration[0]) * 1000 + Math.trunc(Number(span.duration[1]) / 1_000_000); + const status = span.status.code === SpanStatusCode.ERROR ? 'ERROR' : 'ok'; + const msg = span.status.message ? ` (${span.status.message.slice(0, 100)})` : ''; + const attrs = this.formatAttrs(span); + process.stderr.write(`[trace] ${indent}◀ ${span.name} (${durMs}ms, ${status})${attrs}${msg}\n`); + this.open.delete(spanId); + } + + private formatAttrs(span: ReadableSpan): string { + const entries = Object.entries(span.attributes ?? {}); + if (entries.length === 0) return ''; + const shown = entries + .filter(([k]) => !['runId', 'sessionKey'].includes(k)) + .slice(0, 4) + .map(([k, v]) => `${k}=${typeof v === 'string' ? v : String(v)}`) + .join(' '); + return shown ? ` {${shown}}` : ''; + } + + async forceFlush(): Promise {} + + async shutdown(): Promise { + this.open.clear(); + } +} diff --git a/packages/moss-agent/src/observability/file-trace.ts b/packages/moss-agent/src/observability/file-trace.ts new file mode 100644 index 00000000..9a60b614 --- /dev/null +++ b/packages/moss-agent/src/observability/file-trace.ts @@ -0,0 +1,140 @@ +/** + * FileSpanProcessor — serializes SDK ReadableSpans to a local JSONL file. + * + * Attached to the TracerProvider alongside the OTLP exporter, so the same + * spans ship to the receiver AND land on disk. Best-effort: flush failures + * never block the agent. + * + * Path: {workspaceDir}/.moss/analytics/traces.jsonl + */ +import type { SpanProcessor, ReadableSpan } from '@opentelemetry/sdk-trace-base'; +import type { Context, HrTime } from '@opentelemetry/api'; +import { SpanStatusCode } from '@opentelemetry/api'; +import fs from 'node:fs/promises'; +import path from 'node:path'; + +const FLUSH_INTERVAL_MS = 30_000; + +export interface SerializedSpan { + name: string; + startTime: number; + endTime: number; + attributes: Record; + events: Array<{ name: string; time: number; attrs?: Record }>; + status: 'ok' | 'error'; + statusMessage?: string; +} + +export interface TraceStats { + totalSpans: number; + totalErrors: number; + errorRate: number; + byName: Record; + toolSpans: Array<{ toolName: string; count: number; errors: number; avgDurationMs: number }>; +} + +/** HrTime is [seconds, nanoseconds]; reduce to epoch-ms number for the file. */ +function hrToMs(hr: HrTime): number { + return hr[0] * 1000 + Math.trunc(hr[1] / 1_000_000); +} + +/** Convert an SDK ReadableSpan into the JSONL-friendly SerializedSpan shape. */ +export function serializeSpan(span: ReadableSpan): SerializedSpan { + const attrs: Record = {}; + for (const [k, v] of Object.entries(span.attributes ?? {})) { + if (typeof v === 'string' || typeof v === 'number' || typeof v === 'boolean') attrs[k] = v; + } + return { + name: span.name, + startTime: hrToMs(span.startTime), + endTime: hrToMs(span.endTime), + attributes: attrs, + events: (span.events ?? []).map((e) => ({ + name: e.name, + time: hrToMs(e.time), + ...(e.attributes ? { attrs: e.attributes as Record } : {}), + })), + status: span.status.code === SpanStatusCode.ERROR ? 'error' : 'ok', + ...(span.status.code === SpanStatusCode.ERROR && span.status.message + ? { statusMessage: span.status.message } + : {}), + }; +} + +export class FileSpanProcessor implements SpanProcessor { + private buffer: ReadableSpan[] = []; + private timer: ReturnType | null = null; + private readonly file: string; + + constructor(workspaceDir: string) { + this.file = path.join(workspaceDir, '.moss', 'analytics', 'traces.jsonl'); + this.timer = setInterval(() => { void this.flush(); }, FLUSH_INTERVAL_MS); + } + + onStart(_span: ReadableSpan, _parentContext: Context): void { + /* nothing — we serialize on end */ + } + + onEnd(span: ReadableSpan): void { + this.buffer.push(span); + } + + async flush(): Promise { + if (this.buffer.length === 0) return; + const snapshot = this.buffer.splice(0); + const lines = snapshot.map(serializeSpan).map((s) => JSON.stringify(s)).join('\n') + '\n'; + try { + await fs.mkdir(path.dirname(this.file), { recursive: true }); + await fs.appendFile(this.file, lines, 'utf-8'); + } catch { + // Silently ignore — never block the agent. + } + } + + async forceFlush(): Promise { + await this.flush(); + } + + async shutdown(): Promise { + if (this.timer) clearInterval(this.timer); + this.timer = null; + await this.flush(); + } +} + +function emptyStats(): TraceStats { + return { totalSpans: 0, totalErrors: 0, errorRate: 0, byName: {}, toolSpans: [] }; +} + +/** Read aggregated stats from a traces.jsonl file (for CLI reporting). */ +export async function readTraceStats(file: string): Promise { + let lines: string[]; + try { + lines = (await fs.readFile(file, 'utf-8')).split('\n').filter(Boolean); + } catch { + return emptyStats(); + } + const stats = emptyStats(); + for (const line of lines) { + let span: SerializedSpan; + try { span = JSON.parse(line); } catch { continue; } + stats.totalSpans++; + if (span.status === 'error') stats.totalErrors++; + const entry = stats.byName[span.name] ??= { count: 0, errors: 0, avgDurationMs: 0 }; + entry.count++; + if (span.status === 'error') entry.errors++; + const duration = span.endTime - span.startTime; + entry.avgDurationMs = (entry.avgDurationMs * (entry.count - 1) + duration) / entry.count; + if (span.name === 'moss.tool.invoke') { + const toolName = String(span.attributes.toolName ?? 'unknown'); + let tool = stats.toolSpans.find((t) => t.toolName === toolName); + if (!tool) { tool = { toolName, count: 0, errors: 0, avgDurationMs: 0 }; stats.toolSpans.push(tool); } + tool.count++; + if (span.status === 'error') tool.errors++; + tool.avgDurationMs = (tool.avgDurationMs * (tool.count - 1) + duration) / tool.count; + } + } + stats.errorRate = stats.totalSpans > 0 ? stats.totalErrors / stats.totalSpans : 0; + stats.toolSpans.sort((a, b) => b.count - a.count); + return stats; +} diff --git a/packages/moss-agent/src/observability/index.ts b/packages/moss-agent/src/observability/index.ts index a8da9d8b..c4f93b49 100644 --- a/packages/moss-agent/src/observability/index.ts +++ b/packages/moss-agent/src/observability/index.ts @@ -1,15 +1,29 @@ +/** + * Observability public entrypoint. + * + * initObservability() wires up the OTel SDK (trace + metric + local file + * trace) based on env vars; when disabled it is a no-op and withSpan / + * mossMetrics.* run as no-ops. Call once at CLI startup (before agent + * creation); call shutdownObservability() on exit to flush. + */ export { redactSensitiveData, parseTelemetryAllow } from './redact.js'; export type { RedactOptions } from './redact.js'; export { TraceRegistry, setTracer, + setTraceRedactor, getTracer, withSpan, + startSpan, turnAttributes, toolAttributes, llmRequestAttributes, + sessionAttributes, } from './tracing.js'; -export type { Tracer, TraceSpan } from './tracing.js'; +export type { Tracer, TraceSpan, ActiveSpan } from './tracing.js'; +export { mossMetrics } from './metrics.js'; +export { FileSpanProcessor, readTraceStats } from './file-trace.js'; +export type { SerializedSpan, TraceStats } from './file-trace.js'; export { logLLMUsage, readUsageLog, @@ -20,3 +34,51 @@ export { resolveLLMUsageLogPath, } from './llm-usage.js'; export type { LLMUsageRecord, LLMUsageSummary } from './llm-usage.js'; + +import { initObservabilitySdk } from './sdk.js'; +import { propagation, context as otelContext, defaultTextMapSetter } from '@opentelemetry/api'; + +export interface InitOptions { + workspaceDir: string; + serviceName?: string; + otlpUrl?: string; +} + +export function initObservability(opts: InitOptions): void { + const otelEnabled = process.env.MOSS_OTEL_ENABLED === '1' || !!process.env.MOSS_OTEL_URL; + // MOSS_TRACE=console streams spans to stderr for local debugging, and can + // run on its own (no OTLP receiver needed). + const consoleTrace = process.env.MOSS_TRACE === 'console' + || process.env.MOSS_TRACE === '1' + || process.env.MOSS_TRACE === 'true'; + if (!otelEnabled && !consoleTrace) return; + initObservabilitySdk({ + serviceName: process.env.MOSS_OTEL_SERVICE_NAME ?? opts.serviceName ?? 'moss', + otlpUrl: process.env.MOSS_OTEL_URL ?? opts.otlpUrl ?? 'http://localhost:4318', + enabled: otelEnabled, + // tracing 开则 metrics 默认开（纠正 from-remote 两开关易漏配） + metricsEnabled: otelEnabled && process.env.MOSS_METRICS_ENABLED !== '0', + // 本地文件 trace 默认开（仅 OTel 启用时），MOSS_FILE_TRACE=0 关 + fileTraceEnabled: otelEnabled && process.env.MOSS_FILE_TRACE !== '0', + consoleTraceEnabled: consoleTrace, + workspaceDir: opts.workspaceDir, + }); +} + +export { shutdownObservabilitySdk as shutdownObservability } from './sdk.js'; + +/** + * Inject W3C traceparent into outbound fetch headers for the current span. + * Returns headers unchanged when no active span (graceful degradation). + */ +export function propagateHeaders( + headers: Record = {}, +): Record { + try { + const injected: Record = { ...headers }; + propagation.inject(otelContext.active(), injected, defaultTextMapSetter); + return injected; + } catch { + return headers; + } +} diff --git a/packages/moss-agent/src/observability/metrics.ts b/packages/moss-agent/src/observability/metrics.ts new file mode 100644 index 00000000..30eef115 --- /dev/null +++ b/packages/moss-agent/src/observability/metrics.ts @@ -0,0 +1,65 @@ +/** + * OpenTelemetry metrics for Moss — instrument handles. + * + * Instruments are resolved LAZILY from the global MeterProvider on first use, + * so they pick up the real MeterProvider once the SDK is started + * (sdk.ts → setGlobalMeterProvider). When observability is disabled (no + * provider registered), metrics.getMeter() returns a noop meter whose + * instruments are no-ops — business code calls .add()/.record() + * unconditionally at zero cost. + * + * Resolving eagerly at module load would bind to the noop meter before the SDK + * starts; the cached noop instruments would never switch to real ones. + * + * Usage: + * import { mossMetrics } from './observability/index.js'; + * mossMetrics.llmTokens.add(inputTokens, { direction: 'input', model }); + */ +import { metrics } from '@opentelemetry/api'; + +const meter = () => metrics.getMeter('moss-agent'); + +type Counter = { add(value: number, attributes?: Record): void }; +type Histogram = { record(value: number, attributes?: Record): void }; + +function counter(name: string, opts?: { unit?: string }): () => Counter { + let cached: Counter | undefined; + return () => (cached ??= meter().createCounter(name, opts)); +} +function histogram(name: string, opts?: { unit?: string }): () => Histogram { + let cached: Histogram | undefined; + return () => (cached ??= meter().createHistogram(name, opts)); +} + +// Wrap a lazily-resolved instrument in a stable object exposing .add/.record +// so call-sites write `mossMetrics.x.add(...)` unchanged. +function counterHandle(name: string, opts?: { unit?: string }): Counter { + const get = counter(name, opts); + return { + add(value, attributes) { + try { get().add(value, attributes ?? {}); } catch { /* noop */ } + }, + }; +} +function histogramHandle(name: string, opts?: { unit?: string }): Histogram { + const get = histogram(name, opts); + return { + record(value, attributes) { + try { get().record(value, attributes ?? {}); } catch { /* noop */ } + }, + }; +} + +export const mossMetrics = { + // LLM + llmTokens: counterHandle('moss.llm.tokens', { unit: '{token}' }), + llmDuration: histogramHandle('moss.llm.request.duration', { unit: 'ms' }), + // tool + toolInvocations: counterHandle('moss.tool.invocations'), + toolDuration: histogramHandle('moss.tool.invoke.duration', { unit: 'ms' }), + // session + sessionCount: counterHandle('moss.session.count'), + sessionDuration: histogramHandle('moss.session.duration', { unit: 'ms' }), + // 每轮工具数（纠正 from-remote 把它误命名为 session.turns 的错位） + sessionToolCount: histogramHandle('moss.session.tool_count'), +}; diff --git a/packages/moss-agent/src/observability/sdk.ts b/packages/moss-agent/src/observability/sdk.ts new file mode 100644 index 00000000..bb5072de --- /dev/null +++ b/packages/moss-agent/src/observability/sdk.ts @@ -0,0 +1,92 @@ +/** + * OpenTelemetry SDK assembly — single NodeSDK for trace + metric, shared Resource. + * Local file trace attaches as a FileSpanProcessor alongside the OTLP exporter. + * + * initObservabilitySdk reads env to decide what to start. When disabled it + * does nothing (no SDK, no timers). shutdownObservabilitySdk flushes all + * exporters/processors (fire-and-forget send errors never crash the agent). + */ +import { NodeSDK } from '@opentelemetry/sdk-node'; +import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'; +import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; +import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; +import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; +import { resourceFromAttributes } from '@opentelemetry/resources'; +import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions'; +import fs from 'node:fs'; +import path from 'node:path'; +import { FileSpanProcessor } from './file-trace.js'; +import { ConsoleSpanProcessor } from './console-trace.js'; +import type { SpanProcessor } from '@opentelemetry/sdk-trace-base'; + +export interface ObservabilityConfig { + serviceName: string; + otlpUrl: string; + enabled: boolean; + metricsEnabled: boolean; + fileTraceEnabled: boolean; + consoleTraceEnabled: boolean; + workspaceDir: string; +} + +let sdk: NodeSDK | null = null; + +/** Read package version from packages/moss-agent/package.json (not hardcoded). */ +function readPackageVersion(): string { + try { + const pkgPath = path.join(import.meta.dirname, '..', '..', 'package.json'); + return String(JSON.parse(fs.readFileSync(pkgPath, 'utf-8')).version ?? '0.0.0'); + } catch { + return '0.0.0'; + } +} + +export function initObservabilitySdk(cfg: ObservabilityConfig): void { + // Start the SDK if anything is enabled: OTel tracing/metrics, local file + // trace, or console trace (MOSS_TRACE=console runs standalone). Guarded + // against double-init via the `sdk` singleton below. + const wantsStart = cfg.enabled || cfg.consoleTraceEnabled; + if (!wantsStart || sdk) return; + try { + const resource = resourceFromAttributes({ + [ATTR_SERVICE_NAME]: cfg.serviceName, + [ATTR_SERVICE_VERSION]: readPackageVersion(), + }); + + const spanProcessors: SpanProcessor[] = []; + if (cfg.consoleTraceEnabled) { + spanProcessors.push(new ConsoleSpanProcessor()); + } + if (cfg.enabled) { + spanProcessors.push(new BatchSpanProcessor(new OTLPTraceExporter({ url: `${cfg.otlpUrl}/v1/traces` }))); + } + if (cfg.enabled && cfg.fileTraceEnabled) { + spanProcessors.push(new FileSpanProcessor(cfg.workspaceDir)); + } + + // If nothing is enabled (no console, no OTel), don't start the SDK at all. + if (spanProcessors.length === 0 && !cfg.metricsEnabled) return; + + const metricReaders = (cfg.enabled && cfg.metricsEnabled) + ? [new PeriodicExportingMetricReader({ + exporter: new OTLPMetricExporter({ url: `${cfg.otlpUrl}/v1/metrics` }), + exportIntervalMillis: 10_000, + })] + : []; + + sdk = new NodeSDK({ + resource, + spanProcessors, + ...(metricReaders.length > 0 ? { metricReaders } : {}), + }); + sdk.start(); + } catch { + // Best-effort — never block the agent. Failure means no telemetry, not a crash. + } +} + +export async function shutdownObservabilitySdk(): Promise { + if (!sdk) return; + try { await sdk.shutdown(); } catch { /* ignore */ } + sdk = null; +} diff --git a/packages/moss-agent/src/observability/tracing.ts b/packages/moss-agent/src/observability/tracing.ts index 71337e10..957cf500 100644 --- a/packages/moss-agent/src/observability/tracing.ts +++ b/packages/moss-agent/src/observability/tracing.ts @@ -1,183 +1,205 @@ +/** + * Tracing — SDK-backed withSpan + attributes. + * + * Uses the global TracerProvider. When none is registered (observability + * disabled), trace.getTracer() returns a noop tracer and withSpan runs fn + * directly with zero tracing overhead. + * + * Legacy setTracer/setTraceRedactor/TraceRegistry/getTracer are kept as noop + * shims so existing imports (cli-main.ts, moss-agent.ts, agent-loop-llm-call.ts) + * do not break until callers are migrated to initObservability. + */ +import { trace, context, SpanStatusCode } from '@opentelemetry/api'; +import type { Span } from '@opentelemetry/api'; import { errorMessage } from '../errors.js'; +import { redactSensitiveData } from './redact.js'; +// Resolve the tracer lazily so it picks up the real TracerProvider once the +// SDK is started (sdk.ts → setGlobalTracerProvider). Binding eagerly at module +// load would capture the noop tracer before the SDK starts — same failure mode +// as the metrics instruments (which are also lazy). Returns a noop tracer when +// no provider is registered, so withSpan/startSpan are zero-cost when disabled. +const resolveTracer = () => trace.getTracer('moss-agent'); - - - - - - - - - - +/** Public span handle passed to withSpan's fn (mirrors the OTel Span API). */ export interface TraceSpan { - setAttribute(key: string, value: string | number | boolean): void; - addEvent(name: string, attributes?: Record): void; - setStatus(ok: boolean, message?: string): void; - end(): void; } +/** Legacy Tracer interface — kept for typing compatibility. */ export interface Tracer { - startSpan( name: string, attributes?: Record, - parent?: TraceSpan + parent?: TraceSpan, ): TraceSpan; } - - -const noopSpan: TraceSpan = { - setAttribute() {}, - addEvent() {}, - setStatus() {}, - end() {}, -}; - -const noopTracer: Tracer = { - startSpan(_name, _attrs, _parent) { - return noopSpan; - }, -}; - - - -function createConsoleTracer(): Tracer { - return { - startSpan(name, attributes, parent) { - const start = Date.now(); - const attrs = attributes ? ` ${JSON.stringify(attributes)}` : ''; - const parentInfo = parent ? ` (parent)` : ''; - console.error(`[trace] ▶ ${name}${attrs}${parentInfo}`); - return { - setAttribute(key, value) { - console.error(`[trace] ${name}.${key} = ${value}`); - }, - addEvent(eventName, eventAttrs) { - const ea = eventAttrs ? ` ${JSON.stringify(eventAttrs)}` : ''; - console.error(`[trace] ${name} :: ${eventName}${ea}`); - }, - setStatus(ok, message) { - const status = ok ? 'OK' : 'ERROR'; - const msg = message ? ` (${message})` : ''; - console.error(`[trace] ${name} status=${status}${msg}`); - }, - end() { - const ms = Date.now() - start; - console.error(`[trace] ◀ ${name} (${ms}ms)`); - }, - }; - }, - }; -} - - - -export class TraceRegistry { - private tracer: Tracer = noopTracer; - private redactor: ((text: string) => string) | null = null; - - setTracer(tracer: Tracer | 'console'): void { - this.tracer = tracer === 'console' ? createConsoleTracer() : tracer; - } - - setTraceRedactor(fn: (text: string) => string): void { - this.redactor = fn; - } - - getTracer(): Tracer { - return this.tracer; - } - - redactMessage(text: string): string { - return this.redactor ? this.redactor(text) : text; - } -} - -const defaultTraceRegistry = new TraceRegistry(); - - - - - -export function setTracer(tracer: Tracer | 'console'): void { - defaultTraceRegistry.setTracer(tracer); -} - - - - - - -export function setTraceRedactor(fn: (text: string) => string): void { - defaultTraceRegistry.setTraceRedactor(fn); -} - - -export function getTracer(): Tracer { - return defaultTraceRegistry.getTracer(); -} - - - - - +/** + * Run fn inside a span. On success sets OK; on throw records the exception + * (type/message/stack as a span event), sets ERROR with a redacted message, + * and rethrows. Never swallows errors. + */ export async function withSpan( name: string, attributes: Record | undefined, - fn: (span: TraceSpan) => Promise, - parent?: TraceSpan + fn: (span: Span) => Promise, ): Promise { - const span = defaultTraceRegistry.getTracer().startSpan(name, attributes, parent); - try { - const result = await fn(span); - span.setStatus(true); - return result; - } catch (err) { - span.setStatus(false, defaultTraceRegistry.redactMessage(errorMessage(err))); - throw err; - } finally { - span.end(); - } + const span = resolveTracer().startSpan(name, attributes ? { attributes } : undefined); + return context.with(trace.setSpan(context.active(), span), async () => { + try { + const result = await fn(span); + span.setStatus({ code: SpanStatusCode.OK }); + return result; + } catch (err) { + span.recordException(err as Error); + span.setStatus({ + code: SpanStatusCode.ERROR, + message: String(redactSensitiveData(errorMessage(err))), + }); + throw err; + } finally { + span.end(); + } + }); } +/** + * Imperative span lifecycle, for call-sites that cannot use the callback-based + * withSpan (e.g. a loop body whose control flow uses continue/break/throw). + * + * startSpan() creates and activates a span; runInSpanContext() runs a fn within + * that span's context (so child withSpan calls nest under it); endSpan() ends + * it. Always call endSpan in a finally that covers every exit path. + */ +export interface ActiveSpan { + /** The underlying OTel Span. Set attributes / add events on it. */ + readonly span: Span; + /** Run fn within this span's active context (for child-span nesting). */ + runInSpanContext(fn: () => Promise): Promise; + /** + * Drive an async generator within this span's active context. For + * async-generator call-sites (e.g. streamChat) that must yield to the + * caller while keeping the span context active across awaits, so child + * spans created inside the generator nest under this span. + */ + runInSpanContextGen(gen: AsyncGenerator): AsyncGenerator; + /** End the span. ok=false records ERROR. Idempotent. */ + end(ok?: boolean, message?: string): void; +} +export function startSpan( + name: string, + attributes?: Record, +): ActiveSpan { + const span = resolveTracer().startSpan(name, attributes ? { attributes } : undefined); + const active = context.active(); + const spanContext = trace.setSpan(active, span); + let ended = false; + return { + span, + runInSpanContext(fn) { + return context.with(spanContext, fn); + }, + runInSpanContextGen(gen: AsyncGenerator): AsyncGenerator { + // Generators suspend at yield and lose the stack-local active context on + // the next .next(). Drive the generator through a thin wrapper that + // re-activates the span context on each resume, so child spans created + // inside the generator nest under this span. + const ctx = spanContext; + return (async function* delegate() { + while (true) { + const res = await context.with(ctx, async () => gen.next()); + if (res.done) return res.value as T; + yield res.value as T; + } + })(); + }, + end(ok = true, message) { + if (ended) return; + ended = true; + try { + span.setStatus( + ok + ? { code: SpanStatusCode.OK } + : { code: SpanStatusCode.ERROR, ...(message ? { message } : {}) }, + ); + } catch { /* noop */ } + span.end(); + }, + }; +} - - +// ── Attributes constructors (span dimensions, centralized) ────────────── export function turnAttributes( runId: string, turn: number, - model: string + model: string, ): Record { return { runId, turn, model }; } - - - export function toolAttributes( runId: string, toolName: string, - toolCallId: string + toolCallId: string, ): Record { return { runId, toolName, toolCallId }; } - - - export function llmRequestAttributes( runId: string, model: string, - inputTokens: number + inputTokens: number, ): Record { return { runId, model, inputTokens }; } + +export function sessionAttributes( + runId: string, + model: string, + sessionKey: string, +): Record { + return { runId, model, sessionKey }; +} + +// ── Legacy noop shims (do not remove — existing imports depend on them) ── + +const noopSpan: TraceSpan = { + setAttribute() {}, + addEvent() {}, + setStatus() {}, + end() {}, +}; + +const noopTracer: Tracer = { + startSpan() { return noopSpan; }, +}; + +export class TraceRegistry { + setTracer(_tracer: Tracer | 'console'): void { + /* no-op under the SDK model */ + } + setTraceRedactor(_fn: (text: string) => string): void { + /* no-op — redaction handled by redactSensitiveData in withSpan */ + } + getTracer(): Tracer { return noopTracer; } + redactMessage(text: string): string { return text; } +} + +export function setTracer(_tracer: Tracer | 'console'): void { + // No-op under the SDK model. Tracer/MeterProvider is configured via + // initObservability() in observability/index.ts. +} + +export function setTraceRedactor(_fn: (text: string) => string): void { + /* no-op under the SDK model */ +} + +export function getTracer(): Tracer { + return noopTracer; +} diff --git a/packages/moss-agent/src/tools/web-fetch.ts b/packages/moss-agent/src/tools/web-fetch.ts index 2a31aa41..908cb30d 100644 --- a/packages/moss-agent/src/tools/web-fetch.ts +++ b/packages/moss-agent/src/tools/web-fetch.ts @@ -20,6 +20,7 @@ import TurndownService from 'turndown'; import type { Tool, ToolContext } from '../core/tools/tool-types.js'; import { getRootLogger } from '../logger.js'; import { MossError, ErrorCode, errorMessage } from '../errors.js'; +import { propagateHeaders } from '../observability/index.js'; import { ensureKeepAliveDispatcherInstalled } from '../provider/keep-alive-dispatcher.js'; const log = getRootLogger().child('tool:web-fetch'); @@ -632,12 +633,12 @@ export function createWebFetchTool(opts: WebFetchOptions = {}): Tool<{ if (pinnedDispatcher) dispatchersToClose.push(pinnedDispatcher); const fetchInit: RequestInit = { signal: mergedSignal, - headers: { + headers: propagateHeaders({ 'User-Agent': activeUserAgent, Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,application/json;q=0.9,text/plain;q=0.8,*/*;q=0.5', ...(shouldRewriteToIp ? { Host: originalHost } : {}), - }, + }), redirect: 'manual', }; if (pinnedDispatcher) { diff --git a/packages/moss-agent/src/tools/web-search.ts b/packages/moss-agent/src/tools/web-search.ts index 5702666c..7e1bbf01 100644 --- a/packages/moss-agent/src/tools/web-search.ts +++ b/packages/moss-agent/src/tools/web-search.ts @@ -32,12 +32,14 @@ import type { Tool, ToolContext } from '../core/tools/tool-types.js'; import { getRootLogger } from '../logger.js'; import { MossError, ErrorCode , errorMessage} from '../errors.js'; +import { propagateHeaders } from '../observability/index.js'; import { ensureKeepAliveDispatcherInstalled } from '../provider/keep-alive-dispatcher.js'; import { createGoogleNewsRssBackend, createRssSearchBackend, parseUserFeeds, -} from './rss-search.js';import { +} from './rss-search.js'; +import { browseSearchPage, browseSearchPages, type BrowserSearchPageSnapshot, @@ -456,7 +458,11 @@ async function fetchWithTimeout( const timer = setTimeout(() => controller.abort(), timeoutMs); try { await ensureKeepAliveDispatcherInstalled(); - const res = await fetch(url, { ...init, signal: controller.signal }); + const res = await fetch(url, { + ...init, + headers: propagateHeaders((init?.headers ?? {}) as Record), + signal: controller.signal, + }); const text = await res.text(); return { ok: res.ok, status: res.status, text }; } catch (err) { diff --git a/packages/moss-agent/test/observability-file-trace.spec.mjs b/packages/moss-agent/test/observability-file-trace.spec.mjs new file mode 100644 index 00000000..603fc99f --- /dev/null +++ b/packages/moss-agent/test/observability-file-trace.spec.mjs @@ -0,0 +1,61 @@ +#!/usr/bin/env node +// @rdk-moss/agent — FileSpanProcessor buffers spans, flushes to traces.jsonl, +// and readTraceStats aggregates. Uses minimal ReadableSpan-shaped objects. +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import os from 'node:os'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'file-trace.js')).href); +const { FileSpanProcessor, readTraceStats } = mod; + +// 构造一个最小 ReadableSpan 形状 (SDK 的 ReadableSpan 是接口，鸭子类型即可) +function fakeSpan(name, attrs, isError) { + const now = Date.now(); + const sec = Math.trunc(now / 1000); + return { + name, + kind: 0, + spanContext: () => ({ traceId: 't'.repeat(32), spanId: 's'.repeat(16) }), + startTime: [sec, 0], + endTime: [sec, 1_000_000], + attributes: attrs ?? {}, + status: isError ? { code: 2, message: 'boom' } : { code: 1 }, + events: [], + resource: { attributes: [] }, + instrumentationScope: { name: 'moss-agent' }, + duration: [0, 1_000_000], + ended: true, + droppedAttributesCount: 0, + droppedEventsCount: 0, + droppedLinksCount: 0, + links: [], + }; +} + +const tmp = await fs.mkdtemp(path.join(os.tmpdir(), 'moss-trace-')); +const proc = new FileSpanProcessor(tmp); +proc.onStart(fakeSpan('noop'), undefined); +proc.onEnd(fakeSpan('moss.tool.invoke', { toolName: 'read_file' }, false)); +proc.onEnd(fakeSpan('moss.tool.invoke', { toolName: 'read_file' }, true)); +proc.onEnd(fakeSpan('moss.llm.request', { model: 'm' }, false)); +await proc.forceFlush(); + +const file = path.join(tmp, '.moss', 'analytics', 'traces.jsonl'); +const content = await fs.readFile(file, 'utf8'); +const lines = content.split('\n').filter(Boolean); +assert.equal(lines.length, 3, 'should write 3 span lines'); + +const stats = await readTraceStats(file); +assert.equal(stats.totalSpans, 3, 'totalSpans'); +assert.equal(stats.totalErrors, 1, 'totalErrors'); +assert.ok(stats.byName['moss.tool.invoke'], 'tool span aggregated by name'); +assert.equal(stats.byName['moss.tool.invoke'].count, 2, '2 tool spans'); +assert.equal(stats.byName['moss.tool.invoke'].errors, 1, '1 tool error'); +assert.equal(stats.toolSpans[0].toolName, 'read_file', 'tool breakdown by toolName'); + +await proc.shutdown(); +await fs.rm(tmp, { recursive: true, force: true }); +console.error('[spec] observability-file-trace OK'); diff --git a/packages/moss-agent/test/observability-integration.spec.mjs b/packages/moss-agent/test/observability-integration.spec.mjs new file mode 100644 index 00000000..d534b270 --- /dev/null +++ b/packages/moss-agent/test/observability-integration.spec.mjs @@ -0,0 +1,68 @@ +#!/usr/bin/env node +// @rdk-moss/agent — End-to-end integration: enable SDK, run nested withSpan +// (session→turn→llm), verify the local file trace lands the three nested spans. +// Does NOT require a live receiver — OTLP sends are fire-and-forget; the +// FileSpanProcessor writes to disk regardless of network. +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import os from 'node:os'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'index.js')).href); +const { initObservability, shutdownObservability, withSpan, mossMetrics } = mod; +const traceMod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'file-trace.js')).href); +const { readTraceStats } = traceMod; +// api 'metrics' to assert the global MeterProvider got registered on init. +const otelApi = await import('@opentelemetry/api'); + +const tmp = await fs.mkdtemp(path.join(os.tmpdir(), 'moss-int-')); +process.env.MOSS_OTEL_ENABLED = '1'; +process.env.MOSS_OTEL_URL = 'http://localhost:4318'; // receiver 未起，fire-and-forget 不抛 +process.env.MOSS_FILE_TRACE = '1'; + +// Before init: getMeterProvider() is the noop provider (constructor NoopMeterProvider). +// After init: SDK registered a real MeterProvider. This guards against the regression +// where mossMetrics binds to the noop meter at module load and never switches to real +// instruments (instruments must be resolved lazily AFTER setGlobalMeterProvider). +const providerBefore = otelApi.metrics.getMeterProvider(); +assert.equal(providerBefore.constructor.name, 'NoopMeterProvider', + 'no global MeterProvider before init (default noop)'); + +initObservability({ workspaceDir: tmp }); + +const providerAfter = otelApi.metrics.getMeterProvider(); +assert.notEqual(providerAfter.constructor.name, 'NoopMeterProvider', + 'initObservability registered a real MeterProvider (not noop)'); +assert.notStrictEqual(providerAfter, providerBefore, 'provider changed after init'); + +// 三层 span：session → turn → llm（模拟 agent 调用栈） +// 在 llm span 内记一条 metric — 验证 instruments 在 SDK start 后解析为真 +// （lazy 解析；eager 绑定会卡在 noop meter）。 +await withSpan('moss.session', { runId: 'r1', model: 'm', sessionKey: 'sk' }, async () => { + return withSpan('moss.agent.turn', { runId: 'r1', turn: 1, model: 'm' }, async () => { + return withSpan('moss.llm.request', { runId: 'r1', model: 'm', inputTokens: 100 }, async (span) => { + span.setAttribute('outputTokens', 50); + mossMetrics.llmTokens.add(100, { direction: 'input', model: 'm' }); + mossMetrics.llmTokens.add(50, { direction: 'output', model: 'm' }); + mossMetrics.llmDuration.record(42, { model: 'm' }); + return 'done'; + }); + }); +}); + +await shutdownObservability(); + +const file = path.join(tmp, '.moss', 'analytics', 'traces.jsonl'); +const stats = await readTraceStats(file); +assert.equal(stats.totalSpans, 3, 'three nested spans landed in file'); +assert.ok(stats.byName['moss.session'], 'session span present'); +assert.ok(stats.byName['moss.agent.turn'], 'turn span present'); +assert.ok(stats.byName['moss.llm.request'], 'llm span present'); + +await fs.rm(tmp, { recursive: true, force: true }); +delete process.env.MOSS_OTEL_ENABLED; +delete process.env.MOSS_OTEL_URL; +delete process.env.MOSS_FILE_TRACE; +console.error('[spec] observability-integration OK'); diff --git a/packages/moss-agent/test/observability-metrics.spec.mjs b/packages/moss-agent/test/observability-metrics.spec.mjs new file mode 100644 index 00000000..658a11f0 --- /dev/null +++ b/packages/moss-agent/test/observability-metrics.spec.mjs @@ -0,0 +1,30 @@ +#!/usr/bin/env node +// @rdk-moss/agent — mossMetrics instruments export + noop behavior (no provider registered). +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import path from 'node:path'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'metrics.js')).href); +const { mossMetrics } = mod; + +// 未 setGlobalMeterProvider 时返回 noop meter，instruments 仍可调用不抛错。 +assert.ok(mossMetrics, 'mossMetrics should be exported'); +assert.equal(typeof mossMetrics.llmTokens.add, 'function', 'llmTokens.add is a function'); +assert.equal(typeof mossMetrics.llmDuration.record, 'function', 'llmDuration.record is a function'); +assert.equal(typeof mossMetrics.toolInvocations.add, 'function', 'toolInvocations.add is a function'); +assert.equal(typeof mossMetrics.toolDuration.record, 'function', 'toolDuration.record is a function'); +assert.equal(typeof mossMetrics.sessionCount.add, 'function', 'sessionCount.add is a function'); +assert.equal(typeof mossMetrics.sessionDuration.record, 'function', 'sessionDuration.record is a function'); +assert.equal(typeof mossMetrics.sessionToolCount.record, 'function', 'sessionToolCount.record is a function'); + +// noop 调用零成本、不抛 +assert.doesNotThrow(() => { + mossMetrics.llmTokens.add(10, { direction: 'input', model: 'm' }); + mossMetrics.llmDuration.record(123, { model: 'm' }); + mossMetrics.toolInvocations.add(1, { tool: 't', status: 'ok' }); + mossMetrics.sessionCount.add(1, { outcome: 'ok' }); + mossMetrics.sessionToolCount.record(3, { outcome: 'ok' }); +}); + +console.error('[spec] observability-metrics OK'); diff --git a/packages/moss-agent/test/observability-noop.spec.mjs b/packages/moss-agent/test/observability-noop.spec.mjs new file mode 100644 index 00000000..3b54b8cd --- /dev/null +++ b/packages/moss-agent/test/observability-noop.spec.mjs @@ -0,0 +1,27 @@ +#!/usr/bin/env node +// @rdk-moss/agent — initObservability is a noop when disabled; +// propagateHeaders passes through when no active span. +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import path from 'node:path'; +import os from 'node:os'; +import fs from 'node:fs/promises'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'index.js')).href); +const { initObservability, shutdownObservability, propagateHeaders } = mod; + +// 不设任何 env，initObservability 应 noop（不创建文件、不抛） +const tmp = await fs.mkdtemp(path.join(os.tmpdir(), 'moss-noop-')); +delete process.env.MOSS_OTEL_ENABLED; +delete process.env.MOSS_OTEL_URL; +assert.doesNotThrow(() => initObservability({ workspaceDir: tmp })); +await shutdownObservability(); + +// propagateHeaders 无 active span 时原样返回（补一个已存在的 header） +const out = propagateHeaders({ 'x-custom': '1' }); +assert.equal(out['x-custom'], '1', 'passes existing headers through'); +assert.ok(out, 'returns a headers object'); + +await fs.rm(tmp, { recursive: true, force: true }); +console.error('[spec] observability-noop OK'); diff --git a/packages/moss-agent/test/observability-tracing.spec.mjs b/packages/moss-agent/test/observability-tracing.spec.mjs new file mode 100644 index 00000000..855609e8 --- /dev/null +++ b/packages/moss-agent/test/observability-tracing.spec.mjs @@ -0,0 +1,44 @@ +#!/usr/bin/env node +// @rdk-moss/agent — withSpan: success path sets OK, error path records +// exception + rethrows. attributes constructors produce expected shape. +// setTracer/setTraceRedactor/TraceRegistry/getTracer legacy shims exist & are noop. +import assert from 'node:assert/strict'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import path from 'node:path'; + +const dir = path.dirname(fileURLToPath(import.meta.url)); +const mod = await import(pathToFileURL(path.join(dir, '..', 'dist', 'observability', 'tracing.js')).href); +const { + withSpan, turnAttributes, toolAttributes, llmRequestAttributes, + setTracer, setTraceRedactor, getTracer, TraceRegistry, +} = mod; + +// 未注册 tracer provider 时为 noop tracer，withSpan 仍正常执行 fn 并返回结果。 +const result = await withSpan('test.span', { a: 1 }, async (span) => { + assert.equal(typeof span.setAttribute, 'function'); + assert.equal(typeof span.addEvent, 'function'); + assert.equal(typeof span.setStatus, 'function'); + assert.equal(typeof span.end, 'function'); + span.setAttribute('k', 'v'); + span.addEvent('ev', { x: 1 }); + return 42; +}); +assert.equal(result, 42, 'withSpan returns fn result'); + +// 异常路径：rethrow（不吞错） +await assert.rejects( + withSpan('test.err', {}, async () => { throw new Error('boom'); }), + /boom/, +); + +// attributes 构造器形状 +assert.deepEqual(turnAttributes('r1', 3, 'm'), { runId: 'r1', turn: 3, model: 'm' }); +assert.deepEqual(toolAttributes('r1', 'read_file', 'tc1'), { runId: 'r1', toolName: 'read_file', toolCallId: 'tc1' }); +assert.deepEqual(llmRequestAttributes('r1', 'm', 100), { runId: 'r1', model: 'm', inputTokens: 100 }); + +// 旧 API 保留为 noop shim，不抛 +assert.doesNotThrow(() => setTracer('console')); +assert.doesNotThrow(() => setTraceRedactor((s) => s)); +assert.ok(getTracer()); +assert.ok(new TraceRegistry()); +console.error('[spec] observability-tracing OK'); diff --git a/scripts/check-workspace-hygiene.mjs b/scripts/check-workspace-hygiene.mjs index 4b3ab9c2..920a4a8c 100644 --- a/scripts/check-workspace-hygiene.mjs +++ b/scripts/check-workspace-hygiene.mjs @@ -46,14 +46,27 @@ function markdownAnchors(body) { return anchors; } +function stripCodeBlocks(body) { + // Remove fenced code blocks (``` or ~~~) AND their markers, so TS + // computed-key syntax like `[ATTR]: value,` inside an example is not + // mistaken for a markdown reference definition (`[name]: url`). + // CommonMark: a fenced code block opens with up to 3 leading spaces followed + // by ``` or ~~~ (optionally a language string) and closes with a fence of + // the same kind with at least as many ticks. + return body + .replace(/^[ \t]{0,3}(`{3,}|~{3,})[^\n]*\n[\s\S]*?\n[ \t]{0,3}\1[ \t]*$/gm, '') + .replace(/^( {4}|\t).*$/gm, ''); +} + function findMarkdownLinks(body) { + const stripped = stripCodeBlocks(body); const links = []; const inlineLink = /!?\[[^\]]*]\(([^)\s]+)(?:\s+"[^"]*")?\)/g; - for (const match of body.matchAll(inlineLink)) { + for (const match of stripped.matchAll(inlineLink)) { links.push(match[1]); } const refDef = /^\s*\[[^\]]+]:\s+(\S+)/gm; - for (const match of body.matchAll(refDef)) { + for (const match of stripped.matchAll(refDef)) { links.push(match[1]); } return links;