Docs/agent ci guide#15
Open
1-ztc wants to merge 57 commits into
Open
Conversation
…sion for same-file backups
Co-Authored-By: Claude <noreply@anthropic.com>
Sub-project 1 of 5 monitoring sub-projects. Scope: finish trace collection debt (commit session span wiring, batch OTLP export, CI tests, env docs) + add local HTML trend dashboard reading JSONL. Co-Authored-By: Claude <noreply@anthropic.com>
11-task plan: commit existing wiring, batch OTLP export, local tracer setAttribute fix, SessionExporter, session outcome write-back, MOSS_TRACE=file wiring, moss-analytics-dashboard package, moss stats command, un-ignore tests + CI, env docs, final verification. Co-Authored-By: Claude <noreply@anthropic.com>
记录让现有 trace 链路稳健、可操作、文档自洽的加固方案： 4318 端口冲突二选一文档化、清理未使用的 @opentelemetry/* 依赖、 新增 moss 侧启用文档。不改运行时行为。metrics/propagation/采样/ 持久化留作后续 spec。 Co-Authored-By: Claude <noreply@anthropic.com>
5 个任务：删未使用依赖、otel-bridge 注释、moss 侧启用文档、 otel README 二选一改写、全量验证。对应 spec 2026-07-12-otel-trace-hardening-design.md。 Co-Authored-By: Claude <noreply@anthropic.com>
otel-bridge 是纯 fetch + 手写 OTLP JSON 实现，不依赖官方 SDK。 这 4 个 devDependency 全仓库零 import，移除以减小体积、避免误导。 Co-Authored-By: Claude <noreply@anthropic.com>
避免后人误以为该接官方 SDK 而引入 @opentelemetry/* 依赖； 明确仅在 metrics / context propagation 等具体需求出现时才考虑。 Co-Authored-By: Claude <noreply@anthropic.com>
覆盖启用环境变量、两个后端二选一、本地文件导出、排查提示、 链路层级与代码位置引用。与 D:\otel\README.md 互为呼应。 Co-Authored-By: Claude <noreply@anthropic.com>
otel-bridge 引入 OtelSpanState / OTEL_STATE symbol,子 span 从 parent 继承 traceId 并填 parentSpanId;agent-loop 各层透传 parentSpan, moss-agent 创建 session 根 span 作为 parentSpan 并在会话结束后 向 OTLP 后端的 /v1/session-summary 上报会话摘要。 让 session → agent.llm_turn → tool.execute 三级 span 在 Jaeger/面板里真正串联成树,并补全会话级摘要视图。 Co-Authored-By: Claude <noreply@anthropic.com>
新增 PreFlightRouter:用户提问交给 LLM 前,若命中的 Skill 标记 timeSensitive(默认 true,纯方法类用 stable: true opt-out),则自动 跑 web_search+web_fetch 取最新结果注入上下文,再让 LLM 基于最新 数据回答。10s 超时 + 错误降级(失败返回空串不影响正常流程)。 - skills/registry+types: frontmatter 解析 timeSensitive / searchQueryTemplate - builtin: 10 个内置 skill 标 timeSensitive:false - tui/oneshot: buildPreSearchContext 在匹配 skill 后注入预搜索结果 - rdk-model-zoo/rdk-source-map SKILL: 标记时效性 - pre-flight-router.spec.mjs: 覆盖触发逻辑 - openspec/changes/factual-knowledge-web-search-gate: 提案与 spec Co-Authored-By: Claude <noreply@anthropic.com>
承接 43fdc6b:工具调用层把 parentSpan 传给 tool.execute span, 使孙 span 继承 session 根 span 的 traceId。 Co-Authored-By: Claude <noreply@anthropic.com>
提案对齐实际实现（简化范围）：集成点在 CLI 层而非 agent 循环、 预搜索仅由 Skill 标记单路触发（未实现 FactualQuestionClassifier 关键词兜底）、后端用 web_search+web_fetch 而非 knowledge_search、 timeSensitive 默认 true（stable opt-out）。tasks 勾选已做项、 跳过项标 [~] 注明。 Sync: 新建主 spec openspec/specs/pre-flight-router/spec.md。 归档至 openspec/changes/archive/2026-07-12-factual-knowledge-web-search-gate/。 Co-Authored-By: Claude <noreply@anthropic.com>
receiver 落本地 SQLite,重启不丢,面板可查历史,为后续 metrics spec 定义存储模型。不改 Moss 埋点/OTLP 协议/Collector。 metrics+SDK/采样/跨进程传播留作 spec2/3。 Co-Authored-By: Claude <noreply@anthropic.com>
7 个任务:装 better-sqlite3、SQLite 初始化建表、写路径接库、 读路径改从库读、清理定时器+退出 flush、README 补持久化说明、 端到端验证含重启不丢。对应 spec 2026-07-12-otel-trace-persistence-design.md。 Co-Authored-By: Claude <noreply@anthropic.com>
实现期验证发现 WAL 模式 checkpoint 时序依赖进程存活,强杀下 可能丢已提交数据。改 journal_mode=DELETE + synchronous=FULL, 每条 INSERT 同步刷盘,进程被强杀也不丢。spec 同步说明。 Co-Authored-By: Claude <noreply@anthropic.com>
On push/PR to 2026_* branches, install moss-ci from 1-ztc/ci_test, build Moss, and run the capability suite against the freshly-built Moss binary. Gates the PR on capability regression (a previously-passing capability test that now fails blocks merge). Requires the MOSS_API_KEY repo secret (Moss ignores env vars for model settings, so the CI writes a temporary config.json at runtime). Co-Authored-By: Claude <noreply@anthropic.com>
The printf-based config generation lost the API key to shell quoting (apiKey ended up empty). Write /tmp/moss-ci/config.json from Python reading MOSS_API_KEY from the env, which is quoting-proof and validates the JSON. Also fetch examples via a depth-1 clone into RUNNER_TEMP. Co-Authored-By: Claude <noreply@anthropic.com>
给 Moss 加 metrics 指标采集(LLM token/调用、工具耗时/成功率、 会话指标),经 OTel SDK → OTLP 发到现有 receiver → SQLite → 面板; trace 不动,仅加简易采样(默认1.0)。复用 spec1 持久化基建。 跨进程 propagation 留作 spec3,trace 迁 SDK 留作未来。 Co-Authored-By: Claude <noreply@anthropic.com>
9 个任务:装 SDK 依赖、metrics.ts 骨架、业务点记指标、cli-main 接线、 otel-bridge 采样、receiver /v1/metrics+表、/api/metrics+面板、 文档、端到端验证。对应 spec 2026-07-12-otel-metrics-design.md。 Co-Authored-By: Claude <noreply@anthropic.com>
metrics 指标采集将用官方 SDK Meter。这 5 个是真实运行时依赖 (api/sdk-metrics/exporter-metrics-otlp-http/resources/ semantic-conventions),非上轮删的未用死依赖。见 spec2。 Co-Authored-By: Claude <noreply@anthropic.com>
MeterProvider + 三类指标(Counter/Histogram)+ enable/disable + noop 兜底。依赖对齐到 2.x 线(sdk-metrics 2.9.0 + exporter-metrics-otlp-http 0.220.0),用 resourceFromAttributes 替代已废弃的 new Resource 构造。 Co-Authored-By: Claude <noreply@anthropic.com>
- LLM: tokens(in/out)、calls、duration_ms(成功+error 双路径) - 工具: calls、duration_ms(含 isError 状态) - 会话: session.count(按 outcome 维度) metrics 与 trace 独立,不开 metrics 时 noop 零开销。 会话 duration/turns 因 notifyRunObserver 拿不到精确时间暂略, 留待 trace 迁 SDK 后补。 Co-Authored-By: Claude <noreply@anthropic.com>
MOSS_METRICS_URL/MOSS_METRICS_ENABLED 开启,与 trace 独立。 MOSS_METRICS_SERVICE_NAME 默认回退 MOSS_OTEL_SERVICE_NAME。 Co-Authored-By: Claude <noreply@anthropic.com>
按 MOSS_TRACE_SAMPLE_RATIO(默认1.0)采样,traceId-hash 比例。 只影响 sendSpan 发送,不影响 span 树结构——未采样的 span 仍建(保证 parentSpan 链完整),子 span 继承父采样决策。 Co-Authored-By: Claude <noreply@anthropic.com>
metrics 环境变量、与 trace 独立、采样比例、三类指标清单、 receiver 收发端点。 Co-Authored-By: Claude <noreply@anthropic.com>
计划草稿用了 new Resource + SemanticResourceAttributes(1.x API), 实际实现对齐 2.x 用 resourceFromAttributes。同时消除 hygiene 对 [SemanticResourceAttributes.SERVICE_NAME] 的 broken-link 误报。 Co-Authored-By: Claude <noreply@anthropic.com>
After the capability suite, restore the last run's JSON from a per-branch cache, export the current run, diff-files the two into $GITHUB_STEP_SUMMARY (visible on PR Checks), then cache current as next run's baseline. First run on a branch establishes a baseline; subsequent runs report new_failures / fixed. Requires moss-ci with export + diff-files (1-ztc/ci_test@29e0bc5+). Co-Authored-By: Claude <noreply@anthropic.com>
Empty commit — fires the workflow again now that a baseline run is cached, so the job summary should show a diff-files report (expected: No changes, since Moss didn't change). Co-Authored-By: Claude <noreply@anthropic.com>
Moss 调 web_search/web_fetch 时注入 W3C traceparent 头,跨进程 链路不断。手写(不迁 SDK),AsyncLocalStorage 传当前 span, withSpan 包 als.run。只出站、只 web_search/web_fetch。 单测覆盖 propagation 逻辑。四个短板(持久化/metrics+采样/ 传播)的最后一个。 Co-Authored-By: Claude <noreply@anthropic.com>
7 个任务:otel-bridge 加 ALS+getCurrentSpan+injectTraceparent+ runWithContext、tracing.ts withSpan 接 runWithContext、index 导出、 web-search/web-fetch 注入、单测、文档、端到端验证。 对应 spec 2026-07-12-otel-context-propagation-design.md。 Co-Authored-By: Claude <noreply@anthropic.com>
otel-bridge 加 AsyncLocalStorage 存当前 span, getCurrentSpan() + injectTraceparent(); TraceSpan 接口加可选 runWithContext, otel-bridge 实现(als.run), withSpan 检测到则包 fn。旧 tracer 不受影响。 为跨进程 propagation 铺路(spec3)。 Co-Authored-By: Claude <noreply@anthropic.com>
index 导出 getCurrentSpan/injectTraceparent; web_search/web_fetch 出站 fetch 注入 W3C traceparent 头, 下游识别则加入同一 trace, 跨进程链路不断(spec3)。不在 span 内降级不注入。 Co-Authored-By: Claude <noreply@anthropic.com>
5/5 通过: getCurrentSpan(内/外)、injectTraceparent(内/外、 W3C 格式 00-32hex-16hex-2hex、sampled flag=01、保留已有 header)。 Co-Authored-By: Claude <noreply@anthropic.com>
web_search/web_fetch 注入 W3C traceparent,下游加入同一 trace。 依赖 fetch 在 span 异步链路内、不在 span 降级、只覆盖 web 工具、 flags 反映采样。 Co-Authored-By: Claude <noreply@anthropic.com>
New moss-ci (571cc8e) extracts structured tool calls from Moss session jsonl, and the capability suite now has a "工具调用-顺序与参数" test using tool_sequence + tool_args. This run exercises it on CI. Co-Authored-By: Claude <noreply@anthropic.com>
Official Anthropic API (api.anthropic.com) authenticates with x-api-key. Some Anthropic-protocol gateways fronted by a custom baseUrl (e.g. https://llmapi.horizon.auto) use Authorization: Bearer instead — these returned HTTP 401 against Moss. Detect at request time: if baseUrl is not api.anthropic.com, send Authorization: Bearer instead of x-api-key. Applied in both the provider-layer AnthropicLLMProvider and the CLI-layer callAnthropic (which the CLI one-shot path actually uses). Official Anthropic auth is unchanged. Verified: `moss --config-file <horizon-config> "reply horizon-ok"` returns horizon-ok via HORIZON-DeepSeek-Pro, and moss-ci successfully captures structured tool_calls through the same path. Co-Authored-By: Claude <noreply@anthropic.com>
之前 notifyRunObserver 拿不到 session 起始时间,会话只有 count。 现发现 run.params.parentSpan 即 sessionSpan,其 OTEL_STATE.startTime 可读——新增 getSpanStartTime() 公开 API(不戳私有 symbol), 记 session.duration_ms(从 sessionSpan 起始算)与 session.turns(done.result.toolCalls.length,作活跃度代理,turns 本身不在 done.result)。 Co-Authored-By: Claude <noreply@anthropic.com>
ObservableGauge 自动采 process.memoryUsage() 的 rss/heapUsed, 每次 export 记一行,零外部依赖(node 内置)。能看 Moss 长跑时 内存变化。未装 host-metrics(系统级,不对 Moss 症),自采进程级更对症。 Co-Authored-By: Claude <noreply@anthropic.com>
DO NOT MERGE — intentional regression to demonstrate that the moss-ci capability suite + regression diff catches a real degradation. read_file now replaces ZETA-7741-NOVA with CORRUPTED-TOKEN in returned content, so the "读取文件-提取随机token" capability test (which asks for the secret_token value, not present in the prompt) fails. Expected CI result: capability regression diff reports this test as a new_failure. Revert this commit once the demo is observed. Co-Authored-By: Claude <noreply@anthropic.com>
真机冒烟暴露:notifyRunObserver 上报 session-summary 读
process.env.MOSS_OTEL_URL,而 trace 的 url 经 enableOtelTracing({url})设
——编程式只传后者时 session-summary 漏发(sessions:0)。新增 getOtelUrl()
暴露 bridge 已配的 url,notifyRunObserver 优先用它、fallback env,
CLI 与编程式入口都正确。env 路径仍保留(CLI 兼容)。
Co-Authored-By: Claude <noreply@anthropic.com>
skill 匹配发生在 CLI 层(session span 之前),之前无法监控命中了哪些。
新增 getMatchedSkillNames();ChatOptions 加 matchedSkills 字段;
tui/oneshot 匹配后透传;MossAgent 建 session span 时
setAttribute('matchedSkills', names.join(','))。
不动 CLI/agent 边界——skill 命中作为 session span 的 attribute,
面板每轮 session 能看到命中了哪些 skill。真机验证:
matched skills ['rdk-source-map','rdk-model-zoo','rpi-knowledge']
记进 session span attrs。
Co-Authored-By: Claude <noreply@anthropic.com>
Revert the intentional read_file regression (ZETA→CORRUPTED) from the demo commit 401dd59 — Moss is back to normal read_file fidelity. Upgrade the workflow to two layers: quick on push/PR (--tag quick, ~2-3 min), full on workflow_dispatch + nightly schedule (multi-step end-to-end). Per-layer cache keys so regression diffs compare within the same layer. Wire MOSS_CI_JUDGE_API_URL secret into the run step for the llm_judge evaluator (F1). Requires moss-ci with files_modified extraction + --tag + `--` argv fix (1-ztc/ci_test@c154302). Co-Authored-By: Claude <noreply@anthropic.com>
Add docs/agent-ci.md explaining the moss-ci capability regression suite that gates Moss PRs: what the two-layer workflow runs, the 13-test coverage matrix, the required secrets, how tool calls / file changes are extracted from Moss sessions, custom Anthropic-protocol endpoints, and how to run it locally. Co-Authored-By: Claude <noreply@anthropic.com>
Add the verification result: all 13 tests pass against a real Moss build, including C1 autonomous bug-fix running 3× (3/3 pass). Note the C1 fix — maxTurns wasn't the blocker; the tests_pass evaluator just needed case-insensitive matching of Moss's natural phrasing. Co-Authored-By: Claude <noreply@anthropic.com>
The job name was dynamic (Moss capability tests (quick)/(full)/(manual)) via github.event_name interpolation. Branch protection's "Require status checks" matches by exact check name, so a dynamic name can't be reliably selected. Use a stable "Moss capability tests" — the layer is already shown in step names and the summary. Co-Authored-By: Claude <noreply@anthropic.com>
ci: stable job name for branch-protection status check matching
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the workflow job name stable (
Moss capability tests) instead of dynamic (Moss capability tests (quick)/(full)/(manual)).Why
Branch protection's "Require status checks" matches by exact check name. A dynamic name can't be reliably selected. The
layer (quick/full) is already shown in step names + summary.
🤖 Generated with Claude Code