ci: judge (Anthropic/Bearer) + keep diff alive on red runs#3
Open
1-ztc wants to merge 11 commits into
Open
Conversation
Sync from ci_test@1b61266: - Inject MOSS_CI_JUDGE_API_KEY alongside URL so F1 can really score on CI (Anthropic /v1/messages + Bearer, HORIZON-GLM @ llmapi.horizon.auto). - Add if: always() to Restore/Export/diff so the regression-diff report runs even when `moss-ci run` exits 1 on test failure (it was being skipped exactly when most needed), and guard the baseline save/cache so a red run doesn't corrupt the next baseline. Requires new repo secrets: MOSS_CI_JUDGE_API_URL, MOSS_CI_JUDGE_API_KEY. Co-Authored-By: Claude <noreply@anthropic.com>
发现 notifyRunObserver 的 AI 解读代码(getSpanTraceId/generateAiInsight/ generateRuleInsight/aiInsight)和 summary 的 traceId 字段被还原丢失, 导致 session.trace_id 空 → inspect_monitoring 工具拉不到 trace 树, AI 解读也空。重新加回: - otel-bridge 重新导出 getSpanTraceId(index 重新导出) - summary 带 traceId 字段(从 parentSpan 拿) - notifyRunObserver 改 async,调 generateAiInsight(含规则 fallback) 真机:session.trace_id 有值,AI 解读生成(规则 fallback),inspect_monitoring 能拉到完整 trace 树。 Co-Authored-By: Claude <noreply@anthropic.com>
inspect-monitoring.ts 和 builtin 的 import/注册被还原丢失,导致 Moss 找不到 inspect_monitoring 工具(它用 exec/read_file 自己摸)。重建。 Co-Authored-By: Claude <noreply@anthropic.com>
`moss-ci run` only prints the summary table, so a red run showed "1 failed" with no way to see which test failed or what the judge scored (F1) — only diagnosable by re-running locally. Add a step that prints per-test status + judge scores from the exported JSON, and upload current.json as an artifact for full offline inspection (flake runs, eval details). Co-Authored-By: Claude <noreply@anthropic.com>
3/3 通过:正常会话(trace树+工具+AI解读)、空session(提示先跑)、 session.traceId空((无trace)降级)。用 fixture receiver,全程一个server。 覆盖之前真机漏的 case(traceId空 → inspect拉不到trace)。
防止 linter/格式化器静默从 src 删代码,build 却仍绿(之前 inspect- monitoring.ts 和 spec4 解读代码两次被删都没发现)。build 后 grep dist 确认 inspectMonitoringTool/getSpanTraceId/generateAiInsight 等关键 符号真编进去,缺了就报错退出。并入 verify 链(build 之后、typecheck之前)。
Print each flake run's verdict + judge score (F1 runs 3x). Fix artifact name — layer_label has parens/space which upload-artifact rejects; use event_name instead. Co-Authored-By: Claude <noreply@anthropic.com>
F1 judge returns error="" on CI but 5.0 locally. Add temporary debug step reproducing the raw _call_judge with full exception repr/traceback. Co-Authored-By: Claude <noreply@anthropic.com>
Re-run the connectivity check (prior run showed ConnectTimeout from the GitHub runner to llmapi.horizon.auto). DNS + curl + 30s judge call, run BEFORE the 9-min Moss suite so we see the result fast. Co-Authored-By: Claude <noreply@anthropic.com>
记录 trace(3层span)、metrics(3类指标)、session摘要+AI解读、skill命中 的精确埋点位置(文件:行号)和"为什么埋这、不埋别处"。 含调用链图、数据流图、埋点设计原则、已知风险。
Probe step was temporary diagnosis (judge unreachable confirmed). Remove it. Update per-test printer to show skipped evals with the reason, so F1 reads 'llm_judge=SKIP(judge unreachable: ConnectTimeout...)' instead of a bare 'fail' when the runner can't reach the endpoint. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync from ci_test@1b61266:
moss-ci runexits 1 on test failure (it was being skipped exactly when most needed), and guard the baseline save/cache so a red run doesn't corrupt the next baseline.Requires new repo secrets: MOSS_CI_JUDGE_API_URL, MOSS_CI_JUDGE_API_KEY.
What & why
Fixes #
Type of change
Checklist
npm run verifyis green (boundaries + hygiene + build + typecheck + lint + test)@public/@beta/@internalAGENTS.mdScope Guard); robot/vendor specifics live in host adapters, knowledge modules, or platform extensions