diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 7a82cef..092e941 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -21,7 +21,10 @@ jobs: - name: Compile Python helpers run: | python3 -m py_compile \ + tools/skill_observability/*.py \ + tools/skill_observability/adapters/*.py \ skills/entity-router/scripts/*.py \ + skills/entity-pgen/scripts/*.py \ skills/entity-env-build/scripts/*.py \ skills/entity-nt2py/scripts/*.py @@ -31,6 +34,15 @@ jobs: - name: Test entity-env-build run: python3 -m unittest discover -s skills/entity-env-build/tests -v + - name: Test entity-pgen + run: python3 -m unittest discover -s skills/entity-pgen/tests -v + + - name: Validate observability schemas + run: | + for schema in tools/skill_observability/schemas/*.json; do + python3 -m json.tool "$schema" >/dev/null + done + - name: Check skill frontmatter run: | python3 - <<'PY' diff --git a/.gitignore b/.gitignore index 8289ec5..d83edfd 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,6 @@ htmlcov/ # Local scratch and remote smoke-test artifacts test-wtih-cc/ .tmp/ + +# Large evaluation source caches (recorded in MANIFEST.json, not committed) +evals/e2e-neutral-streaming/source-cache/ diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..fcd5135 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,144 @@ +# Changelog + +All notable changes to the Entity skills bundle are documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +The bundle is versioned with semantic versioning, starting at 0.x: the first +version considered production-satisfactory will be released as 1.0.0. Schema +versions (store, plan, GoalSpec, checkpoint, compat checker) are independent +integer compatibility contracts and are not the product version. + +## [0.5.0] - 2026-07-22 + +Observability and reconciliation (Phase 3): out-of-band changes now surface, +drift fails, and skill adoption is measurable. + +### Added + +- `status --live` reconciliation: the report always carries a `divergences` + list classifying out-of-band changes between the store and the scheduler: + - `job_gone`: the recorded job is absent from `squeue`; `confirmed` is true + when `sacct` also has no record, false when `sacct` is unavailable. + - `state_mismatch`: the job reached a terminal scheduler state + (`recorded: submitted`, `observed: `) the router never saw. + - `untracked_job`: a foreign scheduler job runs in the Case run root — + evidence of control-plane bypass. + Live status makes at most three bounded scheduler queries. +- `entityctl apply --refresh`: re-executes a completed `data` Goal plan so a + stale `data-inventory.json` is rebuilt after run artifacts changed. + Rejected for other Goal kinds. This closes the 0.4.0 known limitation. +- `doctor` hard failures (exit 2, `ok: false`, new `failures` list): client + bundle installs that drifted from the runtime bundle, and stored Site + profiles that fail validation. +- `doctor` warnings for leaked active Operations (crash leftovers holding a + Case), each with the exact `entityctl operation cancel ` remedy. +- Skill adoption metrics: the phase-segmentation report + (`tools/skill_observability/adapters/claude_phases.py`) now carries a + `skill_adoption` section counting skill-script invocations + (router/env_build/pgen/nt2py) versus raw equivalents (sbatch/srun/scancel/ + scheduler polls/build tools), direct `sqlite` control-plane surgery, and a + `skill_call_share` ratio — the clean-comparison metric required by the + 1.0.0 candidate criteria. + +### Changed + +- The `data.inventory.v1` executor Step always re-walks the run root instead + of short-circuiting on a previous verified receipt; inventory is a pure + function of the current run root, which is what makes `--refresh` work. + +## [0.4.0] - 2026-07-22 + +### Added + +- New Goal kinds `build` and `data` (Phase 2 e2e coverage): + - `build` registers an env-build-produced build into the identity chain. + Hard gates at plan and apply: the checkpoint must have + `compatibility.status == "pass"` and a `decisions.parameters` confirmation + digest. The registered build identity feeds subsequent run Goals without + an explicit `executable`. + - `data` inventories a run's outputs into a content-hashed + `data-inventory.json` and advances `current.data_id` / readiness. +- Executor Step kinds `build.register.v1` and `data.inventory.v1`. +- Gates added: run Goal planning now requires a simulation-parameter + confirmation record (`.decisions.json` written by + `pgen_preflight.py confirm`) whose `input_sha256` matches the current TOML; + missing or stale records fail planning with `needs_decision` (Phase 1.5). + +### Changed + +- Plans now carry `goal_kind`; `validate_plan` checks per-kind key sets and + Step schemas. Plans written by 0.1.0 remain valid (missing `goal_kind` + reads as `run`) but should be re-planned. + +### Known limitations + +- Re-inventorying a run whose outputs changed produces the same Plan, which + resolves to the completed Operation; a refresh path lands with the + reconciliation work in 0.5.0. +- `analysis` Goal is not implemented yet; analysis stays with entity-nt2py + and router records only. + +## [0.3.0] - 2026-07-22 + +### Added + +- entity-env-build: build parameter confirmation gate (Phase 1.5, hard fail). + `entity_checkpoint.py confirm --checkpoint + --by [--confirm-defaults]` records `decisions.parameters` + `{digest, confirmed_by, confirmed_at, defaults, card}`; the compatibility + check `parameters.confirmation` fails when the record is missing or the + digest no longer matches the current requirements. +- entity-pgen: `pgen_preflight.py card ` prints the simulation + parameter card; `pgen_preflight.py confirm --by + [--confirm-defaults]` writes `.decisions.json`. + +## [0.2.0] - 2026-07-22 + +### Added + +- `entityctl site discover `: enumerates Slurm partitions and QoS via + `sinfo`/`sacctmgr` and suggests `policy.default_*` values (read-only). +- `templates/site-profile.schema.json` documenting the Site profile contract. +- `entityctl submission create/verify`: submission fingerprints are always + recomputed by the tool from the final artifacts; `verify` reports stale and + missing artifacts and exits non-zero on drift. +- Executor anomaly remediation: scheduler rejections (invalid QoS/partition) + now carry the exact repair path (`entityctl site discover` + policy fix). + +## [0.1.0] - 2026-07-22 + +First disciplined release. Baseline: the Entity Router v5 architecture lineage +(store schema 1) with the fixes below. Development plan: +`design/router-development-plan-2026-07-22.md`. + +### Added + +- `entityctl operation cancel `: terminal escape hatch for pending/running + Operations; releases the Case so a different Plan can proceed. +- `entityctl store migrate`: store schema migration entry point (shell; no + schema migrations exist yet). Doctor reports the store schema version and + points here on mismatch. +- `bundle_version` reported alongside `bundle_hash` in `entityctl doctor` and + install receipts; single source at `skills/entity-router/VERSION`. +- entity-env-build: `mpi.openmpi_min_version` compatibility check — OpenMPI + must be >= 5.0.0 when selected (fail below; warn when unrecorded). +- Gates added: re-applying a Plan after a failed Apply reopens the Operation + and resumes committed Steps instead of deadlocking the Case. + +### Fixed + +- entity-router executor sbatch now invokes `srun -input `; + Entity ignores positional arguments and previously opened the default + `input` file, burning the first submitted job. +- A failed Apply no longer leaks an active Operation; it finishes as + `anomaly` and releases the Case (previously forced direct sqlite surgery). +- Missing-store error no longer points at the nonexistent `entityctl migrate`. +- `status --live` degrades to cached controller state with a warning when the + scheduler query fails, instead of exiting 2. + +### Changed + +- User-facing "Router v5" strings renamed to plain "Entity Router"; v5 remains + only as the internal architecture lineage and store schema integer. +- Doctor output keys `v5_store`/`v5_store_available` renamed to + `store`/`store_available`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dccfce5..becaa60 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,11 +29,19 @@ refactor(nt2py): simplify data inventory ```bash python3 -m unittest discover -s tests -v +python3 -m unittest discover -s skills/entity-pgen/tests -v python3 -m unittest discover -s skills/entity-env-build/tests -v python3 -m py_compile \ + tools/skill_observability/*.py \ + tools/skill_observability/adapters/*.py \ skills/entity-router/scripts/*.py \ + skills/entity-pgen/scripts/*.py \ skills/entity-env-build/scripts/*.py \ skills/entity-nt2py/scripts/*.py + +for schema in tools/skill_observability/schemas/*.json; do + python3 -m json.tool "$schema" >/dev/null +done ``` 四个 `SKILL.md` 都必须保留合法 YAML frontmatter,并通过 skill 结构校验。 diff --git a/README.md b/README.md index d7b162f..2c1f25d 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,18 @@ 一套帮助 Agent 使用 [Entity](https://github.com/entity-toolkit/entity) 完成天体物理模拟的 skills package。 -`skills/entity-router/SKILL.md` 是 Router 入口。它根据用户目标和 workspace 状态选择 playbook,并按需调用三个同级子 skill: +`skills/entity-router/SKILL.md` 是受管 simulation 工作流的控制入口。Router v5 +把公共模型收敛为 `Project → Goal → Operation → Evidence`:主 Agent 只提供用户语义, +程序派生 Case/Operation ID、路径、hash、Locator、claim、Step 和恢复状态。边界明确的 +只读或 standalone 领域任务仍可直接调用对应 owner skill。 - `entity-pgen`:PGen、匹配 TOML 和设计记录; - `entity-env-build`:依赖环境与 Entity 编译; - `entity-nt2py`:nt2py 数据访问、绘图和导出。 -Simulation 的准备、运行、续跑和状态恢复由 Router 按自身 `playbooks/` 管理,不单独建立 run skill。 - -初版 Router 将专业工作和 Router-owned run 操作放入 Case-bound Worker;Router 只保留控制上下文。`skills/entity-router/scripts/entity_router_state.py` 负责 Case、Workflow、Action Contract、revision、事件和 stale 状态。 +Simulation run 的计划、提交、续接和状态恢复统一由 `entityctl.py plan/apply/status` +管理,不单独建立 run skill。SQLite `router.db` 是唯一结构化 controller authority; +Local 与 SSH 使用同一个内容寻址 executor 和同一份 StepSpec。 ```text entity-skills/ @@ -20,17 +23,59 @@ entity-skills/ │ │ ├── agents/ │ │ ├── scripts/ │ │ ├── references/ -│ │ ├── playbooks/ │ │ └── templates/ │ ├── entity-pgen/ │ ├── entity-env-build/ │ └── entity-nt2py/ ├── tests/ +├── tools/skill_observability/ +├── evals/e2e-neutral-streaming/ ├── design/ └── legacy/ ``` -当前整体设计见 `design/architecture.md`,workspace 和状态细节见 `design/workspace-and-state.md`。`design/` 和 `legacy/` 不属于 Router 运行时上下文。 +当前实施合同见 `design/router-v5-goal-operation-architecture.md`;改造前基线见 +`design/current-entity-router-architecture-2026-07-19.md`。`architecture-v4.md`、 +`architecture.md` 和 `model-efficient-router-flow.md` 是历史设计,不代表当前公共入口。 +skill 执行观测合同见 `design/skill-observability.md`。`design/` 和 `legacy/` 不属于 +Router 运行时上下文。 + +## 公共控制状态 + +Codex、Claude Code、Kimi Code 和普通 shell 默认共享控制机上的 +`~/.entity-router/router.db`。Operation、identity、事件和 evidence reference 不写入 +客户端私有目录或源码仓库。远端不可用时仍可读取最后一次控制快照,但缓存 evidence +不代表当前远端事实。 + +```bash +python3 skills/entity-router/scripts/entityctl.py doctor +python3 skills/entity-router/scripts/entityctl.py \ + --actor-run-id --actor-provider \ + install --source-root /path/to/entity-skills/skills +python3 skills/entity-router/scripts/entityctl.py \ + --actor-run-id --actor-provider \ + site add --profile /absolute/site-profile.json +python3 skills/entity-router/scripts/entityctl.py site list +python3 skills/entity-router/scripts/entityctl.py export --output /absolute/export.json +python3 skills/entity-router/scripts/entityctl.py plan \ + --project-root /absolute/project --goal goal.json --output plan.json +python3 skills/entity-router/scripts/entityctl.py \ + --actor-run-id --actor-provider \ + apply --plan plan.json +python3 skills/entity-router/scripts/entityctl.py status \ + --project-root /absolute/project --live +``` + +`plan` 只写指定的 Plan artifact,不推进 controller 状态;同一 Plan 重复 `apply` +会根据 Step receipt 恢复,不重复 `sbatch`。`status` 默认只读本地 controller; +`--live` 最多执行三次有界 scheduler 查询:job 状态、job 离开队列后的 `sacct` +确认,以及 Case run root 的 untracked-job 扫描。 + +`entityctl install` 将一个经过 hash 验证的运行版本发布到 +`~/.entity-skills/bundles/`;Codex、Claude Code 和 Kimi Code 的 discovery 目录只保留 +指向同一 bundle 的符号链接投影,不再分别维护三套文件。 + +`entity-pgen` 的直接调用分为只读和 standalone 修改。它在写入前必须运行自身的 preflight;preflight 查询 Router v5 store,target 落在注册 Case 的 source/identity/active-run Locator 内即视为受管。受管写入当前 fail-closed(v5 pgen Goal 尚未实现),只读与 standalone 路径不受影响。Router 控制状态位于独立 control root,不依赖源码祖先目录中的 `_case/` 标记。 ## 仓库与发布 @@ -43,9 +88,95 @@ entity-skills/ 详细协作约定见 `CONTRIBUTING.md`。 +## Skill 运行观测 + +`tools/skill_observability/skill_observer.py` 提供平台无关的 append-only +trace。它记录 skill 身份、关键决策、工具调用、artifact 和外部验证, +不记录隐藏思维链,也不回写 Router 或 owner 状态。 + +创建 run 时必须传入任务、Agent/tool 配置指纹和实际暴露的 skill。 +Entity source、Router Case 和 raw data root 通过 `--protected-root` 显式保护; +skill source 会自动加入保护列表。 + +```bash +python3 tools/skill_observability/skill_observer.py start \ + --task-id \ + --input-ref \ + --input-sha256 \ + --variant full \ + --agent-provider \ + --agent-model \ + --agent-configuration \ + --tool-profile \ + --tool-configuration \ + --skill skills/entity-router \ + --skill skills/entity-pgen \ + --protected-root /absolute/entity-source \ + --protected-root /absolute/router-case \ + --protected-root /absolute/raw-data +``` + +使用返回的 `` 包装命令,小型 JSON 输出可作为脱敏证据保存: + +```bash +python3 tools/skill_observability/skill_observer.py tool \ + --run-dir \ + --name pgen-preflight \ + --capture-json \ + --capture-json-name pgen-preflight \ + --capture-authority entity-pgen-preflight \ + -- python3 skills/entity-pgen/scripts/pgen_preflight.py + +python3 tools/skill_observability/skill_observer.py evidence pgen-preflight \ + --run-dir \ + --result /evidence/pgen-preflight.json \ + --expect allowed + +python3 tools/skill_observability/skill_observer.py finish \ + --run-dir --status completed + +python3 tools/skill_observability/skill_observer.py validate \ + --run-dir +``` + +`evidence` 同时支持历史 `router-action`、v5 `router-operation`、`env-build` +和 `nt2py-inventory`。v5 验证器核对 Plan hash、Operation/Step journal、owner-site +receipts、controller status 和独立 scheduler snapshot。 +完整协议与证据等级见 `design/skill-observability.md`。 + +## 端到端 Skill 对照评测 + +`evals/e2e-neutral-streaming/` 是轻量 A/B 对照:skill/no-skill 两组跑同一个 +模拟任务,对照 trace、token 消耗和完成时间。任务文本、物理参数和启动方式见 +`evals/e2e-neutral-streaming/RUNBOOK.md`。对照 oracle(`oracle/` 下 5 个 gate 与 +`thresholds.json`,由 `tests/test_oracle.py` 覆盖)仍然保留;schema 与 fake Slurm +脚手架已于 2026-07-21 移除,历史版本见 git 记录。 + +已有 Codex、Claude Code 和 Kimi Code 记录均可增量导入。adapter 只保留 tool +call/output 的 hash、大小、顺序、原生 session/agent ID 和平台原始 usage,显式忽略 +对话文本和 reasoning,并排除 observer 自身调用: + +```bash +python3 tools/skill_observability/skill_observer.py import-codex \ + --run-dir \ + --rollout /absolute/path/to/codex-rollout.jsonl + +python3 tools/skill_observability/skill_observer.py import-claude \ + --run-dir \ + --transcript /absolute/path/to/claude-session.jsonl + +python3 tools/skill_observability/skill_observer.py import-kimi \ + --run-dir \ + --session /absolute/path/to/kimi-session-directory +``` + 本地验证: ```bash python3 -m unittest discover -s tests -v +python3 -m unittest discover -s skills/entity-pgen/tests -v python3 -m unittest discover -s skills/entity-env-build/tests -v +python3 -m py_compile \ + tools/skill_observability/*.py \ + tools/skill_observability/adapters/*.py ``` diff --git a/design/architecture-v4.md b/design/architecture-v4.md new file mode 100644 index 0000000..336de79 --- /dev/null +++ b/design/architecture-v4.md @@ -0,0 +1,269 @@ +# Entity Skills 架构 v4:共享控制状态与模型高效入口 + +日期:2026-07-18 +状态:已实施并通过本地验收 + +验收快照(2026-07-18): + +| 项目 | 结果 | +|---|---| +| 公共状态 | `bh-reconnection` 由公共 project binding 解析到 controller-local Case;查询前后 Case revision 65、sha256、size、mtime 全部不变 | +| 三端安装 | Codex、Claude Code、Kimi Code 的 12 个 skill 投影均解析到 bundle `sha256:ea3d59bcac8cb9cfcae46e52d2e809d0ac24cd447cb094e27660142330d301b6` | +| 单写合同 | 并行 writer 被拒绝;显式 handoff 后新 Agent 可继续;lease 过期不阻塞恢复 | +| 查询效率 | 真实 project inspect 为 1 次工具往返、0 次远端调用、0 次审批、2,281 B 输出,正确性硬门通过 | +| 回归 | Router/observability 84 项、PGen 3 项、env-build 18 项全部通过;Python 3.6 grammar、JSON、skill quick validation 和 `git diff --check` 通过 | + +Claude/Kimi 原生 session 已导入共享 trace,用于定位上下文成本;但两者不是同一任务的 +matched-scenario 对照,因此跨 provider token reduction 仍标记为 `not_assessed`。真实 SSH +build/run/nt2 staging 与 m87 live canary 属于长期 flow 计划的后续门,不作为本轮公共状态 +架构完成度的替代证据,也未在本轮触发远端计算。 + +## 1. 问题与目标 + +近期 `bh-reconnection` 实践暴露出四个相互耦合的问题: + +1. Codex、Claude Code 和 Kimi Code 各自安装了完整 skill 副本,版本可以漂移; +2. 项目源码、客户端私有目录、Router Case、远端 `_tools` 和会话日志之间的 + 权威边界不够直观; +3. Case 虽然共享,但事件和 Action 缺少 Agent/session/bundle provenance; +4. Agent 直接驾驶 `show -> start-action -> SSH -> finish-action` 原语,简单查询也会 + 触发大量模型轮次和上下文回灌。 + +v4 的目标不是删除 Router 的安全约束,而是把这些约束下沉到确定性程序中: + +- 所有同机 Agent 从一个公共控制区发现和读取项目状态; +- 控制事实默认保存在 Agent 所在机器,远端不可用时仍可恢复; +- 远端只拥有执行产物和原始数据,控制端只保存 Locator、身份和最后验证证据; +- 每次状态写入都可追溯到 Agent run 和 skill bundle; +- 普通查询只调用一个紧凑只读入口,不创建 Action、不修改 Case; +- 项目源码、控制状态、执行证据和运行 trace 各有且只有一个权威位置。 + +## 2. 文件存放与唯一权威 + +### 2.1 控制机公共区域 + +默认控制根为 `$ENTITY_ROUTER_HOME` 或 `~/.entity-router`。它位于 Agent 所在机器, +不属于任何客户端: + +```text +~/.entity-router/ +├── registry.json # Case UID -> control root,可重建索引 +├── project-bindings.json # project root -> Case UID,唯一项目绑定事实 +├── sites/.json # site/transport/root 描述 +└── cases/-