Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ DeepseekCotexplorations/
| 上传者 ID | 研究主题 | 文件夹 | 环境摘要 |
|---|---|---|---|
| xiaobright | DeepSeek V4 在 DSH 各 preset 下的能力与触发机制 | [`xiaobright-deepseek-v4-harness`](./contributions/xiaobright-deepseek-v4-harness/) | DSH commit 47f9438 · Windows 11 + WSL 24.04 · DeepSeek 官方 API · deepseek-v4-pro/flash · minimal/standard/PTC/anchored-standard |
| MolecularFullerene | DSH 首请求工具 schema 2×2 消融与 Request #2 协议 pilot | [`molecularfullerene-dsh-schema-ablation`](./contributions/molecularfullerene-dsh-schema-ablation/) | DSH 0.1.0-rc.5 / 47f9438 · macOS 26.5.2 arm64 · DeepSeek 官方 API · deepseek-v4-pro · exact Minimal + 自建 2×2 surface |
| 1127353621zxm-netizen | DSH 会话导出(明文 JSONL)与内部存储(zstd 多帧)的格式差异与导入方法 | [`1127353621zxm-netizen-dsh-session-import-export`](./contributions/1127353621zxm-netizen-dsh-session-import-export/) | DSH rc.6 · Windows 11 · opencode go 订阅 · deepseek-v4-pro/flash · 自建 preset · node 22.23.2 |
| 1127353621zxm-netizen | node 版本与 DSH zstd 会话存储的兼容性(22.14 缺 zstd API 启动崩溃) | [`1127353621zxm-netizen-node-zstd-compat`](./contributions/1127353621zxm-netizen-node-zstd-compat/) | DSH rc.6 · Windows 11 · opencode go 订阅 · deepseek-v4-pro/flash · 自建 preset |
| 1127353621zxm-netizen | DSH 插件工具 JSON Schema 合规性修复(字段级 required / type:json) | [`1127353621zxm-netizen-deepwrite-schema-fixes`](./contributions/1127353621zxm-netizen-deepwrite-schema-fixes/) | DSH rc.6 · Windows 11 · opencode go 订阅 · deepseek-v4-flash · 自建 preset |
Expand Down
1 change: 1 addition & 0 deletions contributions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@
| 上传者 ID | 研究主题 | 文件夹 | 环境摘要 |
|---|---|---|---|
| xiaobright | DeepSeek V4 在 DSH 各 preset 下的能力与触发机制 | [`xiaobright-deepseek-v4-harness`](./xiaobright-deepseek-v4-harness/) | DSH commit 47f9438 · Windows 11 + WSL 24.04 · DeepSeek 官方 API · deepseek-v4-pro/flash · minimal/standard/PTC/anchored-standard |
| MolecularFullerene | DSH 首请求工具 schema 2×2 消融与 Request #2 协议 pilot | [`molecularfullerene-dsh-schema-ablation`](./molecularfullerene-dsh-schema-ablation/) | DSH 0.1.0-rc.5 / 47f9438 · macOS 26.5.2 arm64 · DeepSeek 官方 API · deepseek-v4-pro · exact Minimal + 自建 2×2 surface |
65 changes: 65 additions & 0 deletions contributions/molecularfullerene-dsh-schema-ablation/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# DeepSeek V4 Pro:DSH 首请求 Schema 消融与 Request #2 Pilot

- **上传者 ID**:MolecularFullerene
- **研究主题**:模型可见工具 schema 对首轮轨迹指纹的影响,以及 `reasoning_content` / session header 对第二请求的协议影响
- **日期**:2026-08-16

## 结果摘要

在固定官方 Minimal system、`contexts=[]`、首请求恰好两个工具、
`reasoningEffort=max` 的条件下,我们完成了固定样本量的
persistent/one-shot `bash` × editor/read 2×2 消融,共 40 条成功轨迹。

- persistent `bash`:17/20 为社区分类器的 minimal-like
- one-shot `bash`:6/20
- 风险差 +55 个百分点;nominal Fisher 双侧 `p≈0.0011`
- editor:13/20;read:10/20;Fisher `p≈0.523`
- 40/40 首动作均合法且任务相关,全部在 executor dispatch 前取消

因此,shell schema bundle 是本轮词法轨迹偏移中最大的观测边际关联;
file tool 的整体主效应证据较弱。该实验只证明 schema 会改变
`We need / Let me` 轨迹分布,不证明存在两个离散“人格”,也不证明
`Let me` 代表能力失败。

另完成 retain/drop `reasoning_content` × same/new session 的四格真实 API
protocol pilot。四格各 `n=1`,均 HTTP 200 且严格 JSON 正确。它只能排除
“删除 reasoning_content 必然触发协议拒绝或让简单任务立即失败”的过强说法;
不能证明 reasoning passback 或 session header 对复杂 agent 任务无影响。

探索性 screening 还观察到 exact Minimal 为 16/20 minimal-like,
macOS `bash/read` surrogate 为 8/20;该批次在初始 n=3 后看过结果再扩样,
所以只作为探索性证据。

## 实验环境

| 项 | 值 |
|---|---|
| dsh 版本 | `0.1.0-rc.5`, commit `47f943859bef60e4160492346772ded9b24f765a` |
| 操作系统 | macOS 26.5.2, arm64 |
| API 来源 | DeepSeek 官方 API |
| 模型 | `deepseek-v4-pro`, `reasoningEffort=max` |
| harness / preset | 官方 exact Minimal system;自建 persistent/one-shot bash × editor/read 2×2 surface |
| 其他 | Node.js v24.18.0;factorial 使用 `max_tokens=256000`;request #2 pilot 使用 4096 |
| 对照快照 | `dsh-anchored-standard@db4527a2...`; `modeltest@04255b55...` |

模型名为服务端可变 alias,API 未暴露 server build/revision。两个题面与批次、
identity 和时间仍有共变;报告中的 Fisher p 值为 nominal,不能替代跨任务复现。

## 材料清单

- `reports/FACTORIAL_REPORT.md`:固定样本量 2×2 schema 消融
- `reports/REQUEST2_PILOT_REPORT.md`:request #2 四格协议 pilot
- `reports/SCREENING_REPORT.md`:探索性首轮 screening
- `data/factorial-summary.json`:去标识化分格与效应汇总
- `data/request2-pilot-summary.json`:四格 protocol/correctness 汇总
- `data/screening-summary.json`:去标识化探索性汇总

不公开 API key、原始 user/session id、其稳定 hash、完整 reasoning、原始 request
body、工具调用 ID、绝对路径或 Project2 测试内容。

## 备注

`minimal-like` 是对 `We need / Let me` 等词法特征的轨迹标签,不是能力分数。
分类口径引用 [`xiaobright/modeltest@04255b55`](https://github.com/xiaobright/modeltest/tree/04255b55f16c4439e538239fb9783070c4165081),未复制其源码。

实验器实现、统计复核及报告草稿由 OpenAI Codex 辅助;实验运行与公开结论由上传者复核。
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"schemaVersion": 1,
"study": "first-request-shell-by-file-schema-factorial",
"date": "2026-08-16",
"model": "deepseek-v4-pro",
"provider": "deepseek-official",
"harnessCommit": "47f943859bef60e4160492346772ded9b24f765a",
"platform": "darwin-arm64",
"fixedSurface": {
"systemSha256Prefix": "5fab6e32",
"contexts": 0,
"toolCount": 2,
"reasoningEffort": "max",
"maxTokens": 256000
},
"design": {
"tasks": 2,
"cellsPerTask": 4,
"samplesPerCell": 5,
"includedTrajectories": 40,
"freshSessionPerTrajectory": true,
"randomizedWithinTask": true,
"toolExecution": "cancelled-before-dispatch"
},
"outcome": {
"primaryDiagnostic": "minimal-like lexical trajectory label",
"abilityEndpoint": false,
"validFirstAction": "40/40"
},
"cells": [
{"task":"repo-inspection","shell":"persistent","file":"editor","minimalLike":5,"n":5,"meanPromptTokens":1195},
{"task":"repo-inspection","shell":"persistent","file":"read","minimalLike":5,"n":5,"meanPromptTokens":704},
{"task":"repo-inspection","shell":"one-shot","file":"editor","minimalLike":2,"n":5,"meanPromptTokens":1727},
{"task":"repo-inspection","shell":"one-shot","file":"read","minimalLike":3,"n":5,"meanPromptTokens":1236},
{"task":"project2","shell":"persistent","file":"editor","minimalLike":5,"n":5,"meanPromptTokens":1683},
{"task":"project2","shell":"persistent","file":"read","minimalLike":2,"n":5,"meanPromptTokens":1192},
{"task":"project2","shell":"one-shot","file":"editor","minimalLike":1,"n":5,"meanPromptTokens":2215},
{"task":"project2","shell":"one-shot","file":"read","minimalLike":0,"n":5,"meanPromptTokens":1724}
],
"marginalEffects": {
"persistentVsOneShotShell": {
"left": "17/20",
"right": "6/20",
"riskDifference": 0.55,
"fisherTwoSidedNominal": 0.0010645539010388843,
"blockedExactRandomizationTwoSidedApprox": 0.000196
},
"editorVsRead": {
"left": "13/20",
"right": "10/20",
"riskDifference": 0.15,
"fisherTwoSidedNominal": 0.5231070892917388,
"blockedExactRandomizationTwoSidedApprox": 0.413
}
},
"limitations": [
"The shell factor changes a complete schema bundle, not only persistence semantics.",
"Forty trajectories come from two prompt templates and are not forty independent tasks.",
"Model alias exposes no immutable server build identifier.",
"The lexical label is diagnostic and is not a task-success score."
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"schemaVersion": 1,
"study": "request2-reasoning-content-by-session-header-protocol-pilot",
"date": "2026-08-16",
"model": "deepseek-v4-pro",
"provider": "deepseek-official",
"harnessCommit": "47f943859bef60e4160492346772ded9b24f765a",
"platform": "darwin-arm64",
"design": {
"independentRequest1SourcePerTreatment": true,
"samplesPerCell": 1,
"mainSamplesRun": 0,
"reasoningFactor": ["retain", "drop"],
"sessionFactor": ["same", "new"]
},
"cells": [
{"id":"drop-new","reasoning":"drop","session":"new","httpStatus":200,"finishKind":"stop","answerCorrect":true,"request1ReasoningTokens":55,"request2ReasoningTokens":51,"request1StartsWeNeed":true,"request2StartsWeNeed":false,"request2StartsLetMe":false},
{"id":"retain-same","reasoning":"retain","session":"same","httpStatus":200,"finishKind":"stop","answerCorrect":true,"request1ReasoningTokens":61,"request2ReasoningTokens":34,"request1StartsWeNeed":true,"request2StartsWeNeed":false,"request2StartsLetMe":false},
{"id":"retain-new","reasoning":"retain","session":"new","httpStatus":200,"finishKind":"stop","answerCorrect":true,"request1ReasoningTokens":62,"request2ReasoningTokens":0,"request1StartsWeNeed":true,"request2StartsWeNeed":false,"request2StartsLetMe":false},
{"id":"drop-same","reasoning":"drop","session":"same","httpStatus":200,"finishKind":"stop","answerCorrect":true,"request1ReasoningTokens":8,"request2ReasoningTokens":32,"request1StartsWeNeed":true,"request2StartsWeNeed":false,"request2StartsLetMe":false}
],
"interpretation": "Drop was protocol-acceptable for this fixture. Ceiling performance at n=1/cell cannot establish null effects on complex tasks.",
"excludedFromPublicProjection": [
"reasoning text",
"request bodies and hashes",
"header values and hashes",
"user/session identifiers and hashes",
"tool-call identifiers"
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"schemaVersion": 1,
"study": "exploratory-exact-minimal-vs-bash-read-screening",
"date": "2026-08-16",
"model": "deepseek-v4-pro",
"provider": "deepseek-official",
"platform": "darwin-arm64",
"groups": [
{"task":"repo-inspection","surface":"exact-minimal","minimalLike":10,"n":10},
{"task":"repo-inspection","surface":"bash-read-surrogate","minimalLike":6,"n":10},
{"task":"project2","surface":"exact-minimal","minimalLike":6,"n":10},
{"task":"project2","surface":"bash-read-surrogate","minimalLike":2,"n":10}
],
"overall": {
"exactMinimal": "16/20",
"bashReadSurrogate": "8/20",
"riskDifference": 0.4,
"fisherTwoSidedExploratory": 0.0225,
"validFirstAction": "40/40"
},
"limitations": [
"Sample extension from n=3 to n=10 per cell was decided after viewing the initial results.",
"The macOS bash/read surface is not the Windows pwsh/read configuration used in the historical 98/99 runs.",
"The outcome is a lexical trajectory label, not an ability endpoint."
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# V4 Pro 首请求 2×2 Schema 因子消融(2026-08-16)

## 结论摘要

在固定官方 Minimal system、空 runtime context、恰好两个工具的前提下,本轮把首请求目录拆成两个因子:

- shell schema:官方 Minimal persistent `bash` vs Standard one-shot `bash`
- file schema:`str_replace_editor` vs Standard `read`

固定每个 `题面 × shell × file` cell 为 `n=5`,共 40 条成功轨迹;批次在观察结果前固定样本量,没有中途扩样。以社区既有分类器的 `minimal-like` 标签为诊断终点:

- persistent `bash`:17/20(85%,Wilson 95% CI 64.0%–94.8%)
- one-shot `bash`:6/20(30%,Wilson 95% CI 14.5%–51.9%)
- 风险差:+55 个百分点;Fisher 双侧 `p≈0.0011`
- `editor`:13/20(65%)
- `read`:10/20(50%)
- 风险差:+15 个百分点;Fisher 双侧 `p≈0.523`

在这两个固定题面、40 次首轮抽样里,**shell schema bundle 是本 2×2 中最大的观测边际关联;file schema 的整体主效应证据很弱**。这比“工具数越少越好”或“必须是 editor”更精确,但仍然只证明了输出轨迹分布改变,不证明 persistent 机制本身是原因,也不证明模型能力被打开或关闭。

## 设计与完整分格

两个题面分别是英文 repo-inspection 短探针和中文 Project2 V4.1b 正文。四格均保持:

- system 为官方 Minimal 单句,SHA-256 `5fab6e32...`
- `contexts=[]`
- 首请求恰好两个真实 DSH tool schemas
- `reasoning_effort=max`、`max_tokens=256000`
- 同题面批次共享匿名 user id,每次 fresh session
- 四格随机交错
- assistant message 持久化后、任何工具执行前同步取消

运行时间窗为 2026-08-16 18:03–18:07 CST。40 条轨迹均为唯一 fresh session,两个题面批次各使用一个固定匿名 identity;模型名称是服务端可变 alias,没有可见的 server build id。

| 题面 | persistent + editor | persistent + read | one-shot + editor | one-shot + read |
|---|---:|---:|---:|---:|
| repo-inspection | 5/5 | 5/5 | 2/5 | 3/5 |
| Project2 | 5/5 | 2/5 | 1/5 | 0/5 |
| 合计 | 10/10 | 7/10 | 3/10 | 3/10 |

按题面分层,persistent shell 的方向一致:

- repo:10/10 vs 5/10,Fisher 双侧 `p≈0.0325`
- Project2:7/10 vs 1/10,Fisher 双侧 `p≈0.0198`

`editor` 的 Project2 点估计较高(6/10 vs 2/10),但样本很小且 Fisher `p≈0.170`;repo 上方向相反但更小(7/10 vs 8/10)。所以目前最多称为“file-tool interaction 可能随任务变化”,不能称 editor 已被确认是第二个触发器。

## 不只是“多了 41 token”

四格 prompt token 总量如下:

| 题面 | persistent + read | persistent + editor | one-shot + read | one-shot + editor |
|---|---:|---:|---:|---:|
| repo-inspection | 704 | 1195 | 1236 | 1727 |
| Project2 | 1192 | 1683 | 1724 | 2215 |

结果不随 token 总量单调变化。最醒目的近似 token-match 是 Project2:

- persistent + editor:1683 token,5/5 minimal-like
- one-shot + read:1724 token,0/5 minimal-like

两者只差 41 token,却相差 100 个百分点。这削弱了“总 context 长度足以解释一切”的说法。不过这个 corner 对比同时更换了 shell 与 file 两套 schema;persistent 与 one-shot 的总体平均 prompt token 又相差 532,所以尚未定位到长度、语义或某个参数字段中的哪一个才是原因。

两种 `bash` 的模型可见接口差别很大:one-shot 版的 description 和 parameters 更长,包含 `description` 必填参数以及 workdir、timeout、background、sandbox/escalation 等控制面;persistent 版基本只要求 command。下一轮应分别对这些字段做增量消融,而不是继续只比较整个 preset。

## 不能从本轮推出什么

40/40 首动作都合法、任务相关,且全部在 executor dispatch 前被 Harness 阻止:

- minimal-like:23/23 首动作有效
- standard-like:12/12 首动作有效
- ambiguous:5/5 首动作有效

所以本轮没有观察到 `Let me` 轨迹在首步功能上更差。这个分类器实际上主要识别首行 `We need` 与 `Let me`;五个 ambiguous 全来自 repo + one-shot 条件,也不支持一个普适的严格二元状态。`We need` 是一个对 schema 敏感的轨迹代理变量,不是“启动脑子”的已验证因果按钮。

此外,样本以同一题面内的多次随机抽样为主,不能把 40 次当成 40 个独立任务。普通 Fisher 检验近似假定这些轨迹可交换且近独立;同批次共享匿名 identity,仍可能存在后端路由或时序相关。两个题面又与 identity、batch、时间完全共变,因此不能把二者差异命名为纯 task 主效应。这里报告的 `p` 值均为 nominal,未校正多重比较;分题面和 file-tool 分析只作探索性描述。

## 下一轮最有信息量的实验

1. **真实能力终点**:使用 held-out、多步、只读任务,四种首轮 schema 在首次工具调用后晋升到完全相同的工具面;盲评任务成功率、工具序列、token、时延和成本。
2. **CoT 回传因果**:对同一个首轮响应 fork request #2,保留/删除 `reasoning_content`,并交叉 same/new session id。DSH 会显式回传带 tool call 的首轮 reasoning,这个普通的 in-context 自条件化解释必须先排除。
3. **shell 字段消融**:从 persistent schema 逐项加入 one-shot 的 required `description`、workdir、timeout、background、sandbox 等字段,并做等 token padding;定位是语义、参数复杂度还是长度在驱动词法轨迹。
4. **能力与指纹解耦**:在 schema 固定时随机干预 `We need` / `Let me` 前缀,评分者盲于前缀。只有前缀干预稳定改变任务成绩,才有资格称它为触发器。

可公开的去标识化机器汇总见 `../data/factorial-summary.json`。完整逐次轨迹、模型 reasoning、身份哈希与原始请求元数据仅本地保留,不随研究包公开。
Loading
Loading