feat: doc-mode backend — single-file portable idea format + SQLite→doc export#88
Merged
Conversation
Match the header.png banner convention used by flywheel-memory, flywheel-concept, and flywheel-geometry. flywheel-ideas was the only sibling without one. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
idea.ts was 1,340 lines with 12 action handlers in a single switch. Split each handler into its own module under tools/idea/: register.ts — tool registration + zod schema + dispatch + error boundary create.ts — handleCreate read.ts — handleRead + buildReadNextSteps list.ts — handleList transition.ts — handleTransition + buildTransitionNextSteps forget.ts — handleForget freeze.ts — handleFreeze + handleListFreezes lineage.ts — handleAncestry/Descendants/SharedAssumptions export.ts — handleExport report.ts — handleReport tools/idea.ts kept as a one-line re-export shim so existing imports (src/index.ts) are unchanged. Zero behavior or response-shape change; the full test suite (763 core + 175 mcp-server) passes unmodified. Closes the active hardening item from the roadmap: lower review cost before bimodal SQLite/doc-mode handler work lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
schema.ts has carried SCHEMA_SQL_V13/V14/V15 for several releases without
being re-exported from the core barrel — the published package was at
SCHEMA_VERSION=15 but only re-exposed the SQL constants through V12.
Fix:
- Extend the barrel to cover SCHEMA_SQL_V13/V14/V15.
- Add packages/core/test/api-surface.test.ts:
* Snapshots the sorted list of every exported binding so future
drift (added or removed exports) fails loud in code review.
* Asserts SCHEMA_VERSION equals the highest re-exported
SCHEMA_SQL_V* constant — catches the inverse "bumped version,
forgot to re-export the constant" failure mode too.
- Refresh stale module comments: council-parsers.ts no longer
describes codex/gemini as stubs throwing NotYetImplemented (the
parsers shipped in M9); dispatches.ts no longer describes
council.run as a stub milestone.
No runtime behavior change. Test counts: core 763 passed (+2 for new
snapshots), mcp-server 174 passed unchanged.
Closes the API/barrel drift item from the active hardening sprint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
migrations.ts claimed full idempotence against dirty dev databases, but V13 + V14 ran bare `ALTER TABLE ADD COLUMN` statements that would fail with "duplicate column name" when a dev DB had the column applied out-of-band (no recorded schema_version row). V13 (two ALTERs) and V14 (two ALTERs + a CREATE TABLE/INDEX block) now apply each ALTER through a hasColumn guard, with the CREATE statements inline. SCHEMA_SQL_V13 + SCHEMA_SQL_V14 remain exported from schema.ts as the canonical schema text and are still exercised by the snapshot tests. V15 is CREATE-only and stays as a plain db.exec. The module header now states both idempotence mechanisms explicitly (CREATE TABLE/INDEX IF NOT EXISTS for object creation, hasColumn for column-addition) so the claim matches the implementation. Closes the migration idempotence item from the active hardening sprint. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add docs/consistency.md as the single authoritative reference for how flywheel-ideas keeps the SQLite ledger and Markdown notes in sync. Audits every state-changing MCP operation (idea/assumption/outcome/ council/company/import) against one of four mechanisms — strict compensating write, transactional DB + best-effort fs mirror, DB-only, or strict-staging — and notes which doctor issue kinds surface each class of drift. The doc-mode section is normative even though no code implements doc mode yet: it locks the file-format contract (frontmatter + Claim / Test condition / Assumptions / Evidence log / Verdict / Lesson) and the intended doctor invariants (round-trip, state/verdict, monotonic transitions, malformed-section) so the planned single-document wedge ships against an already-agreed-upon policy instead of forcing a rewrite of this doc. Also documents the decision to ship doc mode as a parallel codepath rather than a RepositoryBackend abstraction, with reasoning. Closes the explicit consistency policy item from the active hardening sprint (P2 in roadmap-active). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
outcome.test.ts asserted that ideas_outcome_verdicts rows came back in ORDER BY assumption_id order by sorting the expected array with String.prototype.localeCompare. SQLite's default ORDER BY uses byte (BINARY) collation, while localeCompare is locale- and ICU-version- dependent. On Node 24's ICU build the two diverge for mixed-case nanoid ids (lowercase letters sorted before uppercase under locale-aware compare, but after under SQLite's byte sort), so the test failed intermittently on CI Node 24 ubuntu and passed everywhere else. Replace localeCompare with a plain < / > comparison so the expected order tracks SQLite's collation deterministically across Node versions and runners. Pre-existing flake on main (latest main run failed for the same reason); fixing here so this hardening PR's CI is green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Normative contract for the upcoming doc-mode wedge: one portable .md per idea, no SQLite. File location, frontmatter keys, stable section order (Claim / Test condition / Assumptions / Evidence log / Verdict / Lesson), assumption sub-block shape, evidence-log monotonicity, the verdict block, and the byte-identical round-trip rule that gates every doc-mode handler via packages/core/test/single-doc-format.test.ts. Includes a canonical worked example (NVDA inventory-risk refutation) so the format is concrete from first read. The spec calls out the backend boundary explicitly: doc mode covers the embeddable single-idea lifecycle (create / read / list / transition / assumption.declare / outcome.log verdict). Council, outcome propagation, lineage, SEC tracking — anything that needs cross-idea state — remains SQLite-only and returns not_supported_in_doc_mode. Cross-links to docs/consistency.md for the per-action matrix and reasoning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands the dispatch surface for the upcoming portable single-file
backend without touching any sqlite-mode behavior. Every tool now
accepts an optional `backend: 'sqlite' | 'doc'` field; the default
remains 'sqlite' so existing callers and tests are unchanged.
Per-tool backend matrix (per docs/single-doc-format.md +
docs/consistency.md):
idea create/read/list/transition supported in doc mode;
forget/freeze/list_freezes/ancestry/descendants/
shared_assumptions/export/report reject as
not_supported_in_doc_mode
assumption declare supported; list/lock/unlock/signposts_due/
forget/radar/extension_set/extension_get reject
outcome log supported (verdict-only, no propagation);
undo/list/read/memo_upsert reject
council all actions reject
company all actions reject
Supported doc-mode actions currently return a stable
`doc_mode_in_flight` placeholder with a next_steps hint back to
`backend: "sqlite"`. The B2 commit on this branch replaces the
placeholders with real read/write handlers against the file format
contract in docs/single-doc-format.md.
Unsupported actions return `not_supported_in_doc_mode: <tool>.<action>`
with a next_steps hint to switch to sqlite. The error code is stable
and documented so MCP clients can match on it.
Adds packages/mcp-server/src/next_steps.ts::mcpNotSupportedInDocMode
helper to keep the unsupported-action response shape uniform across
all five tools.
Test coverage: new packages/mcp-server/test/doc-mode-boundary.test.ts
exercises every (tool, action) pair across the boundary — 37 cases.
Confirms placeholders for supported actions, not_supported errors for
unsupported actions, and that the sqlite default path is unchanged.
Full mcp-server suite: 211 passed (was 174).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ce tests Replaces the B1 placeholders with real read/write handlers for the portable single-file backend. Core (packages/core/src/doc-mode/format.ts): - Pure parser + serializer for the doc-mode .md format defined in docs/single-doc-format.md. No DB, no fs, no MCP coupling — these are the primitives every other doc-mode caller orchestrates around. - DocModeIdea struct + DocAssumption / DocEvidenceEntry / DocVerdict. - buildDocIdeaPath / slugifyDocTitle / DOC_MODE_FOLDER for filesystem layout helpers. - DocFormatError carries a `kind` discriminator matching the doctor invariants planned for B3: doc_malformed_section, doc_transition_out_of_order, etc. Format conformance (packages/core/test/single-doc-format.test.ts): - Golden fixtures under test/fixtures/single-doc/ (minimal + full-lifecycle) parse-then-re-render byte-identical to source. - fast-check property over 100 generated DocModeIdea structs asserts the same invariant for arbitrary well-formed inputs. - Negative cases for every DocFormatError kind: missing closing fence, wrong type/backend, invalid state, out-of-order evidence log, malformed assumption block, mismatched H1, missing required section. MCP handlers (packages/mcp-server/src/tools/idea/doc-mode.ts): - docCreate / docRead / docList / docTransition for the `idea` tool. - docDeclareAssumption appends an ### Assumption block to the idea file's ## Assumptions section. - docLogVerdict sets the ## Verdict block + moves frontmatter state to validated/refuted/parked. No cross-idea propagation — doc mode is the embeddable single-file lifecycle, per docs/consistency.md. - locateDocIdea is an O(n) scan of ideas-doc/; an index would be a different product (probably SQLite-backed) so the linear scan is honest about doc mode's scope. Wiring: - idea/register.ts dispatches the four supported actions to the doc handlers in B1's gate; the B1 placeholder is gone. - assumption.ts dispatches `declare` to docDeclareAssumption. - outcome.ts dispatches `log` to docLogVerdict, inferring the verdict state from refutes/validates lists (or accepting an explicit verdict). Doc mode does not store assumption-id references in outcomes — the verdict block stands alone. Test coverage (packages/mcp-server/test/): - doc-mode-boundary.test.ts: 41 cases. Supported actions now route to real handlers (asserts success or validation error); unsupported actions still return not_supported_in_doc_mode. - doc-mode-e2e.test.ts: 5 cases covering the full lifecycle through the MCP surface — create → declare → transition → log → read, list ordering, invalid-transition rejection, missing-verdict rejection. Every mutation asserts the on-disk file remains parse-clean. API surface snapshot refreshed (packages/core/test/__snapshots__/) to include the new doc-mode exports. SCHEMA_VERSION invariant still holds at v15. Test counts: core 775 passed (was 763 before doc-mode tests); mcp-server 216 passed (was 174 before doc-mode tests). Closes B2 + B4 from the wedge plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the read-only doctor to inspect doc-mode files in addition to
the existing SQLite/markdown mirror checks.
Core (packages/core/src/consistency-doctor.ts):
- Add 4 new ConsistencyIssueKind values:
- doc_round_trip_failed (warning) — file is not byte-identical to
renderer output; re-saving normalizes
- doc_state_verdict_mismatch (error) — frontmatter state and the
## Verdict block disagree, or required verdict is missing
- doc_transition_out_of_order (error) — created_at > updated_at, or
latest evidence-log timestamp > updated_at, or evidence-log
bullets non-monotonic (caught at parse)
- doc_malformed_section (error) — file fails parse (missing fence,
missing required section, malformed assumption block, etc.)
- buildConsistencyDoctorReport now accepts ConsistencyDoctorOptions
with mode: 'sqlite' | 'doc' | 'both' (default 'sqlite' for v0.4
backward compat).
- New inspectDocMode walks <vault>/ideas-doc/ and runs round-trip,
state/verdict, monotonic-transition, and parse-shape checks per
file. Files that fail to parse surface the DocFormatError.kind
directly when meaningful.
- Reuses parseDocIdea/renderDocIdea from doc-mode/format.ts — no
reimplementation of format semantics here.
MCP (packages/mcp-server/src/tools/doctor.ts):
- doctor.consistency tool gains optional `mode` arg, forwarded
through buildConsistencyDoctorReport.
- Result includes the resolved mode for client clarity.
Test coverage (packages/core/test/consistency-doctor-doc-mode.test.ts):
- 11 cases: clean vault (empty + canonical), mode default = sqlite
(backward compat), every issue kind hand-fixtured, mode: 'both'
aggregates counts from both inspectors.
- Uses the format-spec golden fixture (minimal.md) as the
round-trip-safe baseline so this test exercises the same bytes the
format conformance test already proves.
Test counts: core 786 (was 775); mcp-server 216 unchanged.
Closes B3 from the wedge plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves the seven `protobufjs <=7.5.5` advisories that block the Security Audit CI gate: - GHSA-q6x5-8v7m-xcrf — overlong UTF-8 decoding - GHSA-2pr8-phx7-x9h3 — DoS from crafted field names - GHSA-66ff-xgx4-vchm — code injection via bytes-field defaults - GHSA-fx83-v9x8-x52w — prototype injection in message constructors - GHSA-75px-5xx7-5xc7 — code-gen gadget after prototype pollution - GHSA-jvwf-75h9-cwgg — process-wide DoS via unsafe option paths - GHSA-685m-2w69-288q — DoS via unbounded protobuf recursion protobufjs is four levels deep: flywheel-ideas (mcp-server) → @velvetmonkey/flywheel-memory → @huggingface/transformers → onnxruntime-web → protobufjs `npm audit fix` resolved cleanly via package-lock.json only — no package.json or breaking-API changes. Full suite stays green: core 786 passed, mcp-server 216 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-way export from the SQLite ledger to a portable single-file doc-mode `.md`. Carries id/title/state/timestamps, claim body, assumptions (decision-clause preferred), and a verdict block derived from the latest non-undone outcome. Council sessions, lineage, cross-idea citations, propagation flags, and private memo context do not carry over by design. Defaults to `ideas-doc/<slug>-<id>.md` with an `overwrite` guard so re-exports cannot silently clobber an edited doc file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Windows CI failed 10 doc-mode tests because Git checkout converted .md
fixtures to CRLF and the parser's `startsWith('---\n')` fence check
rejected them immediately. Two-layer fix:
- parseDocIdea normalizes CRLF→LF at entry so doc-mode (a portable
single-file format) accepts files authored on Windows. Renderer
continues to emit canonical LF, so CRLF round-trips to LF by design.
- .gitattributes pins *.md to eol=lf so the byte-identical golden
fixture round-trip stays platform-independent.
Adds CRLF round-trip regression coverage for both golden fixtures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a single-file portable doc-mode backend for ideas, alongside the existing SQLite ledger.
docs/single-doc-format.md) — one idea per.mdwith structured sections (claim, assumptions, evidence log, verdict, lesson) and frontmatter (id, title, state, timestamps).backend: \"sqlite\" | \"doc\"flag onidea.*actions with a gate that rejects mode-incompatible calls.doctor.consistencysupportsmode: \"sqlite\" | \"doc\" | \"both\"and validates doc files against format invariants.idea.export_doc— one-way SQLite → doc-mode migration so existing ideas can move into the portable format.Wedge scope (export-only, by design)
This branch ships export but deliberately does not ship
import_doc. You can move an idea from SQLite to a doc file; you cannot move it back. If someone edits an exported.mdand expects it to sync into the SQLite ledger, that will not work — there is no read-back path.import_docis its own follow-on ticket. The wedge is intentionally export-only so the format and conformance tests can be reviewed and adopted without committing to a bidirectional sync surface (which has its own conflict-resolution / lineage-merge / outcome-reconciliation questions).Field-by-field exclusions in
idea.export_docWhat the SQLite ledger tracks but doc-mode does not carry over:
.mdwould either denormalize the graph or require a sidecar index. Worth its own design pass.pass/fail/parked) + rationale + lesson. Historical outcomes are not retained in doc-mode.Test plan
packages/mcp-serversuite (locally: 11 files / 221 passed / 1 skipped / 26.31s with--pool=forks)doc-mode-export.test.ts) covers: assumption decision-clause preference, verdict derivation from outcome verdicts,overwriteguard rejects on existing file, default path underideas-doc/<slug>-<id>.mddoctor.consistency({ mode: \"doc\" })validates an exported file without errorsstatus: refutedlands in the doc; export one with no outcomes, confirmverdict: nulldoctor.consistency({ mode: \"sqlite\" })pathFollow-ups (separate tickets)
idea.import_doc— read a doc-mode.mdback into the SQLite ledger (round-trip viability)🤖 Generated with Claude Code