Skip to content

Emit 0.2 build manifests with supplementary file support, plan-driven archive assembly, and filesystem-identity validation - #456

Merged
eXamadeus merged 1 commit into
mainfrom
julian/07-23-emit_0.2_build_manifests_with_supplementary_file_support_plan-driven_archive_assembly_and_filesystem-identity_validation
Jul 24, 2026
Merged

Emit 0.2 build manifests with supplementary file support, plan-driven archive assembly, and filesystem-identity validation#456
eXamadeus merged 1 commit into
mainfrom
julian/07-23-emit_0.2_build_manifests_with_supplementary_file_support_plan-driven_archive_assembly_and_filesystem-identity_validation

Conversation

@eXamadeus

@eXamadeus eXamadeus commented Jul 23, 2026

Copy link
Copy Markdown
Member

Why

The producer was emitting 0.1 build manifests with a partial assets map covering only primary assets. Supporting non-asset supplementary files (skill companions and archive-only files declared in files) requires a complete entry map and a format bump to 0.2.

Details

Build manifest format (0.2): Producers now always emit facetVersion: 0.2 with a files map covering every inner-archive entry — facet.json, all primary assets, and all supplementary files. The assets key is gone from producer output entirely; it remains a legacy consumer-input shape only. There is no runtime flag or dual-format mode.

Supplementary source loading (load-supplementary-sources.ts): A new filesystem-identity validation layer sits between the protocol's pure path-grammar checks and archive assembly. Before any output is touched, every declared skill-companion and archive-only entry is validated against a strict set of disk-identity rules: the file must exist, be a regular file, have no symlinked parent components, not be a hard link (nlink === 1), resolve within the facet root via realpath, and not share an inode with another declared entry. All failures are collected in a single pass and surfaced as structured ValidationError values. If any fail, the pipeline returns early and prior dist/ output is preserved untouched.

Archive plan as single source of truth: collectArchiveEntriesFromPlan replaces collectArchiveEntries, driving both archive assembly and the files hash map from the shared planArchiveEntries result. This ensures the producer and verifier derive membership and ordering from the same derivation, making disagreement structurally impossible. The plan is pre-sorted lexicographically, so tar output is deterministic across machines.

BuildResult.fileHashes replaces assetHashes: The result type now exposes fileHashes (the complete per-entry map) and facetVersion. CLI output, TUI build view, and --json schema version ("2") are updated accordingly — the --json schema bumps from "1" to "2" because the assets array is replaced by files, and consumers pinned to schema 1 must migrate.

TOCTOU acknowledgment: The lstat/realpath/readFile sequence has an irreducible race window. The threat model is malformed or accidental inputs over a trusted source tree, not an active attacker. The final readFile is performed against the validated resolved path so any swap produces a visible hash mismatch rather than a silently unvalidated entry.

Verification

Tasks 10.1–10.4 and 11.1–11.7 are marked complete. A new candidate-archive.e2e.test.ts builds a representative facet (text companion, binary companion, archive-only README) with the compiled CLI and validates the resulting .facet through validateFacetArchive — the same verifier a registry stage uses — confirming exact 0.2 membership and byte-identical output across two independent builds. The load-supplementary-sources.test.ts unit tests cover the full filesystem-identity failure matrix (missing, directory, symlink target, symlinked parent, hard link, out-of-tree escape, resolved-source alias) plus happy-path byte fidelity. Pipeline-level tests in build-pipeline.test.ts cover the complete supplementary success and failure cases, determinism, scoped output paths, and validation-before-cleanup behavior.


Note

Medium Risk
Changes the authoritative build output format and breaks --json schema 1 consumers; validation-before-write reduces risk of corrupt dist output.

Overview
The build producer now always emits facetVersion 0.2 build manifests with a full inner-archive files hash map (manifest, primaries, and declared supplementary paths). The legacy assets map is no longer written; BuildResult exposes fileHashes and facetVersion instead of assetHashes.

A new Loading files stage loads skill companions and archive-only sources via loadSupplementarySources, with filesystem checks (regular files only, no symlink/hard-link escapes, distinct inodes) before dist/ is touched. Archive membership and ordering come from planArchiveEntries through collectArchiveEntriesFromPlan, replacing collectArchiveEntries.

CLI/TUI output and --json schema 2 now report facetVersion, complete files, and entry counts. Install test mocks read manifest.files for registry stubs. Tests add supplementary pipeline coverage, loader failure matrix, and candidate-archive.e2e for registry-grade 0.2 interop and deterministic bytes.

Reviewed by Cursor Bugbot for commit 64d85a2. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • New Features

    • Added support for packaging supplementary files, including binary and archive-only entries.
    • Build archives now use format version 0.2 and report complete file lists and integrity hashes.
    • Added a “Loading files” build stage with validation for missing, unsafe, aliased, or invalid supplementary files.
    • Added deterministic archive generation and structured JSON build results.
  • Bug Fixes

    • Improved protection against symlink, hard-link, path traversal, and undeclared-file packaging issues.
  • Tests

    • Expanded end-to-end coverage for archive contents, validation, determinism, and build output.

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d83f8b8b-5e2c-4f8d-8ef3-037e2430487f

📥 Commits

Reviewing files that changed from the base of the PR and between e62f6c5 and 64d85a2.

📒 Files selected for processing (14)
  • openspec/changes/support-non-asset-files/tasks.md
  • packages/cli/src/__tests__/candidate-archive.e2e.test.ts
  • packages/cli/src/__tests__/create-build.e2e.test.ts
  • packages/cli/src/commands/build.ts
  • packages/cli/src/tui/views/build/build-view.tsx
  • packages/engine/src/__tests__/build-pipeline.test.ts
  • packages/engine/src/build/__tests__/load-supplementary-sources.test.ts
  • packages/engine/src/build/load-supplementary-sources.ts
  • packages/engine/src/build/pipeline.ts
  • packages/engine/src/install/__tests__/run-add.test.ts
  • packages/engine/src/install/__tests__/run-install.chain.test.ts
  • packages/engine/src/install/__tests__/run-install.receipt.test.ts
  • packages/engine/src/install/__tests__/run-install.test.ts
  • packages/engine/src/install/__tests__/run-remove.test.ts

📝 Walkthrough

Walkthrough

The build pipeline now supports supplementary archive files, validates their filesystem identity, preserves their bytes, and includes them in facet 0.2 archives and fileHashes. CLI JSON, text, and TUI outputs expose the new version and complete entry lists. Installation test fixtures use CurrentBuildManifest and manifest.files. New tests verify archive classification, deterministic output, supplementary-file rejection cases, and CLI archive validation.

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant runBuildPipeline
  participant loadSupplementarySources
  participant FacetArchive
  CLI->>runBuildPipeline: build facet source
  runBuildPipeline->>loadSupplementarySources: load and validate supplementary files
  loadSupplementarySources-->>runBuildPipeline: validated file bytes
  runBuildPipeline->>FacetArchive: create facet 0.2 archive
  FacetArchive-->>runBuildPipeline: archive and fileHashes
  runBuildPipeline-->>CLI: build result
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 72.22% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly describes the main change: emitting 0.2 manifests with supplementary file support and plan-driven validation.
Description check ✅ Passed The description follows the template with Why, Details, and Verification and covers the main implementation and test coverage.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch julian/07-23-emit_0.2_build_manifests_with_supplementary_file_support_plan-driven_archive_assembly_and_filesystem-identity_validation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

eXamadeus commented Jul 23, 2026

Copy link
Copy Markdown
Member Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e62f6c56e9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// parent symlink that stays inside the root; walking with `lstat` rejects
// it. Only existing components are checked — a missing parent surfaces as
// a `missing` target below.
const parentSymlink = await firstSymlinkedParent(rootReal, dirname(sourcePath))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Detect parent symlinks under aliased roots

When rootDir has a different lexical spelling from its real path—such as an explicit symlink passed to facet build, or macOS /var resolving under /private/vardirname(sourcePath) does not start with rootReal, so firstSymlinkedParent immediately returns null. A declared path such as linkdir/note.md, where linkdir is a symlink to another directory inside the facet root, is then accepted because lstat follows the parent link and the later containment check still passes. This bypasses the new no-symlinked-parent validation; walk from the lexical root or construct the checked path from rootReal before calling the helper.

Useful? React with 👍 / 👎.

Comment thread packages/engine/src/build/load-supplementary-sources.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
packages/cli/src/__tests__/candidate-archive.e2e.test.ts (1)

144-146: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use descriptive archive-byte names.

Rename a and b to identify the two deterministic-build artifacts. As per coding guidelines, “Use meaningful variable names.”

Proposed refactor
-const a = await build('det-a')
-const b = await build('det-b')
-expect(Array.from(a)).toEqual(Array.from(b))
+const firstArchiveBytes = await build('det-a')
+const secondArchiveBytes = await build('det-b')
+expect(Array.from(firstArchiveBytes)).toEqual(Array.from(secondArchiveBytes))

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2afe0a86-7359-49da-8be0-29e6d36ba480

📥 Commits

Reviewing files that changed from the base of the PR and between 30f2eb4 and e62f6c5.

📒 Files selected for processing (14)
  • openspec/changes/support-non-asset-files/tasks.md
  • packages/cli/src/__tests__/candidate-archive.e2e.test.ts
  • packages/cli/src/__tests__/create-build.e2e.test.ts
  • packages/cli/src/commands/build.ts
  • packages/cli/src/tui/views/build/build-view.tsx
  • packages/engine/src/__tests__/build-pipeline.test.ts
  • packages/engine/src/build/__tests__/load-supplementary-sources.test.ts
  • packages/engine/src/build/load-supplementary-sources.ts
  • packages/engine/src/build/pipeline.ts
  • packages/engine/src/install/__tests__/run-add.test.ts
  • packages/engine/src/install/__tests__/run-install.chain.test.ts
  • packages/engine/src/install/__tests__/run-install.receipt.test.ts
  • packages/engine/src/install/__tests__/run-install.test.ts
  • packages/engine/src/install/__tests__/run-remove.test.ts

Comment on lines +356 to +358
// Complete inner-archive entry listing (includes facet.json + primaries).
expect(Array.isArray(doc.files)).toBe(true)
expect(doc.files).toContain('facet.json')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert at least one primary archive entry.

A regressed files: ['facet.json'] payload still passes, despite this fixture containing the helper skill. Assert its primary path too.

Proposed test update
 expect(Array.isArray(doc.files)).toBe(true)
-expect(doc.files).toContain('facet.json')
+expect(doc.files).toEqual(
+  expect.arrayContaining(['facet.json', 'skills/helper/SKILL.md']),
+)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Complete inner-archive entry listing (includes facet.json + primaries).
expect(Array.isArray(doc.files)).toBe(true)
expect(doc.files).toContain('facet.json')
// Complete inner-archive entry listing (includes facet.json + primaries).
expect(Array.isArray(doc.files)).toBe(true)
expect(doc.files).toEqual(
expect.arrayContaining(['facet.json', 'skills/helper/SKILL.md']),
)

Comment thread packages/engine/src/build/load-supplementary-sources.ts
@greptile-apps

greptile-apps Bot commented Jul 23, 2026

Copy link
Copy Markdown

Greptile Summary

This PR bumps the facet build-manifest format to 0.2, replacing the primary-only assets hash map with a complete files map that covers every inner-archive entry (manifest, primaries, and supplementary files). A new loadSupplementarySources stage validates every declared supplementary file's disk identity before any output is touched, and the shared planArchiveEntries derivation is now the single source of truth for archive membership and ordering.

  • load-supplementary-sources.ts adds a thorough filesystem-identity validation layer (regular-file, no symlinks/hard links, root containment, inode-alias detection) with single-pass failure collection and rich structured errors.
  • BuildResult.fileHashes replaces assetHashes, --json schema bumps to \"2\", and CLI/TUI output is updated consistently.
  • resolve-git.ts (not in the diff) was not updated: it still reads buildManifest.assets from the output of runBuildPipeline, which is now always undefined in 0.2 manifests, breaking git-sourced facet installs at the cache-put step.

Confidence Score: 4/5

Safe to merge with one fix: resolve-git.ts must be updated before git-sourced facets can be installed.

The pipeline, validation, and archive-assembly changes are well-structured and thoroughly tested. However, resolve-git.ts reads buildManifest.assets from the output of runBuildPipeline — a field that no longer exists in 0.2 manifests — so every git-sourced facet install will throw a TypeError inside cachePutVerified on a cache miss. The stale comment on that line explicitly calls out the now-wrong assumption. Fixing it requires a one-line change (.assets to .files with a matching type), but until it lands the git install path is broken.

packages/engine/src/install/commit/resolve-git.ts — not in the diff but broken by the format change; must be patched before git installs work.

Reviews (3): Last reviewed commit: "Emit `0.2` build manifests with suppleme..." | Re-trigger Greptile

Comment on lines +140 to +144
try {
stats = await lstat(sourcePath)
} catch {
failures.push({ code: 'missing', archivePath: entry.path, declarationSite, sourcePath })
continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unreadable Files Report As Missing

When lstat fails with EACCES, ELOOP, or another filesystem error, this catch reports the declaration as missing. A present but inaccessible supplementary file therefore produces the wrong structured failure and tells the author to create a file that already exists; distinguish ENOENT from other errors and use the existing unreadable case for the latter.

Context Used: AGENTS.md (source)

@changeset-bot

changeset-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 64d85a2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@changeset-bot

changeset-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: fc27156

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc27156b1e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

archive: INNER_ARCHIVE_NAME,
integrity,
assets: assetHashes,
files: fileHashes,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Pass the 0.2 file map to the git cache audit

A fresh install from a git source now builds a 0.2 manifest here, but resolve-git.ts still parses that JSON as the legacy BuildManifest and passes buildManifest.assets to cachePutVerified. Since this manifest contains only files, that value is undefined; cachePutVerified immediately calls Object.entries(archive.fileHashes), so every uncached git install crashes before caching or materialization. Use buildResult.fileHashes (or versioned manifest parsing) for that cache audit.

Useful? React with 👍 / 👎.


const outer = parseTar(result.archiveBytes)
const inner = outer.find((e) => e.name === 'archive.tar.gz')
if (!inner?.data) throw new Error('inner archive missing')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Replace the thrown test narrowing guard

This newly added test narrows an optional archive entry with if (...) throw; repository test guidance requires expect.unreachable() for these guards so the assertion intent is explicit and follows the project-wide convention.

AGENTS.md reference: AGENTS.md:L320-L325

Useful? React with 👍 / 👎.

Comment thread packages/engine/src/build/load-supplementary-sources.ts

eXamadeus commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

Merge activity

  • Jul 24, 4:47 AM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Jul 24, 4:53 AM UTC: Graphite rebased this pull request as part of a merge.
  • Jul 24, 4:53 AM UTC: @eXamadeus merged this pull request with Graphite.

@eXamadeus
eXamadeus changed the base branch from julian/07-23-plumb_companion_bytes_owned-path_sets_and_pre-materialization_reconciliation_through_install_receipt_and_cli_rendering to graphite-base/456 July 24, 2026 04:50
@eXamadeus
eXamadeus changed the base branch from graphite-base/456 to main July 24, 2026 04:51
…en archive assembly, and filesystem-identity validation
@eXamadeus
eXamadeus force-pushed the julian/07-23-emit_0.2_build_manifests_with_supplementary_file_support_plan-driven_archive_assembly_and_filesystem-identity_validation branch from fc27156 to 64d85a2 Compare July 24, 2026 04:52
@eXamadeus
eXamadeus merged commit 6e7d938 into main Jul 24, 2026
6 of 7 checks passed
@eXamadeus
eXamadeus deleted the julian/07-23-emit_0.2_build_manifests_with_supplementary_file_support_plan-driven_archive_assembly_and_filesystem-identity_validation branch July 24, 2026 04:53

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 64d85a2. Configure here.

sourcePath: join(rootDir, entry.path),
})
}
return { ok: false, failures }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Root realpath fails silently

Low Severity

If realpath(rootDir) throws and the archive plan has no supplementary entries, loadSupplementarySources returns { ok: false, failures: [] }. The pipeline maps that to a validation failure with zero errors, so asset-only builds can fail at “Loading files” with no actionable message.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 64d85a2. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant