Skip to content

fix(metadata): use canonical JSON library for cross-language hash compatibility#120

Merged
imlk0 merged 1 commit into
masterfrom
fix-go
May 20, 2026
Merged

fix(metadata): use canonical JSON library for cross-language hash compatibility#120
imlk0 merged 1 commit into
masterfrom
fix-go

Conversation

@imlk0
Copy link
Copy Markdown
Collaborator

@imlk0 imlk0 commented May 20, 2026

Summary

  • Root cause: Rust and Go FlatBuffers libraries produce different binary output for identical data (vtable field ordering differs), so the previous calculate_metadata_hash / CalculateMetadataHash that serialized via FlatBuffers returned different SHA-256 hashes across languages.
  • Fix: Both sides now use dedicated canonical JSON libraries (canon-json in Rust, canonicaljson-go in Go) that automatically sort object keys per RFC 8785, eliminating language-specific binary differences.
  • Cleanup: Removed unused metadata_hash.fbs, metadata_hash_generated.rs, and metadata_hash_generated.go.
  • Verification: Added cross-language hash test and CI check:
    • TestInterop_CalculateMetadataHash_CrossLanguage loads Rust-generated fixture and verifies Go's hash matches the committed root hash
    • CI now compares both the .fb binary and the root hash against the committed fixture
  • CLAUDE.md: Added documentation sync requirement and pre-PR test checks (only verity tests for verity changes)

Changed files

  • cryptpilot-verity/Cargo.toml — add canon-json = "0.2.1"
  • cryptpilot-verity/src/metadata/mod.rs — use canon_json::CanonJsonSerialize
  • cryptpilot-verity/build.rs — remove metadata_hash.fbs compilation
  • cryptpilot-verity/src/metadata/metadata_hash.fbs — deleted
  • cryptpilot-verity/src/metadata/metadata_hash_generated.rs — deleted
  • verity-go/go.mod / go.sum — add canonicaljson-go
  • verity-go/metadata/metadata_hash.go — use canonicaljson.Marshal()
  • verity-go/metadata/generated/metadata_hash_generated.go — deleted
  • verity-go/metadata/metadata_hash.fbs — deleted
  • verity-go/metadata/gen_fixture.sh — also write root hash to file
  • Makefile (gen-interop-fixture) — also write root hash to file
  • verity-go/metadata/testdata/rust.metadata.fb.hash — new committed hash file
  • verity-go/metadata/interop_test.go — new TestInterop_CalculateMetadataHash_CrossLanguage
  • .github/workflows/test.yml — verify root hash in CI
  • CLAUDE.md — add documentation sync and pre-PR test requirements

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,您的请求已接收,请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,未检测到有镜像需要构建,如需重新检测请评论 /start

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,您的请求已接收,请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,未检测到有镜像需要构建,如需重新检测请评论 /start

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,您的请求已接收,请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,未检测到有镜像需要构建,如需重新检测请评论 /start

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,您的请求已接收,请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,未检测到有镜像需要构建,如需重新检测请评论 /start

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,您的请求已接收,请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,未检测到有镜像需要构建,如需重新检测请评论 /start

…patibility

Rust and Go FlatBuffers libraries produce different binary output for the same
data (vtable field ordering), causing calculate_metadata_hash to return
different SHA-256 hashes across languages.

Replace manual JSON construction with canonical JSON libraries that
automatically sort object keys per RFC 8785:
- Rust: canon-json crate (serde_json Formatter)
- Go: github.com/gibson042/canonicaljson-go

Add root hash fixture and cross-language verification:
- gen_fixture.sh and Makefile now save root hash alongside the .fb fixture
- New test TestInterop_CalculateMetadataHash_CrossLanguage verifies Go's
  CalculateMetadataHash matches the committed Rust-generated hash
- CI now checks both .fb binary and root hash match when regenerating fixture

Remove unused FlatBuffers files:
- Delete metadata_hash.fbs and metadata_hash_generated.rs (Rust)
- Delete metadata_hash_generated.go (Go)

Update CLAUDE.md:
- Add documentation sync requirement
- Add pre-PR test checks (only verity tests for verity changes)
@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,您的请求已接收,请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

@imlk0 ,您好,未检测到有镜像需要构建,如需重新检测请评论 /start

@imlk0 imlk0 merged commit c71ddc6 into master May 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants