[WIP] feat: mmap-based package summary cache format (internal/meta) by luoliwoshang · Pull Request #1989 · xgo-dev/llgo

luoliwoshang · 2026-06-26T15:09:44Z

Summary

This PR introduces internal/meta, a new binary format and read/write layer for LLGo's whole-program DCE package summary cache (.meta files). It replaces the MVP's uvarint-encoded format with a zero-copy mmap design modeled after Go's own cmd/internal/goobj format.

Zero-copy reads via mmap: PackageMeta is a thin view over a mmap'd byte slice; Edges, TypeChildren, MethodSlots, and IfaceMethods use unsafe.Slice to reinterpret on-disk bytes directly — no deserialization, no allocation.
CSR layout: each per-symbol section uses an offsets[nsyms+1] prefix array and a flat data array (same pattern as Go's BlkRelocIdx + BlkReloc), giving O(1) per-symbol access.
Single allocation on write: Builder.Build() pre-computes all section sizes/offsets in one pass, allocates one []byte, and writes each section directly at its target offset — no intermediate buffers.
Lazy remap in GlobalSummary: merging N packages only interns string tables (O(unique_symbols)); edges, type-children, method slots and interface methods are all translated lazily on query. DCE typically reaches ~500 of ~10000+ types, avoiding thousands of unnecessary translations.
Compile-time layout assertions: const expressions pin sizeof(Edge)==12, sizeof(MethodSlot)==20, sizeof(MethodSig)==12 so any struct drift that breaks zero-copy reinterpretation fails at build time.
Zero-copy string reads: SymbolName() and NameString() use unsafe.String to return strings pointing directly into the mmap region — no allocation on the hot path.

Wire format sections

stringTable | Symbols | Edges (CSR) | TypeChildren (CSR) |
MethodInfo (CSR) | InterfaceInfo (CSR) | ReflectBitmap

Performance

	Old MVP	New mmap (warm)	Improvement
cookiejar (210 pkgs)
meta read total	59.2ms	— (mmap, amortized into merge)
summary merge	42.6ms	~6ms	7.1x
deadcode analyze	6.5ms	~6ms	—
total	108.3ms	~12ms	9.0x
etcd demo (334 pkgs)
meta read total	216.8ms	— (mmap, amortized into merge)
summary merge	114.7ms	~18ms	6.4x
deadcode analyze	33.7ms	~30ms	—
total	365.2ms	~48ms	7.6x

Status

Builder — accumulates facts, serializes to wire format
PackageMeta — mmap view, zero-copy CSR queries
GlobalSummary — lazy remap merge, global Symbol/Name spaces
Wire up DCE (internal/deadcode) to new GlobalSummary
Integrate into build pipeline (internal/build)

Test plan

TestRoundTrip — in-memory Builder → PackageMeta round-trip
TestRoundTripFile — write to disk → ReadMeta (mmap) round-trip
TestTypeChildrenAlignment — padding + zero-copy correctness across 5 string table lengths
TestWireLayout — struct size and field offset assertions
TestGlobalSummaryMerge — cross-package lazy edge translation, method name unification
TestGlobalSummaryLinkonce — first-wins dedup for linkonce type descriptors

🤖 Generated with Claude Code

# Conflicts: # internal/build/build.go # ssa/abitype.go

… format Introduces internal/meta, a new binary format and read/write layer for LLGo's whole-program DCE package summary cache (.meta files). Key design decisions: - Zero-copy reads via mmap: PackageMeta is a thin view over a mmap'd byte slice; Edges, TypeChildren, MethodSlots and IfaceMethods all use unsafe.Slice to reinterpret the on-disk bytes directly, with no alloc. - CSR layout (Compressed Sparse Row): each per-symbol section stores an offsets[nsyms+1] prefix array and a flat data array, matching the design used by Go's own goobj format (BlkRelocIdx + BlkReloc). - Single allocation on write: Builder.Build() pre-computes all section sizes and offsets in one pass, allocates one []byte, then writes every section directly at its target offset — no intermediate buffers, no seek-back. - Lazy remap in GlobalSummary: merging N PackageMetas only interns the string tables (O(unique_symbols)); edges and type-children are translated lazily on query, avoiding the O(total_edges) rewrite cost of the MVP. - NameRef for method names: method short names are stored as {Off, Len} pairs referencing the string table, keeping them distinct from module-level symbols (LocalSymbol) while still enabling cross-package matching via global Name interning in GlobalSummary. - Compile-time layout assertions: const expressions verify sizeof(Edge)==12, sizeof(MethodSlot)==20, sizeof(MethodSig)==12 so any struct drift that would break the zero-copy reinterpretation fails at build time. Sections in the .meta wire format: stringTable | Symbols | Edges (CSR) | TypeChildren (CSR) | MethodInfo (CSR) | InterfaceInfo (CSR) | ReflectBitmap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace the MVP's uvarint-based metadata format with the new mmap-based zero-copy format (internal/meta). All read paths use unsafe reinterpretation of on-disk bytes; write paths use single-allocation serialization. Key changes: - ssa/cl: switch metaBuilder from *metadata.Builder to *meta.Builder; AddIfaceMethod/AddTypeChild now deduplicate internally - internal/build: wire cache miss/hit paths to meta.ReadMeta/Build/WriteMeta - internal/deadcode: analyze.go uses meta.GlobalSummary with GMethodSlot (flat struct, no nested .Sig) - internal/meta: add zero-copy SymbolName/NameString via unsafe.String, add typeChildrenSet map for O(1) dedup, add FormatMeta for golden tests - cl/_testmeta: update 9 golden files to new deterministic sort order Perf (etcd client demo, 334 packages): summary merge: 114.7ms → ~23ms (5x faster via lazy remap + zero-copy strings) DCE analyze: ~34ms unchanged total: ~148ms → ~52ms (2.8x faster) Co-Authored-By: Claude <noreply@anthropic.com>

Remove Phase 2 eager translation — instead of scanning every type in all 334 packages and translating all ~10080 method slots up front, only mark concrete/interface type kinds in Phase 1 (cheap CSR range checks, no slot translation). MethodSlots() and InterfaceMethods() now translate lazily on first access and cache the result. DCE typically reaches only ~500 of ~10000+ types, so lazy translation avoids thousands of unnecessary pm.NameString + internName calls. Perf (etcd client demo, 334 packages): merge: 115ms → ~18ms (6.4x faster) total: ~149ms → ~48ms (3.1x faster) Co-Authored-By: Claude <noreply@anthropic.com>

PackageMeta query methods are now all lowercase (private). External code interacts only through GlobalSummary, Builder, ReadMeta, Bytes, and Close. - Drop IsConcreteType, IsInterface, IsCompositeType — callers use nmethodSlot/ntypeChild/nifaceMethod count methods instead - Add N-prefix count methods (Go goobj convention): nedge, ntypeChild, nmethodSlot, nifaceMethod — single CSR range check, no unsafe pointer - All CSR accessors share csrSlice[T] generic helper, eliminating repeated CSR+unsafe pointer boilerplate - Remove NSyms() method — same-package callers use pm.nsyms field - Remove NewPackageMetaFromBytes — no external callers - Make buildMethodImplKeys fully lazy: concrete type methodImplKeys are computed on-demand in flood's usedInIface path; only methodRefs reverse index is built eagerly - MetaString → String() (fmt.Stringer) — zero additional public API - Drop ConcreteTypes() from GlobalSummary; isConcrete tracking removed from Phase 1 Co-Authored-By: Claude <noreply@anthropic.com>

# Conflicts: # chore/gentests/gentests.go # cl/compile_test.go # internal/build/collect_test.go # ssa/expr.go # ssa/interface.go

luoliwoshang · 2026-06-30T03:19:13Z

TestBCD 这个发现在Iface调用点再emit interfaceInfo会导致定义包没有emit interface，使用包要interface，事实错乱，应该修改为定义包emit这个事实

codecov · 2026-06-30T05:07:19Z

Codecov Report

❌ Patch coverage is 74.09372% with 293 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/meta/format.go	0.00%	118 Missing ⚠️
internal/dcepass/dcepass.go	59.71%	57 Missing and 28 partials ⚠️
internal/meta/global.go	75.00%	33 Missing and 10 partials ⚠️
internal/meta/meta.go	79.61%	11 Missing and 10 partials ⚠️
internal/meta/builder.go	84.70%	10 Missing and 3 partials ⚠️
internal/deadcode/analyze.go	94.87%	4 Missing and 4 partials ⚠️
ssa/abitype.go	96.15%	2 Missing and 1 partial ⚠️
ssa/interface.go	90.47%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

luoliwoshang and others added 18 commits May 20, 2026 15:14

wip: add metadata package summary model

0676680

wip: persist package summary metadata in build cache

3b5392c

wip: emit semantic metadata summaries

b1ea4db

wip: emit type children metadata

4e788ff

wip: extract ordinary metadata edges

ba87854

wip: add metadata cache debug tooling

23cfb43

wip: add metadata-driven deadcode overrides

fde3580

fix deadcode interface method matching

8a1843f

fix deadcode propagation for interface-used type refs

bb16367

remove cache meta read timing log

ed040b9

fix dce unreachable method slots

83b0ae0

fix dce runtime init root

ae572d9

Merge branch 'main' into wip/deadcode-analyzer

7b1bcc3

# Conflicts: # internal/build/build.go # ssa/abitype.go

test: update metadata expectations for method stubs

dfb4ae2

fix: record interface use from pointer boxing

40a546b

luoliwoshang mentioned this pull request Jun 27, 2026

proposal: LLGo Whole-Program Method Reachability Analysis and Package Summary Cache #1853

Open

4 tasks

luoliwoshang force-pushed the wip/deadcode-mmap branch from a21246b to 9861150 Compare June 27, 2026 11:38

luoliwoshang and others added 4 commits June 27, 2026 22:28

style: go fmt

1542da5

style: go fmt collect_test.go

38b5154

Merge remote-tracking branch 'upstream/main' into wip/deadcode-mmap

782c8e8

# Conflicts: # chore/gentests/gentests.go # cl/compile_test.go # internal/build/collect_test.go # ssa/expr.go # ssa/interface.go

luoliwoshang added 2 commits June 30, 2026 11:53

fix(ssa): emit interface metadata from type descriptors

7fd02e9

fix(ssa): align named interface metadata

55a77fe

luoliwoshang added 2 commits June 30, 2026 13:45

fix(ssa): keep interface metadata descriptor stable

400407f

fix deadcode interface method demands

eb81ecb

luoliwoshang added 10 commits June 30, 2026 19:02

fix deadcode interface info for method demands

d2b514d

update deadcode meta golden files

019f13a

Emit interface metadata from defining packages

092af96

Add generic interface metadata coverage

77f72ae

test: add deadcode drop regression suite

6bd134a

meta: split function demands from ordinary edges

fd8d8cb

meta: emit interface info at callsites

9379516

test: fix generic interface meta golden newline

ea939c3

meta: remove legacy metadata package

6e76191

test: drop interface hash patch path experiment

8653a64

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] feat: mmap-based package summary cache format (internal/meta)#1989

[WIP] feat: mmap-based package summary cache format (internal/meta)#1989
luoliwoshang wants to merge 36 commits into
xgo-dev:mainfrom
luoliwoshang:wip/deadcode-mmap

luoliwoshang commented Jun 26, 2026 •

edited

Loading

Uh oh!

luoliwoshang commented Jun 30, 2026

Uh oh!

codecov Bot commented Jun 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

luoliwoshang commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Wire format sections

Performance

Status

Test plan

Uh oh!

luoliwoshang commented Jun 30, 2026

Uh oh!

codecov Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

luoliwoshang commented Jun 26, 2026 •

edited

Loading

codecov Bot commented Jun 30, 2026 •

edited

Loading