kaizen: Closure visited gen#547
Open
sayrer wants to merge 3 commits into
Open
Conversation
Build-cost lever timbray#2 (the map term), de-risked first per plan. The epsilon closure's two map[*faState]uint64 visited sets (closureBuffers.states and .walkVisited) grew to O(N) entries during a build; their pointer-keyed lookups cache-missed at scale and were ~31% of build CPU (mapaccess1_fast64 + mapassign). Replace them with two generation stamps stored on faState itself (walkVisitedGen, closureVisitedGen), compared against a globally monotonic atomic counter (closureGenCounter) so generations are never reused across the pooled buffers or concurrent builds — the reason the per-buffer maps existed. Result: build 15.2s -> 7.7s at 10k shellstyle patterns (~2x), tail 271 -> 135ms. Exponent unchanged (still O(N^2) at slope ~1.0 — that is the spin-back epsilon fan-out, addressed separately by Plan C); this halves the constant. No matching regression (ExactString/SingleShellstyle/CityLots flat); full suite green; race-clean. Cost: +16 bytes per faState (128 -> 144) of build-only state made permanent — the same memory-vs-speed trade nfa.go:358 once resolved the other way. Could be halved to +8 with uint32 stamps (wrap-safe for ~1e9 closures) if the memory matters. Size goldens (TestMcNfaSizes, TestQuaminaMemoryCost, TestPP) updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # epsi_closure.go
research-main.go writes regenerable CSVs into research/; they are build artifacts, not source, and should not be tracked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sayrer
force-pushed
the
closure-visited-gen
branch
from
beb2900 to
159d53f
Compare
Contributor
Author
|
The first commit is from a few weeks ago, before I had fixed the ssh key on my laptop, that's why this isn't verified. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
speeds up build time by 1.77x