refactor: eliminate synchronous I/O and process spawn bottlenecks - #3
Merged
Conversation
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
Owner
Author
|
I have read the CLA Document and I hereby sign the CLA |
Owner
Author
|
I have read the CLA Document and I hereby sign the CLA |
Owner
Author
|
recheck |
Owner
Author
|
I have read the CLA Document and I hereby sign the CLA |
This was referenced May 25, 2026
vilmire
added a commit
that referenced
this pull request
Jun 1, 2026
…s file, pending guard, cooldown sweep appendRemoteLedgerEntries (#1): - dedup now uses tail:1000 instead of full O(n) readLedgerEntries; P2P replication is cursor-based so duplicates appear in the recent tail Archive rotation (#4): - compactLedger now rotates .archive.jsonl to .archive.N.jsonl (max 5) when the archive exceeds 50MB before appending new entries Worker JSON extraction validation (#5): - extractJsonObjectFromSummary now requires at least one mesh worker result field (changedFiles|errors|gitStatus|nextAction|validationResults) before accepting a JSON block; prevents false positives from tool/log JSON in the final summary Archived counts file (#6): - compactLedger writes cumulative counts to <meshId>.archived-counts.json - getLedgerSummary reads and merges archived counts so taskCompleted/Failed/ Stalled and totalEntries are accurate even after compaction Pending events size guard (#2): - queuePendingMeshCoordinatorEvent trims the pending-events.jsonl to the last 50 events when it exceeds 100KB before appending; prevents unbounded growth when coordinator stops draining autoLaunchCooldownUntil cleanup (#3): - sweepExpiredCooldowns() removes expired entries from the cooldown Map whenever a new cooldown is set; inline check at read site also cleans up the checked key; prevents long-lived daemon Map accumulation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
vilmire
added a commit
that referenced
this pull request
Jun 1, 2026
…ntext tail bound Atomic drain (#1): - drainPendingMeshCoordinatorEvents: rename-then-read with atomicDrainFile() so concurrent coordinators cannot both consume the same pending events file. renameSync is atomic on POSIX: only one caller wins; the other returns null and gets an empty result, eliminating duplicate event delivery. Ledger read cache (#3): - readLedgerEntries: 100ms TTL in-process cache via getCachedRawEntries(). Absorbs repeated reads within a single event-processing burst (agent:stopped triggers 4+ ledger reads for the same file in one synchronous cycle). - invalidateLedgerCache() called on every write: appendLedgerEntry, appendRemoteLedgerEntries, compactLedger — ensures readers always see fresh data after any mutation. - Cache is keyed by meshId (not opts), with filters applied to the cached slice. getSessionRecoveryContext tail bound (#2): - Bounded to tail:500 instead of full O(n) scan. task_dispatched is never archived (ARCHIVABLE_KINDS excludes it), so dispatch history is always present in the active file. The 30-min failure window means consecutiveNode- Failures never needs deep history. tail:500 covers all realistic use cases. Tests: 7 new tests (4 cache, 3 atomic drain), all 92 mesh tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
vilmire
added a commit
that referenced
this pull request
Jun 14, 2026
Chat slowed as message count grew because the list is non-virtualized and its per-row React.memo used reference equality (prev.message === next.message), which never held once the tail controller produced fresh message objects every status/tail tick. Every visible row then re-parsed Markdown on every update. Fix #1 (highest impact): - Replace the ChatMessageRow comparator with a field-aware one keyed on a render signature (buildChatMessageRowSignature) that folds in the fields the row actually renders (content via buildChatMessageSignature, kind, sender, meta/visibility/classification flags). Unchanged rows now short-circuit; the streaming last message still updates because its content/signature changes. - Extract the standard-bubble Markdown into a content-keyed ChatMarkdownBody memo so unchanged rows never re-run ReactMarkdown. Fix #3 (defensive): - Cap retained historyMessages at 500 (most-recent kept) so "Load older" can't grow the rendered set unbounded; historyOffset still advances by the full fetched page size so paging stays aligned. - Drop activeConv.messages.length from the controller useMemo deps (it changed every meta append, tearing down/recreating the controller + resubscribing); push the fresh fallbackRecentCount via updateOptions() instead. Fix #2 (virtualization): deferred. The intricate scroll-restore/auto-follow/ snapshot logic reads full content height; partial virtualization would corrupt scroll position. Left a detailed TODO at the map site with integration points. Adds a unit test asserting the signature comparator holds for an unchanged message and breaks on streaming content / kind / meta / visibility changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vilmire
added a commit
that referenced
this pull request
Jun 19, 2026
Regression coverage for false-idle root fix #3. When claude draws its active progress spinner / `esc to interrupt` cue below the `❯` prompt, that line lands in the `footer` section, not `body`. The old body-scoped busy→idle not-spinner clause missed it and the FSM false-idled while generating (with a stale completion footer left in `body`). New tests in fsm-evaluator-claude.test.ts: - the not-spinner clause is whole-screen scoped (no `section`) - section resolution puts the spinner in `footer`, the stale footer in `body` - stays busy when an active spinner sits below the prompt (footer section) - stays busy when only an `esc to interrupt` cue sits below the prompt - still goes idle on a genuine completion (spinner gone) — proves no over-fix - busy→idle stable_ms is 8000 Updated fsm-evaluator.test.ts for the busy→idle stable_ms bump 6000→8000.
vilmire
pushed a commit
that referenced
this pull request
Jun 20, 2026
vilmire
added a commit
that referenced
this pull request
Jun 20, 2026
…s file, pending guard, cooldown sweep appendRemoteLedgerEntries (#1): - dedup now uses tail:1000 instead of full O(n) readLedgerEntries; P2P replication is cursor-based so duplicates appear in the recent tail Archive rotation (#4): - compactLedger now rotates .archive.jsonl to .archive.N.jsonl (max 5) when the archive exceeds 50MB before appending new entries Worker JSON extraction validation (#5): - extractJsonObjectFromSummary now requires at least one mesh worker result field (changedFiles|errors|gitStatus|nextAction|validationResults) before accepting a JSON block; prevents false positives from tool/log JSON in the final summary Archived counts file (#6): - compactLedger writes cumulative counts to <meshId>.archived-counts.json - getLedgerSummary reads and merges archived counts so taskCompleted/Failed/ Stalled and totalEntries are accurate even after compaction Pending events size guard (#2): - queuePendingMeshCoordinatorEvent trims the pending-events.jsonl to the last 50 events when it exceeds 100KB before appending; prevents unbounded growth when coordinator stops draining autoLaunchCooldownUntil cleanup (#3): - sweepExpiredCooldowns() removes expired entries from the cooldown Map whenever a new cooldown is set; inline check at read site also cleans up the checked key; prevents long-lived daemon Map accumulation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
vilmire
added a commit
that referenced
this pull request
Jun 20, 2026
…ntext tail bound Atomic drain (#1): - drainPendingMeshCoordinatorEvents: rename-then-read with atomicDrainFile() so concurrent coordinators cannot both consume the same pending events file. renameSync is atomic on POSIX: only one caller wins; the other returns null and gets an empty result, eliminating duplicate event delivery. Ledger read cache (#3): - readLedgerEntries: 100ms TTL in-process cache via getCachedRawEntries(). Absorbs repeated reads within a single event-processing burst (agent:stopped triggers 4+ ledger reads for the same file in one synchronous cycle). - invalidateLedgerCache() called on every write: appendLedgerEntry, appendRemoteLedgerEntries, compactLedger — ensures readers always see fresh data after any mutation. - Cache is keyed by meshId (not opts), with filters applied to the cached slice. getSessionRecoveryContext tail bound (#2): - Bounded to tail:500 instead of full O(n) scan. task_dispatched is never archived (ARCHIVABLE_KINDS excludes it), so dispatch history is always present in the active file. The 30-min failure window means consecutiveNode- Failures never needs deep history. tail:500 covers all realistic use cases. Tests: 7 new tests (4 cache, 3 atomic drain), all 92 mesh tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
vilmire
added a commit
that referenced
this pull request
Jun 20, 2026
Chat slowed as message count grew because the list is non-virtualized and its per-row React.memo used reference equality (prev.message === next.message), which never held once the tail controller produced fresh message objects every status/tail tick. Every visible row then re-parsed Markdown on every update. Fix #1 (highest impact): - Replace the ChatMessageRow comparator with a field-aware one keyed on a render signature (buildChatMessageRowSignature) that folds in the fields the row actually renders (content via buildChatMessageSignature, kind, sender, meta/visibility/classification flags). Unchanged rows now short-circuit; the streaming last message still updates because its content/signature changes. - Extract the standard-bubble Markdown into a content-keyed ChatMarkdownBody memo so unchanged rows never re-run ReactMarkdown. Fix #3 (defensive): - Cap retained historyMessages at 500 (most-recent kept) so "Load older" can't grow the rendered set unbounded; historyOffset still advances by the full fetched page size so paging stays aligned. - Drop activeConv.messages.length from the controller useMemo deps (it changed every meta append, tearing down/recreating the controller + resubscribing); push the fresh fallbackRecentCount via updateOptions() instead. Fix #2 (virtualization): deferred. The intricate scroll-restore/auto-follow/ snapshot logic reads full content height; partial virtualization would corrupt scroll position. Left a detailed TODO at the map site with integration points. Adds a unit test asserting the signature comparator holds for an unchanged message and breaks on streaming content / kind / meta / visibility changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vilmire
added a commit
that referenced
this pull request
Jun 20, 2026
Regression coverage for false-idle root fix #3. When claude draws its active progress spinner / `esc to interrupt` cue below the `❯` prompt, that line lands in the `footer` section, not `body`. The old body-scoped busy→idle not-spinner clause missed it and the FSM false-idled while generating (with a stale completion footer left in `body`). New tests in fsm-evaluator-claude.test.ts: - the not-spinner clause is whole-screen scoped (no `section`) - section resolution puts the spinner in `footer`, the stale footer in `body` - stays busy when an active spinner sits below the prompt (footer section) - stays busy when only an `esc to interrupt` cue sits below the prompt - still goes idle on a genuine completion (spinner gone) — proves no over-fix - busy→idle stable_ms is 8000 Updated fsm-evaluator.test.ts for the busy→idle stable_ms bump 6000→8000.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1: Async Batch Writer, CPU Lock fix, and Sync API elimination.