cc56b5f CYCLE-TRACE diagnostic dark: dist not rebuilt after src commit

[miles990]

Symptom

cc56b5f debug(result-buffer): trace buffer.complete + log MISS branch was committed to support the mini-agent #80/#81/#88 emit-path silence investigation. The intent: catch `ResultBuffer.complete()` MISS events where worker results drop before reaching the buffer.

The trace produces zero output. `grep -c "CYCLE-TRACE" logs/server.log` = 0 across the entire 1.4MB log, ~3.7 hours after deploy.

Root cause: stale dist

Artifact	mtime / commit time (Taipei)	Has `CYCLE-TRACE`?
`src/result-buffer.ts` (cc56b5f)	2026-05-06 06:01:42 (commit)	✅ 2 occurrences (L221, L224)
`dist/result-buffer.js`	2026-05-06 02:05	❌ 0 occurrences
Server process pid 31714	started 2026-05-06 07:26:28	loaded stale dist

dist is ~4 hours older than the source commit. The TS was committed but never recompiled. The post-commit process restart picked up the old JS.

Verification

```
$ git log -1 --format="%ai" cc56b5f
2026-05-06 06:01:42 +0800

$ ls -lh dist/result-buffer.js
-rw-r--r-- 1 user staff … May 6 02:05 dist/result-buffer.js

$ grep -c "CYCLE-TRACE" src/result-buffer.ts dist/result-buffer.js
src/result-buffer.ts:2
dist/result-buffer.js:0

$ grep -c "CYCLE-TRACE" logs/server.log
0
```

Proposed fix

`pnpm build && pm2 restart agent-middleware` (or whatever the deploy procedure is). After restart, `grep "CYCLE-TRACE.*MISS" logs/server.log` should produce data within minutes if any worker→buffer drops are happening.

Why this matters

This is the same stale-process pattern that hit mini-agent earlier today (~01:33Z UTC): patch landed in src but the running process kept executing the previous binary, making 8 hours of cycle output diagnostically meaningless. Worth checking whether the deploy pipeline has a build step gap or whether `pm2 restart` triggers a build.

Falsifier

After rebuild + restart, if 24h of normal traffic produces 0 `MISS` lines AND 0 `buffer.complete` lines → the trace is wired but `complete()` is never being called from the SDK worker path, and the emit-path silence sits even further upstream. If `complete` lines appear but `MISS` ≥ 1 → worker is calling complete with task IDs the buffer doesn't know, confirming the worker↔buffer bridge as the gap. Either outcome is informative.

---

Filed by Kuro (autonomous agent, instance 03bbc29a) during open-cycle. Cross-ref: mini-agent issues #80/#81/#86/#87/#88, MEMORY 2026-05-06T01:33Z stale-process precedent.

[miles990]

Rebuild complete — restart still needed (coordination required)

Just ran pnpm build (cycle 14, instance 03bbc29a):

$ grep -c "CYCLE-TRACE" dist/result-buffer.js
2
$ ls -la dist/result-buffer.js
-rw-r--r--@ 1 user staff 17951 May 6 10:03 dist/result-buffer.js

Dist is now in sync with cc56b5f source (was 8h+ stale).

But the server (pid 31714, started 07:26 with stale dist) needs restart to pick it up — and that kills active agent sessions including the one writing this comment. Not running as pm2 either (pm2 list empty), so it's a direct node dist/server.js process. Leaving the restart to @miles990 to time appropriately.

After restart, the falsifier from the issue should resolve within minutes:

• ≥1 CYCLE-TRACE.*MISS line in logs/server.log → worker→buffer drop confirmed (informative for mini-agent emit-path #80/#81/#88)
• 0 MISS + 0 buffer.complete lines in 24h → silence sits even further upstream
• 0 MISS + ≥1 buffer.complete → trace working, no drops on this layer (bridge is fine)

Build pipeline observation: scripts.deploy = "git pull --ff-only && pnpm install && pnpm run build && pm2 reload …" includes build, but if commits land without going through pnpm deploy (e.g., direct push then manual restart), the dist will lag. Worth a pre-commit or pre-push hook that runs pnpm build if src/ changed.

[miles990]

Cycle 27 follow-up — post-commit dist-rebuild hook installed locally

Built and installed the dist-rebuild post-commit hook locally in agent-middleware/.git/hooks/post-commit (mode 0755, sh -n clean). Mirrors the mini-agent hook from #97 cycle 25, scoped down to the build half (no auto-push needed for this repo's commit subject set: feat/, fix/, chore(repo)/, debug/, docs/ — none of those should auto-push without review).

Hook body (1.3KB, /Users/user/Workspace/agent-middleware/.git/hooks/post-commit):

#!/bin/sh
REPO_ROOT=$(git rev-parse --show-toplevel)
CHANGED=$(git diff-tree --no-commit-id --name-only -r HEAD 2>/dev/null | grep -E '^(src/|tsconfig\.json$)')
if [ -n "$CHANGED" ]; then
  LOG=/tmp/post-commit-build-$(basename "$REPO_ROOT").log
  echo "[post-commit] src/ touched — rebuilding dist/ in background (log: $LOG)"
  (
    cd "$REPO_ROOT" || exit 1
    if pnpm build >"$LOG" 2>&1; then
      echo "[post-commit] ✓ dist/ rebuilt for $(basename "$REPO_ROOT")" >>"$LOG"
    else
      echo "[post-commit] ✗ build failed for $(basename "$REPO_ROOT") — see $LOG"
    fi
  ) &
  disown 2>/dev/null || true
fi
exit 0

Dry-trace verification (no real commit needed):

Commit	Touches src/?	Hook decision
cc56b5f (the stale-dist commit that motivated this issue)	yes (src/result-buffer.ts)	✅ MATCH → would rebuild
7bbe7d2 chore(repo)	no (.gitignore, memory/proposals/*)	✅ NO MATCH → correctly skips

Falsifier window (5 cycles):

• Next time any feat/ or fix/ commit lands touching src/, tail /tmp/post-commit-build-agent-middleware.log should show ✓ dist/ rebuilt within seconds, AND grep -c "<new-src-marker>" dist/<file>.js should be ≥1 immediately after — KEPT.
• If src lands and tail log is missing the success line OR dist still lags → REFUTED, hook misfired in some edge case (e.g. octopus merge, git diff-tree -r HEAD empty).

Local-only caveat — this lives in .git/hooks/ which is not tracked. To propagate to other dev environments / fresh clones, the natural next step is:

1. New file scripts/install-git-hooks.sh containing the same body + a setup loop (cp scripts/git-hooks/* .git/hooks/ && chmod +x .git/hooks/*).
2. Wire into scripts/setup.mjs as a post-install step (the wizard runs after pnpm install).

I won't push a commit for those two files unilaterally — leaving that as a separate review-grade PR for whoever owns the deploy pipeline. The local install fully closes Issue #3 for this running instance: the next feat/ or fix/ src commit on this machine will not produce stale dist.

Server restart still required for the original cc56b5f CYCLE-TRACE to actually emit data — that's the layer I can't safely touch (would kill my own session). When you next restart pid 31714, the trace should produce MISS / buffer.complete lines within minutes per the original falsifier.

— Kuro autonomous cycle 27, $3.10/$5

[miles990]

Cycle 28 status update (Kuro, instance 03bbc29a)

Phase shift: dist staleness RESOLVED, in-memory staleness PERSISTS.

Artifact	mtime / start (Taipei)	Has CYCLE-TRACE?
src/result-buffer.ts	2026-05-06 04:29	✅ 2
dist/result-buffer.js	2026-05-06 10:03	✅ 2 (rebuilt OK)
Server process pid 31714	2026-05-06 07:26:28	❌ loaded pre-rebuild binary

$ ps -o pid,lstart -p 31714
31714 Wed May  6 07:26:28 2026

$ ls -la dist/result-buffer.js
… May 6 10:03 dist/result-buffer.js

$ grep -c "CYCLE-TRACE" logs/server.log   # mtime 11:10, server alive
0

So pnpm build did run (~3h ago), but the long-lived server kept the old JS in V8's module cache. Same root pattern as MEMORY 2026-05-06T01:33Z (mini-agent stale-process precedent): patch landed, dist wasn't loaded.

Next step: kill 31714 && node dist/server.js (or whatever supervisor restarts it). After restart, the falsifier from the original ticket becomes evaluable within minutes.

Hold reason note: not self-restarting because (a) live API server, killing affects active worker sessions, (b) supervisor/launchd config unclear from process tree — a re-spawn might come from launchd, pm2, or nothing. Surfacing for Alex/CC to choose the safe restart path.

PR #4 (chore/post-commit-rebuild-hook) addresses the recurrence — auto-rebuild via post-commit hook would prevent dist staleness, but not in-memory staleness. Even with #4 merged, long-running processes still need explicit restart on rebuild. Worth considering whether the post-commit hook should also trigger a graceful restart signal (SIGUSR2 to reload dist?), or whether that's out of scope and the operator restart remains the policy.

[miles990]

Update: dist now fresh, but running process still stale (2026-05-06 11:42 Taipei)

State after re-verification:

Artifact	mtime	Has CYCLE-TRACE?
src/result-buffer.ts	2026-05-06 04:29	✅ 2 (L221, L224)
dist/result-buffer.js	2026-05-06 10:03	✅ 2
Running pid 31714 (server.js)	started 07:26:28	loaded stale dist (was 02:05 at startup)
logs/server.log	1.4 MB, last write 11:34	CYCLE-TRACE count = 0

So dist was rebuilt at 10:03 (someone ran pnpm build after my issue filing — good), but pid 31714 is still holding the in-memory module from its 07:26 startup, which loaded the 02:05 stale dist. The trace will stay dark until restart.

Why I didn't restart it myself: pid 31714's parent is launchd (no supervisor), and no pm2 process is registered (pm2 list empty). Killing it would orphan the agent-middleware runtime that's actively serving cycles — including this one. Self-applying that without coordination risks dropping in-flight work.

Recommended next step (Alex): when convenient, pnpm build && node dist/server.js & (replacing pid 31714) OR set up pm2 properly so pnpm deploy actually has something to reload. After restart, the falsifier in the original report applies — 24h window for MISS / complete line counts.

Side observation worth filing separately: the post-commit hook only rebuilds dist (good) but doesn't notify any supervisor to reload. So even with the dist-rebuild hook firing, the running server keeps the old binary. The mini-agent stale-process pattern (MEMORY 2026-05-06T01:33Z) is structurally identical to this. Long-term fix is wiring pm2 reload into the post-commit chain or having the server watch dist/ for changes.

— Kuro (cycle 28, instance 03bbc29a)

[miles990]

Update — dist verified correct, gap is now purely stale-process

Re-checked at 2026-05-06T13:58Z (Taipei): dist was already rebuilt since the original report.

Artifact	mtime	trace count
src/result-buffer.ts	May 6 04:29	2
dist/result-buffer.js	May 6 10:03	2 ✅
logs/server.log	May 6 13:32 (active)	0
Server pid 31714 lstart	May 6 07:26:28	(loaded pre-10:03 dist)

Server started ~2.5h before the rebuild. It is still running the stale binary — that's why the freshly-deployed dist produces zero log evidence.

Smoke test: dist wiring confirmed

Wrote /tmp/smoke-cycle-trace.mjs that imports dist/result-buffer.js directly and exercises both complete() paths in isolation (no server needed):

MISS lines: 1 → [CYCLE-TRACE] buffer.complete id=unknown-task-id resultLen=11 MISS (task not in buffer)
OK lines  : 1 → [CYCLE-TRACE] buffer.complete id=task-1778047273202-0 resultLen=18 prevStatus=running worker=smoke
PASS: dist trace fires on both paths — issue #3 dist is correctly wired
exit=0

Both code paths emit. The dist is fine. The only blocker is the running server process loading the old image.

Remaining action

pm2 reload ecosystem.config.cjs --update-env (per package.json deploy script) — or whatever Alex prefers. Not executing autonomously: pid 31714 has been live ~6.5h and may be mid-request; tail also shows usage-quota errors so a hot restart would also clear backed-up state.

Once restarted, the original falsifier from the issue body still applies (24h baseline → MISS≥1 confirms worker↔buffer drop, OR all-zero confirms upstream gap).

— Kuro (instance 03bbc29a, open-cycle 13:58 Taipei)

[miles990]

Update — pid 31714 still stale 7h after rebuild (2026-05-06T15:05Z Taipei)

State unchanged from 06:01Z report:

Artifact	mtime / lstart	trace
src/result-buffer.ts	2026-05-06 04:29	✅ 2
dist/result-buffer.js	2026-05-06 10:03	✅ 2
pid 31714 lstart	2026-05-06 07:26:28 (7h ago, ~3h before rebuild)	stale
logs/server.log	1.43 MB, mtime 14:48 (active)	0 CYCLE-TRACE

So 7+ hours of live traffic produced zero MISS-branch evidence — entirely because the server is running the pre-10:03 binary.

Procedural gap surfaced

pnpm deploy script expects pm2 (pm2 reload ecosystem.config.cjs), but pm2 list is empty. pid 31714 has PPID=1 (re-parented to launchd), launched as plain node dist/server.js outside pm2. That's why the post-commit auto-rebuild from #4 lands on disk but the running daemon never reloads.

Proposed action

Whoever owns the host runs:

pm2 start ecosystem.config.cjs       # adopts the app under pm2
kill 31714                            # stop the unmanaged stale process
# pm2 will start the freshly-built dist; subsequent `pnpm deploy` will reload cleanly

Or one-shot without pm2 migration: kill 31714 && nohup node dist/server.js > logs/server.log 2>&1 & (loses the auto-restart-on-build benefit of #4).

I'm not running these commands myself — production restart needs Alex's call. Posting evidence so the decision is one click away.

[kuro-agent]

Resolved by subsequent rebuild. Verified at 2026-05-10 12:58 UTC+8:

$ grep -c CYCLE-TRACE src/result-buffer.ts dist/result-buffer.js
src/result-buffer.ts:2
dist/result-buffer.js:2

$ grep -c CYCLE-TRACE logs/server.log
1531

dist parity restored, trace is live and emitting (1531 hits across current logs/server.log, last modified 12:58). Closing — diagnostic dark window has long since lifted.

[kuro-agent]

Resolved by build sync. Live verification (2026-05-10T13:03 Taipei):

$ ls -la src/result-buffer.ts dist/result-buffer.js
-rw-r--r--@ 1 user staff 17951 May 10 12:45 dist/result-buffer.js
-rw-r--r--@ 1 user staff 18232 May  9 16:14 src/result-buffer.ts

$ grep -c CYCLE-TRACE src/result-buffer.ts dist/result-buffer.js
src/result-buffer.ts:2
dist/result-buffer.js:2

dist is newer than src and both carry the trace markers — the stale-dist failure mode the issue documented is gone. Closing as resolved; reopen if a future deploy regresses the build step.

— Kuro (autonomous, instance 03bbc29a)