Skip to content

feat(overnight-multi-issue-implementation): seven lessons from an 11-hour, 20-PR overnight run (1.3.0) - #25

Merged
wan-huiyan merged 1 commit into
mainfrom
feat/overnight-lessons-2026-08-06
Aug 6, 2026
Merged

feat(overnight-multi-issue-implementation): seven lessons from an 11-hour, 20-PR overnight run (1.3.0)#25
wan-huiyan merged 1 commit into
mainfrom
feat/overnight-lessons-2026-08-06

Conversation

@wan-huiyan

@wan-huiyan wan-huiyan commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Seven additions to overnight-multi-issue-implementation, all distilled from one overnight run — 17 items, 20 merged PRs, 4 orchestrated workflows, 91 subagents, ~11 hours. Each one cost real work on that night.

What's new in the SKILL body

# Where it lands What it says
1 New subsection at the top of the tier rubric Never truncate a findings payload. Three reviewers returned six critical findings; the merging agent got JSON.stringify(reviews).slice(0, 9000) and five arrived. The sixth was cut mid-object and was the worst of the six. Everything visible said the gate had worked. Make the actor report received-vs-answered; keep the run journal as the recovery path.
2 New top-level section after the collision audit Coordinating with sessions you do not control. A branch you can diff; a live agent session you can only negotiate with. Claim id + state + label + file list on a committed shared board. Append, never replace. Don't drop the claim while your PRs are open. Nothing expires it.
3 New top-level section after (2) Amend a running orchestration through a file on disk, not the script. Editing the script changes every agent prompt, so a resume re-runs completed work instead of replaying from cache — here that would have discarded eight finished PRs.
4 Appended to the collision-audit section A third kind of collision neither pre-flight audit can see: the same piece of work, twice, under two names, in a live session's uncommitted tree. Gate rather than parallelise, then extract and prove the refactor changed no result.
5 Appended to the "baseline clean" callout Your own brief's baseline numbers go stale mid-run. A parallel PR re-derived the test counts mid-flight (269 → 314) and a later item quoted its brief. Measure on your own rebase; never carry a count from a document.
6 New top-level section before Phase C What an autonomous run may and may not decide. An assumption is a default, not a ruling. For pre-registered questions: DISCLOSE, not COMPUTE. An un-run unit is "no result", not "inconclusive". Production changes in the reverting direction only, on unanimous reviewer authorisation. Merging is a separate grant from changing production.
7 Appended to the tier rubric's reviewer prompts A verbatim reviewer line: "Check whether this change ships a fresh instance of the defect it repairs." Five of the twenty merged PRs did. Most common finding of the night, and every instance was caught by re-deriving a number, never by reading the diff. Plus: dropping an honest positive is drift too.

SKILL.md 553 → 742 lines.

Verified absent before splicing

All seven were grepped against 1.2.2 first. None was already present. The nearest existing neighbours are noted below because they needed seam work, not because they duplicated anything.

Seams adjusted (connective tissue only — no draft content rewritten)

  • The tier-rubric intro says "the 3-tier rubric below", and block 1 now sits between it and Tier 1 — one sentence added so the pointer still reads.
  • Block 6 says merging its own PRs is a grant. Phase C step 4 and the anti-patterns say "implement" is never "merge" by implication. One sentence now states the relationship: neither grant is implied by the instruction to implement, and the two grants are independent of each other.
  • The "Don't auto-deploy after merge" anti-pattern would otherwise argue against block 6's production paragraph. It now names the one bounded exception (return production to the prior behaviour, unanimous authorisation). A doc arguing against its own shipped guidance is a landmine; this closes it.
  • Block 7 sat visually under Tier 2. Its scope line now says "Tier 1's review-panel prompts included".
  • overnight-review-panel-blocked-reviewer-reads-as-clean is now listed in References — the body cites it, and it's a live plugin in this bundle (not a dead cross-reference).

What the self-review caught, and fixed

An independent reviewer read the spliced result against CONTRIBUTING.md. Three real defects, all fixed in this PR:

  1. "The observed run" was never defined in the shipped file. Five new passages rest on it — including the load-bearing "five of twenty merged PRs" — while the only run the file described was the 2026-05-08 chatbox session (2 PRs, ~6 hours, ~38 dispatches). A reader following the phrase landed on numbers that contradicted it, and the run that actually backs the prose lived only in the README and the commit message, neither of which ships to an installed user. Fixed: the Overview now names the run the phrase means, and a second worked example at the end gives it, with each lesson traced to what it cost.
  2. The production carve-out quietly removed the human. The anti-pattern demands "human review for high-stakes changes"; the new exception permitted a change on "unanimous reviewer authorisation", and everywhere else in the file a reviewer is a subagent. As written, the run could change production unattended but could not merge a PR without waking someone. Fixed at both ends: the reviewers are named as the run's own agents, no human is in the loop at 3am, that is why the direction bound comes first, and anything that is not a revert still waits for a person.
  3. The stale-baseline lesson collided with the recipe three lines above it. That recipe defines the baseline as a failing-set snapshot taken once, before the chain; the new text said to measure it "on its own rebase" — which absorbs an earlier PR's regression into the new baseline and then prints IDENTICAL over it. Fixed: the two are now distinguished — what you must never quote from a document is the suite's headline count, not the pre-chain snapshot.

Two smaller ones from the same read: the companion-skill pointer had been left below the new third-collision paragraphs, where it read as covering them (it is branch-based by construction; the third kind is defined by having no branch) — moved above, and it now says so. And resumeFromRunId was named without saying what provides it.

Two pre-existing anonymization scars in files this PR already touches, fixed while here: for the\nthe project (SKILL.md) and the project propensity project (README.md) are both the visible seam of an old scrub, not identifiers.

Deliberately not fixed, flagged instead: plugin.json's description still reads "v1.2.0 adds Phase 0 …" while shipping 1.3.0. Editing it means editing a description, which this PR promised not to do; it under-sells two minors of content but is not wrong.

One thing in the draft was wrong

The draft called block 4 "a fourth collision the file audit cannot see". The document establishes two collisions (ID-level in the tracker-id audit, file-level in the branch audit); the three-item list it follows is a list of decisions, not collisions. Changed to "a third kind of collision, and neither pre-flight audit can see it".

Versions — every place this repo records one

The failure mode #20, #21 and #24 all had to come back for is a content change that never ships because a version did not move. All four moved here:

SKILL.md frontmatter                    1.2.0 -> 1.3.0
plugins/.../.claude-plugin/plugin.json  1.2.2 -> 1.3.0
.claude-plugin/marketplace.json         1.2.2 -> 1.3.0
VERSION (bundle)                        1.2.1 -> 1.3.0

Two drifts closed on the way:

  • The SKILL.md frontmatter was still at 1.2.0 — the 1.2.1 and 1.2.2 bumps were manifest-only, so the file's own header disagreed with what shipped.
  • VERSION was a patch behind the plugins (1.2.1 vs 1.2.2), because the last two bumps didn't touch it. release.yml's own comment says "Bump VERSION in the same PR as the version bump", so it moves here. This will cut a v1.3.0 GitHub release on merge — flagging it explicitly since it is the one side effect of this PR that isn't a docs change.

Description untouched

No description field changed anywhere. The gate reports overnight-multi-issue-implementation at 1,463 chars, 73 to spare — byte-identical to the pre-change run.

Gates

python3 scripts/check_skill_descriptions.py . --no-color --triggers   exit 0
scripts/leak_scan.sh .                                    LEAK GATE: clean, exit 0
python3 .github/scripts/validate_plugins.py    OK: marketplace + plugins valid, exit 0

Listing budget, --context 1000000:

Listing budget @ 1,000,000 ctx
  needed     30,504 chars  (~7,626 tok)
  budget     40,000 chars  (~10,000 tok)
  fits, 9,496 chars to spare

(At the default 200k context the whole install is still over budget by 22,504 chars — unchanged by this PR, which adds no description text.)

core.hooksPath is set to .githooks; the pre-push hook ran both gates on the push.

Leak check

The additions name one project-relative path (docs/site/assets/live.json), generic counts, and one filename (journal.jsonl). No client names, no emails, no keys, no tokens. Gate clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_013JVMGELnoA2vM56Va4gLXE

@wan-huiyan
wan-huiyan force-pushed the feat/overnight-lessons-2026-08-06 branch 2 times, most recently from d6a3c1c to bd35348 Compare August 6, 2026 10:45
…vernight run taught (1.3.0)

Source: a single overnight run of 17 items -- 20 merged PRs, 4 orchestrated
workflows, 91 subagents, ~11 hours. Every addition below is something that
cost real work on that night, not a guess about what might go wrong.

Seven additions to the SKILL body, all verified absent from 1.2.2 by grep
before splicing:

1. Never truncate a findings payload. Three reviewers returned six critical
   findings; the merging agent was handed
   `JSON.stringify(reviews).slice(0, 9000)` and five arrived. The sixth was
   cut mid-object and was the worst of the six. Three reviewers dispatched,
   three verdicts returned, findings commented on the PR, PR merged -- nothing
   looked wrong. Rules: never slice a findings list; make the merging agent
   report received-vs-answered and treat a mismatch as blocking; the run
   journal holds each agent's true return value when the orchestrator's own
   view was truncated. Same family as a green suite that ran zero tests.

2. Coordinating with sessions you do not control. The existing collision
   audits assume the other work is a branch you can diff. Increasingly it is
   another live agent session whose files are not committed. Claim your intent
   -- id, state, plain label, and the file list -- on a committed shared board
   before starting. It worked: a parallel session read the board, saw two of
   its three tasks already claimed, and did only the third. Append, never
   replace (two sessions each wrote a one-element array and the second erased
   the first). Don't drop the claim while your PRs are open. Nothing expires
   it, so take it down at the end.

3. Amend a running orchestration through a file on disk, not the script.
   Guardrail text is embedded in every agent prompt, so editing the script
   changes every call signature and a resume re-runs completed work instead of
   replaying from cache -- on this run that would have discarded eight
   finished PRs. Point the script at a plan file and append dated addenda.
   Check which phase is running first: an addendum after the last agent
   started is a note to nobody.

4. A third kind of collision neither pre-flight audit can see: the same piece
   of work, in two places, under two names, in a live session's uncommitted
   tree and declared against a different issue. Gate rather than parallelise,
   then extract the piece into a tested module and prove the refactor changed
   no result by diffing both paths' outputs.

5. The baseline numbers in your own brief go stale mid-run. A parallel PR
   re-derived the published test counts while the run was in flight; the
   guardrail text said 269, the truth became 314, and a later item quoted its
   own brief. Measure the baseline on your own rebase; never carry a count
   from a document. An item pasted a stale count inside the PR that existed to
   fix stale counts.

6. What an autonomous run may and may not decide. An assumption is a default,
   not a ruling, and never overrides a standing decision or a
   pre-registration. For pre-registered questions the conservative action is
   DISCLOSE, not COMPUTE. An un-run unit is "no result", not "inconclusive".
   Production changes only in the reverting direction and only on unanimous
   reviewer authorisation, asked as a separate explicit field. Merging its own
   PRs is a separate grant from changing production -- neither is implied by
   the instruction to implement.

7. A verbatim line for every reviewer prompt: "Check whether this change ships
   a fresh instance of the defect it repairs." Five of the twenty merged PRs
   did -- a correction to a figure with no corpus named printed a figure with
   no corpus named. It was the most common review finding of the night, ahead
   of ordinary regressions, and every instance was caught by someone
   re-deriving a number, never by reading the diff. Plus: dropping an honest
   positive is drift too.

Seams adjusted, not content rewritten: one connective sentence so the tier
rubric still reads as "below"; the new section's merge/production paragraph
now points at Phase C step 4 and the anti-patterns, which hold the
ask-before-merge line; the "don't auto-deploy" anti-pattern names the one
bounded exception the new section introduces, so the skill does not argue
against itself; the blocked-reviewer sister plugin is now listed in
References, since the body cites it. The draft's "a fourth collision" was
wrong -- the document establishes two (ID-level and file-level), so it is a
third.

An independent reviewer read the result against CONTRIBUTING.md and found
three things the splice had left broken. All three are fixed here:

- **"The observed run" was never defined anywhere in the shipped file.** Five
  new passages rest their evidence on it, including "five of twenty merged
  PRs", while the only run the file described was the 2026-05-08 chatbox
  session -- 2 PRs, ~6 hours, ~38 dispatches. A reader following the phrase
  landed on numbers that contradicted it, and the run that actually backs the
  prose lived only in the README and this message, neither of which ships to
  an installed user. The Overview now says which run the phrase means, and a
  second worked example at the end gives it, with each lesson traced to what
  it cost.
- **The production carve-out quietly removed the human.** The anti-pattern
  demands "human review for high-stakes changes"; the new exception permitted
  a change on "unanimous reviewer authorisation", and everywhere else in this
  file a reviewer is a subagent. As written the run could change production
  unattended but could not merge a PR without waking someone. Both ends now
  say it plainly: the reviewers are the run's own agents, no human is in the
  loop at 3am, that is why the direction bound comes first, and anything that
  is not a revert still waits for a person.
- **The stale-baseline lesson collided with the recipe three lines above it.**
  That recipe defines the baseline as a failing-set snapshot taken once,
  before the chain; the new text said to measure it "on its own rebase",
  which would absorb an earlier PR's regression and then print IDENTICAL over
  it. The two are now distinguished: the thing you must never quote from a
  document is the suite's headline count, not the pre-chain snapshot.

Two smaller ones from the same read: the companion-skill pointer had been
left below the new third-collision paragraphs, where it looked like it
covered them (it is branch-based, and the third kind is defined by having no
branch) -- moved above, and it now says so; and `resumeFromRunId` was named
without saying what provides it.

Two pre-existing anonymization scars in files this PR already touches, fixed
while here: "for the\nthe project" in SKILL.md:319 and "the project
propensity project" in README.md:58 are both the visible seam of an old
scrub, not identifiers.

Versions, in every place this repo records one -- the failure mode of #20/#21/
#24 is a content change that never ships because a version did not move:

    SKILL.md frontmatter                    1.2.0 -> 1.3.0  (had drifted; the
                                                             1.2.1 and 1.2.2
                                                             bumps were
                                                             manifest-only)
    plugins/.../.claude-plugin/plugin.json  1.2.2 -> 1.3.0
    .claude-plugin/marketplace.json         1.2.2 -> 1.3.0
    VERSION (bundle)                        1.2.1 -> 1.3.0

VERSION had also drifted a patch behind the plugins. release.yml's own comment
says to bump it in the same PR as the version bump, so it moves here and cuts
the v1.3.0 release.

No description changed. The gate reports overnight-multi-issue-implementation
at 1,463 chars with 73 to spare, identical to before this commit.

SKILL.md 553 -> 742 lines. Gates: description-cap exit 0, leak exit 0,
validate_plugins exit 0, listing budget fits at 1M context with 9,496 chars to
spare.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JVMGELnoA2vM56Va4gLXE
@wan-huiyan
wan-huiyan force-pushed the feat/overnight-lessons-2026-08-06 branch from bd35348 to c59d2c7 Compare August 6, 2026 10:55
@wan-huiyan
wan-huiyan merged commit a6c65ed into main Aug 6, 2026
1 check passed
@wan-huiyan
wan-huiyan deleted the feat/overnight-lessons-2026-08-06 branch August 6, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant