Tighten operator evidence scoping to default own-version only

[exisz]

Context\nRecent score maintenance over-carried historical upstream issues into newer version evidence blocks. Evidence should default to the version it was observed/reported for, unless there is explicit evidence that it still affects later versions.\n\n## DoD\n- Operator pod instructions say evidence defaults to its own version only.\n- Carry-forward into later versions requires explicit still-affects/reproduced/version-range proof.\n- Launcher prompt carries the same critical guard.\n- Run one subagent audit on a recent version using the new instruction, from latest/backward.\n\nRequested by Exis in #isitstable-com on 2026-06-13.

[exisz]

Partial v2026.6.5 evidence-scope audit result (strict own-version default):\n\nConfirmed likely over-carried / should exclude from stable v2026.6.5 under strict reading unless later stable proof is found:\n- openclaw/openclaw#90418 — 2026.6.5-beta.2 only; no explicit stable v2026.6.5 repro.\n- openclaw/openclaw#91433 — reproduced on 2026.6.1 and 2026.6.5-beta.2 only; no explicit stable v2026.6.5 repro.\n- openclaw/openclaw#85439 — 2026.6.5-beta.2 only / asks for stable fix.\n- openclaw/openclaw#88233 — range through 2026.6.5-beta.2 only.\n- openclaw/openclaw#91890 — appears already implemented/fixed in v2026.6.5, not negative evidence.\n- openclaw/openclaw#91454 — beta-only connector-path repro, no stable repro.\n- openclaw/openclaw#91067 — beta-only condition unchanged.\n- openclaw/openclaw#91216 — beta-only repro; fixed on main.\n- openclaw/openclaw#91401 — fix not in beta yet; no explicit stable broken proof.\n\nConfirmed keep examples:\n- #91804, #91873, #91815, #91772, #91902, #91872, #91865, #91816, #91806, #91861 — explicit stable v2026.6.5 evidence.\n- #88332 — explicit still reproduces on fleet machines updated to 2026.6.5.\n- #91009 — explicitly says v2026.6.1..v2026.6.5 behaves the same.\n- #91756 — borderline, but mentions v2026.6.5 tag/release source carrying same surface.\n\nNote: #136 has not yet been rewritten. Follow-up should finish full evidence classification and then update #136 + trigger sync if changes are applied.

[exisz]

Final v2026.6.5/#136 cleanup result under strict evidence-scope rule:\n\n- Updated #136 body at 2026-06-13T00:43Z.\n- Kept: 48 evidence items with explicit stable v2026.6.5 still-affects/reproduced/tag/release-source proof.\n- Removed: 17 beta-only / fixed-context / migration-candidate items without stable v2026.6.5 negative proof.\n- New evidence penalty: 149.\n- Estimated raw score with current formula: 80 base - 149 evidence + ~9 survival = about -60.\n- Labels were narrowed to kept affected/risk areas. Score label remains score:0-9 because current label taxonomy has no negative bucket; follow-up needed for negative score labels.\n- Sync workflow completed successfully: https://github.com/exisz/IsItStable/actions/runs/27451298049\n\nRemoved examples/reasons are now recorded in #136 under "Operator refresh notes — 2026-06-13". This confirms the failure mode: not a separate issue database; the overreach came from operator-written evidence blocks carrying beta/older issues into the stable version issue.

[exisz]

v2026.6.1 strict evidence-scope cleanup — done

Performed a one-shot strict v2026.6.1 evidence-scope sweep on #133 per OPERATOR_POD.md §2.1 (default-EXCLUDE for ambiguous / carry-forward / same-component / still-open-but-not-confirmed evidence).

Result

• Before: 104 evidence items, evidence penalty -280
• After: 103 evidence items, evidence penalty -279
• Removed: 1 item

Removed list with reasons

Issue	Area	Why excluded
openclaw/openclaw#90822	discord	Reporter's environment is OpenClaw 2026.5.6; they explicitly state "tried updating to 2026.6.1 but gateway didn't restart with new version". Maintainer review note confirms: "could not confirm the gateway was actually running v2026.6.1 after restart, so current shipped behavior is not yet tied to their observed runtime." Per Gate B rule (ambiguity defaults to EXCLUDE), this is not valid v2026.6.1 evidence.

Methodology

1. Parsed current isitstable:v1 JSON block → 104 unique evidence items.
2. Ran scripts/isitstable_evidence_gate.py --target 2026.6.1 against every upstream openclaw/openclaw issue:
   • include: 99
   • needs_review: 5 (#90047, #90443, #90277, #91363, #91016)
   • exclude: 0
3. Manually applied Gate B prompt review to the 5 needs_review items — all 5 passed (explicit "after upgrade to 2026.6.1", explicit OpenClaw version: 2026.6.1 field, or maintainer-confirmed still-affects on v2026.6.1).
4. Hand-audited the 3 include items whose OpenClaw version field was NOT 2026.6.1 (potential carry-forward risk):
   • #90134: ✅ INCLUDE — title body and field both list 2026.5.28, 2026.6.1; explicit affected range.
   • #90506: ✅ INCLUDE — title says "fail at runtime on 2026.5.28 and 2026.6.1"; body says "Also broken on: 2026.6.1 (2e08f0f)".
   • #90822: ❌ EXCLUDE — reporter is on 2026.5.6, unverified on 2026.6.1.
5. Audited all 32 closed items for "fixed in 2026.6.1" patterns that would invalidate them as v2026.6.1 negative evidence — none found; all closed items are bug fixes that originally affected v2026.6.1 and remain valid v2026.6.1 evidence.

Affected components

Recomputed from kept evidence — set unchanged (other Discord items remain, so affected:discord is still valid).

Score / labels

• Kept score bucket: score:0-9 (evidence penalty -279 produces a deeply negative raw score regardless of formula).
• ⚠ Follow-up needed: The IsItStable label taxonomy currently has no negative-score buckets (score:0-9 is the floor). For v2026.6.1 — and likely other deeply-penalized versions — a score:negative or score:<0 bucket would surface "score floor reached" more honestly. Per OPERATOR_POD.md §5 instruction, this run keeps score:0-9 and flags the gap explicitly.
• No other label changes (affected:* set unchanged; risk:* unchanged).

Estimated raw score

Using website-style estimation (base 80 − evidence penalty − vote penalty + survival days):

• 80 − 279 − (votes) + (survival) → strongly negative, will floor to whatever the website does for sub-zero raw scores. Bucket display = score:0-9.
• Actual final score computed by the website using data/settings.json.

Sync status

• ✅ gh workflow run sync.yml -R exisz/IsItStable triggered after body edit.
• ✅ Run #27451699861 completed: success.

Constraints honored: scope limited to issue #133 (v2026.6.1 only), no new version issues created, no other versions touched, no YES/NO/rollback/stay recommendation.

[exisz]

Clean verification summary for v2026.5.28/#113 evidence-scope cleanup:\n\n- #113 updated at 2026-06-13T01:03Z.\n- Before: 46 evidence items, evidence penalty 117.\n- After: 35 evidence items, evidence penalty 92.\n- Removed: 11 items under strict OPERATOR_POD.md §2.1 scope rule.\n- Dropped affected components that were only carried by removed items: plugin-system, browser, node-surface.\n- Current affected count: 23.\n- Current score label remains score:0-9 because negative score labels do not exist yet.\n- Sync workflow completed successfully: https://github.com/exisz/IsItStable/actions/runs/27451789191\n\nThe removed-list table is recorded directly in #113 under "Operator refresh notes — 2026-06-13 (evidence-scope cleanup)".

[exisz]

Clean verification summary for v2026.5.27/#111 evidence-scope cleanup:\n\n- #111 updated at 2026-06-13T05:47Z.\n- Before: 67 evidence items, evidence penalty 182.\n- After: 62 evidence items, evidence penalty 168.\n- Removed: 5 items under strict OPERATOR_POD.md §2.1 scope rule.\n- Removed issues: openclaw/openclaw#72574, #87045, #87332, #87357, #87690.\n- Rationale: four were fixed/backported into release/2026.5.27; one was duplicate evidence already represented by #84376.\n- Current score label remains score:0-9 because negative score labels do not exist yet.\n- Sync workflow completed successfully: https://github.com/exisz/IsItStable/actions/runs/27458236861\n\nThis was a targeted writeback of the prior audit's clear EXCLUDE list; broader ambiguous evidence was left unchanged.

[exisz]

v2026.5.26 (#109) strict evidence-scope cleanup — retry result

Outcome: zero EXCLUDE — all 27 evidence items pass strict v2026.5.26 gating.

The previous run aborted before completing classification (zsh variable-name collision). This retry classifies every item via scripts/isitstable_evidence_gate.py --target 2026.5.26 (Gate A) plus per-item human review against operator-pod §2.1 strict rules:

• Evidence defaults to its own explicitly reported/reproduced version only.
• Carry-forward to v2026.5.26 requires explicit v2026.5.26 affected/reproduced/broken text, version range covering 26, or maintainer/user still-affects proof.
• Ambiguous, same-component, still-open, recently-updated, later-version-only, or beta-only evidence defaults to EXCLUDE.

Counts

Field	Value
Items before	27
Kept	27
Removed	0
Deferred (ambiguous)	0
Total penalty	-71 (unchanged)
Score label	score:20-29 (unchanged; estimated raw ~80 base − 71 evidence + small survival/vote adjustments → low 20s, well within bucket)

Per-item gate verdict (all INCLUDE)

#	Issue	Why it stays
1	openclaw/openclaw#87320	"OpenClaw stable v2026.5.26 published…" reproduction trace on 26
2	openclaw/openclaw#87382	Title + Env both pin "macOS app 2026.5.26" + "Gateway 2026.5.26 (10ad3aa)"
3	openclaw/openclaw#87395	Title "becomes unavailable on 2026.5.26"; Env "OpenClaw 2026.5.26 (10ad3aa)"
4	openclaw/openclaw#87397	Title "(2026.5.26)"; Env "OpenClaw: 2026.5.26 (npm)"
5	openclaw/openclaw#87352	Version field OpenClaw 2026.5.26 (10ad3aa)
6	openclaw/openclaw#87332	"Doctor in 2026.5.26 reports…"; Env "openclaw 2026.5.26 (10ad3aa)"
7	openclaw/openclaw#87387	OpenClaw version field = 2026.5.26 (10ad3aa)
8	openclaw/openclaw#87359	Env "OpenClaw 2026.5.26 (10ad3aa)"; title pins 2026.5.26
9	openclaw/openclaw#87302	Title "Docker container broke after 2026.5.26 upgrade" + version field 2026.5.26
10	openclaw/openclaw#87321	Env "OpenClaw version: 2026.5.26"
11	openclaw/openclaw#87392	Env "OpenClaw: 2026.5.26 (10ad3aa)" + "v2026.5.26 tag" review
12	openclaw/openclaw#87217	"Fresh 2026.5.26 file lock stale evidence" — explicit user still-affects on 26
13	openclaw/openclaw#87312	OpenClaw version field = 2026.5.26 (10ad3aa)
14	openclaw/openclaw#87368	Version "OpenClaw 2026.5.26 (10ad3aa)"; maintainer review confirms "v2026.5.26 both suppress"
15	openclaw/openclaw#87326	Env "OpenClaw: 2026.5.26 (10ad3aa) — also reproducible on main"
16	openclaw/openclaw#87381	Env "OpenClaw version: 2026.5.26"
17	openclaw/openclaw#87329	Env table "OpenClaw version: 2026.5.26"
18	openclaw/openclaw#83184	"Confirming this is still live on 2026.5.26 (latest stable)" — explicit still-affects
19	openclaw/openclaw#87609	Title "regression 2026.5.26"; version field 2026.5.26
20	openclaw/openclaw#87472	Title "compaction timeout in 2026.5.26"; version field 2026.5.26
21	openclaw/openclaw#87595	Title "still reproducing on 2026.5.26"; version field 2026.5.26
22	openclaw/openclaw#87665	"Concrete repro (local, OpenClaw 2026.5.26 + @openclaw/discord 2026.5.26 + @openclaw/codex 2026.5.26)"
23	openclaw/openclaw#87704	Same concrete 2026.5.26 repro stack as #87665
24	openclaw/openclaw#87491	Title "2026.5.26: upgrade can create phantom workdir…"; env 2026.5.26
25	openclaw/openclaw#87990	Range covering 26: "verified statically via npm view for 2026.5.22, 2026.5.24, 2026.5.26, 2026.5.27, 2026.5.28-beta.1"
26	openclaw/openclaw#87604	Gateway 2026.5.22→2026.5.26; node host on 2026.5.26; plugin "version 2026.5.26"
27	openclaw/openclaw#87651	"In OpenClaw 2026.5.26, long-running tools…" — explicit on 26

Removed

None. The previous operator's gating was already strict; nothing to drop under the retry's stricter rules either. No items fall into the "ambiguous / same-component / still-open without 26 evidence / later-version-only / beta-only" exclude buckets.

Issue body / labels

• Body unchanged (already consistent: affected array, JSON evidence, human table, total -71 all match).
• Labels unchanged. All 17 affected:* labels match the 17 components in the JSON; risk labels (risk:channel, risk:tool, risk:kernel, risk:runtime) match evidence types. score:20-29 is the right bucket given -71 evidence.

Sync

Triggered gh workflow run sync.yml -R exisz/IsItStable post-audit:

• Run id 27458375008
• Conclusion: success

Method (for reproducibility)

• Script: nebula/scripts/classify_v109.py (Python; avoids zsh variable-name pitfalls).
• Steps: parse [v2026.5.26] [OpenClaw] Is it stable? #109 body → extract isitstable:v1 JSON → for each evidence ref, gh issue view -R openclaw/openclaw --json title,body,comments → pipe to scripts/isitstable_evidence_gate.py --target 2026.5.26 (Gate A) → human review of target mentions for ambiguous-vs-explicit context.
• Raw classifier output: nebula/scripts/v109_classified.json.

Follow-ups / no-ops

• No new score-bucket label needed; score:20-29 covers raw ~20 with margin.
• Negative-bucket labels were not required; nothing crossed below 0 in this run.
• If website recompute drops below score:20-29 after sync settle, file a follow-up to add score:10-19 / score:0-9 buckets — currently nearest existing bucket is fine.