STYLE-PASS BAKE-OFF v2 (#1556): 6 challengers vs Gemini 3.1 — incumbent holds, MAI 2.5 the near-miss candidate#1558
Conversation
…— incumbent holds Evidence + registry rows only; NO pipeline switch (owner/orchestrator call). Arms over the fixed crypt base (asset_BKn1kiX2c8BaifYj559yKx7Y, flux.1-dev depth-CN, seed 12345), 111/120 CU. Blind 5-scorer panel, Gemini anchor reproduced 8.0. Panel (unanimous ordering): Gemini 3.1 8.0 > MAI 2.5 Edit 7.0 > GPT-Image-2 6.0 > El Diablo 3.0. - MAI 2.5 Edit: CANDIDATE — defect-clean, artifact-free, perfect lock (0.9999), cheapest (12 CU); did NOT beat the 8.0 incumbent this single-seed round. - GPT-Image-2: REJECT (clean but mid, 45 CU). Access = Scenario ONLY (codex CLI has no image-gen connector, only -i vision input; the '~free Codex connector' path does not exist here). - Seedream 5.0 Pro: REJECT (invented arch doorway — ADDITIONS-LOCK violation). - Kontext Blockout->Render: REJECT (0.854 recall < 0.95 — geometry-aware not geometry-locked + invented water pool). - El Diablo: REJECT (under-cooked). Ideogram V4: INELIGIBLE (txt2img-only, no img2img edit). Gate calibration (honest): gate1 = recall vs fixed BASE (adopted incumbent is only 0.878 vs raw greybox, so >=0.95-vs-greybox is unreachable); gate2 = NET-NEW invented vs base+incumbent baseline (base itself flags 7 crypt cells; the #1540 detector was calibrated on camp). Control was off-subject (jungle) so instrument validity rests on the Gemini anchor this round — disclosed in RANKED.md. Adds qa/evidence/1556/ (arms + overlays + gate.py + gate_results.json + blind panel + panel_verdict + RANKED.md); updates docs/MODEL-REGISTRY.md + qa/model_registry.json (evaluated_1556, non-gating).
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 34 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (16)
📒 Files selected for processing (12)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
evaOS review status: completedPR: #1558 - STYLE-PASS BAKE-OFF v2 (#1556): 6 challengers vs Gemini 3.1 — incumbent holds, MAI 2.5 the near-miss candidate evaOS review completed for this PR head. Automation note: agents should wait for this comment to reach PR URL: #1558 Review URL: #1558 (review) |
![evaos-code-review-bot[bot]](https://avatars.githubusercontent.com/in/4184532?s=60&v=4){kind=small}
There was a problem hiding this comment.
Walkthrough
PR: #1558 - STYLE-PASS BAKE-OFF v2 (#1556): 6 challengers vs Gemini 3.1 — incumbent holds, MAI 2.5 the near-miss candidate
Head: d60c8914230a33f82e5c43f4be2d75847ebf7f81 into main. Review event: COMMENT.
Provider: GLM/Z.ai through ZCode (zcode-glm, zcode, model GLM-5.2).
Estimated review effort: 5/5 (~70 min)
Changed Files
| File | Status | Churn | Purpose | Risk |
|---|---|---|---|---|
qa/evidence/1556/RANKED.md |
added | +85/-0 | Documentation | Low |
qa/evidence/1556/arms/arm1_gemini31.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/arms/arm2_gptimage2.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/arms/arm3_mai25edit.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/arms/arm4_seedream5pro.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/arms/arm6_kontext_blockout.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/arms/arm7_eldiablo.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/gate.py |
added | +73/-0 | Changed file | Moderate: validated P3 finding |
qa/evidence/1556/gate_results.json |
added | +200/-0 | Changed file | Elevated: large change |
qa/evidence/1556/kontext_prompt.txt |
added | +1/-0 | Changed file | Low |
qa/evidence/1556/overlays/contact_sheet.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/overlays/pr_best_frames.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/overlays/zoom_base_rightwall.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/overlays/zoom_mai_gpt.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/overlays/zoom_seedream_arch.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/overlays/zoom_seedream_vs_base.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/panel/SCORER_PROMPT.md |
added | +41/-0 | Documentation | Low |
qa/evidence/1556/panel/blind/.gitignore |
added | +1/-0 | Changed file | Low |
qa/evidence/1556/panel/blind/README.md |
added | +11/-0 | Documentation | Low |
qa/evidence/1556/panel/blind/image_1.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/panel/blind/image_2.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/panel/blind/image_4.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/panel/blind/image_5.png |
added | +0/-0 | Changed file | Low |
qa/evidence/1556/panel/panel_mapping.json |
added | +42/-0 | Changed file | Low |
qa/evidence/1556/panel/panel_verdict.json |
added | +66/-0 | Changed file | Moderate: validated P3 finding |
2 additional changed files omitted from this walkthrough.
Review Signal
Validated inline findings: 2 (P0: 0, P1: 0, P2: 0, P3: 2).
Dropped findings before posting: 0. High-severity findings: 0.
Risk Taxonomy
- Proof gap: 2
Validation and Proof
1 required validation/proof recommendation(s) selected from changed files.
- required: Unity editor or Play Mode smoke - WorldOS repo profile implies Unity runtime risk. Proof: Unity editor smoke; Play Mode log; scene/prefab screenshot or recording.
Proof status: missing - 1 required validation/proof recommendation(s) missing from PR metadata.
Profile validation hints: Prefer correctness, persistence, CI, release, and regression findings over style-only feedback.
Profile proof expectations: Look for Unity editor, play-mode, fixture, or focused smoke evidence when runtime behavior changes.
Related Context
Related issues/PRs: #1556, #1540.
Suggested labels: docs.
Suggested reviewers: none from current metadata.
Review Settings Preview
- Profile: assertive
- Enabled sections: Review summary (inline_review); Walkthrough (inline_review); Changed-files table (walkthrough); Effort estimate (walkthrough); Related issues/PRs (walkthrough); Suggested labels (suggestion_only); Review status comment (sticky_status)
- Path instructions:
Assets/**- Prioritize scene, prefab, save-state, asset-reference, and gameplay regressions. - Path instructions:
ProjectSettings/**- Treat build, platform, input, graphics, and release behavior changes as high risk. - Label suggestions: unity, gameplay, regression-hardening
- Reviewer suggestions: none
- Suggestion behavior: suggestions only; labels and reviewers are not auto-applied.
- Roadmap-only settings: auto-apply labels; auto-request reviewers; required status checks
Pre-merge checklist
- Inline comments target current RIGHT-side diff lines.
- No secret-like content survived into posted inline comments.
- REQUEST_CHANGES is only used when eligible P0/P1 findings survive validation.
- Required behavior proof is present or not applicable.
- Labels and reviewers are suggestions only; the bot did not auto-apply them.
| "Gemini 3.1 (anchor)": 8 | ||
| }, | ||
| "anchor": "image_5 Gemini 3.1 = 8.0 (reproduces its documented adopted incumbent score of 8.0 -> instrument valid on the subject-matched anchor)", | ||
| "control_check": "image_3 (real PoE2) median 2.0 \u2014 OUT of the [6.8,9.2] band, BIMODAL [1,1,2,8,9]. CAUSE: the control I chose is an outdoor jungle-temple EXTERIOR, not a crypt interior; 3/5 scorers applied the not_a_crypt/off-brief penalty (1,1,2) while 2 scored pure craft (8,9). This is a CONTROL-SELECTION flaw (off-subject), NOT an instrument failure: the craft reads (8,9) confirm real art reads top-tier on craft, and the subject-matched Gemini ANCHOR reproduced 8.0 exactly. Panel RELATIVE ordering is unanimous across all 5 scorers and trustworthy; the band-validation is deferred to the anchor rather than the control this round.", |
There was a problem hiding this comment.
P3: 'unanimous across all 5 scorers' overstated — scorer 4 tied Gemini and MAI at 8
The verdict string claims 'Panel RELATIVE ordering is unanimous across all 5 scorers' and RANKED.md repeats 'unanimous across all 5 scorers (Gemini > MAI > GPT-Image-2 > El Diablo)'. Checking per_scorer row 4: image_1 (MAI) = 8 and image_5 (Gemini) = 8 — that is a TIE at the top, not a strict Gemini > MAI ordering. Rows 1,2,3,5 do rank Gemini > MAI strictly (8>7, 9>8, 8>7, 8>7), so 4/5 strictly prefer Gemini and 1/5 ties them. 'Unanimous' is accurate for the GPT>El-Diablo tail and for 'Gemini >= MAI', but not for the strict 'Gemini > MAI' head. Suggest rewording to '4/5 scorers ranked Gemini strictly above MAI; scorer 4 tied them at 8; all 5 agreed on the Gemini>=MAI>GPT>El-Diablo ordering' to match the data. This does not change the verdict (median 8 > 7 holds).
Category: Proof gap
Why this matters: The unanimity claim is load-bearing for the 'no switch this PR' conclusion and is repeated verbatim in the human-readable RANKED.md. An operator re-evaluating adoption (especially of MAI, the named CANDIDATE) deserves the accurate split — 4/5 strict + 1 tie — because the tie is the single strongest data point in MAI's favor and is currently hidden behind the word 'unanimous'.
| return {"plate": str(plate), | ||
| "recall_vs_base": round(recall_base, 4), "recall_pass": recall_base >= MIN_RECALL, | ||
| "recall_vs_greybox": round(recall_grey, 4), | ||
| "invented": ic["invented"], "invented_pass": ic["invented"] == 0, |
There was a problem hiding this comment.
P3: invented_pass uses the raw ==0 bar the PR's own evidence declares miscalibrated
gate.py's invented_pass is ic["invented"] == 0 — the raw inverse-coherence flag count must be exactly zero. But both RANKED.md ("Gate 2 'invented == 0' is unreachable for the crypt") and gate_results.json's gate2 field ("the raw >=0/==0 bars are miscalibrated for the walled crypt") explicitly document that this bar is invalid: the fixed BASE itself flags 7 cells and the gold incumbent flags 7. The honest signal the PR argues for is NET-NEW flags vs the base+incumbent baseline (the invented_net_new column in gate_results.json). As written, anyone running python3 qa/evidence/1556/gate.py arm1_gemini31.png gets invented=6 (pass=False) for the incumbent that the PR concludes HOLDS, and the same false pass=False for GPT/Seedream/Kontext — only MAI and El Diablo (raw 0) pass. The harness output contradicts its own documented verdict. Either compute net-new inside gate.py (subtract the documented baseline_falsepos_cells / base flags) for the invented_pass boolean, or rename the field to invented_raw_pass and add an invented_net_new_pass derived against the baseline so the CLI's pass/fail matches the PR's conclusion.
Category: Proof gap
Why this matters: The gate harness is the reusable artifact future bakes invoke. Shipping it with a pass/fail flag that disagrees with the PR's own calibration reasoning means the next bake-off will reject the incumbent (Gemini) by its own gate, and a reviewer/operator trusting the CLI output over the markdown will reach the opposite verdict from the one documented.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d60c891423
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return {"plate": str(plate), | ||
| "recall_vs_base": round(recall_base, 4), "recall_pass": recall_base >= MIN_RECALL, | ||
| "recall_vs_greybox": round(recall_grey, 4), | ||
| "invented": ic["invented"], "invented_pass": ic["invented"] == 0, |
There was a problem hiding this comment.
For this crypt bake-off the committed evidence recalibrates Gate 2 to use net-new inverse-coherence flags because the fixed base/incumbent have known false positives, but this harness still marks the gate as passing only when the raw inverse_coherence_flags count is zero. When someone reruns the advertised deterministic gate on a valid arm with baseline false positives, it will report a failure that does not reproduce gate_results.json or the ranked table; subtract the baseline cells or report/pass on net-new flags instead.
Useful? React with 👍 / 👎.
| "Gemini 3.1 (anchor)": 8 | ||
| }, | ||
| "anchor": "image_5 Gemini 3.1 = 8.0 (reproduces its documented adopted incumbent score of 8.0 -> instrument valid on the subject-matched anchor)", | ||
| "control_check": "image_3 (real PoE2) median 2.0 \u2014 OUT of the [6.8,9.2] band, BIMODAL [1,1,2,8,9]. CAUSE: the control I chose is an outdoor jungle-temple EXTERIOR, not a crypt interior; 3/5 scorers applied the not_a_crypt/off-brief penalty (1,1,2) while 2 scored pure craft (8,9). This is a CONTROL-SELECTION flaw (off-subject), NOT an instrument failure: the craft reads (8,9) confirm real art reads top-tier on craft, and the subject-matched Gemini ANCHOR reproduced 8.0 exactly. Panel RELATIVE ordering is unanimous across all 5 scorers and trustworthy; the band-validation is deferred to the anchor rather than the control this round.", |
There was a problem hiding this comment.
The visual panel protocol treats the same-panel real-art control as the calibration check; here the control median is 2.0 outside the [6.8,9.2] band, so deferring validation to the Gemini anchor makes the absolute medians uncalibrated. When these medians are used to keep/reject challengers or register MAI as a candidate, the result rests on an invalid panel; rerun with a subject-matched in-band control or mark the medians as advisory only.
Useful? React with 👍 / 👎.
| any re-run. The arm ordering is unaffected (unanimous across scorers). | ||
|
|
||
| ## Artifacts | ||
| - `arms/` — the 6 full-res generated arms · `overlays/` — contact sheet + zoom overlays (Seedream arch, |
There was a problem hiding this comment.
The repo's visual-evidence rule caps committed qa/evidence frames at ≤400KB/frame and ≤6 frames, but this change commits the six full-res arms plus panel/overlay PNGs; 14 of the new PNGs are over 400KB, with arm1_gemini31.png alone about 6.2MB. For visual PR evidence this bloats the repo and violates the evidence contract; keep only a small compressed reviewer-visible set and move full-res artifacts out of the committed tree.
Useful? React with 👍 / 👎.
| "evaluated_1556": { | ||
| "_doc": "STYLE-PASS BAKE-OFF v2 (#1556): challengers run as ONE style pass over the fixed crypt base asset_BKn1kiX2c8BaifYj559yKx7Y (flux.1-dev depth-CN). INFORMATIONAL ONLY \u2014 these ids are deliberately NOT added to approved_bases/models: none beat the Gemini 3.1 incumbent (panel 8.0) so none is adopted; the promotion gate is unchanged. Full evidence: qa/evidence/1556/RANKED.md + panel/panel_verdict.json + gate_results.json.", |
There was a problem hiding this comment.
This adds a new 5-scorer visual panel with medians and cost data, but repo-wide search for #1556/STYLE-PASS BAKE-OFF finds no corresponding qa/scores_ledger.md or scores DB entry. The run contract and RUNBOOK-INDEX require visual-critic panels to land on the visual scores surface, so future rollups and release/accounting queries will miss this measurement unless it is logged through qa/scores_db.py (or explicitly marked non-citable).
Useful? React with 👍 / 👎.
1 participant
Closes #1556 (M-ALIGN). Evidence + registry rows only — NO pipeline switch (the adoption call stays with the orchestrator/owner).
Six owner-surfaced challengers each did ONE style pass over the SAME fixed crypt base (
asset_BKn1kiX2c8BaifYj559yKx7Y, flux.1-dev depth-CN, seed 12345) with identical STRUCTURE-LOCK + ADDITIONS-LOCK prompts; Flux Kontext took the greybox directly (native blockout mode). Blind 5-scorer panel, Gemini as anchor. 111 / 120 CU.Ranked table
No challenger beats the Gemini 3.1 incumbent (8.0); panel ordering unanimous across all 5 scorers. MAI 2.5 Edit (7.0) is the notable near-miss — registered as an evaluated CANDIDATE (multi-seed re-run before adoption). No id added to the promotion allowlist.
Best frames
Base → top arms (Gemini incumbent, MAI, GPT-Image-2, Seedream w/ invented arch, Kontext):
Seedream invented an arched doorway where the base has solid wall (ADDITIONS-LOCK violation):
Owner's two access questions
0.144.0) exposes image only as-i/--image= vision INPUT to the coding agent; there is no image-gen endpoint/MCP among its connectors. The "~free connector" path does not exist for this pipeline; marginal style-pass cost does NOT drop to zero.Honest calibration + caveats (in RANKED.md)
registration_recall, so a "≥0.95 vs greybox" bar is unreachable for any painterly plate.Deterministic gate harness:
qa/evidence/1556/gate.py. Registry:qa/model_registry.json → evaluated_1556(non-gating) +docs/MODEL-REGISTRY.md.