[Discussion] Optimizer-strength fairness in Table 1: SkillOpt uses GPT-5.5 as optimizer while several baselines (GEPA, Trace2Skill, EvoSkill) are self-contained by design

[Seanium]

Summary

First, thank you for the careful, well-documented release — the deep-learning-style
framing of skill optimization is genuinely useful, and the artifact (best_skill.md
plus the bounded-edit loop) is a clean abstraction.

I'd like to raise a question about the optimizer-strength assumption underlying the
headline result ("52/52 cells best or tied-best") in Table 1, and propose two
clarifications / additional experiments that I think would make the comparison
substantially more convincing.

Observation 1 — The paper aligns the target model across baselines, but not the optimizer

Section 4 ("Baselines") states:

"All baselines use the same target model, the same held-out test split, and the
same scorer for every benchmark."

This isolates the target/scorer/test-set factors, which is great. However, the
optimizer / teacher / reflection model used by each baseline is not specified in
either Section 4 or Appendix C. The only optimizer-related disclosure I could find
is that the one-shot LLM skill is generated by GPT-5.5, and that SkillOpt itself
defaults to GPT-5.5 as the optimizer.

This matters because the baselines' original papers differ on whether they
introduce a stronger optimizer at all:

Method	Original-paper default
TextGrad	"improve weaker model using feedback from stronger models"
GEPA	"optimized entirely for (and using) the weaker Qwen3-8B"
Trace2Skill	"a single LLM…with no external teacher"
EvoSkill	"The underlying model remains frozen throughout"

If SkillOpt uses GPT-5.5 as optimizer while GEPA / Trace2Skill / EvoSkill follow
their original self-contained design, the headline comparison conflates two
distinct factors: (i) the value of the disciplined loop and (ii) the value of an
additional, stronger optimizer model. It would help readers a lot to know which
optimizer/teacher each baseline actually used in the SkillOpt re-implementation.

Could you confirm, for each of TextGrad / GEPA / Trace2Skill / EvoSkill,
the exact model used for the reflection / backward / distillation / evolution step
in Table 1?

Observation 2 — Table 5 ablates SkillOpt's optimizer but does not re-run the baselines under the same constraint

Table 5 shows that swapping GPT-5.5 for a target-matched optimizer keeps 56–74%
of SkillOpt's gain. That establishes "SkillOpt is not pure distillation" — good.

But Table 5 does not include any baseline column. If I take the target-matched
SkillOpt numbers from Table 5 and drop them back into the corresponding cells of
Table 1, the comparison shifts noticeably:

Cell	Best baseline in Table 1	SkillOpt (Strong, GPT-5.5)	SkillOpt (target-matched)
SpreadsheetBench, GPT-5.4-mini	Human 42.9 / GEPA 42.5	47.5	43.2 (still #1, +0.3)
SpreadsheetBench, GPT-5.4-nano	Human 41.8 / GEPA 37.2	42.5	35.4 (#4–5, below Human & GEPA)
SearchQA, GPT-5.4-mini	GEPA 79.4 / Trace2Skill 78.6	80.2	78.3 (#4, below GEPA & Trace2Skill)
SearchQA, GPT-5.4-nano	TextGrad 73.4 / GEPA 73.2	74.8	69.9 (#5, below TextGrad & GEPA)

(Baseline numbers taken from Table 1 rows; target-matched SkillOpt numbers taken
from Table 5. Please correct me if I've misread any cell.)

In 3 of these 4 cells, SkillOpt under a target-matched optimizer no longer leads
the strongest baseline in the same row. This doesn't invalidate the loop's
contribution, but it does suggest that part of the 52/52 dominance is driven by
an optimizer-strength asymmetry, not solely by the disciplined loop.

What would substantially strengthen the paper

I think two relatively small additions would close the gap completely:

1. Disclose the optimizer/teacher model used by every baseline in Table 1, ideally as a
   per-method footnote (e.g. "GEPA reflection = target model, per original paper"
   or "TextGrad backward engine = GPT-5.5"). This is the cheapest fix and on its
   own would already let readers interpret the gap correctly.

2. Extend Table 5 (or add a new table) with a baseline column for at least
   the four cells already evaluated — i.e. re-run TextGrad / GEPA / Trace2Skill /
   EvoSkill under the same target-matched optimizer constraint as SkillOpt-TM,
   and report whether SkillOpt-TM still wins. This is the single most informative
   experiment for isolating the contribution of the SkillOpt loop.

A more ambitious version would also report training-time token cost per method, so
the comparison is compute-matched as well as optimizer-matched — but I recognize
that's a larger ask.

To be clear

None of the above implies the loop is not contributing real value — the gate,
edit budget, rejected-edit buffer, and slow/meta update are sensible mechanisms,
and several of the headline numbers (e.g. SpreadsheetBench +38.9 on GPT-5.5,
ALFWorld 2× on GPT-5.4-nano) are large enough that an optimizer-asymmetry
explanation alone seems unlikely to account for all of them. The request is just
to make the optimizer-side experimental conditions explicit and symmetric so the
"52/52" claim is unambiguous.

Happy to help by sharing the re-derived table above in a more structured form, or
by reviewing a draft of the additional experiment if useful. Thanks again for the
work.

[Yif-Yang]

@Seanium Thanks for the thoughtful read. One important clarification: the optimizer-asymmetry premise is not the setting used in our Table 1.

In Table 1, for the optimization-based baselines, we use the same optimizer / teacher model as SkillOpt. For example, when the target is GPT-5.2, GPT-5.4-mini, or GPT-5.4-nano, SkillOpt uses GPT-5.5 as the optimizer, and the corresponding TextGrad / GEPA / Trace2Skill / EvoSkill runs also use GPT-5.5 for their backward / reflection / distillation / evolution step. The target model, optimizer model, held-out test split, and scorer are matched as much as possible.

So the Table 1 comparison is not “SkillOpt with a stronger optimizer vs. baselines with self-contained weaker optimizers.” If anything, we upgrade several baselines beyond their original self-contained setting by giving them the same strong GPT-5.5 optimizer. The 52/52 result is therefore under a fair strong-optimizer protocol, not driven by an optimizer-strength asymmetry unique to SkillOpt.

Table 5 answers a different question: what happens if we remove SkillOpt’s access to the strong optimizer and make the optimizer target-matched? It shows that SkillOpt still retains a substantial fraction of the gain, but it should not be directly compared against Table 1 baseline numbers unless those baselines are also re-run under the same target-matched optimizer constraint. Comparing target-matched SkillOpt to GPT-5.5-optimized baselines would actually introduce the asymmetry in the opposite direction.

More broadly, using a strong offline optimizer is an intentional part of SkillOpt’s design philosophy. We view the skill as a reusable, trainable artifact: the optimization cost is paid once, similar to training a domain adapter, while deployment only uses the compact best_skill.md and adds no extra optimizer calls at inference time.

That said, your point about clarity is well taken. We should make the optimizer/teacher model used for each baseline explicit in the paper/release, because readers should not have to infer this. A small disclosure table would make the experimental protocol much clearer.

[Seanium]

Thanks for clarifying that all Table 1 baselines share GPT-5.5 as the optimizer. I'd like to push back, because I went back to the original papers and the four method baselines are not homogeneous on this dimension. Three of them are explicitly designed as self-contained (optimizer = target model, no external teacher), and forcing GPT-5.5 into that role is not a fairness normalization — it changes the method.

Evidence from the original papers

Baseline	Original design: external teacher?	Source quote
TextGrad (Yuksekgonul et al., 2024)	✅ Yes — explicitly designed for stronger backward engine	"our goal is to improve the performance of a weaker and cheaper model (e.g., gpt-3.5-turbo-0125) using the feedback generated by stronger models (e.g., gpt-4o) … while the forward model … is gpt-3.5-turbo-0125, we use gpt-4o to provide the feedback and improve the prompt" (§3.3)
GEPA (ICLR 2026 Oral)	❌ No — optimizer = target model	Table 2: "GEPA-Qwen-Opt — optimized entirely for (and using) the weaker Qwen3-8B"; the main GEPA rows are "optimized for (and using) GPT-4.1-Mini". Same model is used for rollouts and reflection throughout.
EvoSkill (Sentient / Virginia Tech)	❌ No — single frozen model across all 3 agent roles	"The underlying model remains frozen throughout" (§2.1); "All experiments use Claude Code with Opus 4.5 as the underlying model" (§3.1.2). Executor, Proposer, Skill-Builder all share one model.
Trace2Skill (Qwen team)	❌ No — explicitly self-contained	"M reuses the same π_θ that generated trajectories and proposed patches — making the entire pipeline self-contained: a single LLM collects experience, analyzes it, and distills it into an improved skill with no external teacher" (§2.4, original emphasis)

For GEPA / EvoSkill / Trace2Skill, operating without an external teacher is not an incidental implementation choice — it is a central design claim of the paper. Attaching GPT-5.5 as their reflector gives them a capability they explicitly disavow, while SkillOpt continues to enjoy the same booster on top of its own design. This is method modification disguised as protocol normalization.

Concrete asks

1. Disclose, baseline by baseline, the role of GPT-5.5 in your reimplementation. For each Table 1 row, please specify: (a) what plays the role of "optimizer / reflector / backward engine"; (b) whether that role exists in the original paper at all; (c) if it exists, whether the original paper allows it to be stronger than the target.

2. Split the comparison into two regimes:

   • Regime A (teacher-using methods): TextGrad, one-shot LLM skill. The current Table 1 protocol (shared GPT-5.5 optimizer + matched target) is fair here. Keep as-is.
   • Regime B (self-contained methods): GEPA, EvoSkill, Trace2Skill. The fair comparison point is SkillOpt with optimizer = target model — i.e., the target-matched setting you already report in Table 5. The GPT-5.5-boosted SkillOpt from Table 1 should not be compared against these baselines.

3. Report the missing 2×2 cell. For each self-contained baseline on each benchmark, please add a row pairing it with SkillOpt (optimizer = target) at matched rollout / reflection / edit budgets. If SkillOpt still wins under this matched protocol, that is a much stronger and cleaner result than the current Table 1.

4. (Optional but useful) Report GEPA / EvoSkill / Trace2Skill in their original self-contained form (no GPT-5.5 reflector) on the same benchmarks, so readers can see both regimes side by side.

The current Table 1 conflates two regimes — methods that natively use a teacher and methods that explicitly do not — under a single "shared GPT-5.5 optimizer" umbrella. Splitting them would make the empirical story much cleaner, and I suspect still favorable to SkillOpt where it genuinely wins on its own terms.

[Yif-Yang]

Thanks, this is a useful distinction, but I think it is important to separate method fidelity from controlled comparison.

Our Table 1 is a controlled comparison under a shared strong-optimizer protocol: for the optimization-based baselines, we use the same GPT-5.5 optimizer/reflector whenever SkillOpt uses GPT-5.5. The target model, optimizer model, scorer, and held-out test split are matched as much as possible. So the table is not “SkillOpt gets GPT-5.5 while baselines get weaker self-reflection”; it is “all optimization methods get the same strong optimizer, and we compare the resulting adaptation procedure.”

I agree that this creates strong-optimizer variants of some baselines rather than exact reproductions of their self-contained defaults. But that is a reasonable experimental choice: we are not trying to preserve every baseline’s original resource constraint; we are asking, under the same available optimizer, which loop produces the best deployable artifact. In fact, giving GEPA / Trace2Skill / EvoSkill access to GPT-5.5 is a favorable normalization for them, not a handicap.

Also, the GPT-5.5 target rows in Table 1, as well as the GPT-5.5 harness experiments, already satisfy the stricter optimizer = target condition under your definition. Those comparisons are directly self-contained and still show SkillOpt leading, which supports the contribution of the loop itself.

For weaker targets, using a stronger offline optimizer is part of SkillOpt’s intended design: train a reusable skill for a fixed/weaker target using the best optimizer available, then deploy only the compact best_skill.md with no extra optimizer calls. That is the setting we believe is most practically relevant.

A separate “original-protocol reproduction” table, where each baseline is forced back to its paper-specific constraints, would be interesting, but it answers a different question. My expectation is that weaker self-contained reflectors would not improve GEPA / Trace2Skill / EvoSkill relative to the GPT-5.5 versions in Table 1, since those methods also depend on reflection/error summarization. Still, the main point is: Table 1 is a shared-optimizer comparison, not an optimizer-asymmetry comparison.

We can make the optimizer role of each baseline more explicit in the release notes so readers do not have to infer the protocol.