Auto-route recognition probes via a route-classification probe - #22
Merged
Conversation
…manual select Symbolic solvers self-route by structural parse; the recognition probes couldn't, because all 7 probe tasks look like "MultipleChoice" and differ only in semantic content — so probe selection was a manual use_probe(task) / demo dropdown. Add a route-classification probe: a last-token hidden-state classifier over the probe tasks whose max-softmax confidence is the COVERAGE GATE — route only when confident, else fall through to pmi_floor. This extends commitment=coverage from the symbolic solvers (gate on parse coverage) to the probes (gate on router confidence). - route.py: RouteProbe (pure-numpy inference, serialized like other components) + calibrate_route_probe (layer sweep, picks best by CV). - ModelProfile.router slot (get/set/merge/persist); bundled into both fp16 + fp32 profiles. - dispatch: Ctx.router/route_lookup; _solve_choice auto-routes when no explicit probe is set (transient — cleared after, can't leak to the next prompt). - DispatchTurnstyle: loads the router on init; calibrate_router(). - demo: snarks/movie examples now use AUTO (showcasing auto-detection); the dropdown is an override. (Also folds in the friendly-label dropdown from #21.) Validation: router 100% 7-way CV @l10; held-out probe prompts auto-route 90% / 86% correct (the rest abstain to pmi_floor at τ=0.9, never misroute); symbolic tasks (arithmetic/date/ordering/table_query/boolean) never misroute. Native BBH unchanged at 92.5% (router doesn't engage where use_probe is explicit). 692 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Probe selection is now automatic — no manual
use_probe/dropdown.Why: symbolic solvers self-route by structural parse, but the 7 recognition-probe tasks all look like "MultipleChoice" and differ only in semantic content, so they needed an explicit task name.
What: a route-classification probe (last-token hidden-state classifier over the probe tasks). Its max-softmax confidence is the coverage gate — route only when confident, else fall through to
pmi_floor. Extendscommitment = coveragefrom the symbolic solvers to the probes.route.py:RouteProbe(pure-numpy, serialized like other components) +calibrate_route_probeModelProfile.routerslot; bundled into both fp16/fp32 profiles_solve_choiceauto-routes when no explicit probe is set (transient, can't leak)DispatchTurnstyle.calibrate_router(); loads router on initValidation: router 100% 7-way CV @l10; held-out probe prompts auto-route 90% / 86% correct (rest abstain at τ=0.9, never misroute); symbolic tasks never misroute; native BBH unchanged at 92.5%; 692 tests pass.
Supersedes #21 (friendly labels folded in here).