fix: derive scene seed from instance_id, not sorted index#24
Merged
jeqcho merged 3 commits intoJul 14, 2026
Merged
Conversation
tasks.py seeded each Scene with its sorted position, and derive_seed hashes only (eval_seed, scene_seed, epoch), so task identity never entered the seed. Instance i of every task therefore drew from a byte-identical PCG64 stream: 21% of the suite's continuous setup draws were exact cross-task duplicates. Seed from instance_id (unique and task-qualified) instead. Scene seeds go from 5 distinct to 50; cross-task duplicate draws go to zero. This CHANGES EVERY REALIZED SCENE and therefore every published number. Landing it needs a version bump and a changelog note that results are not comparable across the change. test_canonical_instruction_matches_epoch_zero re-derived the seed from index rather than reading scene.init_seed, so it pinned the seeding scheme instead of the invariant it meant to check. It now reads the seed off the Scene.
Every Scene stamps spec.version into its metadata, so leaving the versions
alone would ship two different scene sets under identical version strings.
That is the silent non-comparability this change exists to prevent, so the
nine tasks on version=1 go to 2 and scoop_pasta goes 2 -> 3.
The changelog states plainly that results before and after this change are
not comparable, and records the measurement that justified the break.
Note: the PACKAGE version is single-sourced from git tags via hatch-vcs
('nothing is committed to main'), so the minor bump to v0.4.0 has to be a
tag cut on your side rather than something this PR can carry.
jeqcho
added a commit
that referenced
this pull request
Jul 14, 2026
…25) Post-merge cleanups from the #24 review: - test_sim_blueprint's sweep paired scenes with derive_seed(0, index, epoch), seeds that no longer occur at runtime now that init_seed is crc32-derived. Sweep with derive_seed(0, scene.init_seed, epoch) instead. Pinned tests keep their fixed seed literals (they check blueprint/realization consistency at a seed, any seed); the one comment claiming those literals were real epochs is reworded. - TaskSpec.version's default was bumped to "2" in #24, which would mint future new tasks at version 2. Restore the default to "1" and set version="2" explicitly on the nine existing tasks (scoop_pasta already pins "3"). Realized version stamps are unchanged. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3.
Scene seeds now come from a hash of
instance_idinstead of the instance's sortedposition.
derive_seedhashes only(eval_seed, scene_seed, epoch), so passingindexmeant task identity never entered the seed and instance i of every taskdrew from the same PCG64 stream. Distinct scene seeds go from 5 to 50, and
cross-task duplicate draws go from 21% of all continuous setup values to zero.
Results before and after this change are not comparable. Every realized scene is
different. That is the point of the change and it is stated at the top of the
changelog.
What's here
tasks.py: seed fromcrc32(instance_id). Used crc32 rather thanhash()onpurpose, since
hash()is randomized per process byPYTHONHASHSEEDand wouldhave made scenes non-reproducible across runs. No collisions across the 50 ids.
test_canonical_instruction_matches_epoch_zero: readsscene.init_seedinstead ofrecomputing from
index, per your note, so it pins the invariant rather than aseeding scheme.
CHANGELOG.md: new file, with the non-comparability warning and the numbers thatjustified the break.
Two things to check me on
Task versions. I bumped all ten (
TaskSpec.version: nine 1 -> 2,scoop_pasta2 -> 3). This is beyond what you asked for. Reasoning: every
Scenestampsspec.versioninto its metadata, so leaving them alone ships two different scenesets under identical version strings, which is the silent non-comparability the change
exists to prevent. A changelog note is prose; that field is what a consumer can check.
Happy to drop it if you disagree.
Package version.
pyprojectsingle-sources it from git tags via hatch-vcs("nothing is committed to main"), so the minor bump to v0.4.0 has to be a tag cut on
your side. Nothing in this PR touches it.
Gates
ruff checkclean,ruff format --checkclean,mypyclean, 307/307 pytest with the100% coverage gate met.
The repro and impact scripts are in #3. Happy to add them under
scripts/if you wantthem in-tree.