research: version-pinning design review — exact pins vs daily release

[Jammy2211]

Overview

The PyAuto stack pins one exact version string across every library and workspace: __version__ is duplicated in each library's __init__.py, and autoconf.check_version() hard-fails a user's import when the workspace's recorded version disagrees with the installed library. The motivation (stop users pairing a wrong source/workspace combination) is sound, but the check often breaks for legitimate users, the version string is smeared across code and workspaces, and accidentally released versions (later yanked from PyPI) may still linger in source. This task is a design review: assess whether the high-level design makes sense, and how it should interact with the daily automated release about to be reinstated in rehearsal mode.

Autonomy: --auto launch, effective level supervised (research cap). Plan recorded here per the autonomy contract; the run proceeds and judgment forks come back as batched questions on this issue. Deliverable is a report on this issue — no code changes, no PR.

Plan

• Map the current pinned-version mechanism end-to-end: library __version__ duplication, the autoconf workspace check, workspace version records, and how the release pipeline stamps them.
• GitHub archaeology: find the accidentally released versions and the PyPI-yank issue; check whether those version strings still linger in source or tags.
• Catalogue the user-facing failure modes of the exact-match check from issue history.
• Assess the high-level design against alternatives (compatible-range checks, warn-vs-fail, single-source-of-truth versioning).
• Assess the interaction with the reinstated daily release rehearsal (PyAutoBuild release.yml → TestPyPI, then real PyPI).
• Write the design review + recommendation to this issue and propose follow-up prompts for any concrete changes.

Detailed implementation plan

Affected Repositories

• PyAutoBuild (primary — owns release stamping and the reinstated daily release)
• Read-only survey across PyAutoConf, PyAutoFit, PyAutoArray, PyAutoGalaxy, PyAutoLens and the three user workspaces

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoBuild	main	clean

Suggested branch: none — research-only, no code edits; deliverable is the report on this issue.

Implementation Steps

1. Mechanism survey. Document the full pinning pipeline: __version__ = "2026.7.6.649" duplicated in PyAutoConf/autoconf/__init__.py, PyAutoFit/autofit/__init__.py, PyAutoArray/autoarray/__init__.py, PyAutoGalaxy/autogalaxy/__init__.py, PyAutoLens/autolens/__init__.py; check_version() in PyAutoConf/autoconf/workspace.py (hard WorkspaceVersionMismatchError, config/general.yaml version.workspace_version precedence over legacy version.txt, the two bypass paths); setup.py reading VERSION from the environment; how PyAutoBuild release.yml rewrites all of these plus the workspace side at release time; how sibling-library dependency versions are (or are not) pinned in packaging metadata.
2. Archaeology. gh search issues/PR history for the yank incident; enumerate published versions per package via the PyPI JSON API and git tags; identify the accidental versions and grep current source for lingering occurrences.
3. Failure modes. Search library/workspace issue trackers for WorkspaceVersionMismatchError / version-mismatch reports; classify why the check fires on legitimate users (main-branch clones, partial upgrades, copy-pasted scripts, conda/pip skew).
4. Design assessment. Weigh exact-match hard-fail against alternatives: warn instead of raise; compatible-range (date-window or minor-series) matching; deriving the version from package metadata (importlib.metadata) instead of hand-stamped __init__.py strings; single source of truth for the stack version; whether the five-way duplication is load-bearing or incidental.
5. Daily-release interaction. With a weekday release cadence returning (rehearsal → TestPyPI first), assess how exact pinning behaves: how stale a user workspace can be after one day, whether daily stamping amplifies mismatch frequency, what the rehearsal-mode stamping must NOT touch, and what invariants the release pipeline needs so an accidental publish can't happen silently again.
6. Report. Post the design review to this issue: current-state map, incident findings, failure-mode catalogue, recommendation (keep / adjust / replace), and proposed follow-up Mind prompts for any concrete changes.

Key Files

• PyAutoConf/autoconf/workspace.py — check_version, WorkspaceVersionMismatchError, bypasses
• PyAuto{Conf,Fit,Array,Galaxy,Lens}/<pkg>/__init__.py — duplicated __version__ stamps
• PyAutoBuild/.github/workflows/release.yml — daily release + rehearsal mode (currently paused)
• workspace config/general.yaml — version.workspace_version, workspace_version_check
• */setup.py — VERSION env-var wiring

Worktree root

None — research-only task, no worktree.

Original Prompt

Click to expand starting prompt

We have this API which pins version numbers throughout the source code, with the motivation being it stops users from pairing a wrong version of the source code with the workspace. I think this is a good design, albeit it often breaks for users, and it means we have this version number throughout the code and workspace. We also ran into issues with me accidentally releasing some versions which I think are still in the source code — look at the GitHub history to find them, there was an issue about yanking their PyPIs. Can you review and assess if this high level design makes sense or if we can do it better? Also consider how we will basically have nightly build + release reinstated soon.

[Jammy2211]

Design review: the version-pinning scheme, the yank incident, and daily releases

All six plan steps are complete. Findings first, assessment second, recommendation last.

1. How the scheme actually works today

One stack-wide date version YYYY.M.D.<minor> is stamped at release time into seven surfaces:

1. __version__ in all five libraries' __init__.py (sed in release.yml, then tag_and_merge.sh commits "Update version to X" to each library main and tags vX),
2. exact sibling pins in the published wheels (autolens==X requires autogalaxy==X → autoarray==X/autofit==X → autoconf==X) — pyproject.toml declares them unpinned; the pin is injected at build time,
3. git tags on all five repos,
4. Colab URL tag refs in docs ("Release X: bump Colab URL tag refs" commits),
5. workspace config/general.yaml version.workspace_version ("Release X: pin workspace version" commits to every workspace main),
6. legacy version.txt at workspace roots,
7. READMEs / docs/index.md / paper.md install instructions.

At runtime, autoconf.workspace.check_version() (called on import by autofit/autogalaxy/autolens) compares the installed library's __version__ against the workspace-recorded version and raises a hard WorkspaceVersionMismatchError on exact inequality. Bypasses: version.workspace_version_check: False in general.yaml, or PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1. The check resolves the workspace from Path.cwd(), so it silently degrades to a warning when scripts run outside a workspace root.

2. The accidental releases — found, and they are still live in source

The accidental series is 2026.6.25.641 → 2026.7.6.649 (9 releases, all five packages). Mechanism: release.yml computes VERSION="${DATE}.${MINOR_VERSION:-github.run_number}", and a scheduled cron cannot pass workflow_dispatch inputs — so the weekday 02:00 cron ran real releases (rehearsal defaults false) with the workflow run number as the "minor version": 641, 642, … 649. Each one pushed version-bump commits to all five library mains + all workspace mains + Colab-URL bumps, which fired stale test runs and flooded email — leading to the org-wide cron pause on 2026-07-06 (#114), the same day as the last accidental release.

All 9 releases were subsequently yanked on PyPI with no reason string, and no tracked GitHub issue exists for the yank (I searched titles, bodies and comments org-wide; the only tracked artifacts are the cron pause #114 and the separate, deliberate mass-yank campaign of pre-2026.5.29 versions with reason "Drop Python 3.9/3.10/3.11 support").

Consequence, live right now: pip install autolens resolves to 2026.5.29.4 (the latest non-yanked version, all five packages), while every library main and every workspace main still records the yanked 2026.7.6.649. A fresh user following the default path (pip install + clone workspace main) gets a hard WorkspaceVersionMismatchError on import — and the error's advice tells them to pip install --upgrade autolens==2026.7.6.649, i.e. to pin-install a yanked release (pip permits it with a warning, but it is exactly the wrong instruction).

3. User-facing failure-mode catalogue

• Release-state skew (the current incident): yank/rollback on PyPI is invisible to the recorded versions in git; exact match turns any skew into a hard import failure.
• main-branch clones: workspace main legitimately runs ahead of the installed release; the error text itself acknowledges this is "expected and not actionable" — a hard fail whose own message says the situation is normal is a warn, not an exception.
• Partial upgrades: pip install --upgrade autofit alone skews the stack; the wheel pins mitigate top-down installs but not selective upgrades.
• Daily cadence amplification: with weekday releases, any user who upgrades the library without git pull-ing the workspace the same day (or vice versa) mismatches — exact equality makes the expected daily state an error.
• Internal tax: the scheme already needed bypass wiring in release CI (PyAutoBuild#100), directional-advice fixes (PyAutoConf#107), and repeated manual pin-alignment chores (HowToLens#4, euclid pipeline Soemtimes failed with unable to find distribution #13). Seven stamped surfaces means every release mutates ~12 repos.

4. Assessment: is the high-level design sound?

The goal is sound; the enforcement is aimed at the wrong invariant. Two distinct mechanisms are conflated:

• Library↔library lockstep (exact == sibling pins in wheels): keep this. The five libraries are co-developed and co-released; exact pins are what guarantees a coherent installed stack, and they carry no daily-noise cost.
• Workspace↔library exact-match hard fail: replace. The true invariant is not "workspace and library were released the same day" — it is "the installed library is new enough for the workspace's scripts, and not so much newer that examples have drifted". Exact equality is a proxy that is maximally brittle under a daily release cadence, and it forces the release pipeline to commit to every workspace repo every day, which is precisely what caused the June/July incident cascade.

5. Recommendation

R1 — immediate remediation (small, do before anything else): re-align PyPI with source. Either cut one manual release so pip install matches what main records, or roll the recorded versions back to 2026.5.29.4. Until then the new-user path is broken. Also fix the mismatch-error advice to never recommend pinning a yanked version.

R2 — make the workspace check a compatibility window, not equality (medium): workspace records a minimum required library version (bumped deliberately, only when scripts start depending on new API — not by the release cron), check installed >= minimum, and warn (never raise) when the library is much newer than the workspace clone (e.g. >30 days by date-version comparison). This preserves the wrong-pairing protection while eliminating the daily-mismatch storm and removing the release pipeline's need to touch workspace repos at all.

R3 — stop committing version bumps to main on every release (medium): the wheel already gets its version from the VERSION env var at build time; runtime __version__ can come from importlib.metadata. Keep the git tags (they anchor Colab URLs and releases); drop the daily "Update version to X" / "pin workspace version" / Colab-bump commits, which were the noise-and-stale-CI engine of the incident. Colab URLs can point at a latest-release moving tag or be bumped only on deliberate releases.

R4 — guards for the reinstated daily release (small, folds into the morning-status/rehearsal task): (a) never derive a public version from github.run_number — under cron default the minor to 1 (date already makes it unique per day); (b) gate the real-PyPI leg on a repo variable (vars.RELEASE_REHEARSAL) so a schedule event alone can never do a real release — flipping to real PyPI stays a deliberate human act; (c) any future yank gets a tracked issue recording versions + reason.

R2+R3 together make daily releases and the pairing-protection goal compatible: releases become cheap (wheels + tags only), and the workspace check enforces the invariant users actually need.

Proposed follow-up Mind prompts

1. bug/PyAutoBuild/ — R1 remediation: re-align PyPI/source versions + fix yanked-version advice (blocks the user-facing install path today).
2. feature/autoconf/ — R2 compatibility-window check in autoconf/workspace.py + workspace general.yaml schema (minimum_library_version).
3. refactor/PyAutoBuild/ — R3 wheel-only stamping, importlib.metadata runtime version, tags without bump commits.
4. R4 rides the existing release/pyautoheart/morning_status_and_daily_release_rehearsal.md prompt (its plan already proposes the vars.RELEASE_REHEARSAL gate; add the run_number fix).

Per the autonomy contract this run ends here — no code was changed; filing the follow-up prompts and closing this issue are human decisions.

[Jammy2211]

Maintainer decision (2026-07-08): all input supported — R1–R4 endorsed. On hold until the in-flight morning-status/release-rehearsal task (PyAutoLabs/PyAutoHeart#39, which claims PyAutoBuild and covers the R4 ground) completes, to avoid two tasks reworking the release pipeline at once. Follow-up prompts will be filed after it ships.

[Jammy2211]

Batched maintainer questions (supervised --auto run continues meanwhile)

Hold lifted — the rehearsal-mode work shipped (PyAutoHeart#39): weekday cron rehearses to TestPyPI until vars.RELEASE_MODE=live, so R4 is substantially done. R2 is now in flight as PyAutoLabs/PyAutoConf#118 (version-check-compat-floor). Two things need your call; neither blocks R2:

Q1 — R1, realigning PyPI with source (human act either way). Workspace main genuinely depends on post-2026.7.6.649 API (e.g. the oversampled-PSF guides), so new users stay broken until a real release exists that's newer than the recorded versions. Options:

• (a) Recommended: when you're happy with a few clean rehearsals, flip vars.RELEASE_MODE=live (or dispatch release.yml manually with a chosen minor_version) — one real release realigns everything, and R2's floor semantics then keep it aligned.
• (b) Additionally/alternatively un-yank 2026.7.6.649 on PyPI to restore today's pairing immediately (it was yanked for noise, not proven breakage — but Heart#27's release-fidelity failures are worth checking before un-yanking).
• Either way: should future yanks get a tracked issue (versions + reason)? I'd make it a one-line rule in the Build release docs.

Q2 — R3 scope under daily live releases. Core R3 (endorsed): stop committing __init__.py version bumps to library mains (runtime __version__ via importlib.metadata, wheels stamped at build, tags kept) and stop writing workspace version.txt/general.yaml pins (R2 makes them obsolete). The open scope question: release_workspaces also regenerates notebooks, bumps Colab URL tag refs, and commits to every workspace main on each release — under daily live releases that's a large daily diff to 8 repos. Options:

• (a) Recommended: move notebook regen + Colab bumps out of the daily path — run them only on dispatched (deliberate) releases, keeping daily live releases wheels+tags-only.
• (b) Keep them daily (status quo volume).
• (c) Defer — R3 ships only the version-stamping changes, notebook/Colab cadence gets its own prompt later.

One residue from #39 to fold into R3 either way: MINOR_VERSION still falls back to github.run_number on scheduled runs — harmless in rehearsal, but the moment RELEASE_MODE=live flips, scheduled live releases would again get run-number versions (the exact June/July accident). R3 will default it to 1 on schedule.

[Jammy2211]

Maintainer decisions recorded (2026-07-08): R2 merged (PyAutoConf#119) and R3-core merged (PyAutoBuild#121, gated order respected). Q2 answered: notebook regen + Colab bumps keep the same per-release cadence — they ride each release, so their freshness now explicitly relies on releases happening (daily under live mode; no code change needed, #121 already preserved them). Q1 remains the only open item: flipping vars.RELEASE_MODE=live (or a manual dispatch) to cut the first real release — which realigns PyPI with source and unbreaks the new-user install path. Follow-ups now being filed: workspace minimum_library_version adoption; Heart version_skew floor-vs-tag rework; library __version__ via importlib.metadata.

[Jammy2211]

Rehearsal green on the new pipeline — design review complete

Dispatched the first release through the merged R3 pipeline in rehearsal mode (maintainer decision: stay on TestPyPI for now): run 28973384455 — success.

• Version composed as 2026.7.8.1.dev65201 — the new <date>.1 scheme (no run-number in the public part), dev-segment uniqueness for rehearsals working as designed.
• All five packages: built → metadata-checked → published to TestPyPI → pinned-installed → full test suites passed.
• Every live-path job (release / workspaces / notes / Colab / wiki) correctly skipped.

Final state of the review's recommendations

• R1 — deferred by choice: staying on TestPyPI. The standing human action, whenever ready: flip vars.RELEASE_MODE=live or dispatch with rehearsal=false (note: manual dispatches default to live — pass rehearsal=true explicitly when rehearsing by hand). The first live release realigns PyPI, unbreaks the new-user path, and exercises the tag-only/no-pin legs rehearsal can't reach.
• R2 — merged (PyAutoConf#119): compatibility-floor check.
• R3 — merged (PyAutoBuild#121): wheels+tags only, <date>.1 versions; Q2 answered — notebook/Colab cadence stays per-release.
• R4 — shipped via PyAutoHeart#39 + the run_number fix in feat: release stamping slim — wheels+tags only, no daily bump commits (R3-core) #121.
• Follow-ups filed in PyAutoMind: workspace floor adoption (best after first live release), Heart version_skew floor-vs-tag rework.

Closing — the review, its code follow-ups, and the rehearsal validation are done; the live flip is tracked here as the one open human action.