Make unpinned dependencies actually mean latest; fix version-line extraction - #10
Merged
Conversation
papa2 and microscape are unpinned in envs/python.yml, which is right while both are in active development — pinning would mean a second commit here for every improvement there, and the pin goes stale by default. It went stale exactly that way: microscape sat at d9b4645 for a week while its own version PR merged. But an unpinned URL never changes, so Docker restores the cached pip layer and never re-fetches. papa2's version PR merged at 20:39Z, the image built at 20:42Z, and the image still shipped the 2026-04-01 commit. Unpinned was giving neither reproducibility nor currency. CI now resolves both upstream HEADs with `git ls-remote` and passes them as build args that appear in the pip layer's command, so the layer key moves whenever upstream does. They are recorded, not enforced — what was actually installed is still read back from pip's direct_url.json by bin/tool_versions.sh, which is the number that matters and the one that cannot drift from reality. Reproducibility now comes from recording what ran rather than from freezing it in advance, which is the whole point of the manifest and is what makes tracking main safe. Also: `nextflow -v` was never the problem. This container prints "Illegal option --" before its real output for BOTH -v and -version, so swapping the flag fixed nothing. The probe now takes the first line containing something version-shaped rather than the literal first line, which works whichever flag is used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by rebuilding the .sif from #8 and reading what the new manifest reported.
Unpinned was giving neither reproducibility nor currency
papa2 and microscape are unpinned in
envs/python.yml, which is right while both are inactive development — pinning means a second commit here for every improvement there, and
the pin goes stale by default. It did: microscape sat at
d9b4645for a week while itsown version PR merged.
But an unpinned URL never changes, so Docker restores the cached
pip installlayer andnever re-fetches:
fd3cb5aa(merged 20:39:23Z)1bbad5a1— committed 2026-04-01The version PR merged three minutes before the container built, and the build still
installed the April commit. Nothing in the image or its logs said so.
Fix: make the cache key move when upstream does
CI resolves both HEADs with
git ls-remoteand passes them as build args that appear inthe pip layer's command, so the layer invalidates whenever upstream moves.
envs/python.ymlstays unpinned.
They are recorded, not enforced. What was actually installed is still read back from
pip's
direct_url.jsonbybin/tool_versions.sh— that is the number that matters, andthe one that cannot drift from reality.
An earlier revision of this PR pinned both instead. That was the wrong call: reproducibility
now comes from recording what ran rather than freezing it in advance, which is the whole
point of the manifest in #8, and it is what makes tracking main safe during active
development. Pinning is still the right move at a release, when you want frozen
comparability rather than the latest fix.
nextflow: the flag was not the problem
nextflow -vand-versionare both valid and give different output (terse line vsbanner). This container prints
Illegal option --before its real output for both, soswapping the flag fixed nothing — the probe was taking the literal first line and recording
the warning as the version.
It now takes the first line containing something version-shaped, which works whichever flag
is used. Kept at
-v.Worth a separate look: something in this image makes nextflow emit that warning at all. It
does not affect pipeline runs, but it is not expected.
The nextflow warning, traced
nextflowwith no arguments at all prints it, so it was never about the flag:The base image defines a
whichshell function built for GNU which:but Debian 12+ ships debianutils'
which, which accepts only[-as]. The function wasnever updated for the swap, so every
which javain nextflow's launcher emits the errorand continues. Harmless to runs — but it polluted every invocation, and it is what made
the version probe record the error as a version.
Fixed by installing the
whichpackage (GNU which), so the function's flags aresupported. The probe's version-line change stays regardless: a tool that prints anything
before its version should not defeat it.
Verified
Everything else in the rebuilt image reports correctly: cutadapt 5.2, mafft 7.526,
FastTree 2.2.0, R 4.5.3, dada2 1.38.0, DECIPHER 3.6.0, ape 5.8.1, python 3.13.14.
The .sif is already rebuilt and live on fir with #8 and #9 —
MICROSCAPE_GIT_SHAbaked in,and its
main.nfhashes tocecc6abc…, matching commit 04df324. No previous .sif matchedany commit at all. This PR makes the dependency versions inside it current as well, and
needs one more rebuild to take effect.