Skip to content

Make unpinned dependencies actually mean latest; fix version-line extraction - #10

Merged
rec3141 merged 1 commit into
mainfrom
probe-and-pin
Jul 31, 2026
Merged

Make unpinned dependencies actually mean latest; fix version-line extraction#10
rec3141 merged 1 commit into
mainfrom
probe-and-pin

Conversation

@rec3141

@rec3141 rec3141 commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Found by rebuilding the .sif from #8 and reading what the new manifest reported.

Unpinned was giving neither reproducibility nor currency

papa2 and microscape are unpinned in envs/python.yml, which is right while both are in
active development — pinning means a second commit here for every improvement there, and
the pin goes stale by default. It did: microscape sat at d9b4645 for a week while its
own version PR merged.

But an unpinned URL never changes, so Docker restores the cached pip install layer and
never re-fetches:

upstream main in the freshly built image
papa2 fd3cb5aa (merged 20:39:23Z) 1bbad5a1 — committed 2026-04-01

The version PR merged three minutes before the container built, and the build still
installed the April commit. Nothing in the image or its logs said so.

Fix: make the cache key move when upstream does

CI resolves both HEADs with git ls-remote and passes them as build args that appear in
the pip layer's command, so the layer invalidates whenever upstream moves. envs/python.yml
stays unpinned.

They are recorded, not enforced. What was actually installed is still read back from
pip's direct_url.json by bin/tool_versions.sh — that is the number that matters, and
the one that cannot drift from reality.

An earlier revision of this PR pinned both instead. That was the wrong call: reproducibility
now comes from recording what ran rather than freezing it in advance, which is the whole
point of the manifest in #8, and it is what makes tracking main safe during active
development. Pinning is still the right move at a release, when you want frozen
comparability rather than the latest fix.

nextflow: the flag was not the problem

nextflow -v and -version are both valid and give different output (terse line vs
banner). This container prints Illegal option -- before its real output for both, so
swapping the flag fixed nothing — the probe was taking the literal first line and recording
the warning as the version.

It now takes the first line containing something version-shaped, which works whichever flag
is used. Kept at -v.

Worth a separate look: something in this image makes nextflow emit that warning at all. It
does not affect pipeline runs, but it is not expected.

The nextflow warning, traced

nextflow with no arguments at all prints it, so it was never about the flag:

++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot java
Illegal option --

The base image defines a which shell function built for GNU which:

which () { ( alias; declare -f ) | /usr/bin/which --tty-only --read-alias ... }

but Debian 12+ ships debianutils' which, which accepts only [-as]. The function was
never updated for the swap, so every which java in nextflow's launcher emits the error
and continues. Harmless to runs — but it polluted every invocation, and it is what made
the version probe record the error as a version.

Fixed by installing the which package (GNU which), so the function's flags are
supported. The probe's version-line change stays regardless: a tool that prints anything
before its version should not defeat it.

Verified

Everything else in the rebuilt image reports correctly: cutadapt 5.2, mafft 7.526,
FastTree 2.2.0, R 4.5.3, dada2 1.38.0, DECIPHER 3.6.0, ape 5.8.1, python 3.13.14.

The .sif is already rebuilt and live on fir with #8 and #9MICROSCAPE_GIT_SHA baked in,
and its main.nf hashes to cecc6abc…, matching commit 04df324. No previous .sif matched
any commit at all. This PR makes the dependency versions inside it current as well, and
needs one more rebuild to take effect.

@rec3141 rec3141 changed the title Pin both pip dependencies; read the version line, not the first line Pin papa2, bump the microscape pin, and fix version-line extraction Jul 31, 2026
@rec3141 rec3141 changed the title Pin papa2, bump the microscape pin, and fix version-line extraction Make unpinned dependencies actually mean latest; fix version-line extraction Jul 31, 2026
papa2 and microscape are unpinned in envs/python.yml, which is right while both are in
active development — pinning would mean a second commit here for every improvement there,
and the pin goes stale by default. It went stale exactly that way: microscape sat at
d9b4645 for a week while its own version PR merged.

But an unpinned URL never changes, so Docker restores the cached pip layer and never
re-fetches. papa2's version PR merged at 20:39Z, the image built at 20:42Z, and the image
still shipped the 2026-04-01 commit. Unpinned was giving neither reproducibility nor
currency.

CI now resolves both upstream HEADs with `git ls-remote` and passes them as build args
that appear in the pip layer's command, so the layer key moves whenever upstream does.
They are recorded, not enforced — what was actually installed is still read back from
pip's direct_url.json by bin/tool_versions.sh, which is the number that matters and the
one that cannot drift from reality.

Reproducibility now comes from recording what ran rather than from freezing it in
advance, which is the whole point of the manifest and is what makes tracking main safe.

Also: `nextflow -v` was never the problem. This container prints "Illegal option --"
before its real output for BOTH -v and -version, so swapping the flag fixed nothing. The
probe now takes the first line containing something version-shaped rather than the literal
first line, which works whichever flag is used.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rec3141
rec3141 merged commit 8911d76 into main Jul 31, 2026
@rec3141
rec3141 deleted the probe-and-pin branch July 31, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant