Record what actually ran: commit SHA, script id, real tool versions - #8
Merged
Conversation
Write a run manifest at workflow completion into ${outdir}/viz alongside
provenance.json: pipeline version/revision, Nextflow version, command line,
per-process container images (tool versions via tags), reference databases, and
the resolved analysis parameters (primer error rate, maxEE/truncQ/truncLen,
min_overlap, min_reads/samples/seqs, denoise engine). Guarded by try/catch so it
never affects pipeline success.
Lets omc-platform's AI Methods drafting + agents state the real workflow, versions,
and parameters instead of 'not specified in the pipeline outputs'.
Closes #5
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A 363-sample dataset analysed on 2026-07-23 could not say what produced it. Its report gave "microscape 1.0.0" — a manifest string unchanged across the repo's history — and a null commit. The .sif that ran it was overwritten two days later, and neither that container's main.nf nor the current one matches ANY commit in this repository. Three versions in play, no way to tell them apart. Three causes, three fixes: workflow.commitId is null for every run launched from the .sif, because there is no git repo inside it. The Dockerfile now takes GIT_SHA/GIT_REF/BUILD_DATE build args and bakes them into env vars and OCI labels; CI already knew the SHA (it tags the image with it) and now passes it in. The manifest falls back to that and records which source the commit came from, so "unknown" is visible rather than silent. script_id (the MD5 of main.nf) is recorded even when both of the above are missing. It is what let this be diagnosed at all. containers/ recorded image names, whose tags give versions only if tags are pinned. It now also runs bin/tool_versions.sh, which ASKS each binary — cutadapt, mafft, FastTree, vsearch, DADA2, DECIPHER, ape, papa2, microscape, python, R — instead of reading the conda spec. The spec says what was requested; only the binary knows what was resolved, and a claim resting on DADA2's error model is not reproducible against "bioconductor-dada2". The probe never fails: an absent tool reports null, because a manifest with a hole is better than no manifest.
COPY . /pipeline/ puts main.nf, bin/ and nextflow.config inside the image, but the build only triggered on Dockerfile and envs/. So editing the pipeline shipped nothing, and the .sif drifted from git until neither the container's main.nf nor the running one matched any commit. This is the mechanism behind the provenance gap fixed in the previous commit: baking the SHA in is only meaningful if the image is rebuilt when the code changes.
papa2 and microscape are installed from git URLs and report a version that never changes — microscape has said 0.1.0 across every commit — so the previous probe would have written exactly the meaningless version this whole change was meant to remove. PEP 610 puts the real answer in direct_url.json: microscape 0.1.0 resolves to commit 765d9f90baa5. These two are also the tools most likely to change the numbers a claim rests on, so they are the ones whose provenance matters most. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Probed inside the real .sif rather than a dev machine, which is the only way to learn this. Merging stderr made a shell 'command not found' look like a version banner, so vsearch — which is in neither env spec and is genuinely not installed — was reported as 'present but not answering'. Now the shell is asked first, and absent is null. Also: nextflow -v is not a flag (it is -version), and vsearch is dropped from the list rather than left to report a permanent null. Everything else in the container does answer: cutadapt 5.2, mafft 7.526, FastTree 2.2.0, R 4.5.3, dada2 1.38.0, DECIPHER 3.6.0, ape 5.8.1, python 3.13.14. The only tools whose version string means nothing are papa2 and microscape, both pinned by commit instead.
main.nf conflicted in the manifest block: this branch rewrote the same lines to fall back to the SHA baked into the container, so the resolution keeps this side, which is a superset of main's for those fields. Also records the two parameters main added while this branch was open — primer_detect_reads and min_group_samples. A manifest that silently omits a parameter is the same failure as one that reports a stale version, and merging without noticing is exactly how it would happen.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-on to #6, which added
run_manifest.json. That manifest had three holes, all ofwhich made a real dataset untraceable.
What went wrong
A 363-sample dataset analysed 2026-07-23 could not say what produced it. Its report gave
microscape 1.0.0— a manifest string unchanged across this repo's entire history — anda null commit. The
.sifthat ran it had been overwritten two days later. Neither thatcontainer's
main.nfnor the current one matches any commit here. Three versions inplay, no way to tell them apart.
Three causes, three fixes
workflow.commitIdis null for every run launched from the.sif, because there isno git repo inside it — and that is how this pipeline actually runs. The Dockerfile now
takes
GIT_SHA/GIT_REF/BUILD_DATEbuild args into env vars and OCI labels; CI alreadyknew the SHA (it tags the image with it) and now passes it in. The manifest falls back to
that and records which source the commit came from, so "unknown" is visible rather
than silent.
script_id(the MD5 ofmain.nf) is now recorded unconditionally. It pins the sourceeven when both of the above are missing, and it is the only reason this was diagnosable.
containersrecorded image names, whose tags give versions only if tags are pinned.It now also runs
bin/tool_versions.sh, which asks each binary — cutadapt, mafft,FastTree, DADA2, DECIPHER, ape, papa2, microscape, python, R, Nextflow. The env spec says
what was requested; only the binary knows what was resolved, and a claim resting on
DADA2's error model is not reproducible against "bioconductor-dada2".
Two things found while testing it in the real container
papa2andmicroscapearepip install git+…and both report a hardcoded0.1.0thathas never changed — exactly the meaningless version this PR exists to remove. PEP 610
writes the real commit into
direct_url.json, so the probe now records that too(microscape
0.1.0→d9b4645d3cd4). Companion PRs make the version strings themselvesmeaningful: rec3141/microscape#1 (merged) and rec3141/papa2#1.
The container build only triggered on
Dockerfileandenvs/**, whileCOPY . /pipeline/putsmain.nf,bin/andnextflow.configinside the image. Editingthe pipeline shipped nothing — which is the mechanism behind the drift above. Build
triggers now cover what the image actually contains. Baking the SHA in is only meaningful
if the image rebuilds when the code changes.
Verified
bin/tool_versions.shwas run inside the actual.sifon fir, not just a dev machine.Everything answers — cutadapt 5.2, mafft 7.526, FastTree 2.2.0, R 4.5.3, dada2 1.38.0,
DECIPHER 3.6.0, ape 5.8.1, python 3.13.14 — and the only two tools whose version string
means nothing are pinned by commit instead. That probe run also caught two bugs in it: a
shell "command not found" was being reported as "present but not answering", and
nextflow -vis not a flag. Both fixed;vsearchdropped, since it is a danaSeq tool andappears nowhere in this pipeline.