Skip to content

Three R-comparison tests pass without calling papa2, and the suite never runs in CI #2

Description

@rec3141

Three tests record PASS without invoking papa2

In tests/compare_with_r.py:

test line body
test_seqtab (make_sequence_table) 256 reads R's CSV, prints its shape, record(name, True, ...)
test_chimera (remove_bimera_denovo) 268 same
test_collapse (collapse_no_mismatch) 359 same

Each loads the R reference, reports its dimensions, and records True. papa2 is never
called and nothing is compared. Two carry honest comments ("we can't perfectly reproduce
this without identical mergers", "shape check only — depends on upstream"), but they
still record PASS, so the summary counts them as green.

Compare test_nwalign immediately below test_chimera, which does the real thing:
computes papa2.nwalign(...) and compares to R's output.

The suite does not run in CI

.github/workflows/ci.yml runs pytest tests/ -v, which collects 7 tests — all from
test_*.py files. compare_with_r.py matches neither test_*.py nor *_test.py, so
pytest skips the file entirely and none of its ~14 comparisons execute.

This is understandable rather than neglectful: the suite needs R reference outputs from
tests/r_reference.R, and tests/r_outputs/ is not committed. It is a deliberate manual
two-step (Rscript tests/r_reference.R then python tests/compare_with_r.py). The
consequence is still that dada2-equivalence is only ever checked when someone remembers,
and three of those checks would not verify anything even then.

The rest of the suite is sound — real assertions (r_val == py_val, max_diff < 0.01,
n_diff == 0), the pytest files are tight (atol=1e-12, exact equality, no skips), and
the exit code is correct (return 0 if n_fail == 0 else 1).

Why chimera is the one that matters most

is_bimera is not a reimplementation — src/chimera.cpp is DADA2's own C++, wrapped by
src/dada2_capi.cpp and reached through ctypes, so the kernel is identical to R's.

But remove_bimera_denovo — the driver that selects candidate parents, applies
min_fold/min_abund, and tallies across samples — is a reimplementation, and it is
where a divergence from R would live. microscape-nf runs the R path through its own
sparse reimplementation of removeBimeraDenovo and the Python path through this one, so
the two pipelines' chimera calls could differ while both correctly call the same
isBimera.

test_chimera is the test that would catch that, and it is one of the three that always
passes.

Suggested

  1. Make the three tests compare, or record FAIL/SKIP rather than PASS — a test that
    cannot verify should not report success.
  2. Consider committing small reference outputs (or generating them in a container that
    has R + dada2) so the suite can run in CI. remove_bimera_denovo in particular is
    worth gating.

Found while tracing pipeline provenance for omc-platform, which drafts Methods sections
from microscape-nf output and needs to state whether the R and Python paths are
equivalent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions