Add CI and unit tests for Python utilities

.github/workflows/ci.yml

@@ -0,0 +1,38 @@
+name: Smoke Test
+
+on:
+  push:
+    branches: [master, add_tests]
+  pull_request:
+    branches: [master]
+
+jobs:
+  smoke-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install biopython numpy gffutils pyfaidx
+
+      - name: Verify Python scripts parse without errors
+        run: |
+          find pipelines -name '*.py' | while read f; do
+            echo "Checking $f ..."
+            python -c "import py_compile; py_compile.compile('$f', doraise=True)"
+          done
+
+      - name: Verify Nextflow config is valid
+        uses: nf-core/setup-nextflow@v2
+        with:
+          version: "latest-stable"
+
+      - name: Nextflow smoke test
+        run: nextflow run main.nf -help

.github/workflows/tests.yml

@@ -0,0 +1,26 @@
+name: Unit Tests
+
+on:
+  push:
+    branches: [master, add_tests]
+  pull_request:
+    branches: [master]
+
+jobs:
+  unit-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install biopython numpy gffutils pyfaidx pytest pytest-cov
+
+      - name: Run unit tests with coverage
+        run: pytest tests/ -v --cov=pipelines --cov-report=term-missing --cov-report=xml

README.md

@@ -1,22 +1,117 @@
+[![Smoke Test](https://github.com/CAMI-challenge/CAMISIM/actions/workflows/ci.yml/badge.svg)](https://github.com/CAMI-challenge/CAMISIM/actions/workflows/ci.yml)
+[![Unit Tests](https://github.com/CAMI-challenge/CAMISIM/actions/workflows/tests.yml/badge.svg)](https://github.com/CAMI-challenge/CAMISIM/actions/workflows/tests.yml)
+
 # CAMISIM
 
-CAMISIM is a software to model abundance distributions of microbial communities and to simulate corresponding shotgun metagenome datasets.\
-It was mainly developed for the [Critical Assessment of Metagenome Annotation (CAMI)](http://microbiome-cosi.org/cami) challenge, but should be suitable for general use. Please don't hesitate to [open a new issue](https://github.com/CAMI-challenge/CAMISIM/issues) if you run into problems or need help.
+CAMISIM simulates realistic shotgun metagenome and metatranscriptome datasets from microbial communities. It models abundance distributions, generates reads using established simulators, and produces gold standard assemblies and binning references for benchmarking.
+
+Originally developed for the [Critical Assessment of Metagenome Interpretation (CAMI)](http://microbiome-cosi.org/cami) challenge, CAMISIM is suitable for general use. Please [open an issue](https://github.com/CAMI-challenge/CAMISIM/issues) if you run into problems.
+
+## What's new in v2.0
+
+CAMISIM 2.0 is a complete rewrite using [Nextflow](https://www.nextflow.io/) (DSL2), replacing the previous Python-based pipeline. Key changes:
+
+- Nextflow-managed workflow with support for local, SLURM, and cloud executors
+- Conda/Mamba-based dependency management
+- Metatranscriptomic simulation pipeline (new)
+- Configurable read simulators: ART (Illumina), NanoSim3 (Nanopore), WGSIM, pbsim3 (PacBio)
+
+The previous standalone Python version is available via the git tag `1.31-final` but will not receive updates. Use `convert_config.py` to migrate v1 config files to v2 format.
+
+## Quick start
+
+### Requirements
+
+- [Nextflow](https://www.nextflow.io/) (DSL2)
+- [Conda](https://docs.conda.io/) or [Mamba](https://mamba.readthedocs.io/) (for automatic environment management)
+
+### Run a metagenomic simulation
+
+```bash
+nextflow run main.nf --pipeline metagenomic
+```
+
+### Run a metatranscriptomic simulation
+
+```bash
+nextflow run main.nf --pipeline metatranscriptomic
+```
+
+### Use a custom config
+
+```bash
+nextflow run main.nf --pipeline metagenomic --config path/to/your.config
+```
+
+Both pipelines ship with default example data under `nextflow_defaults/` so you can do a test run out of the box.
+
+## Key parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `size` | `0.05` | Sample size in Gbp |
+| `type` | `art` | Read simulator (`art`, `nanosim3`, `wgsim`, `pbsim3`) |
+| `number_of_samples` | `2` | Number of samples to simulate |
+| `gsa` | `true` | Generate gold standard assembly |
+| `anonymization` | `true` | Anonymize read and contig IDs |
+| `seed` | `632741178` | Random seed for reproducibility |
+| `biom_profile` | `""` | BIOM profile for community design (optional) |
+
+See [Configuration File Options](https://github.com/CAMI-challenge/CAMISIM/wiki/Configuration-File-Options) for the full parameter reference.
+
+## Input files
+
+| File | Format | Description |
+|------|--------|-------------|
+| `genome_locations.tsv` | TSV | `genome_ID<tab>path/to/genome.fa` |
+| `meta_data.tsv` | TSV with header | `genome_ID<tab>OTU<tab>NCBI_ID<tab>novelty_category` |
+| `distribution_N.txt` | TSV | `genome_ID<tab>abundance_fraction` (one per sample) |
+| `gene_annotation_locations.tsv` | TSV | `genome_ID<tab>path/to/annotations.gff3` (metatranscriptomic only) |
+
+## Project structure
+
+```
+CAMISIM/
+  main.nf                          # Entry point
+  nextflow.config                   # Global config (selects pipeline)
+  pipelines/
+    metagenomic/                    # Metagenomic simulation pipeline
+      metagenomic.nf                #   Main workflow
+      sample_wise_simulation.nf     #   Per-sample read simulation
+      read_simulators/              #   ART, NanoSim3, WGSIM modules
+      scripts/                      #   Python utilities
+      config/                       #   Default parameters
+    metatranscriptomic/             # Metatranscriptomic simulation pipeline
+      metatranscriptomic.nf         #   Main workflow (+ gene expression)
+      read_simulators/              #   ART, NanoSim3, pbsim3 modules
+      scripts/                      #   Python utilities
+      config/                       #   Default parameters
+    shared/                         # Shared modules (both pipelines)
+      anonymization.nf              #   Read/contig anonymization
+      binning.nf                    #   Read-to-genome mapping
+      distribution.nf               #   Abundance normalization
+      scripts/                      #   Shared Python utilities
+  nextflow_defaults/                # Example input data for test runs
+  tools/                            # Bundled simulators and reference data
+  tests/                            # Unit tests (pytest)
+```
+
+## Documentation
+
+- [User manual](https://github.com/CAMI-challenge/CAMISIM/wiki/User-manual)
+- [Configuration File Options](https://github.com/CAMI-challenge/CAMISIM/wiki/Configuration-File-Options)
+- [File Formats](https://github.com/CAMI-challenge/CAMISIM/wiki/File-Formats)
+
+## Citation
+
+If you use CAMISIM, please cite:
 
-### CAMISIM 2.0
-CAMISIM received a major update to version 2.0, using nextflow. You can still use the old python standalone using the tag 1.31-final, it will not receive updates however.\
-You can use the script "convert_config.py" to convert your CAMISIM1 config file to CAMISIM2 (no guarantees for correctness, please check yourself).\
-This version has been tested, but if you encounter any unforeseen difficulties or differences from what you expect your simulation to look like, please raise an Issue.
+> Fritz\*, Hofmann\*, et al. (2019). **CAMISIM: Simulating metagenomes and microbial communities.** *Microbiome*, 7:17. doi:[10.1186/s40168-019-0633-6](https://doi.org/10.1186/s40168-019-0633-6)
 
-### Documentation 
-* [User manual](https://github.com/CAMI-challenge/CAMISIM/wiki/User-manual)
-* [Configuration File Options](https://github.com/CAMI-challenge/CAMISIM/wiki/Configuration-File-Options)
-* [File Formats](https://github.com/CAMI-challenge/CAMISIM/wiki/File-Formats)
+You may also cite the CAMI benchmark paper:
 
-### Citation
+> Sczyrba\*, Hofmann\*, Belmann\*, et al. (2017). **Critical Assessment of Metagenome Interpretation — a benchmark of metagenomics software.** *Nature Methods*, 14(11):1063–1071. doi:[10.1038/nmeth.4458](https://doi.org/10.1038/nmeth.4458)
 
-If you use CAMISIM, please cite the publication at *Microbiome*:
-* Fritz*, Hofmann*, et al. (2019). **CAMISIM: Simulating metagenomes and microbial communities.** *Microbiome*, 2019, 7:17. doi:[10.1186/s40168-019-0633-6](https://doi.org/10.1186/s40168-019-0633-6)
+## License
 
-A part of CAMISIM's functionality was also described in the CAMI manuscript, thus you may also cite:
-* Sczyrba*, Hofmann*, Belmann*, et al. (2017). **Critical Assessment of Metagenome Interpretation—a benchmark of metagenomics software.** *Nature Methods*, 14, 11:1063–1071. doi:[10.1038/nmeth.4458](https://doi.org/10.1038/nmeth.4458)
+Apache License 2.0. See [LICENSE.txt](LICENSE.txt).

tests/conftest.py

@@ -0,0 +1,10 @@
+"""Shared fixtures and path setup for CAMISIM tests."""
+
+import sys
+import os
+
+# Add script directories to Python path so we can import modules
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, os.path.join(ROOT, "pipelines", "shared", "scripts"))
+sys.path.insert(0, os.path.join(ROOT, "pipelines", "metagenomic", "scripts"))
+sys.path.insert(0, os.path.join(ROOT, "pipelines", "metatranscriptomic", "scripts"))

tests/test_anonymizer.py

@@ -0,0 +1,120 @@
+"""Tests for pipelines/shared/scripts/anonymizer.py — sequence anonymization."""
+
+import io
+import pytest
+from anonymizer import Anonymizer
+
+
+@pytest.fixture
+def anon():
+    return Anonymizer()
+
+
+def _make_fasta(*seqs):
+    """Create a FASTA-formatted StringIO from (id, seq) tuples."""
+    lines = []
+    for sid, seq in seqs:
+        lines.append(f">{sid}\n{seq}\n")
+    return io.StringIO("".join(lines))
+
+
+class TestAnonymizeSequences:
+    def test_basic_anonymization(self, anon):
+        inp = _make_fasta(("seq1", "ACGT"), ("seq2", "TTTT"))
+        out = io.StringIO()
+        mapping = io.StringIO()
+        anon.anonymize_sequences(mapping, input_stream=inp, output_stream=out, file_format="fasta")
+
+        mapping.seek(0)
+        lines = mapping.read().strip().split("\n")
+        assert len(lines) == 2
+        assert lines[0].startswith("seq1\t")
+        assert lines[1].startswith("seq2\t")
+
+    def test_prefix(self, anon):
+        inp = _make_fasta(("seq1", "ACGT"))
+        out = io.StringIO()
+        mapping = io.StringIO()
+        anon.anonymize_sequences(mapping, input_stream=inp, output_stream=out, sequence_prefix="S0R", file_format="fasta")
+
+        mapping.seek(0)
+        assert "S0R0" in mapping.read()
+
+    def test_sequential_ids(self, anon):
+        inp = _make_fasta(("a", "A"), ("b", "C"), ("c", "G"))
+        out = io.StringIO()
+        mapping = io.StringIO()
+        anon.anonymize_sequences(mapping, input_stream=inp, output_stream=out, file_format="fasta")
+
+        mapping.seek(0)
+        ids = [line.split("\t")[1] for line in mapping.read().strip().split("\n")]
+        assert ids == ["0", "1", "2"]
+
+    def test_output_contains_sequences(self, anon):
+        inp = _make_fasta(("seq1", "ACGT"))
+        out = io.StringIO()
+        mapping = io.StringIO()
+        anon.anonymize_sequences(mapping, input_stream=inp, output_stream=out, file_format="fasta")
+
+        out.seek(0)
+        content = out.read()
+        assert "ACGT" in content
+
+    def test_invalid_format_raises(self, anon):
+        with pytest.raises(AssertionError):
+            anon.anonymize_sequences(io.StringIO(), file_format="bam")
+
+
+class TestAnonymizeSequencePairs:
+    def test_paired_ids(self, anon):
+        inp = _make_fasta(("r1/1", "ACGT"), ("r1/2", "TTTT"), ("r2/1", "GGGG"), ("r2/2", "CCCC"))
+        out = io.StringIO()
+        mapping = io.StringIO()
+        anon.anonymize_sequence_pairs(mapping, input_stream=inp, output_stream=out, file_format="fasta")
+
+        mapping.seek(0)
+        new_ids = [line.split("\t")[1] for line in mapping.read().strip().split("\n")]
+        # pair 0: 0/1, 0/2; pair 1: 1/1, 1/2
+        assert new_ids == ["0/1", "0/2", "1/1", "1/2"]
+
+    def test_prefix_with_pairs(self, anon):
+        inp = _make_fasta(("a", "A"), ("b", "C"))
+        out = io.StringIO()
+        mapping = io.StringIO()
+        anon.anonymize_sequence_pairs(mapping, input_stream=inp, output_stream=out, sequence_prefix="X", file_format="fasta")
+
+        mapping.seek(0)
+        new_ids = [line.split("\t")[1] for line in mapping.read().strip().split("\n")]
+        assert new_ids == ["X0/1", "X0/2"]
+
+    def test_six_sequences_three_pairs(self, anon):
+        inp = _make_fasta(
+            ("r1f", "AAAA"), ("r1r", "TTTT"),
+            ("r2f", "GGGG"), ("r2r", "CCCC"),
+            ("r3f", "ACGT"), ("r3r", "TGCA"),
+        )
+        out = io.StringIO()
+        mapping = io.StringIO()
+        anon.anonymize_sequence_pairs(mapping, input_stream=inp, output_stream=out, file_format="fasta")
+
+        mapping.seek(0)
+        new_ids = [line.split("\t")[1] for line in mapping.read().strip().split("\n")]
+        assert new_ids == ["0/1", "0/2", "1/1", "1/2", "2/1", "2/2"]
+
+
+class TestAnonymizerEdgeCases:
+    def test_empty_input(self, anon):
+        inp = io.StringIO("")
+        out = io.StringIO()
+        mapping = io.StringIO()
+        anon.anonymize_sequences(mapping, input_stream=inp, output_stream=out, file_format="fasta")
+        assert out.getvalue() == ""
+        assert mapping.getvalue() == ""
+
+    def test_original_ids_in_mapping(self, anon):
+        inp = _make_fasta(("my_original_id", "ACGT"))
+        out = io.StringIO()
+        mapping = io.StringIO()
+        anon.anonymize_sequences(mapping, input_stream=inp, output_stream=out, file_format="fasta")
+        mapping.seek(0)
+        assert "my_original_id" in mapping.read()