SPOT-BGC

A Snakemake Pipeline to Output meTagenomics-derived Biosynthetic Gene Clusters

Author: Vi Varga

Current version: 3.1.2

Last Major Update: 2026.02.04

Introduction

The SPOT-BGC pipeline is a workflow designed to process metagenomic FASTQ read data, eliminate human contamination, and assemble bacterial genomes; in order to predict Biosynthetic Gene Clusters (BGCs) present in the data. Metagenome-Assembled Genomes (MAGs) are produced during an intermediate step of this pipeline.

This pipeline can process both paired-end (PE) and single-end (SE) FASTQ input. In order to preserve as much data as possible, assembly is performed on both a per-sample and per-cohort basis.

Pipeline structure

The SPOT-BGC pipeline performs the following on input metagenomic FASTQ reads:

1. Quality assessment: FastQC, MultiQC
2. Quality trimming: Trimmomatic
3. Human read filtration by mapping to the human genome: Bowtie2
4. Normalization of read counts: BBNorm
5. For the per-sample assemblies, sample IDs with <100k reads remaining are filtered out
6. Assembly: per-sample with (Meta)SPAdes, per-cohort with MEGAHIT (Note that if the complexity of a file causes MetaSPAdes to be unable to process it, MEGAHIT will be used, instead.)
7. Human contig elimination (sanity check): Kraken2
8. Binning of contigs into MAGs: MetaBAT
9. Quality assessment of the MAGs: CheckM
10. Taxonomic assignments of the MAGs: CheckM
11. BGC predictions: GECCO, AntiSMASH

flowchart TD
    X(Set up working directory with snakemake_setup.sh) --> A[Read QC]
    A --> B[Quality Trimming]
    B --> C[Human Read Filtration]
    C --> D[Read Count Normalization]
    D --> E[>= 100k Reads Filtration]
    E --> F{Per-Cohort Assembly}
    E --> G{Per-Sample Assembly}
    F --> H[Human Contig Elimination]
    G --> H[Human Contig Elimination]
    H --> I[Binning Contigs into MAGs]
    I --> J[MAG QC]
    J --> K[Taxonomic Assignment]
    K --> L[BGC Prediction]
    L --> |Optional| M(Clean working directory with snakemake_cleanup.sh)

Loading

Dependencies

The primary dependencies of the SPOT-BGC pipeline are:

• Snakemake
• Apptainer

Running SPOT-BGC

Owing to a temporary bug in Snakemake at the time of this pipeline's creation, the programs used must be run out of containers, rather than via conda environments (source). All files necessary to generate the containers are included in this repository.

To build the containers with Apptainer, run the following code from the workflow/ directory:

# create the workflow/containers/ directory
mkdir containers
cd containers/
# then build the containers with Apptainer
# the read QC container
apptainer build --build-arg ENV_FILE=../envs/env-QualityChecking.yml env-QualityChecking.sif ../scripts/conda_environment_args_ubuntu.def
# read trimming container
apptainer build --build-arg ENV_FILE=../envs/env-trimmomatic.yml env-trimmomatic.sif ../scripts/conda_environment_args_ubuntu.def
# human read removal container
apptainer build --build-arg ENV_FILE=../envs/env-bowtie2.yml env-bowtie2.sif ../scripts/conda_environment_args_ubuntu.def
# normalization via BBTools container
apptainer build --build-arg ENV_FILE=../envs/java-11.yml bbtools.sif ../scripts/conda_environment_args_ubuntu-bbtools.def
# assembly container
apptainer build --build-arg ENV_FILE=../envs/metagenome_assembly.yml metagenome_assembly.sif ../scripts/conda_environment_args_ubuntu.def
# human contig elimination with Kraken2
apptainer build --build-arg ENV_FILE=../envs/env-kraken.yml --build-arg REF_FILE=../../resources/Ref/{REFERENCE} env-kraken2db.sif ../scripts/conda_environment_args_ubuntu-kraken2db.def
# replace the {REFERENCE} placeholder with the name of your reference file
# note that this container will take some time to assemble, owing to the need to download & install databases
# QC of MAGs & taxonomic profiling
apptainer build --build-arg ENV_FILE=../envs/mag_assembly_qc.yml mag_assembly_qc.sif ../scripts/conda_environment_args_ubuntu.def
# BGCs with AntiSMASH
apptainer build --build-arg ENV_FILE=../envs/env-antismash.yml env-antismash.sif ../scripts/conda_environment_args_ubuntu-antismash.def
# BGCs with GECCO
apptainer build --build-arg ENV_FILE=../envs/env-gecco.yml env-gecco.sif ../scripts/conda_environment_args_ubuntu.def

Set up your project file structure as follows:

├── LICENSE
├── README.md
├── config
│   └── config.yaml
├── resources
│   ├── RawData
│   │     ├── {COHORT_ID}
│   │     │    └── {SAMPLE_ID}
│   │     │        ├── {SAMPLE_1.fastq}
│   │     │        └── {SAMPLE_1.fastq}
│   │     └── {COHORT_ID}
│   │         └── {SAMPLE_ID}
│   │             └── {SAMPLE.fastq}
│   └── Ref
|       └── {REFERENCE}
└── workflow
    ├── containers
    ├── envs
    └── scripts

For the setup above, please note the following:

• The COHORT_ID is intended to be the NCBI BioProject number, but can be designated by the user however you wish. However, a cohort ID must be provided for every sample.
• The sample raw FASTQ files should be in directories with the sample name. This is the structure that results if the data is downloaded directly from the NCBI with SRA (Sequence Read Archive) fetch.
• The REFERENCE should be a DNA reference genome. Modify the config.yaml file located in the config/ directory with your reference genome name, or overwrite the name on the command line with --config human_genome=resources/Ref/REFERENCE.FASTA.
• As is illustrated, the SPOT-BGC pipeline works with both PE and SE input FASTQ files.

Once you have organized your project as illustrated above, run the pipeline setup script from your SPOT-BGC working directory, like so ¹:

# make the script executable if needed
chmod +x workflow/scripts/snakemake_setup.sh
# this script will create some key indexing files for the SPOT-BGC pipeline
bash workflow/scripts/snakemake_setup.sh

Running on a regular Linux server

After running the snakemake_setup.sh script, you can run the pipeline from your terminal from the main directory (i.e., the directory where config/, resources/ and workflow/ are located) like so:

# activate your conda snakemake environment
conda activate snakemake
# standard usage
snakemake --use-singularity --cores 20
# note that you must call the --use-singularity flag, as all programs are installed in Apptainer containers
# note that when working on a server, 
# you can specify the specific cores used with `taskset`

To modify the number of threads used by various programs, do the following:

# program thread modification
snakemake --use-singularity --cores 20 --config threads_{PROGRAM}={NUMBER}
# for example: 
snakemake --use-singularity --cores 20 --config threads_trimming=6

Please check the config.yaml file located at config/config.yaml to see the list of programs whose thread allocation can be modified in this way.

In addition to the thread count allocation, the following settings can be modified directly from the command line:

• The bin size used by MetaBat
• The memory allocation usable by BBnorm
• The maximum memory allocation to be used for normalization & assembly
• Email to use to notify user of completion ²

Paired-End-only bug recovery workflow (v.3.1.2)

A bug was noticed wherein if only paired-end files were provided to the workflow, mapping and and normalization did not proceed, causing the pipeline to fail. Single-end-only and mixed single- and paired-end cohort structures (the latter of which was used for testing) have no issues.

This bug is proving difficult to resolve, so a recovery workflow was designed using a new script, PEonly_REPAIR.sh

If only paired-end reads are used, the pipeline should be run in 3 steps:

snakemake --use-singularity --omit-from filt_100k
bash workflow/scripts/PEonly_REPAIR.sh threads reference_name
# Please see the PEonly_REPAIR.sh scipt for full details on usage
snakemake --use-singularity

Running on a SLURM HPC

If you wish to run the SPOT-BGC pipeline in a SLURM environment, it is recommended to use an interactive session in an instance of zellij, screen, tmux, etc. Snakemake will automatically submit jobs and request resource allocation for you. While it is possible to include all pertinent information in the Snakemake command line call, I have included an example SLURM profile config.yaml file in the profiles/slurm/ directory. Once modified with your HPC project ID, etc., it can be used to run the SPOT-BGC pipeline like so:

# open an interactive, detachable terminal session
zellij -s snakemake-SPOT-BGC
# zellij attach snakemake-SPOT-BGC to reattach later
module purge
module load snakemake/8.27.0 # or whatever version of Snakemake is supported on your HPC
# note that NO OTHER MODULE should be loaded other than Snakemake
chmod +x workflow/scripts/snakemake_setup.sh
bash workflow/scripts/snakemake_setup.sh
snakemake --executor slurm --jobs 10 --profile=profiles/slurm --use-singularity -w 1000
# it is recommended to use -w 1000 or similar to allow for some latency that can occur on SLURM clusters
# --jobs can be set however the user prefers - 10 was simply what I used

Pipeline status

Current release version

Bug hotfix, 2026.02.04: Build 3.1.2

• A bug was noticed wherein if only paired-end files were provided to the workflow, mapping and and normalization did not proceed, causing the pipeline to fail
  • Single-end-only and mixed single- and paired-end cohort structures (the latter of which was used for testing) have no issues
  • This bug is proving difficult to resolve, so a recovery workflow was designed using a new script, PEonly_REPAIR.sh
  • If only paired-end reads are used, the pipeline should be run in 3 steps: snakemake --use-singularity --omit-from filt_100k → bash PEonly_REPAIR.sh threads reference_name → snakemake --use-singularity
• The reference file was left in the Snakefile by accident - this has been replaced by the corresponding config.yaml parameter, as originally intended
• Simplified contents of rule all

Ongoing work for future versions

Potential future plans (no set date):

• Branching pipeline: Allow user to choose which programs in the later parts of the pipeline actually need to be run.

Logs of previous major updates

Minor QoL updates & bug fixes, 2026.01.06: Build 3.1.1

• config.yaml, Snakefile and various scripts updated to allow user to specify the MAG bin size and associated MAG file extension
• Fixed a minor bug in the Snakefile where the per-Cohort BGC prediction inputs were incorrectly specified to be the per-Sample binning outputs

Wildcard patch & minor updates, 2025.10.22: Build 3.1.0

• Patch wildcard usage: Resolved issues relating to wildcard inconsistency that was causing the pipeline to occasionally crash
  • I received help from StackOverflow contributors in this process:
    • Second Question (which enabled me to track down the true source of the errors): https://stackoverflow.com/questions/79792470/how-can-i-use-wildcard-paths-from-a-pandas-dataframe-as-required-rule-inputs-and/79792740#79792740
    • First Question: https://stackoverflow.com/questions/79790165/snakemake-wildcard-issues-using-touch-to-avoid-premature-pipeline-progression
• Quality of life updates:
  • GECCO: writing out GFF results files by default for easier parsing
  • Working directory cleaning: The workflow/scripts/snakemake_cleanup.sh script can now be used to clean the SPOT-BGC working directory prior to rerunning the pipeline (i.e., removing old results & log files, deleting the files in resources/RawData/ etc.)
  • Additions to config.yaml file: MetaSPAdes timeout time, domain selection for CheckM taxonomy

Second major update, 2025.08.02: Build 3.0.0

• Ensured SLURM HPC environmental compliance & functionality.
  • A change made for version 2.0.0 has been rolled back, and the snakemake_setup.sh script must once again be run by the user prior to running the SPOT-BGC pipeline. This was done due to circumvent complications with running Snakemake in an HPC environment.
• The configuration file at config/config.yaml includes some additional parameters that can be modified from the command line beyond program thread counts, such as the memory allocation to BBnorm.
• Usage notes/disclaimers:
  • At this stage, most users will need to modify the config.yaml file manually in order to change the reference genome, as well as modify the Snakefile or bash scripts manually in order to change the settings/arguments of the various programs.

Minor update, 2025.01.18: Build 2.0.1

• Hotfix for AntiSMASH functionality in the case of multiple MAGs predicted from an assembly
• MetaBat minimum MAG size has been lowered from program default, and the setting has been added to the config.yaml file for easier user modification

First major update, 2025.01.17: Build 2.0.0

• Kraken2 has replaced BLASTN for the human contig elimination step, to improve speed
• A safety measure has been added to the per-sample MetaSPAdes/SPAdes assembly: If a sample does not successfully assembly within 6 hours, the process will terminate & MEGAHIT will be used to assemble the sample, instead
• Taxonomic profiling is now done with CheckM
• The threads allocated for most rules can be modified from the command line with --config thread_{PROGRAM}={NUMBER}. Please see the configuration file at config/config.yaml for the full list of programs whose thread count can be modified in this manner.
• If a cohort contains both SE and PE reads, the per-cohort assembly will be able to use all three together.
• The initial files needed to run the entire pipeline are now generated as part of the workflow.
• Usage notes/disclaimers:
  • At this stage, most users will need to modify the config.yaml file manually in order to change the reference genome, as well as modify the Snakefile or bash scripts manually in order to change the settings/arguments of the various programs.
  • The workflow has not yet been tested on a SLURM HPC environment, only on a server.

Initial publication, 2024.11.12: Build 1.0.0-beta

• The SPOT-BGC pipeline is functional, with the human genome as the reference genome.
• Note that while it has been successfully run for the analysis it was designed for, extensive testing of the pipeline has not been carried out.
• At this stage, most users will need to modify the config.yaml file manually in order to change the reference genome, as well as modify the Snakefile manually in order to change the settings/arguments of the various programs.
• Note that the containers used to run the programs are not included in this repository, but should be created by the user as described below.

Footnotes

1. Note that the above is a change from SPOT-BGC v. 2.0.0 in order to comply with SLURM processing requirements. Previously, the running of the snakemake_setup.sh script was included in the Snakefile, but this raised errors when run in a SLURM environment. ↩

2. Note that the email feature requires the mail (https://linux.die.net/man/1/mail) Linux program to be installed. ↩