more features to align interleaved files (with parallelization) by DiegoBrambilla · Pull Request #25 · erikrikarddaniel/biomakefiles

DiegoBrambilla · 2020-11-16T11:19:17Z

Hi,
There was no feature in makefile.diamond for running DIAMOND against specific databases other than RefSeq, so I created two new makefiles, makefile.diamond_bacmet and makefile.diamond_deeparg.
I have also developed rules in makefile.bowtie2 and makefile.diamond to run interleaved files.

Alsio, I have added separate recipies in makefile.blast to run both BLASTn and BLASTx with GNU Parallel.

Please check if changes conform to your standards.

erikrikarddaniel

I think it would be better to keep the python scripts in their own repositories, but document how to install in relevant makefiles.
I have refrained from using the .fasta file ending in rules in preference for the more specific .fna, .faa and some others. I think it would be good to stick to that and change .fasta to .fna.
It looks like you deleted the header from makefile.diamond. There are more things in the diff for this one that looks like large chunks of things were deleted.
You can't have an include in a library makefile, especially not one which has an absolute path! Instead, write in the documentation that one has to include some other makefiles as well.
I'm not sure about having a diamond_bacmet and diamond_deeparg. It would fit better to have a makefile.bacmet and makefile.deeparg for fetching data etc. and then just continue with specifying the path to databases in individual Makefiles. (Here, you're also including makefiles...)

…met and makefile.trimgalore

…iles

…files

DiegoBrambilla · 2021-12-08T15:10:25Z

Hi,
Back after more than one year, addressing your previous requests:

I think it would be better to keep the python scripts in their own repositories, but document how to install in relevant makefiles.

The python scripts were removed from bin/. Instead, I have created dedicated gists for bacmet.py and deeparg.py, consistent with other makefiles, see for example the target interleave_all_fastq.gzs in makefile.misc. Documentation and specific make rules have been created in the dedicated makefiles to fetch such gists (see below).

I have refrained from using the .fasta file ending in rules in preference for the more specific .fna, .faa and some others. I think it would be good to stick to that and change .fasta to .fna.

Done. Now rules and targets in makefile.bacmet and makefile.deeparg refer to either .fna or .faa files

It looks like you deleted the header from makefile.diamond. There are more things in the diff for this one that looks like large chunks of things were deleted.

The aforementioned changes were reverted. Now the header of makefile.diamond is back to the way it was before.

You can't have an include in a library makefile, especially not one which has an absolute path! Instead, write in the documentation that one has to include some other makefiles as well.

Done. include statements no longer present.

I'm not sure about having a diamond_bacmet and diamond_deeparg. It would fit better to have a makefile.bacmet and makefile.deeparg for fetching data etc. and then just continue with specifying the path to databases in individual Makefiles. (Here, you're also including makefiles...)

makefile.bacmet and makefile.deeparg have been created. They have rules to fetch databases and parse DIAMOND output.
Mosty notably, they report some rules already present in makefile.diamond, namely those for the database formatting into .dmnd and the DIAMOND annotation.
The choice behind such redundancy was to generate output files that were referring to a functional database (either diamond or bacmet) through their names.
Also, you need to include only makefile.bacmet or makefile.deeparg this way if you are only interested into performing a functional annotation against BacMet or Deeparg.

erikrikarddaniel

Thanks for the contribution!
Many targets are duplicated and could go into e.g. makefile.misc and, I think, makefile.diamond. You can, and should, include more than one makefile and it's nothing wrong with requiring inclusion of a dependent makefile.

erikrikarddaniel · 2021-12-08T15:29:11Z

 NCBI_BLAST_DB_PATH  = YOU NEED TO SET THE NCBI_BLAST_DB_PATH VARIABLE (see makefile.blast)
 BLAST_PROTEIN_DB    = YOU NEED TO SET THE BLAST_PROTEIN_DB VARIABLE (see makefile.blast)
 BLAST_NUCLEOTIDE_DB = YOU NEED TO SET THE BLAST_NUCLEOTIDE_DB VARIABLE (see makefile.blast)
+BLAST_NUCLEOTIDE_DB_PATH = .


Why do you need this (and the next)? Can't you use the BLAST_NUCLEOTIDE_DB variable?

Agreed, will change accordingly.

erikrikarddaniel · 2021-12-08T15:30:20Z

 # *** Targets ***

 # UNTESTED and assumes you've also included makefile.misc that can convert a fastq.gz to an fna:
 all_mega.blastn6s: $(subst .fna,.mega.blastn6.tsv.gz,$(wildcard *.fastq.gz))


This will not work. You're trying to replace .fna in a list of files that ends with .fastq.gz.

Good point, better stick with $(wildcard *.fna)

erikrikarddaniel · 2021-12-08T15:30:43Z

+# BLASTn-specific macros
+#
+# *Don't redefine!*
+MAKECALL_BLASTn_VERSION     = echo "$$(date +"%Y%m%d %H:%M:%S"): $@ was made with $$(blastn -version|grep 'blastn:')" > $@.makecall


I prefer only capitals in macro names.

I understand, consistency is important.

erikrikarddaniel · 2021-12-08T15:31:26Z

+# full path of the database file, without the ".dmnd" suffix.
+DIAMOND_DB =
+
+# deeparg.py specific macro: define minimum a.a. alignemnt length threshold


Could be documented better.

erikrikarddaniel · 2021-12-08T15:33:03Z

+# IMPORTANT: databases are updated recurrently, so always check the availability of the http
+
+# Download HTTPS URL for the bacmet.py script, necessary for the downstream analyses
+HTTPS = https://gist.github.com/08ab264f0ea127cc242d99bc3844c743.git 


It would be better if you could package your script in Conda and instruct users to install that way. See e.g. https://github.com/erikrikarddaniel/hmmrank

Agreed, but I am short on time right now. Can we make the github gists do for the time being, please?

Fine, but I'd prefer if the instructions are to put it in ~/bin rather than in the working directory.

No problem, thanks.

erikrikarddaniel · 2021-12-08T15:42:32Z

 	bowtie2 $(BOWTIE_ALIGN_OPTS) -x $(BOWTIE_INDEX) -1 $(word 1,$^) -2 $(word 2,$^) 2> $@.err | samtools view -Sb | samtools sort > $@
 	@echo "`date +"%Y%m%d %H:%M:%S"`: DONE" >> $@.makecall

+%.pe.bowtie2.sorted.bam: %.r1.fastq %.r2.fastq


You might have a use case for this, but I avoid working with unzipped fastq files as much as I can and since bowtie2 handles zipped files, I don't see why one needs this. Better to zip any files that are not zipped before starting the mapping.

Rules and targets with unzipped FASTQ files will be removed.

erikrikarddaniel · 2021-12-08T15:43:34Z

+ALENGTH = 37.5
+
+# Set this to suffixes corresponding to the types of files you wish to fetch with some
+# of the t$(DIAMOND_DB)ets.


erikrikarddaniel · 2021-12-08T15:44:52Z

+# and link the file "features.fasta" into the DIAMOND work directory
+# 2) install deeparg, e.g. via conda
+# $ conda create -n deeparg_env python=2.7.18
+# $ source activate deeparg_env


Should be conda activate deeparg_env.

erikrikarddaniel · 2021-12-08T15:45:39Z

+# *** misc targets ***
+
+# compress all .fastq files in the work directory.
+compress_all_fastqs: $(subst .fastq,.fastq.gz, $(wildcard *.fastq))


Typical target for makefile.misc.

Removed: better left on makefile.misc

erikrikarddaniel · 2021-12-08T15:46:20Z

+# Dependencies will automatically be unzipped if not present in unzipped format
+# thanks to the pattern above. Unzipped files will be deleted after
+# interleaving.
+interleave_all_fastq.gzs: $(subst .R1.fastq.gz,.intlv.fastq.gz,$(wildcard *.R1.fastq.gz))


Also makefile.misc. (Also in another file I reviewed above. Define every target only once.)

Same as above reply.

…kefile.diamond

DiegoBrambilla and others added 6 commits November 16, 2020 11:28

more recipies for interleaved files with Bowtie2 and DIAMOND

0f01769

Merge https://github.com/erikrikarddaniel/biomakefiles

6678b07

more recipies for parallel BLASTn & BLASTx

484050e

adding GNU Parallel recomendations in makefile.last

b825119

restoring makefile.diamond to version from commit 5ff8ec

dc95d89

parsing makefile.blast

fc3d740

erikrikarddaniel reviewed Nov 19, 2020

View reviewed changes

DiegoBrambilla added 4 commits December 8, 2021 15:05

improving documentation and syntax for makefile.deeparg, makefile.bac…

17b4776

…met and makefile.trimgalore

Merge branch 'master' of https://github.com/DiegoBrambilla/biomakefiles

887c356

Merge branch 'master' of https://github.com/erikrikarddaniel/biomakef…

0bab141

…iles

merging with erikrikarddaniel/biomakefiles master, removingredundant …

8243fd8

…files

Update makefile.bacmet

1f7033b

erikrikarddaniel reviewed Dec 8, 2021

View reviewed changes

DiegoBrambilla added 5 commits December 9, 2021 15:29

fixed macros and targets of makefile.blast

d809e89

moved DIAMOND targets from makefile.deeparg and makefile.bacmet to ma…

892efc2

…kefile.diamond

removed targets with unzipped FASTQs from makefile.bowtie2

f0a58d3

instructions on how to fetch DeepARG and BacMet are in /bin

21bc41b

resolving conflicts with makefile.bacmet

8073dca

Conversation

DiegoBrambilla commented Nov 16, 2020

Uh oh!

erikrikarddaniel left a comment

Choose a reason for hiding this comment

Uh oh!

DiegoBrambilla commented Dec 8, 2021

Uh oh!

erikrikarddaniel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants