Custom

.gitignore

@@ -185,4 +185,5 @@ oryza-terrier.final.TEs.csv
 drosophila-terrier.final.TEs.csv
 lightning_logs/
 comparison-test-data/get-rules.ipynb
-compare.sh
+compare.sh
+drosophila.final.TEs.fa.gz

README.rst

@@ -78,23 +78,29 @@ To run inference on a FASTA file, run this command:
 
 .. code-block:: bash
 
-    terrier --file INPUT.fa --output-fasta OUTPUT.fa
+    terrier --input INPUT.fa --output-fasta OUTPUT.fa
 
 That will add the classification to after the sequence ID in the `OUTPUT.fa` FASTA file.
 
 If you want to save the probabilities for all classes run this:
 
 .. code-block:: bash
 
-    terrier --file INPUT.fa --output-csv OUTPUT.csv
+    terrier --input INPUT.fa --output-csv OUTPUT.csv
 
 The columns will be the probability of each classification and the rows correspond to each sequence in ``INPUT.fa``.
 
+You can also use a URL as the input:
+
+.. code-block:: bash
+
+    terrier --input https://example.com/INPUT.fasta.gz --output-fasta OUTPUT.fa
+
 If you want to output a visualization of the prediction probabilities:
 
 .. code-block:: bash
 
-    terrier --file INPUT.fa --image-dir OUTPUT-IMAGES/
+    terrier --input INPUT.fa --image-dir OUTPUT-IMAGES/
 
 The outputs for the above can be combined together. For more options run 
 

docs/preprocessing.rst

@@ -28,7 +28,7 @@ These two files can be generated from the Repbase database using the ``terrier-t
 
 .. code-block:: bash
 
-    terrier-tools preprocess --repbase $REPBASE_DIR --seqbank $REPBASE_DIR/Repbase-seqbank.sb --seqtree $REPBASE_DIR/Repbase-seqtree.st
+    terrier-tools preprocess --input $REPBASE_DIR --seqbank $REPBASE_DIR/Repbase-seqbank.sb --seqtree $REPBASE_DIR/Repbase-seqtree.st
 
 This will create a SeqBank file called ``Repbase-seqbank.sb`` and a SeqTree files called ``Repbase-seqtree.st`` and place them the ``$REPBASE_DIR``. 
 
@@ -46,13 +46,13 @@ Now you are ready to train Terrier using the SeqBank and SeqTree files you have
 Optional: Display the SeqTree
 ------------------------------
 
-You can list the number of accessions for each node in the SeqTree file with this command:
+You can list the number of sequences for each node in the SeqTree file with this command:
 
 .. code-block:: bash
 
     seqtree render $REPBASE_DIR/Repbase-seqtree.st --print --count
 
-That will output a tree with the number of accessions like this:
+That will output a tree with the number of sequences like this:
 
 .. code-block:: text
 
@@ -124,4 +124,33 @@ This will create an HTML file with the Sunburst chart of the SeqTree like this:
 
 You can open the HTML file in a browser to view the chart.
 
-You can also output the SeqTree with a .png, .svg, or .pdf extension by changing the extension of the output file.
+You can also output the SeqTree with a .png, .svg, or .pdf extension by changing the extension of the output file.
+
+Custom Datasets
+----------------
+
+You can create a custom repeat library in FASTA format, with the classification of each sequence like this:
+
+.. code-block:: text
+
+    >SeqID#DNA/Academ
+    ACTGACTGACTG...
+
+Or with the classification separated with a tab character like this:
+
+.. code-block:: text
+
+    >SeqID	LTR/Caulimovirus
+    ACTGACTGACTG...
+
+Then preprocess like this:
+
+.. code-block:: bash
+
+        terrier-tools preprocess --input custom.fasta --seqbank custom-seqbank.sb --seqtree custom-seqtree.st
+
+You can include Repbase with your custom dataset like this:
+
+.. code-block:: bash
+
+        terrier-tools preprocess --input $REPBASE_DIR --input custom.fasta --seqbank combined-seqbank.sb --seqtree combined-seqtree.st

docs/reproduction.rst

@@ -15,15 +15,12 @@ Fruit Fly Genome
 
 Bickmann et al. (2023) provide Transposable Elements (TE) models of a fruit-fly genome. Download it with the following command:
 
-.. code-block:: bash
-
-   wget https://raw.githubusercontent.com/IOB-Muenster/TEclass2/refs/heads/main/tests/Drosophila_melanogaster.fasta
 
 Run inference using Terrier like this:
 
 .. code-block:: bash
 
-   terrier --file Drosophila_melanogaster.fasta \
+   terrier --input https://raw.githubusercontent.com/IOB-Muenster/TEclass2/refs/heads/main/tests/drosophila.final.TEs.fa \
         --output-csv drosophila-terrier.final.TEs.csv \
         --min-length 0 \
         --threshold 0
@@ -108,15 +105,12 @@ Rice Genome
 
 Bickmann et al. (2023) also provide Transposable Elements (TE) models of a rice genome. Download it with the following command:
 
-.. code-block:: bash
-
-   wget https://raw.githubusercontent.com/IOB-Muenster/TEclass2/refs/heads/main/tests/Oryza_sativa.fasta
-
 Run inference using Terrier like this:
 
 .. code-block:: bash
 
-   terrier --file Oryza_sativa.fasta --output-csv oryza-terrier.final.TEs.csv --threshold 0
+   terrier --intput https://raw.githubusercontent.com/IOB-Muenster/TEclass2/refs/heads/main/tests/oryza.final.TEs.fa \
+       --output-csv oryza-terrier.final.TEs.csv --threshold 0
 
 .. note::
 
@@ -203,7 +197,7 @@ Run inference using Terrier like this:
 
 .. code-block:: bash
 
-   terrier --file Homo_sapiens.fasta --output-csv Terrier-human.csv --threshold 0
+   terrier --input Homo_sapiens.fasta --output-csv Terrier-human.csv --threshold 0
 
 
 Now evaluate the results with the following command:
@@ -243,7 +237,7 @@ Run inference using Terrier like this:
 
 .. code-block:: bash
 
-   terrier --file Mus_musculus.fasta --output-csv Terrier-mouse.csv --threshold 0
+   terrier --input Mus_musculus.fasta --output-csv Terrier-mouse.csv --threshold 0
 
 
 Now evaluate the results with the following command:

docs/training.rst

@@ -7,7 +7,7 @@ After performing the instructions on the :ref:`preprocessing:Preprocessing` page
 
 To train Terrier, you will need to use the `terrier-tools` CLI utility.
 
-To use the same hyperparameters as in the main release of Terrier, you can run the following command:
+To train with the default settings of Terrier, you can run the following command:
 
 .. code-block:: bash
 
@@ -17,6 +17,19 @@ To use the same hyperparameters as in the main release of Terrier, you can run t
         --seqtree $SEQTREE \
         --seqbank $SEQBANK
 
+If you want to train using the pretrained Terrier model weights as a starting point, you can add the ``--pretrained`` flag:
+
+.. code-block:: bash
+
+    SEQBANK=$REPBASE_DIR/Repbase-seqbank.sb
+    SEQTREE=$REPBASE_DIR/Repbase-seqtree.st
+    terrier-tools train         \
+        --seqtree $SEQTREE         \
+        --seqbank $SEQBANK         \
+        --pretrained default
+
+You can replace the word ``default`` with a path to a checkpoint file if you have one or to a URL to a checkpoint file.
+
 You can see other command-line options by running:
 
 .. code-block:: bash

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "bio-terrier"
-version = "0.3.3"
+version = "0.3.4"
 description = "Transposable Element Repeat Result classifIER"
 authors = ["Robert Turnbull <robert.turnbull@unimelb.edu.au>"]
 license = "Apache-2.0"
@@ -26,7 +26,7 @@ python = ">=3.10,<3.13"
 numpy = "<2.0.0"
 pandas = "<=2.2.0"
 h5py = ">=3.8.0"
-bio-corgi = ">=0.5.0a2"
+bio-corgi = ">=0.5.1"
 pyfastx = ">=1.1.0"
 toml = ">=0.10.2"
 

terrier/apps.py

@@ -219,47 +219,39 @@ def get_prediction_probability(row):
     def checkpoint(self, checkpoint:Path=None) -> str:
         return checkpoint or "https://github.com/rbturnbull/terrier/releases/download/v0.2.0/terrier-0.2.0.ckpt"
 
-    @ta.tool
-    def create_repeatmasker_seqtree(self, output:Path, repbase:Path, label_smoothing:float=0.0, gamma:float=0.0, partitions:int=5):
-        from .repeatmasker import create_repeatmasker_seqtree
-        return create_repeatmasker_seqtree(
-            output=output,
-            repbase=repbase,
-            label_smoothing=label_smoothing,
-            gamma=gamma,
-            partitions=partitions,
-        )
-
     @ta.tool
     def preprocess(
         self, 
-        repbase:Path=ta.Param(..., help="The path to the RepBase fasta directory."), 
-        seqbank:Path=ta.Param(..., help="The path to save the new SeqBank file."), 
-        seqtree:Path=ta.Param(..., help="The path to save the new SeqTree file."), 
+        input:list[str]=ta.Param(..., help="The path to a FASTA file, URL to a FASTA file, multiple FASTA files or a directory of FASTA files (e.g. the RepBase FASTA directory)"),
+        seqbank:Path=ta.Param(None, help="The path to save the new SeqBank file."), 
+        seqtree:Path=ta.Param(None, help="The path to save the new SeqTree file."), 
         label_smoothing:float=0.0, 
         gamma:float=0.0, 
         partitions:int=5,
     ):
         from seqbank import SeqBank
         from .repeatmasker import create_repeatmasker_seqtree
 
-        seqbank = SeqBank(path=seqbank, write=True)
-        assert repbase is not None
-        repbase = Path(repbase)
-        assert repbase.exists()
+        assert seqbank or seqtree, "You must provide either a --seqbank or --seqtree output path (usually both)."
 
-        # Create the seqbank from the FASTA files with .ref extension
-        files = list(repbase.glob('*.ref'))
-        seqbank.add_files(files, format="fasta")
+        fasta_paths = self.find_fasta_paths(input)
+
+        # Create the seqbank from the FASTA files
+        if seqbank:
+            seqbank = SeqBank(path=seqbank, write=True)
+            seqbank.add_files(fasta_paths, format="fasta")
 
         # Create the seqtree
-        return create_repeatmasker_seqtree(
-            output=seqtree,
-            repbase=repbase,
-            label_smoothing=label_smoothing,
-            gamma=gamma,
-            partitions=partitions,
-        )
+        if seqtree:
+            seqtree_path = Path(seqtree)
+            seqtree = create_repeatmasker_seqtree(
+                fasta_paths=fasta_paths,
+                label_smoothing=label_smoothing,
+                gamma=gamma,
+                partitions=partitions,
+            )
+            seqtree.save(seqtree_path)
+            seqtree.render(print=1, count=True)
 
     @ta.tool
     def evaluate(