Hi,
I'm running IQ-TREE MPI 3.1.3 on an HPC cluster (SLURM) and have a question regarding checkpointing during the NONREV root test.
IQ-TREE MPI version 3.1.3 for Linux x86 64-bit
Command
iqtree3
-s alignment.fasta
--model-joint NONREV
--root-test
-zb 1000
-au
-te input.treefile
-pre output_prefix
The analysis proceeds normally through parameter optimisation:
Optimal log-likelihood: ...
Parameters optimization took ...
BEST SCORE FOUND : ...
Testing root positions...
After entering the "Testing root positions..." stage, the log continues to report improved roots, for example:
Better root: -534590.228
Better root: -534590.025
Better root: -534588.621
...
Better root: -534586.289
However, I have noticed the following behaviour:
The checkpoint file (.ckp.gz) is updated normally before the root test.
Once the root test begins, the checkpoint file modification time no longer changes, even after several days of computation.
The log also provides no indication of overall progress through the root search other than printing a new "Better root" when one is found.
Because I am running on a cluster with a maximum walltime of 4 days, this is important. If the job reaches the walltime during the root test, I need to know whether the root search can actually resume from the checkpoint.
The documentation states:
--cptime NUM Specify the minimum checkpoint time interval in seconds.
Based on this, I expected the checkpoint file to continue updating throughout the run.
My questions are:
Is it expected that the checkpoint file is not updated during the --root-test stage?
If a job is interrupted during "Testing root positions...", should restarting with the same command resume the root search from where it stopped, or does it restart the root-testing phase from the beginning?
Is there any option to force checkpoint updates during the root-test stage?
Thank you for your help.
Hi,
I'm running IQ-TREE MPI 3.1.3 on an HPC cluster (SLURM) and have a question regarding checkpointing during the NONREV root test.
IQ-TREE MPI version 3.1.3 for Linux x86 64-bit
Command
iqtree3
-s alignment.fasta
--model-joint NONREV
--root-test
-zb 1000
-au
-te input.treefile
-pre output_prefix
The analysis proceeds normally through parameter optimisation:
Optimal log-likelihood: ...
Parameters optimization took ...
BEST SCORE FOUND : ...
Testing root positions...
After entering the "Testing root positions..." stage, the log continues to report improved roots, for example:
Better root: -534590.228
Better root: -534590.025
Better root: -534588.621
...
Better root: -534586.289
However, I have noticed the following behaviour:
The checkpoint file (.ckp.gz) is updated normally before the root test.
Once the root test begins, the checkpoint file modification time no longer changes, even after several days of computation.
The log also provides no indication of overall progress through the root search other than printing a new "Better root" when one is found.
Because I am running on a cluster with a maximum walltime of 4 days, this is important. If the job reaches the walltime during the root test, I need to know whether the root search can actually resume from the checkpoint.
The documentation states:
--cptime NUM Specify the minimum checkpoint time interval in seconds.
Based on this, I expected the checkpoint file to continue updating throughout the run.
My questions are:
Is it expected that the checkpoint file is not updated during the --root-test stage?
If a job is interrupted during "Testing root positions...", should restarting with the same command resume the root search from where it stopped, or does it restart the root-testing phase from the beginning?
Is there any option to force checkpoint updates during the root-test stage?
Thank you for your help.