Current issues with the Tutorial

[sroet]

Hey @alncat,

I started with walking through the tutorial. These are the issues I ran into:

• Tutorial should tell which subsets of the data are needed from the empiar link: seems like 3. or 6. (Tilt series stacks)
• The scripts for the tutorial that are currently in the google drive should be hosted on github as well
• What is the deal with the subm command? (unclear if needed or not)
• The recon.sh file is very system specific (both in the modules and the actual command) and full with commented out code
• The recon.sh file seems to expect to be ran in the main directory, not the TS_028_Imod subdirectory. But this isn't mentioned anywhere.
• The AreTomo2 command consumes more than 128 GB of RAM and gets killed on my test machine during the Flip volume from xzy view to xyz view. stage
• The AreTomo2 command does not actually update the TS_028_Imod directory, but instead creates a new TS_028_st_Imod directory

Unfortunately, that last point makes it unable for me to complete the tutorial

[alncat]

@sroet Thank you very much for your feedback! I will look into these issues and get back shortly~

[alncat]

@sroet

1. Tutorial should tell which subsets of the data are needed from the empiar link: seems like 3. or 6. (Tilt series stacks)
   We should use the 3. Tilt series stacks and metadata of cryo-electron tomograms acquired with defocus-only (DEF) on S. pombe cryo-FIB lamellae, namely, the defocus-only dataset.
2. The AreTomo2 command consumes more than 128 GB of RAM and gets killed on my test machine during the Flip volume from xzy view to xyz view. stage
   You may drop the --FlipVol 1 argument and store the tomogram as xzy view first, then use clip from imod to perform the flipping. https://bio3d.colorado.edu/imod/doc/man/clip.html , clip has the option flip[xyz]. For example, clip flipyz TS_026_st.mrc TS_026_st.mrc
3. What is the deal with the subm command? (unclear if needed or not)
   subm command can execute imod related scripts generated by aretomo.
   Lastly, sorry about the system dependent script.

[sroet]

The next issue I ran into (will also update the original post, but that will not ping you): Aretomo2 does not update the TS_028_Imod directory. It instead creates a TS_028_st_Imod directory:

$ ls TS_28_st_Imod
newst.com  tilt.com  TS_028_st_st.tlt  TS_028_st_st.xf  TS_028_st_st.xtilt
$ ls TS_028_Imod
newst.com  TS_028_st.ali

so

1. I don't see a .aln file at all (I assume this is a mistype of .ali)
2. I don't know if/how to combine the two directories

[alncat]

@sroet Thank you very much for continuing running and reporting issues!

The extra st suffix might be caused by the name of input tilt series stack TS_028_st.ali, but that's just a filename issue. As for aln, it was the alignment file output by aretomo, and might be in the folder where you are running aretomo.
Actually, the folder TS_28_st_Imod, which is created by aretomo, is self contained. For example, you can create aligned tilt series stack using that folder. You can check the newst.com, see if the inputfile, outputfile and transformfile are properly set,
then just execute subm newst.com, IMOD can then reconstruct aligned tilt series stack! If you have performed tomogram reconstruction using aretomo with its automatic alignment capacity. It is a nice way to get the corresponding aligned tilt series stack for CTF estimation.

However, in this tutorial, in order to compare against the particle picking result deposited by deepict, we chose to reconstruct the tomogram using the prealigned tilt series stack TS_028_st.ali and aretomo without alignment. So we don't need the folder output by aretomo. Just continuing estimate the CTF with TS_028_st.ali in the TS_028_Imod folder would be sufficient.

[alncat]

Hi, @sroet ! I was wondering if you have walked through this tutorial? I have recently updated the training code, it has a new command, train_tomo_hvd, which can be invoked by python -m cryodrgn.commands.train_tomo_hvd . This training script uses horovod for training, which is significantly faster and costs less memory.

[sroet]

hey @alncat, thanks for the question. I have not had time yet to retry the tutorial and the test-machine is currently in use for something with higher priority.
It is still on my list to complete, but I unfortunately can't guarantee any date by which this will be done at the moment

[alncat]

hi @sroet , it is ok! Take your time! The current training code can process 10,000 subtomograms in 10 minutes on 4 v100 gpus. I will update the tutorial as well.

[alncat]

Actually, the current default training code called by dsd train_tomo running at the same speed!

[alncat]

@sroet , kudos to the template matching from PyTOM! We can actually get a set better than human annotation on the 80S ribosome! You may check https://www.biorxiv.org/content/10.1101/2024.06.30.601442 and this figure

[alncat]

@sroet

1. Tutorial should tell which subsets of the data are needed from the empiar link: seems like 3. or 6. (Tilt series stacks)
   We should use the 3. Tilt series stacks and metadata of cryo-electron tomograms acquired with defocus-only (DEF) on S. pombe cryo-FIB lamellae, namely, the defocus-only dataset.
2. The AreTomo2 command consumes more than 128 GB of RAM and gets killed on my test machine during the Flip volume from xzy view to xyz view. stage
   You may drop the --FlipVol 1 argument and store the tomogram as xzy view first, then use clip from imod to perform the flipping. https://bio3d.colorado.edu/imod/doc/man/clip.html , clip has the option flip[xyz]. For example, clip flipyz TS_026_st.mrc TS_026_st.mrc
3. What is the deal with the subm command? (unclear if needed or not)
   subm command can execute imod related scripts generated by aretomo.
   Lastly, sorry about the system dependent script.

The clip flipyz operation will result in the flip of handedness! We should do clip flipz TS_026_st.mrc TS_026_st.mrc following the operation clip flipyz to restore the original handedness! reference: https://bio3d.colorado.edu/imod/doc/man/clip.html