segment void takes a long time

[Phaips]

Hey Mart,

Testing the amazing easymode and when running easymode segment void --data in.mrc --output out.mrc --gpu 0 --tta 2 it takes quite a long time to finish running, even when ran on multiple --gpu's.
On the website I see:

The model is relatively fast. It works at 30 Å/px and most tomograms are segmented in a few seconds.

but it's getting scaled to 20 Å/px and starting the interference step alone takes >30 minutes . Maybe I am doing something wrong i.e. I assume --tta is not really needed so I set it to 2 and also the model is 3D not sure 2D would make a difference here? and is there a flag for manually scaling the data to 30 Å/px to make it faster?

The void outputs are great nevertheless! :)

Thanks for your help!
Cheers

full output:

void: using 3D model
easymode segment
feature: void
data_patterns: ['../Position_1mrc']
output_directory: 1_bmask.mrc
output_format: int8
gpus: [0]
tta: 2
overwrite: False
batch_size: 1

Found 1 tomograms to segment. 

Using model: /scicore/home/engel0006/vysamu91/easymode/void.h5, inference at 20.0 Å/px. 

GPU 0 - loading model (/scicore/home/engel0006/vysamu91/easymode/void.h5).
GPU 0 - starting inference.
1/1 (on GPU 0) - void - Position_1.mrc - etc 00:00:00 (34:50 per tomo)

[mgflast]

Hey Philippe,

Try running it again but without the '-gpu 0' argument. My guess is your tensorflow/Python/CUDA combination isn't working but because of the '-gpu 0' argument the auto-GPU-detection doesn't get to fail, and you're not being shown an error.

The 20 vs 30 Å/px thing: for some models there is both a 2D and a 3D version. The 30 A/px would be the 2D one, which is the better one, but you're being served the 3D model because it turns out I set a flag wrong. For now you can run the same command again but with the explicit '--2d' flag. I'll be back in the office properly next week to fix some of these annoying little things, sorry :)

You're right that --tta is not really needed. Outputs can get a bit better with higher values (same concept as in Membrain-seg) but you can run with --tta 1 if you're in a hurry.

Main problem I think is going to be your tensorflow/Python/CUDA combination - do you have a module system with a CUDA 11.2 / 11.8 / 12.4 version that you could load?

[Phaips]

Thanks Mart, you were exactly right.

The issue was that TensorFlow was CUDA-enabled, but it could not find the CUDA/cuDNN libraries from my conda env. The env has tensorflow 2.9.0, cudatoolkit 11.2.2, and cudnn 8.1.0, but $LD_LIBRARY_PATH did not include $CONDA_PREFIX/lib.

After that, TensorFlow sees the GPU and easymode segment void --2d runs properly. Runtime is now ~2:30 per tomo instead of ~35 min.

One minor note: even when I pass --tta 1 it reports tta: 4 (forced), and also it is still at 20 apix but speed is now fine.

Thanks for the help!

I quickly ran it like this:

export LD_LIBRARY_PATH="$CONDA_PREFIX/lib:$LD_LIBRARY_PATH"
export CUDA_VISIBLE_DEVICES=1

easymode segment void \
  --data Position_1.mrc \
  --output 1_bmask.mrc \
  --tta 1 \
  --2d
void: using 2D model
easymode segment
model_path: void
input_data: ['Position_1.mrc']
output_directory: 1_bmask.mrc
gpus: 0
tta: 4 (forced)
overwrite: False
ais_2d_nets: True

Found 1 tomograms to segment.

ais segment -m /scicore/home/engel0006/vysamu91/easymode/void.scnm -apix 20.0 -d ../warp_tiltseries_cm/reconstruction_miss/Position_1_9.68Apx.mrc -ou 1_bmask.mrc -tta 4 -p 1 --overwrite 0 -gpu 0 --stride 1 --use-depth 1.0

GPU 0 - starting inference with 2D model 'void' at 20.0 A/px (n_workers = 16, tta = 4).
1/1 (GPU 0) - void.scnm - Position_1_9.68Apx.mrc - eta 00:00:00 (02:30 this tomo)