GPU Utilization during 2DTM（2D template matching） Running

[goodhaibin]

Dear cisTEM Team,

I am encountering an issue regarding GPU utilization while running 2D Template Matching (2DTM) on a dataset of 3,000 micrographs. Below is the run profile we have configured:
CUDA_VISIBLE_DEVICES=0 /data03/Lijing/cisTEM_pvalue_250530/2dtm_postprocess_combined_release/je_dev3/$command
CUDA_VISIBLE_DEVICES=1 /data03/Lijing/cisTEM_pvalue_250530/2dtm_postprocess_combined_release/je_dev3/$command
CUDA_VISIBLE_DEVICES=2 /data03/Lijing/cisTEM_pvalue_250530/2dtm_postprocess_combined_release/je_dev3/$command
CUDA_VISIBLE_DEVICES=3 /data03/Lijing/cisTEM_pvalue_250530/2dtm_postprocess_combined_release/je_dev3/$command

All four GPU configurations mentioned above were entered into the 'Command to Run' field，as following：

The Problem:
Despite this configuration, nvidia-smi indicates that only a single GPU is actively performing computations. The other assigned GPUs remain idle.

Under this configuration, the total 2DTM runtime reaches over 30,000 hours

Could you please advise if there is an error in our setup or if additional parameters are required to enable multi-GPU parallelization for this process?

Looking forward to your guidance.

[bHimes]

The current config expects only 1 process per gpu with several (2-4 threads is typically more than enough to saturate a gpu.) Sorry, this could be something that is both better documented and i guess we could block on run profile introspection in MatchTemplatePanel.

So you would have four lines in your run profile manager, each with 1 process 2-4 threads and each with a unique cuda visible devices. I realize this is a little clunky and is something on the radar to clean up.

FYI at full sampling 3000 images on 4 gpus is going to take a very very long time. Which version are you using?

edit i see you have 8 gpus, i would suggest using all but even then 3000 images is likely not a realistic experimental design. Happy to help you think through that a bit more.