CUDA kernel failed : no kernel image is available for execution on the device in clusters.

[arthurlirui]

set up the whole virtual environment in clusters, GPU is Nvida v100. But I receive this error,

CUDA kernel failed : no kernel image is available for execution on the device
void aabb_intersect_point_kernel_wrapper(int, int, int, float, int, const float*, const float*, const float*, int*, float*, float*) at L:371 in fairnr/clib/src/intersect_gpu.cu
CUDA kernel failed : no kernel image is available for execution on the device
void aabb_intersect_point_kernel_wrapper(int, int, int, float, int, const float*, const float*, const float*, int*, float*, float*) at L:371 in fairnr/clib/src/intersect_gpu.cu
Traceback (most recent call last):
  File "train.py", line 20, in <module>
    cli_main()
  File "/ibex/scratch/lir0b/NSVF/fairnr_cli/train.py", line 356, in cli_main
    nprocs=torch.cuda.device_count(),
  File "/home/lir0b/.conda/envs/nsvf/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/home/lir0b/.conda/envs/nsvf/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 112, in join
    (error_index, exitcode)
Exception: process 0 terminated with exit code 255

scripts are:

#!/bin/bash
#SBATCH -N 1
#SBATCH --partition=batch
#SBATCH -J table11_d4
#SBATCH -o table11_d4.%J.out
#SBATCH -e table11_d4.%J.err
#SBATCH --mail-user=rui.li@kaust.edu.sa
#SBATCH --mail-type=ALL
#SBATCH --time=23:00:00
#SBATCH --mem=40G
#SBATCH --gres=gpu:v100:2
#SBATCH --cpus-per-task=6
#SBATCH --constraint=[gpu]

#run the application:
#DATASET=./Synthetic_NSVF/gun
#SAVE=./exp/gun

#export PATH=$PATH:/ibex/scratch/lir0b/NSVF/fairnr/clib
#export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/ibex/scratch/lir0b/NSVF/fairnr/clib
#export LD_LIBRARY_PATH=/ibex/scratch/lir0b/NSVF:$LD_LIBRARY_PATH
#conda activate nsvf
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/ibex/scratch/lir0b/NSVF/fairnr/clib:/ibex/scratch/lir0b/NSVF
export PATH=$PATH:/ibex/scratch/lir0b/NSVF/fairnr/clib:/ibex/scratch/lir0b/NSVF
pip install --editable /ibex/scratch/lir0b/NSVF
python setup.py build_ext --inplace
python -u train.py ./Synthetic_NSVF/table11_d4 \
    --user-dir fairnr \
    --task single_object_rendering \
    --train-views "1..35" --view-resolution "384x512" \
    --max-sentences 1 --view-per-batch 4 --pixel-per-view 2048 \
    --no-preload \
    --sampling-on-mask 1.0 --no-sampling-at-reader \
    --valid-views "35..44" --valid-view-resolution "384x512" \
    --valid-view-per-batch 1 \
    --transparent-background "1.0,1.0,1.0" --background-stop-gradient \
    --arch nsvf_base \
    --initial-boundingbox ./Synthetic_NSVF/table11_d4/bbox.txt \
    --raymarching-stepsize-ratio 0.125 \
    --discrete-regularization \
    --color-weight 128.0 --alpha-weight 1.0 \
    --optimizer "adam" --adam-betas "(0.9, 0.999)" \
    --lr 0.001 --lr-scheduler "polynomial_decay" --total-num-update 150000 \
    --criterion "srn_loss" --clip-norm 0.0 \
    --num-workers 0 \
    --seed 2 \
    --save-interval-updates 500 --max-update 150000 \
    --virtual-epoch-steps 5000 --save-interval 1 \
    --half-voxel-size-at  "5000,25000,75000" \
    --reduce-step-size-at "5000,25000,75000" \
    --pruning-every-steps 2500 \
    --keep-interval-updates 5 --keep-last-epochs 5 \
    --log-format simple --log-interval 1 \
    --save-dir ./exp/table11_d4 \
    --tensorboard-logdir ./exp/table11_d4/tensorboard \
    | tee -a ./exp/table11_d4/train.log

To Reproduce
Steps to reproduce the behavior:

1. setup virtual environment
2. upload script on clusters
3. check error log
4. See error

Expected behavior
normal training

Screenshots

[CUDA kernel failed : no kernel image is available for execution on the device
void svo_intersect_point_kernel_wrapper(int, int, int, float, int, const float*, const float*, const float*, const int*, int*, float*, float*) at L:384 in fairnr/clib/src/intersect_gpu.cu
CUDA kernel failed : no kernel image is available for execution on the device
void svo_intersect_point_kernel_wrapper(int, int, int, float, int, const float*, const float*, const float*, const int*, int*, float*, float*) at L:384 in fairnr/clib/src/intersect_gpu.cu
CUDA kernel failed : no kernel image is available for execution on the device
void svo_intersect_point_kernel_wrapper(int, int, int, float, int, const float*, const float*, const float*, const int*, int*, float*, float*) at L:384 in fairnr/clib/src/intersect_gpu.cu
CUDA kernel failed : no kernel image is available for execution on the device
void svo_intersect_point_kernel_wrapper(int, int, int, float, int, const float*, const float*, const float*, const int*, int*, float*, float*) at L:384 in fairnr/clib/src/intersect_gpu.cu
Traceback (most recent call last):
  File "train.py", line 20, in <module>
    cli_main()
  File "/ibex/scratch/lir0b/NSVF/fairnr_cli/train.py", line 356, in cli_main
    nprocs=torch.cuda.device_count(),
  File "/home/lir0b/.conda/envs/nsvf/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/home/lir0b/.conda/envs/nsvf/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 112, in join
    (error_index, exitcode)
Exception: process 1 terminated with exit code 255](url)

[MultiPath]

Hi, sorry for replying late. Do you still have the issues?

[arthurlirui]

Yes, I try to use the cluster to train model, still has such issue, and what about the training time for a single scene？