When I trained the R-GCN model with dataset 'FB-Toutanova', using setting 'gcn_basis.exp', I used nvtop to monitor the GPU utilization and found that the GPU utilization was mostly 0% during the training process, and would not exceed 50%. Instead, memory usage is staggering, with frequent OOMs on large datasets. The training process is very slow, taking close to a day for 10,000 iterations. Perhaps there are many work for performance improvements.
When I trained the R-GCN model with dataset 'FB-Toutanova', using setting 'gcn_basis.exp', I used nvtop to monitor the GPU utilization and found that the GPU utilization was mostly 0% during the training process, and would not exceed 50%. Instead, memory usage is staggering, with frequent OOMs on large datasets. The training process is very slow, taking close to a day for 10,000 iterations. Perhaps there are many work for performance improvements.