File "/opt/conda/lib/python3.10/site-packages/torch/cuda/streams.py", line 214, in elapsed_time
return super().elapsed_time(end_event)
RuntimeError: Both events must be recorded before calculating elapsed time.
when training stops before commpeling kimg round and resume from stop point
File "/opt/conda/lib/python3.10/site-packages/torch/cuda/streams.py", line 214, in elapsed_time
return super().elapsed_time(end_event)
RuntimeError: Both events must be recorded before calculating elapsed time.
when training stops before commpeling kimg round and resume from stop point