RuntimeError("The program has been running off.")

[kizunasunhy]

In Epoch:16/Step:17296, got best dev performance:
Seq2SeqSpanMetric: f=0.0, rec=0.0, pre=0.0, em=0.1836
....
File "/root/work/BARTNER/model/callbacks.py", line 107, in on_valid_end
raise RuntimeError("The program has been running off.")
RuntimeError: The program has been running off.

Does this mean the model stops by itself because of convergence failure?
What can I do now? To train it again?
Thanks

[yhcc]

Just train it again, it is because of the convergence failure.

[summer-la]

Did the results converge after re-running? @kizunasunhy I'm having the same issue now.