Running Kwola on dockerized ADAE project. The run produced bugs folder and videos, at somepoint this console error was thrown and the run stopped after.
Running kwola again on the run folder produced, this error was looping without stopping.
last log output:
2020-04-26 21:13:43.601079 [15220] TaskProcess: Waiting for input from stdin
2020-04-26 21:13:43.601216 [15220] Running process with following data:
{
"configDir": "kwola_run_014",
"trainingSequenceId": "kmbo",
"trainingStepIndex": 0,
"gpu": null
}
2020-04-26 21:13:43.602209 [15220] Starting Training Step
2020-04-26 21:13:48.182703 [15310] Starting initialization for batch preparation sub process.
2020-04-26 21:13:48.188535 [15310] Starting loading of execution trace weight datas.
2020-04-26 21:13:48.190342 [15310] Finished loading of weight datas for 0 execution traces.
2020-04-26 21:13:48.190381 [15310] Finished initialization for batch preparation sub process.
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
2020-04-26 21:13:48.442085 [15310] Resetting batch prep process pool. Cache full state. New workers: 1
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
2020-04-26 21:13:48.465900 [15309] Starting initialization for batch preparation sub process.
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
2020-04-26 21:13:48.503050 [15309] Starting loading of execution trace weight datas.
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
Running Kwola on dockerized ADAE project. The run produced bugs folder and videos, at somepoint this console error was thrown and the run stopped after.
Running kwola again on the run folder produced, this error was looping without stopping.
last log output:
2020-04-26 21:13:43.601079 [15220] TaskProcess: Waiting for input from stdin
2020-04-26 21:13:43.601216 [15220] Running process with following data:
{
"configDir": "kwola_run_014",
"trainingSequenceId": "kmbo",
"trainingStepIndex": 0,
"gpu": null
}
2020-04-26 21:13:43.602209 [15220] Starting Training Step
2020-04-26 21:13:48.182703 [15310] Starting initialization for batch preparation sub process.
2020-04-26 21:13:48.188535 [15310] Starting loading of execution trace weight datas.
2020-04-26 21:13:48.190342 [15310] Finished loading of weight datas for 0 execution traces.
2020-04-26 21:13:48.190381 [15310] Finished initialization for batch preparation sub process.
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
2020-04-26 21:13:48.442085 [15310] Resetting batch prep process pool. Cache full state. New workers: 1
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
2020-04-26 21:13:48.465900 [15309] Starting initialization for batch preparation sub process.
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
2020-04-26 21:13:48.503050 [15309] Starting loading of execution trace weight datas.
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue