When running the tuner, the autoscaler triggers a request for the worker pod which requires a GPU. If the cluster doesn't provide a GPU in an hour, the worker pod, that is still waiting, suddenly scales down due to the autoscaler config. However the tuner api still reports the pending trials.
The cluster is the root problem since the tuner becomes unusable if it cannot even allocate a GPU within that hour long window.
When running the tuner, the autoscaler triggers a request for the worker pod which requires a GPU. If the cluster doesn't provide a GPU in an hour, the worker pod, that is still waiting, suddenly scales down due to the autoscaler config. However the tuner api still reports the pending trials.
The cluster is the root problem since the tuner becomes unusable if it cannot even allocate a GPU within that hour long window.