Maintain a pool of machines

[justinsb]

For OpenStack gating "We need to pull nodes from a pool of machines that have been spun up ahead of time for speed."

This sounds like a custom NodeProvisioner implementation. The current implementation is optimized around minimizing CPU hours, by doing load prediction. OpenStack has a different criteria, based on minimizing launch latency. It looks like it should be relatively easy to implement a new algorithm, although perhaps a bit tricky to figure out how to plug it in.

[justinsb]

Instead of replacing NodeProvisioner, it might be easier just to have a periodic task within the cloud instance, just like NodeProvisionerInvoker.

Each image type could then have a minimum number of instances. Each cloud could have a defined max # of instances.

I'm not sure whether the NodeProvisioner would interfere with us, though. Perhaps we could ignore messages from it?