[serve] Fix proxy update loop getting stuck when a proxy's node is removed

[nadongjun]

Description

When a Serve proxy actor is pinned to a node that later gets removed (autoscaler consolidation, spot reclaim, or abrupt termination), calling check_health on it raises ActorUnschedulableError.

ActorProxyWrapper.is_shutdown() only catches RayActorError and GetTimeoutError, so this exception propagates up through kill -> ProxyState.shutdown -> ProxyStateManager.update. The controller's proxy update loop then hangs every cycle before it can start a new proxy, so proxies stays empty and all ingress returns 503. Restarting the head node does not help, because the detached proxy state is restored from GCS.

We hit this in production(ray 2.56) with HAProxy mode (RAY_SERVE_ENABLE_HA_PROXY=1) under repeated head/proxy node churn: the fallback proxy ended up pinned to a removed node, so update() got stuck and never reached _start_proxies_if_needed. As a result the proxy actor's setup() was never called, HAProxyManager never started the HAProxy process, port 8000 had no listener, and external requests got 503.

• Fix: catch ActorUnschedulableError in is_shutdown() and treat the actor as ready for shutdown, so the controller can drop the stale proxy and start a new one.

Related issues

Additional information

ActorUnschedulableError is a RayError but not a RayActorError, which is why the existing except RayActorError misses it. Only the unschedulable path changes; the dead, pending, and alive paths behave as before.

[gemini-code-assist]

Code Review

This pull request updates the proxy state management in Ray Serve to handle ActorUnschedulableError when checking if a proxy actor is shut down. Specifically, in proxy_state.py, the is_shutdown method now catches ActorUnschedulableError in addition to RayActorError, treating permanently unschedulable proxy actors as ready for shutdown. Additionally, comprehensive unit tests have been added in test_proxy_state.py to verify this behavior under various actor states. There are no review comments, so no further feedback is provided.