[serve] Add max_request_retries to bound router retry loops

[HrushiYadav]

Description

The router's while True retry loop in route_and_send_request and _pick_and_reserve_replica retries indefinitely when replicas reject requests. max_queued_requests is only checked once at the entry gate, not during retries. Under sustained load with large payloads (~50MB), requests pile up in the retry loop causing OOM -- num_queued_requests can reach 119 when max_queued_requests=20.

This PR adds max_request_retries to RequestRouterConfig:

• Default -1 (unlimited) preserves backwards compatibility
• When set to a non-negative integer, the router raises BackPressureError (503) after exhausting retries
• Both route_and_send_request (replica rejection retry) and _pick_and_reserve_replica (capacity queue retry) are bounded
• Configurable via RequestRouterConfig(max_request_retries=N) or env var RAY_SERVE_ROUTER_MAX_REQUEST_RETRIES

Related issues

Fixes #61017

Additional information

Files changed:

• constants.py -- new RAY_SERVE_ROUTER_MAX_REQUEST_RETRIES env var (follows existing backoff env var pattern)
• config.py -- new max_request_retries field on RequestRouterConfig with validator, __eq__, __hash__
• router.py -- _check_retry_limit() helper, retry counter in both loops
• test_backpressure.py -- config validation test + integration test with max_request_retries=3

Semantics:

• max_request_retries=-1: unlimited (default, no behavior change)
• max_request_retries=0: no retries (fail on first rejection)
• max_request_retries=N: retry up to N times, then 503

[gemini-code-assist]

Code Review

This pull request introduces a configurable limit on the maximum number of times a request can be retried by the router before being dropped with a BackPressureError. It adds max_request_retries to RequestRouterConfig and implements retry tracking and limit checking in AsyncioRouter. The review feedback suggests avoiding accessing the private _deployment_config attribute of RouterMetricsManager inside _check_retry_limit by instead storing and updating _max_queued_requests directly on AsyncioRouter.

[gemini-code-assist]

To avoid accessing the private _deployment_config attribute of RouterMetricsManager inside _check_retry_limit, we should store _max_queued_requests directly on AsyncioRouter and initialize it here.

Suggested change

	self._initial_backoff_s: Optional[float] = None
	self._backoff_multiplier: Optional[float] = None
	self._max_backoff_s: Optional[float] = None
	self._max_request_retries: int = -1
	self._initial_backoff_s: Optional[float] = None
	self._backoff_multiplier: Optional[float] = None
	self._max_backoff_s: Optional[float] = None
	self._max_request_retries: int = -1
	self._max_queued_requests: int = -1

[gemini-code-assist]

Update self._max_queued_requests when the deployment config is updated.

Suggested change

	self._max_request_retries = (
	deployment_config.request_router_config.max_request_retries
	)
	self._max_request_retries = (
	deployment_config.request_router_config.max_request_retries
	)
	self._max_queued_requests = deployment_config.max_queued_requests

[gemini-code-assist]

Use the newly stored self._max_queued_requests directly instead of accessing the private _deployment_config attribute of RouterMetricsManager.

    def _check_retry_limit(self, num_retries: int) -> None:
        """Raise BackPressureError if the retry limit has been exceeded."""
        max_retries = self._max_request_retries
        if max_retries >= 0 and num_retries > max_retries:
            raise BackPressureError(
                num_queued_requests=self._metrics_manager.num_queued_requests,
                max_queued_requests=self._max_queued_requests,
            )