Deferred from #164 (comment) (review discussion with @merceod).
Today a PUSH send to a stalled peer blocks at the peer's high-water mark indefinitely. #164 scoped the blast radius (an HWM-blocked send stalls only that peer, not the process's control plane), but deliberately did NOT add a send timeout, because:
- no call site today handles a send error — a timeout would turn a stalled send into an unhandled exception mid-loop, which is worse than the stall;
- adding SNDTIMEO only on the Rust side would break send-semantics parity between the two transports (the drop-in guarantee).
The right home is the worker-liveness / failure-propagation work, where a timed-out send has somewhere to go: fail the affected requests, mask the sick peer, surface the condition to serving health. When that lands, add the timeout to BOTH transports (pyzmq SNDTIMEO + the Rust set_sndtimeo) behind one knob, with the error path wired into the failure handling.
Deferred from #164 (comment) (review discussion with @merceod).
Today a PUSH send to a stalled peer blocks at the peer's high-water mark indefinitely. #164 scoped the blast radius (an HWM-blocked send stalls only that peer, not the process's control plane), but deliberately did NOT add a send timeout, because:
The right home is the worker-liveness / failure-propagation work, where a timed-out send has somewhere to go: fail the affected requests, mask the sick peer, surface the condition to serving health. When that lands, add the timeout to BOTH transports (pyzmq
SNDTIMEO+ the Rustset_sndtimeo) behind one knob, with the error path wired into the failure handling.