[zmqclient]: Add send-path back-pressure counters and a retry-ladder cap#1233
Open
deepak-singhal0408 wants to merge 1 commit into
Open
[zmqclient]: Add send-path back-pressure counters and a retry-ladder cap#1233deepak-singhal0408 wants to merge 1 commit into
deepak-singhal0408 wants to merge 1 commit into
Conversation
ZmqClient::sendMsg() drives the ZMQ socket in non-blocking mode and, on a full socket (EAGAIN) or a transiently-unready socket (EINTR/EFSM), retries with an exponential back-off ladder. Today that back-pressure signal is only logged and then discarded, and the ladder length is fixed (MQ_MAX_RETRY -> ~41s worst case on a single message), so a producer that owns an outer retry loop has no way to observe congestion or bound the inner wait. Add four process-local, per-instance atomic counters incremented at the branch points that already fire, exposed via inline getters: - getSendEagainTotal() EAGAIN / HWM back-pressure (leading signal) - getSendRetryTotal() retry iterations (EAGAIN + EINTR/EFSM) - getSendBackoffMaxMs() running max of the clamped back-off delay - getSendLadderExhaustedTotal() inner ladder exhausted -> throw The class stays DB-agnostic: it only counts and exposes getters; the owner decides where/how to publish. Also add setSendRetryConfig(maxRetries, maxBackoffMs) to cap the inner ladder. Defaults (maxRetries<0, maxBackoffMs<0) preserve the exact historical behavior for every existing producer; a producer with its own outer retry/coalescing loop can shorten the ladder so control returns quickly instead of blocking on one message. Getters and atomics are appended as trailing members; ZmqClient is only ever held by reference/pointer, so the change is ABI-additive. Unit tests: fill a peerless PUSH socket's send queue to force deterministic EAGAIN -> retry -> ladder-exhaustion and assert each counter advances, plus a guard that the accessors read zero before any send. Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40aec520-9438-497d-9f28-f9f2c3034318 Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
This PR enhances swss::ZmqClient’s send-path observability by adding per-instance atomic counters for ZMQ back-pressure/retry behavior and introducing an optional configuration API to cap the internal retry ladder, with accompanying unit tests to validate counter behavior under forced retry conditions.
Changes:
- Added
setSendRetryConfig(maxRetries, maxBackoffMs)to optionally bound the internal retry/backoff ladder. - Added four per-instance atomic counters with inline getters to expose EAGAIN/retry/backoff/ladder-exhaustion signals.
- Added unit tests to force a retry/ladder-exhaustion scenario and validate counter increments and initialization.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
common/zmqclient.h |
Adds retry-cap API, send-path counter getters, and new member state for caps/counters. |
common/zmqclient.cpp |
Implements retry-cap setter and increments/records counters on retry/backoff and ladder exhaustion. |
tests/zmq_state_ut.cpp |
Adds unit tests to validate counter behavior and default initialization. |
Comment on lines
219
to
224
| else if (zmq_err == EAGAIN) | ||
| { | ||
| // EAGAIN: ZMQ is full to need try again | ||
| m_sendEagainTotal.fetch_add(1, std::memory_order_relaxed); | ||
| SWSS_LOG_WARN("zmq is full, will retry in %d ms, endpoint: %s, error: %d", retry_delay, m_endpoint.c_str(), zmq_err); | ||
| } |
Comment on lines
+153
to
+157
| void ZmqClient::setSendRetryConfig(int maxRetries, int maxBackoffMs) | ||
| { | ||
| m_sendMaxRetries = (maxRetries < 0) ? MQ_MAX_RETRY : maxRetries; | ||
| m_sendMaxBackoffMs = maxBackoffMs; | ||
| } |
Comment on lines
+466
to
+468
| std::string deadEndpoint = "tcp://127.0.0.1:65123"; // nothing bound here | ||
| ZmqClient client(deadEndpoint, 0); | ||
| client.setSendRetryConfig(/*maxRetries=*/3, /*maxBackoffMs=*/5); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What I did
ZmqClient::sendMsg()drives the ZMQ socket in non-blocking mode and, when the socket is full (EAGAIN) or transiently unready (EINTR/EFSM), retries with an exponential back-off ladder. Today that back-pressure signal is only logged and then discarded, and the ladder length is fixed (MQ_MAX_RETRY→ ~41 s worst case on a single message).This PR makes that signal observable and the inner wait boundable, without changing default behavior:
getSendEagainTotal()— EAGAIN / HWM back-pressure (the leading indicator)getSendRetryTotal()— retry iterations (EAGAIN + EINTR/EFSM)getSendBackoffMaxMs()— running max of the clamped back-off delaygetSendLadderExhaustedTotal()— inner ladder exhausted → throwsetSendRetryConfig(maxRetries, maxBackoffMs)— optional cap on the inner ladder. Defaults (maxRetries < 0,maxBackoffMs < 0) preserve the exact historical behavior for every existing producer. A producer that owns an outer retry/coalescing loop can shorten the ladder so control returns quickly instead of blocking on one message.ZmqClientstays DB-agnostic — it only counts and exposes getters; the owner decides where/how to publish.Why
This is the transport-layer half of the fix for sonic-buildimage #28369 (fpmsyncd ZMQ route drop under burst). The send-path back-pressure that precedes a drop is currently invisible; these counters let a producer surface it as a leading indicator, and the retry cap lets a producer with its own outer loop avoid the ~41 s inner block that would otherwise defeat coalescing. The consuming fpmsyncd change is a follow-up sonic-swss PR.
ABI
Getters and atomics are appended as trailing members;
ZmqClientis only ever held by reference/pointer (never embedded by value), and the class is already non-copyable, so the change is ABI-additive.How I tested
New
ZmqClientSendPathCountersunit tests intests/zmq_state_ut.cpp:EAGAIN → retry → ladder-exhaustion, and assert each of the four counters advances to its expected value (with a smallsetSendRetryConfigcap so the test runs in <50 ms instead of the default ~41 s ladder).Full swss-common ZMQ unit-test suite: 18/18 pass, 0 regressions.