Skip to content

[zmqclient]: Add send-path back-pressure counters and a retry-ladder cap#1233

Open
deepak-singhal0408 wants to merge 1 commit into
sonic-net:masterfrom
deepak-singhal0408:fix-28369-prA-sendpath-counters
Open

[zmqclient]: Add send-path back-pressure counters and a retry-ladder cap#1233
deepak-singhal0408 wants to merge 1 commit into
sonic-net:masterfrom
deepak-singhal0408:fix-28369-prA-sendpath-counters

Conversation

@deepak-singhal0408

Copy link
Copy Markdown

What I did

ZmqClient::sendMsg() drives the ZMQ socket in non-blocking mode and, when the socket is full (EAGAIN) or transiently unready (EINTR/EFSM), retries with an exponential back-off ladder. Today that back-pressure signal is only logged and then discarded, and the ladder length is fixed (MQ_MAX_RETRY → ~41 s worst case on a single message).

This PR makes that signal observable and the inner wait boundable, without changing default behavior:

  • Four process-local, per-instance atomic counters incremented at the branch points that already fire, exposed via inline getters:
    • getSendEagainTotal() — EAGAIN / HWM back-pressure (the leading indicator)
    • getSendRetryTotal() — retry iterations (EAGAIN + EINTR/EFSM)
    • getSendBackoffMaxMs() — running max of the clamped back-off delay
    • getSendLadderExhaustedTotal() — inner ladder exhausted → throw
  • setSendRetryConfig(maxRetries, maxBackoffMs) — optional cap on the inner ladder. Defaults (maxRetries < 0, maxBackoffMs < 0) preserve the exact historical behavior for every existing producer. A producer that owns an outer retry/coalescing loop can shorten the ladder so control returns quickly instead of blocking on one message.

ZmqClient stays DB-agnostic — it only counts and exposes getters; the owner decides where/how to publish.

Why

This is the transport-layer half of the fix for sonic-buildimage #28369 (fpmsyncd ZMQ route drop under burst). The send-path back-pressure that precedes a drop is currently invisible; these counters let a producer surface it as a leading indicator, and the retry cap lets a producer with its own outer loop avoid the ~41 s inner block that would otherwise defeat coalescing. The consuming fpmsyncd change is a follow-up sonic-swss PR.

ABI

Getters and atomics are appended as trailing members; ZmqClient is only ever held by reference/pointer (never embedded by value), and the class is already non-copyable, so the change is ABI-additive.

How I tested

New ZmqClientSendPathCounters unit tests in tests/zmq_state_ut.cpp:

  • Fill a peerless PUSH socket's send queue to force deterministic EAGAIN → retry → ladder-exhaustion, and assert each of the four counters advances to its expected value (with a small setSendRetryConfig cap so the test runs in <50 ms instead of the default ~41 s ladder).
  • Guard that the accessors read zero before any send (ABI-additive accessors compile/link and initialize correctly).

Full swss-common ZMQ unit-test suite: 18/18 pass, 0 regressions.

ZmqClient::sendMsg() drives the ZMQ socket in non-blocking mode and, on a
full socket (EAGAIN) or a transiently-unready socket (EINTR/EFSM), retries
with an exponential back-off ladder. Today that back-pressure signal is only
logged and then discarded, and the ladder length is fixed (MQ_MAX_RETRY ->
~41s worst case on a single message), so a producer that owns an outer retry
loop has no way to observe congestion or bound the inner wait.

Add four process-local, per-instance atomic counters incremented at the
branch points that already fire, exposed via inline getters:
  - getSendEagainTotal()          EAGAIN / HWM back-pressure (leading signal)
  - getSendRetryTotal()           retry iterations (EAGAIN + EINTR/EFSM)
  - getSendBackoffMaxMs()         running max of the clamped back-off delay
  - getSendLadderExhaustedTotal() inner ladder exhausted -> throw

The class stays DB-agnostic: it only counts and exposes getters; the owner
decides where/how to publish.

Also add setSendRetryConfig(maxRetries, maxBackoffMs) to cap the inner
ladder. Defaults (maxRetries<0, maxBackoffMs<0) preserve the exact historical
behavior for every existing producer; a producer with its own outer
retry/coalescing loop can shorten the ladder so control returns quickly
instead of blocking on one message.

Getters and atomics are appended as trailing members; ZmqClient is only ever
held by reference/pointer, so the change is ABI-additive.

Unit tests: fill a peerless PUSH socket's send queue to force deterministic
EAGAIN -> retry -> ladder-exhaustion and assert each counter advances, plus a
guard that the accessors read zero before any send.

Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 40aec520-9438-497d-9f28-f9f2c3034318
Signed-off-by: Deepak Singhal <deepsinghal@microsoft.com>
Copilot AI review requested due to automatic review settings July 23, 2026 23:01
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances swss::ZmqClient’s send-path observability by adding per-instance atomic counters for ZMQ back-pressure/retry behavior and introducing an optional configuration API to cap the internal retry ladder, with accompanying unit tests to validate counter behavior under forced retry conditions.

Changes:

  • Added setSendRetryConfig(maxRetries, maxBackoffMs) to optionally bound the internal retry/backoff ladder.
  • Added four per-instance atomic counters with inline getters to expose EAGAIN/retry/backoff/ladder-exhaustion signals.
  • Added unit tests to force a retry/ladder-exhaustion scenario and validate counter increments and initialization.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
common/zmqclient.h Adds retry-cap API, send-path counter getters, and new member state for caps/counters.
common/zmqclient.cpp Implements retry-cap setter and increments/records counters on retry/backoff and ladder exhaustion.
tests/zmq_state_ut.cpp Adds unit tests to validate counter behavior and default initialization.

Comment thread common/zmqclient.cpp
Comment on lines 219 to 224
else if (zmq_err == EAGAIN)
{
// EAGAIN: ZMQ is full to need try again
m_sendEagainTotal.fetch_add(1, std::memory_order_relaxed);
SWSS_LOG_WARN("zmq is full, will retry in %d ms, endpoint: %s, error: %d", retry_delay, m_endpoint.c_str(), zmq_err);
}
Comment thread common/zmqclient.cpp
Comment on lines +153 to +157
void ZmqClient::setSendRetryConfig(int maxRetries, int maxBackoffMs)
{
m_sendMaxRetries = (maxRetries < 0) ? MQ_MAX_RETRY : maxRetries;
m_sendMaxBackoffMs = maxBackoffMs;
}
Comment thread tests/zmq_state_ut.cpp
Comment on lines +466 to +468
std::string deadEndpoint = "tcp://127.0.0.1:65123"; // nothing bound here
ZmqClient client(deadEndpoint, 0);
client.setSendRetryConfig(/*maxRetries=*/3, /*maxBackoffMs=*/5);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants