Bug: fpmsyncd drops route updates (zmq send failed, EAGAIN) under route burst when swss_zmq is enabled

[deepak-singhal0408]

Is it platform specific

generic

Importance or Severity

High

Description of the bug

SYSTEM_DEFAULTS|swss_zmq is a single knob that enables ZMQ mode on both the northbound route path (fpmsyncd -> orchagent RouteOrch) and the southbound path (orchagent -> syncd). When it is enabled (CONFIG_DB SYSTEM_DEFAULTS|swss_zmq = enabled), fpmsyncd forwards route updates to orchagent over the northbound ZMQ channel (tcp://localhost:8101/8102) instead of via Redis keyspace notifications. This bug is on that northbound route path.

Under a burst of route programming (e.g. config reload, a large route add/withdraw, or simultaneous route churn on multiple ASICs' bgp containers), the orchagent route-ZMQ consumer cannot drain the queue fast enough. The ZMQ send buffer reaches its high-water mark, and fpmsyncd repeatedly fails to send:

ERR bgpN#fpmsyncd: :- sendMsg: zmq send failed, endpoint: tcp://localhost:8101, zmqerrno: 11: Resource temporarily unavailable

fpmsyncd retries for a bounded period and then gives up on the send. Because the ZMQ send is attempted before the route is persisted, the affected update is neither delivered to orchagent nor written to APP_DB, and there is no automatic retry/replay — so the route update can be silently lost until the next full route re-sync.

This is functionally harmful (potential dataplane route loss), not just log noise. The code path is generic (same producer/consumer code on virtual and real hardware); the risk grows with route scale.

Steps to Reproduce

1. Bring up a SONiC switch (observed on a multi-ASIC T2 topology; the ZMQ path is platform-generic).
2. Enable ZMQ mode: set CONFIG_DB SYSTEM_DEFAULTS|swss_zmq status=enabled and restart swss (this enables both the northbound route ZMQ and southbound syncd ZMQ; the route producer/consumer path is the one exercised here).
3. Generate a large route burst — e.g. config reload, or add/remove a large batch of routes, or drive route churn on each ASIC's bgp/fpmsyncd simultaneously.
4. Watch fpmsyncd in syslog (/var/log/syslog, bgpN#fpmsyncd).

Actual Behavior and Expected Behavior

Actual: under the burst, fpmsyncd logs sendMsg: zmq send failed ... zmqerrno: 11 (EAGAIN / Resource temporarily unavailable), the send is abandoned, and the affected route update is dropped (not delivered to orchagent, not persisted to APP_DB) with no automatic recovery.

Expected: route updates are reliably delivered and persisted even under burst load; fpmsyncd must not silently drop route updates when the ZMQ channel is temporarily full.

Relevant log output

ERR bgp0#fpmsyncd: :- sendMsg: zmq send failed, endpoint: tcp://localhost:8101, zmqerrno: 11: Resource temporarily unavailable
ERR bgp1#fpmsyncd: :- sendMsg: zmq send failed, endpoint: tcp://localhost:8102, zmqerrno: 11: Resource temporarily unavailable

Output of show version, show techsupport

Observed on a SONiC master-based image (multi-ASIC T2). Full techsupport not captured from the observing run;
reproduces wherever SYSTEM_DEFAULTS|swss_zmq is enabled and a route burst is applied.

[deepak-singhal0408]

@venkit-nexthop could you take this issue?
your sonic-net/SONiC#2328 and relevant PRs should address this? Thanks,

[deepak-singhal0408]

Adding to my note above : SONiC 2328 (+ sonic-swss-common 1187, sonic-swss 4564) already covers the consumer-side handling (faster drain + burst coalescing), but I think we also need complementary producer-side hardening — don't drop on a terminal EAGAIN; backpressure/requeue and re-drive — so enabling swss_zmq route mode is safe regardless of the redesign's progress. Scoping separately (WIP).

[deepak-singhal0408]

Producer-side hardening plan for #28369 — status update

Following up on my note above. I've scoped the producer-side work (fpmsyncd / swss-common ZMQ send path) into 4 small, single-purpose PRs, phased so each one is independently reviewable and the earlier ones instrument the later ones. This is complementary to — not overlapping with — the consumer-side redesign in SONiC PR 2328 / sonic-swss-common PR 1187 / sonic-swss PR 4564.

Root cause (confirmed): with swss_zmq enabled, fpmsyncd sends one ZMQ message per route. The ZMQ HWM is per-message (SND/RCVHWM=10000), so a burst that puts >10k messages in flight hits the cliff → the retry/backoff loop in ZmqClient::sendMsg exhausts → system_error(io_error) is thrown, unwinds through ProducerStateTable::set(), and is swallowed in Select::poll_descriptors. The route is silently lost (present in neither ZMQ nor APP_DB), no crash, no signal.

The 4 PRs

#	PR	Repo	What	Status
1	sonic-swss PR 4751	sonic-swss	Observability — make the silent drop visible (per-ASIC counters + last-drop prefix in STATE_DB FPMSYNCD_ROUTE_STAT_TABLE). No behavior change.	✅ Pushed — in review
2	[zmqclient]: Add ZMQ send-path pressure counters	sonic-swss-common (+ fpmsyncd fold)	Count the EAGAIN / retry / backoff signal that's generated every burst but discarded today → leading indicator ahead of a drop.	Validated, draft ready
3	[fpmsyncd]: Batch route updates per netlink drain	sonic-swss	The actual fix — accumulate distinct KCOs across one FPM socket drain, flush once via set(vector<KCO>). N routes → 1 message, so the per-message HWM cliff is no longer hit.	Validated, draft ready
4	[fpmsyncd]: Coalesce same-key route updates within a drain window	sonic-swss	Churn-only complement to PR-3 — last-writer-wins dedup of same-key updates within a drain (e.g. ECMP nexthop-group fill on clear ip bgp *).	Validated, draft ready

Delivery order matters: PR-1/PR-2 ship the instrumentation, then PR-3/PR-4's before/after impact is measured as two reads of the same funnel table under an identical burst — no bespoke tooling.

Validation (KVM T2, 2 line-cards × 2 ASICs = 4 ASIC instances, swss_zmq=enabled, per-ASIC reads)

• PR-1 — before: FPMSYNCD_ROUTE_STAT_TABLE empty on all 4 ASICs (operator blind). After: populated on all 4; the named last_drop_prefix is present in zebra RIB but absent from APPL_DB — the exact silently-dropped route pinpointed. UT 188/188.
• PR-2 — send-path counters confirm eagain rises as a leading indicator before the terminal drop (terminal_drop == PR-1's drop count, 4/4 ASICs); worst backoff reached 20480 ms. UT 20/20 (swss-common) + 189/189 (fpmsyncd).
• PR-3 (the fix) — under an identical burst: drops → 0 on all 4 ASICs, 30k+ previously-stuck routes/ASIC recovered, ~34× fewer ZMQ messages, no EAGAIN / no backoff. UT 193/193 + 18/18.
• PR-4 — converged route set byte-identical to PR-3 (0 routes lost); dedup logic UT-proven (5 same-key updates → 1 survivor, converged state == replay). UT 196/196. On the KVM fabric the live dedup_ratio stays ~1.0 (vEOS spaces per-prefix updates wider than fpmsyncd's drain cadence, so same-key rarely repeats in one drain) — the benefit needs HW/production FPM velocity; shipping as correct + no-harm.

Deferred: correctness backstop (item 1)

Whether to also add a producer-side backstop (bounded replay-over-ZMQ of a failed batch, then controlled fail-fast instead of an uncaught throw) is gated on PR-1 telemetry: once PR-3 lands, if the residual terminal_drop tail is ~0, item 1 may reduce to "loud counter only". Revisiting after Phases 1–3 land.

I'll link PR-2/3/4 here as they go up.

[venkit-nexthop]

@deepak-singhal0408 FWIW, here is my take on this issue.

I think your 1st PR of counting dropped prefixes is not very useful. When some prefixes get dropped, it is a catastrophic failure. This router is no longer forwarding traffic consistently and the ideal course of action is to fail and let redundancy in the network take over and forward packets correctly. So IMO, even if we report the counters and syslog the failure, there is no course of action that an operator can take. This is just a test debugging tool at that point.

The failure in fpmsyncd can be of the following two types

1. Transient failure where orchagent is backed up and is busy
2. Non-transient and orchagent is in some kind of crash loop

I think when (2) above happens, even your 3rd PR above will not help. It will still run into the same problem. The ideal course of action is to crash and fail early.

I think we should think about how to effectively solve the transient failure above. For that I think the effective solution is to avoid having a lot of in-flight state in ZMQ. This is because ZMQ is also non-coalescing and we should keep it small. Rather, we should have a local coalescing data structure in fpmsyncd. Then any fast updates coming into fpmsyncd can be absorbed there and we can write into ZMQ asynchronously. Any transient ZMQ buffer full can be handled by retrying. If the problem persists, again as before crashing and failing is the best course of action.

[deepak-singhal0408]

Thanks @venkit-nexthop for your valuable inputs.
Had a call with @venkit-nexthop on this. Capturing the discussion outcome here for reference..

@deepak-singhal0408 FWIW, here is my take on this issue.

I think your 1st PR of counting dropped prefixes is not very useful. When some prefixes get dropped, it is a catastrophic failure. This router is no longer forwarding traffic consistently and the ideal course of action is to fail and let redundancy in the network take over and forward packets correctly. So IMO, even if we report the counters and syslog the failure, there is no course of action that an operator can take. This is just a test debugging tool at that point.

[DS]: In Typical DC fabrics, we always have default routes, so if a prefix gets dropped, that is not necessarily mean dropping traffic. Letting redundancy take over means crash recovery (means swss etc restarts), which means the whole router (all links) goes down/comes up (while addressing live traffic). This will have traffic impact (more so at upper layers, where traffic throughput is might higher along with prefix scale).

The purpose of telemetry/visibility in these scenarios would help in following ways;
a. Give external monitoring services an opportunity to alert on this, and take safe mitigative action (TSA+recovery).
b. Over a period of time, we could collect/study enough telemetry data over variety of Device roles/load patterns/prefix scale/Platforms of different capacity(CPU/mem) and come up with next set of improvements.

The failure in fpmsyncd can be of the following two types

1. Transient failure where orchagent is backed up and is busy
2. Non-transient and orchagent is in some kind of crash loop

I think when (2) above happens, even your 3rd PR above will not help. It will still run into the same problem. The ideal course of action is to crash and fail early.

[DS]: yes, for non-transient cases none of these optimizations would help.
But PR#3 still has lot of value since it optimizes this path by reducing system calls(zmq send()/receive()), ZMQ envelope overhead(reduce from per route->route batch)

I think we should think about how to effectively solve the transient failure above. For that I think the effective solution is to avoid having a lot of in-flight state in ZMQ. This is because ZMQ is also non-coalescing and we should keep it small. Rather, we should have a local coalescing data structure in fpmsyncd. Then any fast updates coming into fpmsyncd can be absorbed there and we can write into ZMQ asynchronously. Any transient ZMQ buffer full can be handled by retrying. If the problem persists, again as before crashing and failing is the best course of action.

[DS]: Thanks and agree to this. Will implement a local coalescing datastructure in fpmsyncd and have infra to write into zmq async. This way we get real coalescing benefits when there is back pressure.