Is it platform specific
generic
Importance or Severity
High
Description of the bug
ZmqConsumerStateTable::m_receivedOperationQueue is an unbounded std::queue that grows without limit during BGP route churn, consuming all available memory and triggering kernel OOM kills.
The queue has no deduplication — every SET and DEL for the same route key is queued individually, even though only the latest operation per key matters. While the main thread blocks on synchronous SAI bulk calls (1–2.5s per batch), the ZMQ poll thread continues pushing to the queue with no backpressure.
The Redis-based ConsumerStateTable path is not affected because Redis key-value semantics naturally deduplicate. The ZMQ path bypasses Redis for performance but loses this property.
We first observed this during test_bgp_update_replication.py against VPP, where ~10K routes produced ~70K queue entries (~7x duplication, ~68 MB). Recently on production hardware with ~27K routes and 31 ECMP nexthops, the same pattern caused 25.7 GB growth and kernel panics every ~2 hours.
Related: #28245 — fpmsyncd glibc heap fragmentation on the same device under the same route churn scenario (separate root cause)
Steps to Reproduce
- Enable ZMQ northbound route path (orch_northbond_route_zmq_enabled=true)
- Establish iBGP session advertising ~27K IPv6 routes with 31 ECMP nexthops
- Induce sustained route churn (same prefixes cycling DEL→SET every ~2 seconds)
- orchagent crashes with OOM within ~2 hours
Also reproducible at smaller scale with test_bgp_update_replication.py against VPP (~10K routes, queue grows to ~70K entries).
Actual Behavior and Expected Behavior
Actual: orchagent RSS grows to 25.7 GB and triggers kernel OOM kill. Four kernel panics in 8 hours, crashing every ~2 hours. GDB confirmed the queue held 5,226,644 entries — 5.2 million redundant operations on ~27K unique routes. jemalloc stats showed 6.67 GB of 6.81 GB resident memory consumed by queue entries (KFV tuples, field-value vectors, nexthop strings, shared_ptr control blocks). The main thread was blocked in bulkCreate(route_entry) waiting on syncd while the poll thread kept filling the queue.
Expected: The ZMQ consumer should deduplicate operations by key (matching Redis key-value semantics), keeping memory bounded by the number of unique keys rather than growing per-message.
Proposed fix: Replace the unbounded FIFO queue with an in-memory key-value staging area using last-writer-wins semantics, preserving insertion order. Uses a double-buffer design with O(1) lock hold time regardless of queue depth.
Scope: zmqconsumerstatetable.cpp and .h in sonic-swss-common only. No changes to orchagent, fpmsyncd, or restart/reload protocols. We have a working fix initially done for vpp which we validated on hardware when we hit the oom issue. If the design approach looks good, we can raise the PR for review.
HLD attached below.
Fix validation (same device, same route churn):
- orchagent RSS: 25.7 GB → OOM in 2 hrs (before) → 545 MB after 14 hours (after)
- Queue entries: 5.2M unbounded (before) → dedup active, bounded by unique keys (after)
- Kernel panics: 4 in 8 hours (before) → zero (after)
Relevant log output
2026_07_11_05_43_18 Kernel Panic - Out of memory [Fri Jul 10 10:42:20 PM PDT 2026]
2026_07_11_03_44_29 Kernel Panic - Out of memory [Fri Jul 10 08:43:31 PM PDT 2026]
2026_07_11_01_40_44 Kernel Panic - Out of memory [Fri Jul 10 06:39:44 PM PDT 2026]
2026_07_10_23_45_59 Kernel Panic - Out of memory [Fri Jul 10 04:45:01 PM PDT 2026]
docker invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP)
[ 4518] 0 4518 7123459 6746811 6746811 0 0 54550528 0 0 orchagent
Output of show version, show techsupport
Platform: x86_64-arista_7060x6_64pe_b
HwSKU: Arista-7060X6-64PE-B-O128
ASIC: Broadcom TH5
Memory: 30 GB
SONiC: 202511 branch (nightly 2026.07.06 / 2026.07.15)
Build: 20251110.23
Attach files (if any)
See HLD for full design and analysis:
https://github.com/rejithomas-arista/zmq-queue-deduplication-hld/blob/main/zmq-queue-deduplication-hld.md
Is it platform specific
generic
Importance or Severity
High
Description of the bug
ZmqConsumerStateTable::m_receivedOperationQueue is an unbounded std::queue that grows without limit during BGP route churn, consuming all available memory and triggering kernel OOM kills.
The queue has no deduplication — every SET and DEL for the same route key is queued individually, even though only the latest operation per key matters. While the main thread blocks on synchronous SAI bulk calls (1–2.5s per batch), the ZMQ poll thread continues pushing to the queue with no backpressure.
The Redis-based ConsumerStateTable path is not affected because Redis key-value semantics naturally deduplicate. The ZMQ path bypasses Redis for performance but loses this property.
We first observed this during test_bgp_update_replication.py against VPP, where ~10K routes produced ~70K queue entries (~7x duplication, ~68 MB). Recently on production hardware with ~27K routes and 31 ECMP nexthops, the same pattern caused 25.7 GB growth and kernel panics every ~2 hours.
Related: #28245 — fpmsyncd glibc heap fragmentation on the same device under the same route churn scenario (separate root cause)
Steps to Reproduce
Also reproducible at smaller scale with test_bgp_update_replication.py against VPP (~10K routes, queue grows to ~70K entries).
Actual Behavior and Expected Behavior
Actual: orchagent RSS grows to 25.7 GB and triggers kernel OOM kill. Four kernel panics in 8 hours, crashing every ~2 hours. GDB confirmed the queue held 5,226,644 entries — 5.2 million redundant operations on ~27K unique routes. jemalloc stats showed 6.67 GB of 6.81 GB resident memory consumed by queue entries (KFV tuples, field-value vectors, nexthop strings, shared_ptr control blocks). The main thread was blocked in bulkCreate(route_entry) waiting on syncd while the poll thread kept filling the queue.
Expected: The ZMQ consumer should deduplicate operations by key (matching Redis key-value semantics), keeping memory bounded by the number of unique keys rather than growing per-message.
Proposed fix: Replace the unbounded FIFO queue with an in-memory key-value staging area using last-writer-wins semantics, preserving insertion order. Uses a double-buffer design with O(1) lock hold time regardless of queue depth.
Scope: zmqconsumerstatetable.cpp and .h in sonic-swss-common only. No changes to orchagent, fpmsyncd, or restart/reload protocols. We have a working fix initially done for vpp which we validated on hardware when we hit the oom issue. If the design approach looks good, we can raise the PR for review.
HLD attached below.
Fix validation (same device, same route churn):
Relevant log output
2026_07_11_05_43_18 Kernel Panic - Out of memory [Fri Jul 10 10:42:20 PM PDT 2026] 2026_07_11_03_44_29 Kernel Panic - Out of memory [Fri Jul 10 08:43:31 PM PDT 2026] 2026_07_11_01_40_44 Kernel Panic - Out of memory [Fri Jul 10 06:39:44 PM PDT 2026] 2026_07_10_23_45_59 Kernel Panic - Out of memory [Fri Jul 10 04:45:01 PM PDT 2026] docker invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP) [ 4518] 0 4518 7123459 6746811 6746811 0 0 54550528 0 0 orchagentOutput of
show version,show techsupportAttach files (if any)
See HLD for full design and analysis:
https://github.com/rejithomas-arista/zmq-queue-deduplication-hld/blob/main/zmq-queue-deduplication-hld.md