Is it platform specific
generic
Importance or Severity
Low
Description of the bug
orchagent intermittently crashes in jemalloc (heap corruption / use-after-free) while exiting on the Graceful Shutdown path. The root cause is a teardown ordering problem in the global static ClientServerSai object in libsairedis.
orchagent's SAI handle is a global static ClientServerSai, which owns a ServerSai (which in turn owns the syncd-facing Sai/ZMQ channel and a northbound ZMQ server thread). When main() returns on the graceful path, normal exit() runs the C++ global destructors, tearing down:
ClientServerSai → ServerSai → ~ZeroMQSelectableChannel → zmq_ctx_term()
At that point other threads are still alive and still referencing the global object — the swss ZmqServer poll thread, libzmq I/O threads, and the Logger thread. Destroying the global while those threads are running (and calling zmq_ctx_term() out from under them) causes a use-after-free → heap corruption → jemalloc crash.
This is non-functional; all shutdown work has already completed when the crash occurs.
Steps to Reproduce
- Run orchagent normally on a SONiC device.
- Run a graceful shutdown of orchagent (config reload works)
- Intermittently, orchagent crashes in jemalloc during teardown, its a rare statistical issue
Actual Behavior and Expected Behavior
Actual: After shutdown work completes, orchagent intermittently crashes in jemalloc with heap corruption, because the global ClientServerSai teardown runs while dependent threads still reference it.
Expected: orchagent exits cleanly on the graceful path, with dependent threads and the ZMQ context/channel torn down before the ClientServerSai/ServerSai objects are destroyed.
Relevant log output
2026-06-07 19:58:56,300 INFO waiting for supervisor-proc-exit-listener, rsyslogd, portsyncd, orchagent to die
2026 Jun 7 19:58:52.271761 sonic INFO swss#supervisord: message repeated 2 times: [ orchagent ]
2026 Jun 7 19:58:57.004721 sonic INFO swss#supervisord: orchagent AsyncSwssRecorderStats signal=11 pending=0 high_watermark=0 enqueued=0 drained=0
2026 Jun 7 19:58:57.007159 sonic WARNING kernel: [15652.808499] show_signal: 100 callbacks suppressed
2026 Jun 7 19:58:57.007182 sonic INFO kernel: [15652.808514] traps: orchagent[776939] general protection fault ip:7f19ae2199c6 sp:7f19a96f96c0 error:0 in libjemalloc.so.2[869c6,7f19ae19c000+a7000]
Output of show version, show techsupport
Attach files (if any)
No response
Is it platform specific
generic
Importance or Severity
Low
Description of the bug
orchagent intermittently crashes in jemalloc (heap corruption / use-after-free) while exiting on the Graceful Shutdown path. The root cause is a teardown ordering problem in the global static ClientServerSai object in libsairedis.
orchagent's SAI handle is a global static ClientServerSai, which owns a ServerSai (which in turn owns the syncd-facing Sai/ZMQ channel and a northbound ZMQ server thread). When main() returns on the graceful path, normal exit() runs the C++ global destructors, tearing down:
ClientServerSai → ServerSai → ~ZeroMQSelectableChannel → zmq_ctx_term()
At that point other threads are still alive and still referencing the global object — the swss ZmqServer poll thread, libzmq I/O threads, and the Logger thread. Destroying the global while those threads are running (and calling zmq_ctx_term() out from under them) causes a use-after-free → heap corruption → jemalloc crash.
This is non-functional; all shutdown work has already completed when the crash occurs.
Steps to Reproduce
Actual Behavior and Expected Behavior
Actual: After shutdown work completes, orchagent intermittently crashes in jemalloc with heap corruption, because the global ClientServerSai teardown runs while dependent threads still reference it.
Expected: orchagent exits cleanly on the graceful path, with dependent threads and the ZMQ context/channel torn down before the ClientServerSai/ServerSai objects are destroyed.
Relevant log output
Output of
show version,show techsupportAttach files (if any)
No response