Skip to content

[orchagent]: _exit() on graceful shutdown to avoid destructor-chain memory corruption#4755

Closed
theasianpianist with Copilot wants to merge 1 commit into
masterfrom
copilot/fix-orchagent-crash-jemalloc
Closed

[orchagent]: _exit() on graceful shutdown to avoid destructor-chain memory corruption#4755
theasianpianist with Copilot wants to merge 1 commit into
masterfrom
copilot/fix-orchagent-crash-jemalloc

Conversation

Copilot AI commented Jul 14, 2026

Copy link
Copy Markdown

On SIGTERM/SIGINT, OrchDaemon::start() returns and main() falls through to return 0, running the full ~OrchDaemon destructor chain. This is unsafe: FlexCounterManager destruction makes blocking SAI calls through sairedis's ZMQ channel while libzmq I/O threads are still alive, and orchs torn down earlier in the reverse-order loop have already freed buffers those threads still reference — corrupting the heap and causing intermittent jemalloc crashes.

Changes

  • orchagent/orchdaemon.cpp — adds exit_if_graceful_shutdown_requested(): drains the async swss recorder (setAsync(false) joins the worker and flushes pending records), then calls _exit(0) to bypass the destructor chain entirely
  • orchagent/orchdaemon.h — declares the function with _exit as the default; accepts an injectable exit_fn for unit tests
  • orchagent/main.cpp — calls exit_if_graceful_shutdown_requested() immediately after orchDaemon->start() returns on the graceful path
  • tests/mock_tests/orchdaemon_ut.cpp — unit tests covering the no-shutdown no-op and the drain+exit path
// After orchDaemon->start() returns on SIGTERM/SIGINT:
exit_if_graceful_shutdown_requested();  // drains recorder, _exit(0) — skips ~OrchDaemon

return 0;  // only reached on non-graceful exit

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI changed the title [WIP] Fix orchagent crash in jemalloc during graceful shutdown [orchagent]: _exit() on graceful shutdown to avoid destructor-chain memory corruption Jul 14, 2026
Copilot AI requested a review from theasianpianist July 14, 2026 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants