Skip to content

Memory pools may not be released when a node crashes or is dynamically removed #2881

Description

@SaitejaKommi

Summary

While reviewing the daemon lifecycle, I noticed that memory pools registered by a node appear to remain tracked when that node is removed from a running dataflow or disconnects unexpectedly.

If this observation is correct, repeated node crashes or dynamic node removal could leave stale memory pool registrations until the entire dataflow is destroyed.

Evidence

During RemoveNode cleanup, the daemon removes the node's runtime state and bookkeeping, but I could not find corresponding cleanup for memory pools registered by that node.

Relevant locations:

  • binaries/daemon/src/lib.rs

    • DaemonCoordinatorEvent::RemoveNode
    • node cleanup path
  • MemoryPoolManager

    • memory pool lifecycle

cleanup_orphans appears to operate at the dataflow level rather than the individual node level.

Possible Impact

If memory pools remain registered after node removal:

  • /dev/shm segments may accumulate
  • stale pool registrations may persist
  • long-running daemons could eventually exhaust the configured pool limit

Suggested Direction

If this behavior is unintended, node removal and unexpected disconnect paths could free any memory pools registered by that node before the remaining bookkeeping is discarded.

I'm happy to investigate further and prepare a PR if this is considered a bug.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions