Skip to content

[Bugfix] Clean distributed state after worker initialization failure#49834

Closed
AndreasKaratzas wants to merge 1 commit into
mainfrom
agent/worker-init-distributed-cleanup
Closed

[Bugfix] Clean distributed state after worker initialization failure#49834
AndreasKaratzas wants to merge 1 commit into
mainfrom
agent/worker-init-distributed-cleanup

Conversation

@AndreasKaratzas

Copy link
Copy Markdown
Member

Summary

  • Tear down model-parallel and distributed state when a spawned worker fails before WorkerProc construction completes.
  • Add a focused regression for cleanup order on the partial-initialization path.

With spawn, constructor failure can occur after global groups and their shared-memory queue are initialized but before a WorkerProc exists to own normal shutdown.

Validation

  • Focused worker cleanup regression: 1 passed.
  • Two-GPU invalid Tensorizer shard path: 1 passed in 24.89 seconds, with no surviving worker or GPU allocation in the lifecycle control.
  • Changed-file pre-commit passed.

Buildkite: Model Executor

Duplicate check: #34816 handles parent-death signals, not constructor failure; no open PR addresses this ownership path.

AI assistance: Codex assisted with investigation, implementation, and validation; Andreas directed the ownership design and reviewed the resulting changes.

Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@mergify mergify Bot added v1 bug Something isn't working labels Jul 25, 2026
@AndreasKaratzas
AndreasKaratzas deleted the agent/worker-init-distributed-cleanup branch July 25, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working v1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant