Skip to content

Fix unsafe actor state mutation in ProcessIndex deletion path - #1952

Open
stasimus wants to merge 4 commits into
ing-bank:masterfrom
stasimus:fix/process-index-deletion-thread-safety
Open

Fix unsafe actor state mutation in ProcessIndex deletion path#1952
stasimus wants to merge 4 commits into
ing-bank:masterfrom
stasimus:fix/process-index-deletion-thread-safety

Conversation

@stasimus

@stasimus stasimus commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Problem: in the ProcessIndex deletion path for stopped actors (Cassandra cleanup), cleanup.deleteAllEvents(...).map { ... } mutated the actor's mutable index and replied to the client from the future callback — i.e. on a dispatcher thread concurrently with the actor's own thread. The callback also had no failure handling, and the deletion itself was not crash-safe: a node crash at the wrong moment left the journal and index permanently diverged (either ActorDeleted journaled with the instance events never removed, or the events removed with the instance still recovering as alive).

Reproduce:

  • with a Cassandra journal (supportsCleanupOfStoppedActors), delete a passivated recipe instance while deleteAllEvents fails — the deleteRecipeInstance call times out and the instance stays undeletable in memory until restart;
  • or crash the node between the event deletion and the ActorDeleted journal write — the instance resurfaces after recovery as a live instance with an empty journal and the deletion is never retried.

Fix: deletion is now a durable two-phase operation running entirely on the actor thread. ActorDeletionStarted is journaled first (new persisted event, instance enters the new Deleting status), then the events are deleted asynchronously with the result piped back to the actor as a message, and ActorDeleted is journaled only after the events are actually gone. A deletion interrupted by a crash is resumed on recovery; a failed deletion replies Status.Failure to the caller and is retried by the retention sweep or a client retry. While Deleting, the instance cannot be reactivated or recreated, and the sweep does not launch duplicate concurrent deletions.

Backward compatible for upgrades: the new proto fields are optional and old journals/snapshots deserialize unchanged. Not rollback-safe once the new version has run: older Baker versions cannot deserialize ActorDeletionStarted events found in the ProcessIndex journal, so a downgrade after deletions have been requested requires those journal entries to age out via snapshots.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants