fix(adv): classify ADV-layer 'workflow not found' as completed-workflow evidence#329
Merged
Merged
Conversation
…ow evidence The COMPLETED_WORKFLOW_EVIDENCE_RE regex recognized Temporal SDK error names (WorkflowNotFoundError, WorkflowExecutionAlreadyCompleted) but not the ADV tool-layer wrapper message 'workflow not found for ID: ...' produced when a workflow execution has been deleted. When a poisoned workflow was terminated and deleted at the Temporal level (TMPRL1100 during replay), ADV tools received 'workflow not found' but the classifier could not match it, preventing automatic disk- projection recovery. Tools errored out instead of falling through to recover_via_disk. Add 'workflow not found' and 'workflow execution not found' to the regex so the classifier recognizes both SDK and ADV-layer error text. Recovery context: fixOpsResolutionProjection had a 12-hour-old workflow that lived across 5 worker builds, producing nondeterministic history. After Temporal-level termination + deletion, the ADV tools could not self-recover because the error text was unclassifiable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a poisoned workflow (TMPRL1100) is terminated and deleted at the Temporal level, ADV tools receive
"workflow not found for ID: ..."from the tool layer. TheCOMPLETED_WORKFLOW_EVIDENCE_REregex only recognized Temporal SDK error names (WorkflowNotFoundError,WorkflowExecutionAlreadyCompleted) but not the ADV-layer wrapper, so the recovery classifier could not match it. Tools errored out instead of falling through to disk-projection recovery.Fix
Add
workflow not foundandworkflow execution not foundtoCOMPLETED_WORKFLOW_EVIDENCE_REinplugin/src/temporal/recovery-classification.ts.Context
Recovery hotfix during
fixOpsResolutionProjection— a 12-hour-old workflow lived across 5 worker builds, producing nondeterministic history. After Temporal-level termination + deletion, ADV tools could not self-recover.Verification
pnpm run buildpasses