Skip to content

Fix: Executor can fail to send error body#61

Merged
johncmerfeld merged 4 commits into
developmentfrom
fix/executor-safer-shutdown
May 29, 2026
Merged

Fix: Executor can fail to send error body#61
johncmerfeld merged 4 commits into
developmentfrom
fix/executor-safer-shutdown

Conversation

@johncmerfeld
Copy link
Copy Markdown
Collaborator

We've observed cases where a job fails, and the app is aware, but the executor fails to send an error report. Essentially any time we have an uncaught exception in the finally block of execute, this is a risk.

This change hardens the graceful shutdown process to catch potential sources of these additional exceptions, and also fixes two bugs that Claude found that could have triggered the similar failures.

johncmerfeld and others added 3 commits May 14, 2026 14:54
The except branch in execute() did its cleanup (upload_remaining_artifacts,
update_failure) before send_error, so any exception raised in cleanup would
escape the branch with the diagnostic payload never reaching the app. The
finally still sent DONE(failure), but the app was left with no /error detail.

Restructure the branch to lock self.error in immediately, then run cleanup
guarded with try/except so it can't derail reporting, then send_error last.
update_failure stays before send_error to preserve the status-update ordering
its docstring describes.

While here, fix three latent bugs in upload_artifact that would have hit this
same trap: a typoed self.error_obj attribute, a nonexistent self.bucket_out_path
reference, and an ArtifactS3UploadError constructed with the artifact object
where the name string was expected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@johncmerfeld johncmerfeld merged commit 27ce6df into development May 29, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants