ref: Stop arroyo ChildProcessTerminated from creating Sentry issues

[phacops]

Summary

SNUBA-BA7, SNUBA-BA8, SNUBA-BA9, SNUBA-BAD, and SNUBA-B1T are all the same underlying event: a multiprocessing worker in the transactions consumer was killed by a signal (ChildProcessTerminated: 17, where 17 is SIGCHLD — i.e. the parent was notified a child died, not the child's own exit code).

Root cause

arroyo's RunTaskWithMultiprocessing installs a SIGCHLD handler. When a worker child terminates unexpectedly while the strategy is still running, the handler raises ChildProcessTerminated. In StreamProcessor.run() arroyo does:

except Exception:
    logger.exception("Caught exception, shutting down...")  # ERROR level, with exc_info

...and then re-raises. So a single worker death fans out into multiple Sentry issues:

• 4 issues (SNUBA-BA7/BA8/BAD/B1T, handled: yes, logger: arroyo.processing.processor) — the ERROR-level "Caught exception, shutting down..." log, captured by LoggingIntegration. They're separate issues only because Sentry groups them by wherever the SIGCHLD happened to interrupt the run loop (submit, pickle.dumps, buffer_callback, poll, …).
• 1 issue (SNUBA-BA9, handled: no, mechanism: excepthook) — the same exception, re-raised and uncaught, crashing the process.

The worker death itself is almost always an OOM-kill of the worker (sometimes a native crash). The consumer then shuts down and is restarted by the orchestrator — it recovers on its own. These have 0 users impacted and super_low/low Seer actionability.

These are ERROR-level, so the WARN → log policy from #8077 does not cover them, which is why they surfaced as top unresolved issues the day after #8077 shipped.

Change

Extend before_send in snuba/environment.py to drop ChildProcessTerminated by exception type (an isinstance check walking the __cause__/__context__ chain — the robust, non-string approach #8077's review preferred), alongside the existing AllocationPolicyViolations / RPCAllocationPolicyException filtering. Both the logging path and the excepthook path flow through before_send with hint["exc_info"] set, so this suppresses all 5 issues at once.

The events are still captured as logs/breadcrumbs, and the underlying worker death remains observable via arroyo's sigchld.detected metric.

Note / trade-off

This stops the noise; it does not stop workers from dying. If the worker deaths are OOM-kills (the most likely cause), the real mitigation is operational — worker memory limits, --processes, and input_block_size/output_block_size — which live in the deployment config, not this repo. Worth keeping an eye on the sigchld.detected metric for the transactions consumer; a sustained spike there would indicate a genuine crash-loop rather than occasional worker churn.

Test

Adds tests/test_environment.py covering before_send: pass-through (no/None/unrelated exc), dropping ChildProcessTerminated (top-level and nested in a cause chain), dropping the allocation-policy exceptions, and safe termination on a cyclic exception chain.

Fixes SNUBA-BA7
Fixes SNUBA-BA8
Fixes SNUBA-BA9
Fixes SNUBA-BAD
Fixes SNUBA-B1T

🤖 Generated with Claude Code

https://claude.ai/code/session_01NmYA6zVesfHV8aXRSFGkjB

---

Generated by Claude Code

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 503f013. Configure here.}