Skip to content

part of: ENG-3645 - flush OTEL spans on SIGTERM/SIGINT so killed OpenCode runs keep child spans - #3

Merged
Yanir-D merged 1 commit into
mainfrom
feature/eng-3645-opencode-readwrite-spans-are-silently-lost-on-every-killed
Jul 21, 2026
Merged

part of: ENG-3645 - flush OTEL spans on SIGTERM/SIGINT so killed OpenCode runs keep child spans#3
Yanir-D merged 1 commit into
mainfrom
feature/eng-3645-opencode-readwrite-spans-are-silently-lost-on-every-killed

Conversation

@Yanir-D

@Yanir-D Yanir-D commented Jul 20, 2026

Copy link
Copy Markdown

What

Register once-guarded SIGTERM/SIGINT handlers that forceFlush() the batched OTEL spans and shut the exporter down before the OpenCode process is torn down — so a killed readwrite still exports its child generation/span observations. Also routes the existing server.instance.disposed event through the same guarded routine so shutdown happens exactly once regardless of trigger.

This is PART A of ENG-3645 — the ticket's own "actual span-loss fix".

Why

OpenTelemetry batches spans and (before this change) only exported them on the graceful session.idle / server.instance.disposed lifecycle events. Every investigation create_pull_request readwrite is killed before either fires:

  • pod soft-timeoutSIGTERM to the process group (opencode.py _signal_process_tree_sigterm), or
  • client-disconnect cancellation_kill_process_tree (killpg(SIGTERM) → grace → SIGKILL).

Neither path emits an OpenCode lifecycle event, so forceFlush() was never called and the whole batch died with the process. Result today: create_pull_request spans with zero child observations on every killed run (baseline inv-fc6f1ae7, traces 5ba603cc / 3843ca06, children=0).

The pod already reserves an OPENCODE_KILL_GRACE_S (10s) window after SIGTERM (ENG-3512) — used to salvage OpenCode's stdout NDJSON. This change makes the plugin flush its span batch inside that same window.

Design notes

  • No process.exit() — deliberate. OpenCode owns the process and still needs the grace window to flush its own stdout NDJSON (the source of the salvaged diff); exiting here would truncate it. The pod's post-grace SIGKILL remains the terminator.
  • Once-guarded, module-scoped so repeated plugin inits / a second instance cannot stack duplicate listeners or double-flush; server.instance.disposed shares the same guard.
  • Never throws — a failed flush/shutdown must not mask the original teardown.

Tests

bun test — 17 pass (5 new): handler registration, flush-then-shutdown, once-guard across repeated triggers, no-op when credentials are missing, and the no-process.exit contract. Full gate green: typecheck, lint, format:check, build.

Deploy chain (this PR alone ships nothing)

The plugin is pinned by commit in the code-intel image build (discovery-code services/tree-sitter-service/Dockerfile, git checkout <sha>). After this merges:

  1. Bump that pin to the merged SHA / release tag → discovery-code PR.
  2. discovery-code CI rebuilds + pushes tree-sitter-service (automatic).
  3. ArgoCD Image Updater bumps the digest in discovery-code-helm-chart → sync → pod redeploy (automatic).

Acceptance (ENG-3645 #1) — provable post-deploy only

The span-flush effect is observable only on a killed run. Post-deploy: trigger a create_pull_request readwrite that times out / is cancelled and confirm the investigation trace now shows child GENERATION/SPAN observation(s) under the create_pull_request span (model, prompt/output sizes, timed_out/resolution) — where today there are zero.

part of: ENG-3645

🤖 Generated with Claude Code

…child spans

OpenTelemetry batches spans and only exported them on the graceful
session.idle / server.instance.disposed lifecycle events. Every
investigation readwrite is killed (pod soft-timeout SIGTERM or
client-disconnect cancellation) before either event fires, so the
batch died with the process and every killed run silently lost all of
its child generation/span observations.

Register once-guarded SIGTERM/SIGINT handlers that forceFlush() the
batch then shut the exporter down within the OPENCODE_KILL_GRACE_S
window the pod already reserves after SIGTERM. Route the existing
server.instance.disposed event through the same guarded routine so
shutdown runs exactly once regardless of trigger.

Deliberately does not call process.exit(): OpenCode owns the process
and still needs the grace window to flush its own stdout NDJSON (the
source of the salvaged diff); the pod's post-grace SIGKILL remains the
terminator.

part of: ENG-3645

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Yanir-D Yanir-D self-assigned this Jul 21, 2026
@Yanir-D
Yanir-D merged commit beaaf93 into main Jul 21, 2026
5 checks passed
@Yanir-D
Yanir-D deleted the feature/eng-3645-opencode-readwrite-spans-are-silently-lost-on-every-killed branch July 21, 2026 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant