df_engine: all pipeline cores die under real OTLP load (boo panic in metrics record encoder + arrow DictionaryKeyOverflowError), process stays alive and reports healthy

[henrikrexed]

Pre-filing checklist

• I searched existing issues and didn't find a duplicate

Component(s)

Rust OTAP dataflow (rust/otap-dataflow/)

Bug Description

Under sustained real-world OTLP traffic, df_engine panics on every pipeline core and
the engine never recovers. The process stays alive and keeps accepting connections while
processing nothing, so orchestration sees a healthy pod.

Two distinct panic sites fired in one 30-second window:

1. crates/pdata/src/encode/record/metrics.rs:266 — panic!("boo {lens:?}") in the
   metrics record encoder's column-length check(). This appears to be already known:
   draft PR fix: stop metrics otap encode panic #2984 ("fix: stop metrics otap encode panic") root-causes it as a shared
   protobuf-cursor misuse producing inconsistent column row counts. That PR is open,
   draft and unmerged, so the fix is not in main as of the commit above. Reporting it
   here only as an independent real-world reproduction — a data point for fix: stop metrics otap encode panic #2984, not a
   new bug.

2. arrow-data-58.3.0/src/transform/mod.rs:680 — MutableArrayData::new is infallible: DictionaryKeyOverflowError. I could not find any existing issue for this, and it
   was the majority failure — 3 of 4 cores. It is not obviously metrics-specific;
   MutableArrayData::new is the generic array-merge path, so ordinary high-cardinality
   span or log attributes look sufficient to trigger it.

The most operationally serious part is neither panic but the aftermath: a core that dies
is never restarted and the failure is invisible to every process-level health signal.

Steps to Reproduce

1. Build otap-df at 7502e7d (submodules recursive,
   upstream Dockerfile unmodified):
   cargo build --locked --release --target=x86_64-unknown-linux-gnu
   with RUSTFLAGS=-C target-cpu=x86-64-v2; runtime gcr.io/distroless/cc-debian13:nonroot.
2. Run it with 4 pipeline cores and the config in the Configuration field — one shared
   chain for all three signals, so a metrics panic takes traces and logs with it.
3. Point sustained OTLP traffic at it. I used the opentelemetry-demo Helm chart
   0.40.10 (appVersion 2.2.0, ~20 services) with every service's
   OTEL_EXPORTER_OTLP_ENDPOINT set to the engine, plus a service mesh emitting
   access-log spans and a second demo application ( Google Micro service demo). Ordinary smoke-test volume was
   enough — we never reached our intended load level.
4. Watch the engine log for panicked at and watch the pod's readiness.

Deterministic: reproduced on two independent pods. Time-to-death shortens with
backlog, because upstream SDKs queue telemetry and retry it as a burst — run 1 survived
3m08s, run 2 survived ~48s. That rules out periodic restarts as a mitigation, since each
restart meets a larger backlog than the last.

Expected Behavior

1. Encoding a batch of ordinary OTLP telemetry should not panic.
2. If a pipeline core does panic, pipeline_runtime_failed should be recoverable —
   restart the core or start a new generation — rather than terminal.
3. An engine whose cores have all died should FAIL its health/readiness surface. Reporting
   Running/Ready while processing nothing is the failure mode that costs downstream
   users the most.

On (3) there may be an easy win: the admin server already has the data. GET /api/v1/metrics?format=json&keep_all_zeroes=true returns the internal metric set per
node per core, so a dead core shows as its receiver.otlp / processor.* counters
going flat while sibling cores advance — no log grepping needed.

Actual Behavior

Each panic kills its pipeline core and logs pipeline_runtime_failed. There is no
restart, no new generation, no recovery. Once all four cores are down:

• the process is still running and still accepting connections,
• Kubernetes reports the pod Running, Ready, restarts=0,
• throughput counters keep their last healthy cumulative values, so a dashboard built on
  cumulative counters shows no drop.

I nearly banked a two-hour benchmark run on an engine that had been dead for six
minutes, because every liveness signal derived from the process was green. The only
thing that caught it was grepping the log for panic.

OTel-Arrow Version

7502e7d (main, 2026-07-20)

Environment

• Deployment: Kubernetes, 4-core pod, gcr.io/distroless/cc-debian13:nonroot
• Build: cargo build --locked --release --target=x86_64-unknown-linux-gnu,
  RUSTFLAGS=-C target-cpu=x86-64-v2
• arrow-data 58.3.0
• Load: opentelemetry-demo chart 0.40.10 (appVersion 2.2.0), ~20 services, traces +
  logs + metrics into one pipeline, plus mesh access-log spans and a second application
• Backend: an OTLP/HTTP endpoint that returns 400 on cumulative Sum/Histogram/Summary
  (relevant only as corroboration that the metrics payload was independently judged
  malformed at the same moment the encoder panicked)

Configuration

version: otel_dataflow/v1
engine: {}
groups:
  default:
    pipelines:
      main:
        policies:
          channel_capacity:
            control: { node: 100, pipeline: 100 }
            pdata: 128
        nodes:
          otlp_in:
            type: receiver:otlp
            config:
              protocols:
                grpc: { listening_addr: 0.0.0.0:4317 }
                http: { listening_addr: 0.0.0.0:4318 }
          enrich:
            type: processor:attribute
            config:
              actions:
                - { key: some.static.key, action: upsert, value: some-value }
          parity:
            type: processor:transform
            config:
              kql_query: "logs | extend severity_text = 'ERROR'"
          batch:
            type: processor:batch
            config:
              otap: { min_size: 1000, sizer: items }
              max_batch_duration: 3s
          out:
            type: exporter:otlp_http
            config:
              endpoint: https://<otlp-endpoint>/api/v2/otlp
              client_pool_size: 4
              http:
                headers:
                  Authorization: "<redacted>"
        connections:
          - { from: otlp_in, to: enrich }
          - { from: enrich,  to: parity }
          - { from: parity,  to: batch }
          - { from: batch,   to: out }

Log Output

thread 'pipeline-default-main-core-3-gen-0' (14) panicked at crates/pdata/src/encode/record/metrics.rs:266:13:
boo [21, 21, 21, 7, 21, 21, 21, 21, 21, 21]
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2026-07-23T10:53:34.869Z  ERROR otap-df-controller::controller.pipeline_runtime_failed: Pipeline terminated with a runtime error [core_id=3]

thread 'pipeline-default-main-core-1-gen-0' (12) panicked at /usr/local/cargo/registry/src/index.crates.io-.../arrow-data-58.3.0/src/transform/mod.rs:680:31:
MutableArrayData::new is infallible: DictionaryKeyOverflowError
2026-07-23T10:53:52.252Z  ERROR otap-df-controller::controller.pipeline_runtime_failed: Pipeline terminated with a runtime error [core_id=1] entity/pipeline.attrs: pipeline.id=main pipeline.group.id=default deployment.generation=0 core.id=1 numa.node.id=0

thread 'pipeline-default-main-core-0-gen-0' (11) panicked at /usr/local/cargo/registry/src/index.crates.io-.../arrow-data-58.3.0/src/transform/mod.rs:680:31:
MutableArrayData::new is infallible: DictionaryKeyOverflowError
2026-07-23T10:53:58.263Z  ERROR otap-df-controller::controller.pipeline_runtime_failed: Pipeline terminated with a runtime error [core_id=0]

thread 'pipeline-default-main-core-2-gen-0' (13) panicked at /usr/local/cargo/registry/src/index.crates.io-.../arrow-data-58.3.0/src/transform/mod.rs:680:31:
MutableArrayData::new is infallible: DictionaryKeyOverflowError
2026-07-23T10:53:59.582Z  ERROR otap-df-controller::controller.pipeline_runtime_failed: Pipeline terminated with a runtime error [core_id=2]

# after this point: process alive, pod Ready, restarts=0, zero records processed

Additional Context

Relationship to existing issues (from the pre-filing search):

• Panic site 1 looks like the bug draft PR fix: stop metrics otap encode panic #2984 already fixes. The boo string is a
  column-length consistency check() and the reported lengths [21, 21, 21, 7, 21, ...]
  show exactly the one-column row-count mismatch that PR describes. Treat this report as
  corroboration that it still reproduces on main and does so under ordinary load.
• Issue Controller extension panic bypasses fail-fast shutdown and wedges the engine in run_forever #3401 ("Controller extension panic bypasses fail-fast shutdown and wedges the
  engine in run_forever") is the same theme as our recovery complaint but a different
  path — that one is a controller extension, ours is a pipeline core. Whatever policy
  comes out of Controller extension panic bypasses fail-fast shutdown and wedges the engine in run_forever #3401 would ideally cover both.
• I found nothing for DictionaryKeyOverflowError / MutableArrayData in this engine.

Workaround I adopted, in case it helps others: insert processor:type_router
immediately after the receiver and terminate the metrics port in exporter:noop, so the
metrics encoder is never reached:

router:       { type: processor:type_router, outputs: [logs, metrics, traces], config: {} }
metrics_noop: { type: exporter:noop, config: {} }
# connections: otlp_in -> router; router["metrics"] -> metrics_noop; router["logs"|"traces"] -> ...

With that plus max_batch_duration reduced 3s -> 1s, the same workload ran 41 minutes
with zero panics where it previously died in 30 seconds. ⚠️ I changed two things at
once and cannot attribute the improvement: cutting the batch window also reduces how many
records merge into each array, which plausibly suppresses the dictionary overflow on its
own. So this is not evidence that excluding metrics is sufficient — panic site 2 was
never addressed and we believe it can still fire on high-cardinality traces or logs.

Separately, --validate-and-exit accepts configurations that cannot work at runtime.
Happy to split these out if you'd prefer separate issues:

• processor:filter accepts config: {__bogus__: 1} and prints Configuration is valid.
• processor:type_router with config: {} and no outputs: declared at all validates
  clean, although nothing would be routed by name.
• the KQL processor:transform validates predicates it cannot evaluate at runtime —
  contains, ==, matches regex and replace_regex all fail at runtime against a
  config that validates clean.

[henrikrexed]

Correction and new data: the workaround delays the dictionary overflow, it does not prevent it

Correcting my own report above, and adding a panic site I hadn't seen when I filed.
In the original Additional Context I wrote that with type_router + metrics to noop, plus max_batch_duration cut 3s → 1s, "the same workload ran 41 minutes with zero panics". I flagged that i had changed two things at once and couldn't attribute the improvement. That caveat was right, and it has now been settled the unhappy way — please don't read the workaround as a fix.
That 41-minute run was on a deliberately reduced reproducer (no service mesh, fewer applications). Running the same corrected configuration against the full original workload, the engine died the same way:

	original config	corrected config (router → noop, batch 1s)
boo — crates/pdata/src/encode/record/metrics.rs:266	T+5.6s	never — 0 occurrences
first DictionaryKeyOverflowError	T+23.0s	T+26m10.5s
all four cores dead	T+30.3s	T+33m34.2s
What that settles:

1. Routing metrics away genuinely removes the boo site — zero occurrences, as expected by construction, since the metrics encoder is no longer reachable. That part of the workaround works.
2. The dictionary overflow is accumulation-driven, not batch-size-gated. Cutting the batch window gives ~68× more life and then the identical failure. Smaller batches merge fewer distinct values into each output array, so the key space is exhausted later, not never. Anyone adopting the workaround should size their expectations in minutes, not treat it as a resolution.
3. Consequently the overflow is reachable on traces and logs alone, with the metrics path entirely disconnected. It is not metrics-specific.

New: a second overflow site, inside otap-dataflow itself — and it fires first

The original report only ever saw the arrow-data call site. In this run the earliest death was in df_engine's own code:

thread 'pipeline-default-main-core-0-gen-0' (12) panicked at crates/pdata/src/otap/transform/concatenate.rs:150:43:
Compatible schemas: DictionaryKeyOverflowError

Two of four cores died there (T+26m10.5s, the earliest), the other two at the previously reported arrow-data-58.3.0/src/transform/mod.rs:680. So the overflow surfaces at two independent concatenation points, one of them an expect() inside this repository rather than in a dependency. That seems worth knowing: it isn't a single unlucky call site in arrow-data, and a fix confined to the dependency boundary would likely leave concatenate.rs:150 exposed.
Everything else in the original report stands — including the part I'd still most like your view on: all four cores were dead while the process stayed up, the pod stayed Ready with restarts=0, and the cumulative counters held their last healthy values. In this second run the engine sat there for a further ~40 minutes, resident memory still climbing, looking healthy to every process-level signal.

[AaronRM]

@henrikrexed Thank you for the detailed issue report(s)! We are going to use this issue (#3561) as the umbrella incident tracker and split concrete work into child sub-issues so each bug/fix path can be triaged and closed independently.

This keeps the original report intact while giving us cleaner execution and ownership per problem area. We will link each child here and use this thread for roll-up status.

We will keep #3561 open until the child issues are resolved and the end-to-end failure mode described here is verified fixed.

[AaronRM]

@henrikrexed Thanks again for the detailed report. In my attempt at a local repro, I noted that the admin readiness endpoint (/api/v1/readyz) reports failure (503) with a runtime/readiness reason (not memory pressure). That suggests the “pod stayed Ready while all cores were dead” behavior is likely from Kubernetes probe wiring (for example, probing process/TCP/liveness rather than /api/v1/readyz).

Could you share the exact Pod readinessProbe / livenessProbe config used in the failing runs? That will let us determine whether this is an engine readiness bug, a deployment default mismatch, or both.

[AaronRM]

The 'KQL processor:transform validates predicates it cannot evaluate at runtime' issue is tracked by #1634