chore: benchmark ClickHouse exporter on dedicated runner

[cijothomas]

Summary

• add an OTAP logs pipeline performance suite backed by a pinned ClickHouse 25.6 container
• build df_engine with the experimental clickhouse-exporter feature
• validate persisted ClickHouse rows and report produced/written rates plus normalized engine/backend resources
• temporarily narrow the label-triggered dedicated workflow to this suite while results are iterated

Validation

• python3 tools/sanitycheck.py
• orchestrator schema/template expansion
• exporter-enabled df_engine Docker build
• two local end-to-end suite runs
  • 1,374,304 persisted log rows
  • 1,406,720 persisted log rows
• benchmark JSON generated and all managed containers removed

Iteration notes

The suite requests 100k logs/sec with synchronous inserts for a deterministic correctness baseline. The local workstation sustained roughly 54k-63k ClickHouse rows/sec; the labeled dedicated run will establish behavior on the benchmark host.

No changelog entry: this is performance-test infrastructure only.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.51%. Comparing base (20f107a) to head (101d096).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3512      +/-   ##
==========================================
+ Coverage   86.23%   86.51%   +0.27%     
==========================================
  Files         766      782      +16     
  Lines      301992   313105   +11113     
==========================================
+ Hits       260437   270883   +10446     
- Misses      41031    41698     +667     
  Partials      524      524              

Components	Coverage Δ
otap-dataflow	87.54% <ø> (+0.28%)	⬆️
query_engine	89.57% <ø> (ø)
otel-arrow-go	52.45% <ø> (ø)
quiver	92.19% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[clhain]

This is probably way too low with sync inserts & 100k lrps

[cijothomas]

thanks. Will tune it to get highest perf/cost balance.

[cijothomas]

Updated in d0c9785: the suite now runs two isolated 60-second scenarios, a 50k logs/sec synchronous correctness baseline and a 100k logs/sec async-insert throughput case. Each uses a separate ClickHouse database and report window. The labeled dedicated run will provide the authoritative comparison.

[cijothomas]

Dedicated runner result: first synchronous baseline

• Workflow: https://github.com/open-telemetry/otel-arrow/actions/runs/29588900790
• Result: success
• Persisted rows: 1,389,504
• Requested rate: 100k logs/sec
• Produced rate: 62,267 logs/sec
• ClickHouse-written rate: 55,467 rows/sec
• Dataflow engine CPU: 35.5% of one allocated core
• ClickHouse CPU: 68.0% normalized across four allocated cores
• Dataflow engine max RAM: 31.9 MiB
• ClickHouse max RAM: 1,439.9 MiB

The pipeline is feasible and stable, but this is not yet a true 100k offered-load benchmark. The next iteration should either lower the synchronous fixed rate to an achievable baseline or enable async_insert and measure the production-oriented path.

[cijothomas]

Dedicated runner result: sync vs. acknowledged async inserts

• Workflow: https://github.com/open-telemetry/otel-arrow/actions/runs/29609270201
• Result: success

Scenario	Requested	Produced	Written	Persisted	Engine CPU	ClickHouse CPU	ClickHouse max RAM
Sync correctness	50k/s	50,203/s	50,209/s	3,185,840	32.4% of 1 core	67.7% of 4 cores	2,480.6 MiB
Async + wait	100k/s	8,089/s	7,249/s	513,024	5.3% of 1 core	26.1% of 4 cores	823.5 MiB

Both scenarios used isolated databases and 60-second observation windows. The synchronous baseline now holds its requested rate.

The async result is reproducibly slower locally and on the dedicated host. The exporter enables both async_insert=1 and wait_for_async_insert=1; with the current small batches, each request waits for ClickHouse async-buffer flush. This scenario is therefore an acknowledged-async regression signal, not fire-and-forget maximum throughput. Changing that behavior belongs in exporter configuration/semantics rather than masking it in the benchmark harness.

[cijothomas]

ClickHouse-backed loss reporting added

Commit e0f1a7c now calculates loss using the same observation-window semantics as the existing fake-backend suites:

1. Snapshot traffic generator logs_produced_total and ClickHouse count() at observation start.
2. Snapshot both again at observation stop.
3. Report (produced_delta - persisted_delta) / produced_delta * 100 as dropped_logs_percentage.
4. Query ClickHouse once more after generator and engine drain and retain logs_persisted_final as a separate diagnostic.

Local validation:

• Sync: 2,998,480 produced, 2,884,832 persisted by window end, 3,036,032 final; 3.79% observation-window loss.
• Async: 412,320 produced, 406,496 persisted by window end, 536,032 final; 1.41% observation-window loss.

The final count is intentionally not used in the window loss percentage because it also includes warm-up and shutdown-adjacent rows.

[clhain]

Dedicated runner result: sync vs. acknowledged async inserts

• Workflow: https://github.com/open-telemetry/otel-arrow/actions/runs/29609270201
• Result: success

Scenario Requested Produced Written Persisted Engine CPU ClickHouse CPU ClickHouse max RAM
Sync correctness 50k/s 50,203/s 50,209/s 3,185,840 32.4% of 1 core 67.7% of 4 cores 2,480.6 MiB
Async + wait 100k/s 8,089/s 7,249/s 513,024 5.3% of 1 core 26.1% of 4 cores 823.5 MiB
Both scenarios used isolated databases and 60-second observation windows. The synchronous baseline now holds its requested rate.

The async result is reproducibly slower locally and on the dedicated host. The exporter enables both async_insert=1 and wait_for_async_insert=1; with the current small batches, each request waits for ClickHouse async-buffer flush. This scenario is therefore an acknowledged-async regression signal, not fire-and-forget maximum throughput. Changing that behavior belongs in exporter configuration/semantics rather than masking it in the benchmark harness.

Ugh yes - sorry I didn't realize that the suggestion to add that wait=1 as the default got accepted after it was pointed out that this was a bad idea for the engine's architecture.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Status last refreshed: 2026-07-22 19:57:40 UTC.

• Waiting on: Author
• Next step: Move out of draft to request review.

This automated status or its linked feedback items may be incorrect. If something looks wrong, please report it with the result you expected.

[cijothomas]

Loss metric correction

Commit 101d096 supersedes the observation-boundary loss calculation from e0f1a7c. Boundary deltas can go slightly negative when ClickHouse flushes pre-window backlog during the observation window.

The corrected metric compares two exact totals after both pipelines drain:

• traffic generator terminal cumulative logs_produced snapshot retained by orchestrator telemetry
• final SELECT count() FROM <scenario>.otel_logs against the fresh per-scenario ClickHouse database

The validation CSV is deleted before startup, so failed runs cannot report stale counts.

Local exact-total validation:

• Sync: 2,351,344 generated, 2,259,872 persisted, 91,472 lost (3.89%).
• Async: 654,304 generated, 525,792 persisted, 128,512 lost (19.64%).

The new dedicated run will provide authoritative host values.