Skip to content

chore(pipeline-perf): add container CPU-seconds metric#3515

Draft
cijothomas wants to merge 1 commit into
open-telemetry:mainfrom
cijothomas:perf/cpu-time-per-log
Draft

chore(pipeline-perf): add container CPU-seconds metric#3515
cijothomas wants to merge 1 commit into
open-telemetry:mainfrom
cijothomas:perf/cpu-time-per-log

Conversation

@cijothomas

@cijothomas cijothomas commented Jul 17, 2026

Copy link
Copy Markdown
Member

Summary

Adds a cgroup-based CPU-seconds metric to the pipeline-perf report:
cpu_seconds — the CPU time a component consumes during the observation window,
derived from a new cumulative container.cpu.time gauge (docker stats
total_usage).

Unlike the existing container.cpu.usage (an instantaneous, wall-clock-normalized
utilization ratio), cpu_seconds is an absolute work measure integrated over
the observation window. In a rate-limited test, fixed input work => fixed
CPU-seconds, which makes it a clean, stable regression signal. It needs no
perf/PMU, so it is available on GitHub-hosted runners as well as dedicated
hardware.

Changes

  • Docker monitor (docker_component.py): record a cumulative
    container.cpu.time gauge (seconds) alongside the existing
    container.cpu.usage.
  • Report (integration_report_logs.yaml): add a component_cpu_time table
    (MAX(value) - MIN(value) over the observation window) and emit a cpu_seconds
    row in gh_actions_benchmark.

Validation

Three label-triggered runs on the dedicated bare-metal runner (full suite).
cpu_seconds run-to-run CoV (df-engine):

Scenario CoV
OTLP-ATTR-OTLP 0.05%
OTLP-ATTR-OTAP 0.09%
OTAP-ATTR-OTAP 0.19%
OTAP-ATTR-OTLP 0.04%
OTLP-BATCH-OTLP 0.11%
OTAP-BATCH-OTAP 0.06%

All under 0.2% — a solid, low-variance signal for regression detection. Also
verified via a DuckDB dry-run of the report SQL and python3 tools/sanitycheck.py.

Notes / scope

  • This is intentionally minimal: one absolute CPU-work metric. It complements the
    existing container.cpu.usage/throughput metrics without adding overlapping
    per-log or hardware-counter columns.
  • On GitHub-hosted runners cpu_seconds carries ~1% run-to-run noise (shared-VM
    frequency scaling / neighbors), so it is suitable there only as a coarse gate
    for large regressions; precise tracking belongs on the dedicated/nightly runs.

Kept as draft.

@github-actions github-actions Bot added the ci-repo Repository maintenance, build, GH workflows, repo cleanup, or other chores label Jul 17, 2026
@cijothomas cijothomas added the pipelineperf Trigger for the pipeline performance job on PRs label Jul 17, 2026
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.51%. Comparing base (601c953) to head (17009b2).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3515      +/-   ##
==========================================
+ Coverage   86.25%   86.51%   +0.25%     
==========================================
  Files         767      782      +15     
  Lines      302274   313105   +10831     
==========================================
+ Hits       260729   270876   +10147     
- Misses      41021    41705     +684     
  Partials      524      524              
Components Coverage Δ
otap-dataflow 87.53% <ø> (+0.26%) ⬆️
query_engine 89.57% <ø> (ø)
otel-arrow-go 52.45% <ø> (ø)
quiver 92.19% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cijothomas cijothomas added pipelineperf Trigger for the pipeline performance job on PRs and removed pipelineperf Trigger for the pipeline performance job on PRs labels Jul 17, 2026
@cijothomas cijothomas added pipelineperf Trigger for the pipeline performance job on PRs and removed pipelineperf Trigger for the pipeline performance job on PRs labels Jul 17, 2026
Record a cumulative container.cpu.time gauge from cgroup CPU accounting
(docker stats total_usage) and surface per-window cpu_seconds in the report.

Unlike container.cpu.usage (an instantaneous, wall-clock-normalized utilization
ratio), cpu_seconds is an absolute work measure integrated over the observation
window. In a rate-limited test it is a clean, stable regression signal, and it
needs no perf/PMU so it is available on GitHub-hosted runners as well as
dedicated hardware. Validated on the dedicated runner: run-to-run CoV 0.04-0.19%.
@cijothomas
cijothomas force-pushed the perf/cpu-time-per-log branch from 8c2e99d to 17009b2 Compare July 18, 2026 07:14
@cijothomas cijothomas changed the title chore(pipeline-perf): add CPU-time-per-log metric chore(pipeline-perf): add container CPU-seconds metric Jul 18, 2026
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Jul 18, 2026

Copy link
Copy Markdown

Pull request dashboard status

Status last refreshed: 2026-07-22 19:57:43 UTC.

  • Waiting on: Author
  • Next step: Move out of draft to request review.

This automated status or its linked feedback items may be incorrect. If something looks wrong, please report it with the result you expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-repo Repository maintenance, build, GH workflows, repo cleanup, or other chores pipelineperf Trigger for the pipeline performance job on PRs

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant