Summary
When a test run is instrumented with distributed tracing, the entire run collapses into a single trace. That trace is long-lived (many minutes) and very large (tens of thousands of spans), which makes it hard to use in practice and, on common backends, causes spans to be silently dropped. This proposes scoping traces per test so each test is its own bounded, queryable trace.
Where this comes from
We run Inferno with OpenTelemetry auto-instrumentation (Faraday, Net::HTTP, Sidekiq) added around the worker. With that in place we saw a single trace per run: roughly 15 minutes and about 10,000 spans covering the worker, the validator, and the terminology server, with the root span not yet closed while the run was still going.
The cause is the execution model rather than the instrumentation. Inferno::Jobs::ExecuteTestRun#perform runs the whole run in one Sidekiq job via TestRunner#start, which recurses through every group and test in a single call stack (lib/inferno/test_runner.rb). So whatever span sits at the top of that job becomes the trace root, and every downstream HTTP call the run makes nests under it. Any tracing setup added on top inherits this shape.
Why one trace per run is a problem
This is independent of any particular vendor:
- Long-running, very large traces are hard to parse. A trace is meant to represent a single logical operation. A multi-minute waterfall with thousands of spans is not something a trace UI renders well or a human reads usefully.
- Backends drop the overflow. Trace stores impose a per-trace size limit (for example Grafana Tempo's
max_bytes_per_trace, 5 MB by default) and silently discard spans past it, so the tail of a large run is simply lost.
The net effect is that once tracing is added, the telemetry is close to unusable.
Proposal
Make test execution trace-aware so each test is its own trace (or at least its own root span), correlated back to the run. Two options, not mutually exclusive:
- Native, optional OpenTelemetry instrumentation in inferno-core that opens a root span per test and tags it with the run id. Off unless configured.
- A lightweight, overridable hook around test execution (for example an
around_test style callback in TestRunner#run_test) so external instrumentation can scope traces per test without reaching into internals.
Correlation across the tests of a run can use a shared attribute (a test_run_id) or, more idiomatically, OpenTelemetry span links back to a run level context.
What we do downstream today
As a stopgap we prepend TestRunner#run_test to start a fresh root span per test:
module PerTestTraceRoot
def run_test(test, scratch)
tracer = OpenTelemetry.tracer_provider.tracer('inferno')
# Detach from the enclosing run/job span so the test span starts a new trace.
OpenTelemetry::Context.with_current(OpenTelemetry::Context.empty) do
tracer.in_span("inferno.test #{test.id}",
attributes: { 'inferno.test_run_id' => test_run.id, 'inferno.test_id' => test.id }) do
super
end
end
end
end
Inferno::TestRunner.prepend(PerTestTraceRoot)This turns the single per-run trace into one bounded trace per test, each carrying its downstream calls as children and a shared inferno.test_run_id. It works, but depending on internal method signatures is fragile, which is why we think per-test scoping (or a supported hook) belongs in inferno-core.
Happy to raise a PR for whichever direction maintainers prefer.
Summary
When a test run is instrumented with distributed tracing, the entire run collapses into a single trace. That trace is long-lived (many minutes) and very large (tens of thousands of spans), which makes it hard to use in practice and, on common backends, causes spans to be silently dropped. This proposes scoping traces per test so each test is its own bounded, queryable trace.
Where this comes from
We run Inferno with OpenTelemetry auto-instrumentation (Faraday, Net::HTTP, Sidekiq) added around the worker. With that in place we saw a single trace per run: roughly 15 minutes and about 10,000 spans covering the worker, the validator, and the terminology server, with the root span not yet closed while the run was still going.
The cause is the execution model rather than the instrumentation.
Inferno::Jobs::ExecuteTestRun#performruns the whole run in one Sidekiq job viaTestRunner#start, which recurses through every group and test in a single call stack (lib/inferno/test_runner.rb). So whatever span sits at the top of that job becomes the trace root, and every downstream HTTP call the run makes nests under it. Any tracing setup added on top inherits this shape.Why one trace per run is a problem
This is independent of any particular vendor:
max_bytes_per_trace, 5 MB by default) and silently discard spans past it, so the tail of a large run is simply lost.The net effect is that once tracing is added, the telemetry is close to unusable.
Proposal
Make test execution trace-aware so each test is its own trace (or at least its own root span), correlated back to the run. Two options, not mutually exclusive:
around_teststyle callback inTestRunner#run_test) so external instrumentation can scope traces per test without reaching into internals.Correlation across the tests of a run can use a shared attribute (a
test_run_id) or, more idiomatically, OpenTelemetry span links back to a run level context.What we do downstream today
As a stopgap we prepend
TestRunner#run_testto start a fresh root span per test:This turns the single per-run trace into one bounded trace per test, each carrying its downstream calls as children and a shared
inferno.test_run_id. It works, but depending on internal method signatures is fragile, which is why we think per-test scoping (or a supported hook) belongs in inferno-core.Happy to raise a PR for whichever direction maintainers prefer.