[BUG]: Baggage accumulates across unrelated Hangfire jobs

[davewebscale]

Tracer Version(s)

3.44.0

Operating system and platform

Linux arm64

Instrumentation Mode

automatic

TFM

net8.0

Bug Report

Baggage.Current accumulates across sequential Hangfire jobs, causing oversized baggage headers on downstream HTTP calls

Summary

When using the Hangfire integration with W3C baggage propagation enabled (the default in v3.x),
Baggage.Current appears to grow monotonically across sequential jobs processed by the same worker. Each job's baggage is merged into whatever ambient state already exists rather than replacing it, so jobs accumulate entries from all prior jobs on that worker.
The result is that outbound HTTP calls made by those jobs carry a baggage header that grows without bound,
composed of entries from many prior, unrelated jobs, until it is large enough to be rejected by downstream services enforcing header size limits.

Environment

• Datadog.Trace v3.x (W3C baggage propagation enabled by default; not present in v2.x)
• Hangfire background job workers with full APM attached (CORECLR_ENABLE_PROFILING=1)
• Any downstream HTTP service enforcing a maximum combined request header size

What we believe is happening

1. Baggage.Current is a mutable object stored in an AsyncLocal

Baggage.cs backs Baggage.Current with a single static AsyncLocal<Baggage>, where Baggage is a mutable reference type. AsyncLocal<T> isolation is copy-on-write on the reference — it does not isolate in-place mutations to the object the reference points to. Calling MergeInto on Baggage.Current mutates the shared instance's internal list directly.

2. OnPerforming merges extracted baggage into the existing Baggage.Current

DatadogHangfireServerFilter.OnPerforming calls MergeBaggageInto(Baggage.Current), which adds entries from the extracted job context into Baggage.Current without clearing it first.

3. OnPerformed does not reset Baggage.Current

DatadogHangfireServerFilter.OnPerformed disposes the span scope only. The AsyncLocalScopeManager uses a completely separate AsyncLocal<Scope> with no interaction with Baggage.AsyncStorage. Disposing a scope has no effect on Baggage.Current.

4. Hangfire workers reuse the same ExecutionContext across sequential jobs

The job dispatch loop in BackgroundProcessDispatcherBuilder.ExecuteProcess runs on a dedicated background Thread (see BackgroundProcessExtensions.cs) in a plain while loop calling Worker.Execute once per job, with no ExecutionContext operation between iterations. AsyncLocal values written during job N are still present when job N+1 begins.

5. Hangfire's own AsyncLocal mitigation does not cover server filters

CoreBackgroundJobPerformer.InvokeSynchronously
shows Hangfire is already aware of this class of problem — it wraps synchronous job method invocation in ExecutionContext.Run specifically to prevent AsyncLocal mutations inside the job body from persisting to future jobs, noting in a comment that without this, such mutations "can result in memory leaks and possibly affect future background job method executions in unexpected ways."

However, DatadogHangfireServerFilter.OnPerforming fires before InvokeSynchronously, in BackgroundJobPerformer.InvokeServerFilter, which calls OnPerforming before recursing to the inner performer. The MergeBaggageInto mutation therefore occurs on the worker thread's own ambient context, outside any ExecutionContext.Run sandbox.

Current workarounds

We have applied baggage item and byte caps via DD_TRACE_BAGGAGE_MAX_ITEMS and DD_TRACE_BAGGAGE_MAX_BYTES. These reduce the maximum size of any single extraction but do not prevent accumulation across jobs — repeated merges can still push the cumulative total past the cap over time.

We have also tested disabling baggage from the propagation style via DD_TRACE_PROPAGATION_STYLE_EXTRACT and DD_TRACE_PROPAGATION_STYLE_INJECT, which stops the growth but loses baggage-based cross-service correlation.

Reproduction Code

No response

[chojomok]

@davewebscale - hey there! I'm working on a fix for this. In the meantime, would you be able to open a support ticket with us. Please mention this github issue. Thank you!