User Tracing

[mar-cf]

Overview

This adds a user-facing span pipeline to foundations, parallel to the existing internal tracing pipeline. Application code can emit spans into a separate USER_HARNESS that exports OTLP HTTP over a Unix domain socket (gRPC unsupported) to an OTLP endpoint, with per-trace routing metadata (an application-defined RoutingMetadata trait) carried on the wire and W3C traceparent continuation in/out.

The design is deliberately a mirror of internal tracing onto a second harness: the user API lives in a dedicated tracing::user_tracing module whose entry points (user_tracing::start_trace, user_tracing::span, user_tracing::add_span_tags!, …) mirror the top-level internal-tracing functions (start_trace, span, add_span_tags!, …), plus a top-level dual_span that records into both pipelines at once. Two hard rules shape the surface:

• The user and internal pipelines are independent — separate harnesses, scope stacks, and exporters
• Public surface speaks W3C.

There are two layers to keep separate:

• init-time: stand up USER_HARNESS + the exporter, pointed at the OTLP endpoint's socket.
• per-request activation: user_tracing::start_trace(...) opens a root. Without a root, every user_tracing::span / dual_span() / #[span_fn(user = true)] / user_tracing::add_span_tags! is a no-op.

---

User guide

Settings (init-time)

User tracing is configured via one optional block, gated by the user-tracing feature and fed to the existing foundations::telemetry::init:

TelemetrySettings.user_tracing: Option<UserTracingSettings>   // None = off

UserTracingSettings {
    enabled: bool,                       // default: false  (user tracing is opt-in)
    max_queue_size: Option<NonZeroUsize>,// default: 1_000_000; None = unbounded (mirrors internal)
    output: UserTracesOutput,            // only variant today: OtlpUds(..)
}

OtlpUdsOutputSettings {
    socket_path: String,        // ← path to the OTLP endpoint's UDS (required, no default)
                                //   e.g. "/path/to/otlp-receptor.sock"
    routing_header_name: String,// ← header the endpoint reads each span's encoded routing from
                                //   (required, no default) e.g. "cf-trace-config"
    num_tasks: usize,           // default: 1    (concurrent export workers)
    max_batch_size: usize,      // default: 512  (spans drained per export batch)
}

The two settings a consumer must choose are socket_path (the OTLP endpoint's UDS) and routing_header_name (the header the endpoint reads routing from). Unlike internal tracing, user tracing is off by default (enabled: false) and must be turned on explicitly; max_queue_size mirrors its internal-tracing equivalent; num_tasks and max_batch_size tune the exporter. All others have defaults and can be configured as needed.

There is deliberately no sampling configuration here. Unlike internal tracing, the user pipeline is not sampled inside foundations — the inbound user_tracing control header drives the activation (and therefore sampling) decision upstream.

Instrumentation APIs

Toy example covering the whole surface:

use foundations::telemetry::tracing::{self, RoutingMetadata, TraceparentContext};

// `RoutingMetadata` is a trait — you define the concrete routing type. `group_key` batches
// spans that share it into one request; `encode` produces the routing header's value.
#[derive(Debug)]
struct MyRouting { zone_id: u64, account_id: u64 }

impl RoutingMetadata for MyRouting {
    fn group_key(&self) -> String { format!("{}|{}", self.zone_id, self.account_id) }
    fn encode(&self) -> String { format!("zone={};account={}", self.zone_id, self.account_id) }
}

// Per request: open the root. `routing` is required and fixed at construction (inherited by all
// descendants); `inbound` continues an upstream W3C trace, or None for a fresh one.
let _root = tracing::user_tracing::start_trace(
    "example_span_name",
    MyRouting { zone_id, account_id },
    inbound,                       // Option<TraceparentContext>
);

// Children — pick whichever fits:
let _child = tracing::user_tracing::span("lookup"); // explicit standalone user child
let _s     = tracing::dual_span("db");              // parallel internal + user child

#[tracing::span_fn("handle_request", user = true)]  // whole-fn dual span (sync or async)
async fn handle_request() { /* ... */ }

// Annotate the current user span (no-op if no user trace is active):
tracing::user_tracing::add_span_tags!("cache.status" => "HIT");

// Across .await / spawn / separate hooks (UserSpanScope is !Send): hold the context.
let ctx = tracing::user_tracing::start_trace("req", routing, None).into_context(); // -> TelemetryContext
ctx.apply(async { let _c = tracing::user_tracing::span("work"); }).await;

// Outbound propagation to the next hop:
let traceparent: Option<String> = tracing::user_tracing::w3c_traceparent();

// Inbound parsing (strict W3C):
let inbound = TraceparentContext::parse(header_bytes); // -> Option<TraceparentContext>

Key points for users:

• The context carries the user span: into_context() / TelemetryContext::current() / #[span_fn] all propagate it across .await, even through an internal span's context — no manual threading.
• dual_span() and #[span_fn(user = true)] open a user span in parallel with an internal span — this matches the guideline to create an internal span for every user span, so a single call feeds both pipelines. dual_span names both spans from the same argument, so the user span keeps its name even when the internal trace is sampled out (the two pipelines sample independently).
• Forking a trace leaves the user trace intact — TelemetryContext's trace fork only forks the internal trace; user spans created in the forked context stay in the current user trace.

[mar-cf]

@inikulin PTAL

[mar-cf]

Will update the PR description shortly

[TheJokr]

LGTM now. Please rebase/squash or resolve conflicts with main, bump the cf-rustracing version, and add PipelineType::User for the metrics, then we can merge this.

[mar-cf]

@TheJokr All done, ready to go