feat(sidecar): OpenTelemetry spans + structured audit log + Jaeger compose

[VibhorGautam]

Summary

fills the Phase 2 telemetry scaffolding in sidecar/internal/telemetry/. adds async, non-blocking OpenTelemetry spans across the 4 pipeline stages and a structured JSON audit log, without touching the Stage interface or any stage-level code. an opt-in docker compose --profile observability up -d brings up a local OTel collector and a Jaeger instance so spans are inspectable at http://localhost:16686

closes the scaffolding otel.go and audit.go left behind in phase02. schema declared in the audit.go doc block is preserved exactly: hook_type, decision, score, signals, provenance, session_id, policy_version, trace_id

What changed

• sidecar/internal/telemetry/otel.go - tracer provider wired to OTLP HTTP, ParentBased sampler, graceful Shutdown(ctx). empty endpoint installs a noop provider so dev and prod code paths stay identical
• sidecar/internal/telemetry/audit.go - async audit writer. buffered channel, 1 drain goroutine, atomic drop counter. Emit never blocks the caller
• sidecar/internal/pipeline/pipeline.go - root span per run, child span per stage. duration, signals-added, hard_block, decision and score land on the spans. 1 audit line emitted after the run
• sidecar/cmd/sidecar/main.go - reads the new telemetry: block from sidecar.yaml, opens the audit file (or stdout), flushes both on SIGTERM
• docker-compose.yml - opt-in observability profile adds the OTel collector and Jaeger. sidecar service unaffected unless the profile is enabled
• config/otel-collector.yaml - receives OTLP HTTP on :4318 and forwards to Jaeger on :4317
• docs/observability.md - privacy rules, sampling guidance, what the audit writer deliberately never records (raw payload, canonical text)

new tests:

• sidecar/internal/telemetry/audit_test.go - JSON shape, drop accounting under backpressure, concurrent Emit, emit-after-close no-op
• sidecar/internal/telemetry/otel_test.go - noop fallback, exporter init with a dead endpoint does not panic, scheme stripping
• sidecar/internal/pipeline/pipeline_telemetry_test.go - full span tree, OPA input/output attributes, audit trace correlation, strict short-circuit and OPA error fallback
• sidecar/internal/transport/drain_test.go - graceful drain waits for the serve loop and closes stalled connections
• sidecar/internal/pipeline/bench_test.go - 3 benchmark configurations

0 existing tests changed. all 4 CI jobs are green, including ubuntu and windows integration

Design notes

non-invasive integration. the Stage interface is untouched. Pipeline.Run wraps stage.Run with a child span and reads rc.Signals before and after the call to compute signals.added. stages stay pure; only the pipeline dispatcher knows about spans

fail-open on the observability path. neither the OTel exporter nor the audit writer can block enforcement. audit is a buffered channel with a background drain goroutine; when the queue is full, entries are dropped and the drop counter increments. if the OTel collector is down the exporter buffers internally and drops silently. telemetry failures must not take down the PDP

privacy. the audit writer never records rc.Payload or rc.CanonicalText. only decision metadata and named signals reach the sink. span attributes are the same controlled set. same idea as the LLM-05 prompt-leakage guidance, adapted to the PDP path

compatibility. old pipeline.New(cfg, stages) still works and keeps the existing threshold fallback behaviour. new pipeline.NewWithOptions takes a Tracer and AuditSink plus a PolicyVersion string stamped onto every audit entry

parent context. Pipeline.Run(rc) stays unchanged and kicks off a fresh root span. Pipeline.RunContext(ctx, rc) is a new entry point that takes a caller-supplied context so the ParentBased sampler honours any upstream sampling decision and spans link into an existing trace. the transport listener does not yet propagate context because the SDK frame does not carry traceparent, but when it does the listener can swap to RunContext without any pipeline-level change

OTel version. pinned to go.opentelemetry.io/otel v1.40.0 and semconv v1.26.0 (aligned with the version OPA already pulls in). tested on go 1.25.6

Benchmarks

go test -run '^$' -bench '^BenchmarkPipeline_' -benchmem -benchtime=2s -count=5 -cpu=1 ./internal/pipeline on Apple M4:

BenchmarkPipeline_NoTelemetry          7508 ns/op   1776 B/op   29 allocs/op
BenchmarkPipeline_WithAudit            8617 ns/op   2202 B/op   30 allocs/op
BenchmarkPipeline_WithTracerAndAudit  12716 ns/op   5824 B/op   45 allocs/op

audit adds about 1.11 µs per run. real span recording plus audit adds about 5.21 µs over the no-telemetry path, with tracing itself adding about 4.10 µs over audit-only. this measures in-process span recording and audit queueing, not OTLP network or collector cost

Audit line sample

{"ts":"2026-04-13T09:35:12.521Z","trace_id":"6e8a...","span_id":"9f1...","hook_type":"on_prompt","decision":"block","score":0.9,"signals":["jailbreak_pattern"],"provenance":"user","session_id":"sess-42","policy_version":"v1","blocked_at":"scan","duration_ms":1.7}

How to run locally

# start everything
docker compose --profile observability up -d

# point the sidecar at the collector
cat >> config/sidecar.yaml <<'YAML'
telemetry:
  otel_endpoint: http://otel-collector:4318
  sample_ratio: 1.0
  service_name: acf-sidecar
  insecure: true
YAML

# open the UI
open http://localhost:16686

trace hierarchy should look like:

pipeline.Run (1.7 ms)
├── stage.validate    (0.1 ms)
├── stage.normalise   (0.4 ms)
├── stage.scan        (0.8 ms)  signals.added=1, hard_block=false
├── stage.aggregate   (0.3 ms)
└── opa.evaluate      (0.2 ms)  opa.output.decision=block

Scope decisions and non-goals

• no metrics added. spans + audit cover the current PDP observability scope
• signals are a span attribute, not span events. event cardinality on a busy sidecar would explode. signals stay in a stable controlled vocabulary
• no SDK frame changes. the SDK frame does not carry a traceparent yet; happy to open a follow-up issue if useful
• no kernel or SDK code touched. pure sidecar addition
• no test infra or CI changes. new tests run under the existing go test ./...
• Stage interface unchanged, on purpose, so the 4 existing stages stay frozen

[VibhorGautam]

quick follow-up while poking at the phase 02 docs for context on the otel span naming in this pr

one thing worth flagging for phase 3: docs/pipeline.md signal table lists hmac_invalid: 1.0 # always hard block, but sidecar/internal/pipeline/aggregate.go computes score = maxSignalWeight * provenanceWeight in AggregateStage.Run and always returns hardBlock=false. so if phase 3's opa path emits hmac_invalid for a memory-provenance payload (trust weight 0.6), the score ends up 1.0 × 0.6 = 0.6, which is sanitise under the default 0.85 block threshold, not block. since memory is exactly where hmac_invalid is supposed to be strict (tampered stored state), either it needs a validate-style hardBlock return before aggregate runs, or the "always hard block" note should say it's provenance-weighted like every other signal

not blocking for this otel pr, just flagging since i was in those files already. happy to split it out as a separate issue if useful