Skip to content

DDSketch: negative/zero value tracking (deferred design question, split from #70) #73

Description

@zzylol

What

DDSketch is currently positive-only: add silently drops non-positive
values (v.is_finite() && v > 0.0 guard). DataDog/sketches-go tracks these
instead — a zeroCount scalar plus a full mirrored negativeValueStore
(same index mapping, keyed by Index(-v)), so metrics with legitimate
zero or negative samples (deltas, signed measurements) do not silently
undercount.

Filed separately from #70 (which bundled this as item 3) and scoped
narrowly to just this question, per user request — deliberately deferred
out of #70/sketchlib-go#72s scope since it needs a real design decision,
not a mechanical fix.

Why deferred (not urgent)

Every current ASAPCollector use case for DDSketch is a naturally
non-negative metric (latency, size, byte counts), so the positive-only
restriction has not been a practical problem. No immediate driver to add
this — tracked here so the design question does not get lost, revisit when
a real negative/zero-valued metric use case shows up.

Design question to resolve before implementing

DataDog's approach (two independent dense stores + a scalar) is the
straightforward option, but doubles the store/wire footprint even for
sketches that never see a negative value. Options to weigh when this is
picked up:

  1. Mirror DataDog exactly: zero_count field + a second Buckets
    store for negative values, keyed by key_for(-v). Simple, proven, but
    pays the second-store cost unconditionally.
  2. Single store, encoded sign: fold negative values into the same
    index space via a disjoint offset/range convention (e.g.
    neg_index = -1 - key_for(-v)), avoiding a second store. Needs care:
    key_for(v) for v < 1 can itself already be negative (depending on
    alpha), so a naive "negative index = negative value" convention can
    collide with legitimate positive-value indices near v=1 — would need an
    explicit disjoint range or a side flag, not just index sign.
  3. Lazy second store: only allocate the negative store on first
    negative/zero insert, so the common (always-positive) case pays zero
    extra cost — this is close to option 1 but avoids the unconditional
    memory/wire overhead for sketches that never need it.

Whatever is chosen here should stay consistent with the same decision on
the Go side (sketchlib-go) — see the companion issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions