What
DDSketch is currently positive-only: add silently drops non-positive
values (v.is_finite() && v > 0.0 guard). DataDog/sketches-go tracks these
instead — a zeroCount scalar plus a full mirrored negativeValueStore
(same index mapping, keyed by Index(-v)), so metrics with legitimate
zero or negative samples (deltas, signed measurements) do not silently
undercount.
Filed separately from #70 (which bundled this as item 3) and scoped
narrowly to just this question, per user request — deliberately deferred
out of #70/sketchlib-go#72s scope since it needs a real design decision,
not a mechanical fix.
Why deferred (not urgent)
Every current ASAPCollector use case for DDSketch is a naturally
non-negative metric (latency, size, byte counts), so the positive-only
restriction has not been a practical problem. No immediate driver to add
this — tracked here so the design question does not get lost, revisit when
a real negative/zero-valued metric use case shows up.
Design question to resolve before implementing
DataDog's approach (two independent dense stores + a scalar) is the
straightforward option, but doubles the store/wire footprint even for
sketches that never see a negative value. Options to weigh when this is
picked up:
- Mirror DataDog exactly:
zero_count field + a second Buckets
store for negative values, keyed by key_for(-v). Simple, proven, but
pays the second-store cost unconditionally.
- Single store, encoded sign: fold negative values into the same
index space via a disjoint offset/range convention (e.g.
neg_index = -1 - key_for(-v)), avoiding a second store. Needs care:
key_for(v) for v < 1 can itself already be negative (depending on
alpha), so a naive "negative index = negative value" convention can
collide with legitimate positive-value indices near v=1 — would need an
explicit disjoint range or a side flag, not just index sign.
- Lazy second store: only allocate the negative store on first
negative/zero insert, so the common (always-positive) case pays zero
extra cost — this is close to option 1 but avoids the unconditional
memory/wire overhead for sketches that never need it.
Whatever is chosen here should stay consistent with the same decision on
the Go side (sketchlib-go) — see the companion issue.
What
DDSketch is currently positive-only:
addsilently drops non-positivevalues (
v.is_finite() && v > 0.0guard). DataDog/sketches-go tracks theseinstead — a
zeroCountscalar plus a full mirrorednegativeValueStore(same index mapping, keyed by
Index(-v)), so metrics with legitimatezero or negative samples (deltas, signed measurements) do not silently
undercount.
Filed separately from #70 (which bundled this as item 3) and scoped
narrowly to just this question, per user request — deliberately deferred
out of #70/sketchlib-go#72s scope since it needs a real design decision,
not a mechanical fix.
Why deferred (not urgent)
Every current ASAPCollector use case for DDSketch is a naturally
non-negative metric (latency, size, byte counts), so the positive-only
restriction has not been a practical problem. No immediate driver to add
this — tracked here so the design question does not get lost, revisit when
a real negative/zero-valued metric use case shows up.
Design question to resolve before implementing
DataDog's approach (two independent dense stores + a scalar) is the
straightforward option, but doubles the store/wire footprint even for
sketches that never see a negative value. Options to weigh when this is
picked up:
zero_countfield + a secondBucketsstore for negative values, keyed by
key_for(-v). Simple, proven, butpays the second-store cost unconditionally.
index space via a disjoint offset/range convention (e.g.
neg_index = -1 - key_for(-v)), avoiding a second store. Needs care:key_for(v)forv < 1can itself already be negative (depending onalpha), so a naive "negative index = negative value" convention can
collide with legitimate positive-value indices near v=1 — would need an
explicit disjoint range or a side flag, not just index sign.
negative/zero insert, so the common (always-positive) case pays zero
extra cost — this is close to option 1 but avoids the unconditional
memory/wire overhead for sketches that never need it.
Whatever is chosen here should stay consistent with the same decision on
the Go side (sketchlib-go) — see the companion issue.