feat(kll): ASAPv1 wire payload for KLL and KLLDynamic

[GordonYuanyc]

What

Self-describing ASAPv1 wire serialization for the two KLL quantile sketches, following the HLL/CMS pattern from #65:

• compact KLL → kind_id 0x06 0x00
• KLLDynamic → kind_id 0x06 0x01

Both share one payload [levels, items, coin]. serialize_to_bytes / deserialize_from_bytes now emit/read the ASAPv1 envelope (replacing the old rmp form), authored in per-sketch wire submodules. The nested serde Serialize/Deserialize on KLL (used by HydraCounter) is preserved unchanged.

Design decisions (docs/asapv1_wire_format.md §3.3 + Decisions)

• Q-KLL — no hash-spec metadata. KLL never hashes (it orders raw values with total_cmp), so the hash-spec group (profile id, seed_list, …) has no truthful value. KLL metadata is structural-only: {metadata_version, k, m, item_type, seed?}, and is not HashProfile-derived.
• item_type (f64/i64) is a metadata param, not a separate kind_id (mirrors CMS counter_type). The two variants share the payload shape and differ only by kind_id.
• Top-most-level-first item order, byte-for-byte matching sketchlib-go's KLLState. Compact KLL grows its buffer leftward with L0 reverse-input, so its encoder reverses L0 back to input order (reusing wire_levels/wire_items); KLLDynamic stores this layout natively. Distinct from the proto KLLState path (unchanged).
• Q-KLL-SEED — the reproducible compaction seed is recorded as the one optional metadata key. It's construction config (so metadata, not payload), present only when the sketch carries one, omitted otherwise. The payload's coin already carries the RNG's current position (enough to resume compaction); seed is what a later clear() re-seeds from, so carrying it keeps clear()-reproducibility across a serialize/restore. KLLDynamic has no seed concept and never emits the key — the variants are deliberately not forced symmetric. A consumer that doesn't use it (incl. Go) must still preserve it verbatim on re-encode.

Fail-closed decode

kind_id + metadata + item_type checks; level-layout consistency (levels[0]==0, monotonic, levels[last]==items.len(), num_levels<=61); coin remaining_bits<=64; 2 <= m <= k <= MAX_CACHEABLE_K (so crafted k/m can't drive compute_max_capacity into a huge allocation); and a weighted-count guard (so a structurally-valid but adversarial level distribution can't overflow count()/rank()/cdf() at query time). The same bound was also applied to the nested KLLWire serde path (reachable via HydraCounter).

Tests

• Golden byte-vectors asapv1_golden/kll_f64_k200 / kll_i64_k200 (input 1..=50, seed 42, below compaction threshold → deterministic; seed now recorded in metadata) + cross-language assertions in tests/asapv1_golden.rs.
• Round-trip (byte-stable + quantile-preserving) for both variants; empty-sketch; i64; kind_id cross-rejection; deny_unknown_fields; item-type cross-rejection; inconsistent-levels; out-of-range k/m; weighted-count overflow; nested-serde crafted-k rejection; seed present-when-seeded / omitted-when-unseeded; seed survives round-trip so clear() stays deterministic; unseeded decode + byte-stable re-serialize.
• Full suite green; clippy --all-targets clean; fmt clean.

Review

Three independent adversarial code-review passes (subagents). Findings fixed:

• Round 1 — HIGH: unbounded allocation from attacker-controlled k/m (ASAPv1 decoder). Fixed + regression test.
• Round 2 — HIGH: same k/m vector still open on the nested KLLWire serde path (HydraCounter); MEDIUM: weighted-count overflow on crafted level layouts. Both fixed (harden(kll)) + regression tests.
• Round 3 — seed change confirmed sound, no regression; closed one test-coverage gap (unseeded end-to-end decode).

Commits

1. feat(kll) — ASAPv1 payload for KLL and KLLDynamic
2. harden(kll) — fail closed on crafted dimensions and level layouts
3. feat(kll) — record reproducible compaction seed in KLL metadata
4. test(kll) — assert unseeded compact KLL decodes and round-trips byte-stable

Deferred follow-ups

• Tumbling-window KLL is non-deterministic across processes — tracked in Tumbling-window KLL is non-deterministic across processes (unseeded init; KLLConfig has no seed) #77. tumbling.rs from_config uses unseeded KLL::init; KLLConfig has no seed. Framework-side fix, independent of serialization. (KLLDynamic is not wired into tumbling.)
• KLLDynamic golden — needs a seeded constructor (init_kll_with_seed on KLLDynamic, an algorithm change) to be deterministic.
• Nested KLLWire/HydraCounter serde path still drops seed — intentional; seed survives only the ASAPv1 path. Revisit if Hydra sketches need clear()-reproducibility after a round trip.
• Portable KllSketch still double-wraps ({k, sketch_bytes} where sketch_bytes is ASAPv1). HLL's portable emits bare ASAPv1; KLL not migrated (hydra_kll nests KllSketchData). Separate cleanup.
• sketchlib-go must mirror the 0x06 0x00/0x06 0x01 payload (incl. the optional seed key: preserve, don't interpret) and copy the kll_*.hex fixtures.

🤖 Generated with Claude Code