feat(kll): ASAPv1 wire payload for KLL and KLLDynamic

asapv1_golden/README.md

@@ -29,11 +29,22 @@ golden tests the **wire encoding**, isolated from the hash functions.
 | `hll_hip_p12` | HLL HIP, P12 | `01 03` | same registers + `hip_kxq0=1.5, hip_kxq1=2.5, hip_est=3.0` |
 | `cms_i64_regular_2x3` | Count-Min i64, RegularPath | `02 00` | 2×3 row-major `[[0,1,127],[128,300,65536]]` |
 | `cms_f64_fast_2x3` | Count-Min f64, FastPath | `02 00` | 2×3 row-major `[[0.0,1.5,2.25],[3.75,4.125,5.0625]]` |
+| `kll_f64_k200` | KLL f64, k=200 | `06 00` | integers `1..=50`, compaction seed 42 (recorded in metadata as `seed`) |
+| `kll_i64_k200` | KLL i64, k=200 | `06 00` | integers `1..=50`, compaction seed 42 (recorded in metadata as `seed`) |
 
 The i64 fixture deliberately spans the msgpack integer width boundaries
 (positive fixint / uint8 / uint16 / uint32) to lock the "non-negative integer →
 uint family, minimal width" rule (`docs/asapv1_wire_format.md` §5).
 
+The KLL fixtures are a special case of "state is fixed, not hashed": KLL never
+hashes — it orders raw numeric values — so inserting `1..=50` places exactly
+those retained samples. `k=200` keeps the input below the level-0 capacity, so no
+compaction fires (`num_levels = 1`, one level `[1..50]`) and the state is fully
+deterministic. The fixed compaction seed (42) pins the carried coin state, which
+must match `sketchlib-go`'s coin for the same input. Only the compact KLL
+(`06 00`) has a golden; the dynamic variant (`06 01`) shares the payload shape but
+lacks a seeded constructor, so its cross-language golden is deferred.
+
 ## Tests that consume these
 
 - Rust: `tests/asapv1_golden.rs` — builds each fixture from known state,

asapv1_golden/kll_f64_k200.hex

@@ -0,0 +1 @@
+415341507631010206000000002e000001cd85b06d657461646174615f76657273696f6e01a16bccc8a16d08a96974656d5f74797065a3663634a4736565642a93920032dc0032cb3ff0000000000000cb4000000000000000cb4008000000000000cb4010000000000000cb4014000000000000cb4018000000000000cb401c000000000000cb4020000000000000cb4022000000000000cb4024000000000000cb4026000000000000cb4028000000000000cb402a000000000000cb402c000000000000cb402e000000000000cb4030000000000000cb4031000000000000cb4032000000000000cb4033000000000000cb4034000000000000cb4035000000000000cb4036000000000000cb4037000000000000cb4038000000000000cb4039000000000000cb403a000000000000cb403b000000000000cb403c000000000000cb403d000000000000cb403e000000000000cb403f000000000000cb4040000000000000cb4040800000000000cb4041000000000000cb4041800000000000cb4042000000000000cb4042800000000000cb4043000000000000cb4043800000000000cb4044000000000000cb4044800000000000cb4045000000000000cb4045800000000000cb4046000000000000cb4046800000000000cb4047000000000000cb4047800000000000cb4048000000000000cb4048800000000000cb4049000000000000932a0000

asapv1_golden/kll_i64_k200.hex

@@ -0,0 +1 @@
+415341507631010206000000002e0000003d85b06d657461646174615f76657273696f6e01a16bccc8a16d08a96974656d5f74797065a3693634a4736565642a93920032dc00320102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132932a0000

docs/asapv1_wire_format.md

@@ -12,7 +12,7 @@ If the doc feels long, these sections carry the key points:
 
 - [**Section 1, Envelope**](#section-1-envelope): the Layout table.
 - [**Section 2, Metadata**](#section-2-metadata): the fields table, the hash-spec table, the structural-params table.
-- [**Section 3, Payload**](#section-3-payload): the HLL payload and the Count-Min payload.
+- [**Section 3, Payload**](#section-3-payload): the HLL payload, the Count-Min payload, and the KLL payload.
 
 ## Terms
 
@@ -26,7 +26,7 @@ If the doc feels long, these sections carry the key points:
 
 ## Status
 
-- **Implementing (Rust).** HLL and Count-Min serialize through the shared `message_pack_format::envelope` module per this spec.
+- **Implementing (Rust).** HLL, Count-Min, and KLL (compact + dynamic) serialize through the shared `message_pack_format::envelope` module per this spec.
 - **Self-describing.** The hash-spec metadata is derived from the hasher's `HashProfile` (read live, never hardcoded), so the bytes truthfully describe how a sketch was hashed; custom hash profiles are supported (Section 2).
 - **Byte-level encoding** is pinned in Section 4; the resolved decisions are summarized at the end.
 - **`sketchlib-go`** is aligned separately (see Cross-language contract).
@@ -73,6 +73,7 @@ erDiagram
     PAYLOAD ||--o| HLL_PAYLOAD : kind-0x0101-0x0102
     PAYLOAD ||--o| HLL_HIP_PAYLOAD : kind-0x0103
     PAYLOAD ||--o| COUNTMIN_PAYLOAD : kind-0x0200
+    PAYLOAD ||--o| KLL_PAYLOAD : kind-0x0600-0x0601
     ENVELOPE {
         bytes magic
         u8 version
@@ -88,8 +89,8 @@ erDiagram
         string seed_derivation
         string input_encoding
         array seed_list
-        u32 seed_index "hash-spec group, per-sketch"
-        mixed structural_params "structural group, per-sketch: precision(HLL) or rows+cols+counter_type+mode(CMS)"
+        u32 seed_index "hash-spec group, per-sketch; absent for non-hashing sketches (KLL)"
+        mixed structural_params "structural group, per-sketch: precision(HLL); rows+cols+counter_type+mode(CMS); k+m+item_type(KLL)"
     }
     PAYLOAD {
         msgpack_array raw_state
@@ -106,6 +107,11 @@ erDiagram
     COUNTMIN_PAYLOAD {
         array counts
     }
+    KLL_PAYLOAD {
+        array levels
+        array items
+        array coin
+    }
 ```
 
 ---
@@ -166,7 +172,7 @@ Today `kind_id` is `[family, variant]` and names the sketch's **algorithm** (its
 ### kind_id registry (single source of truth, mirrored verbatim in `sketchlib-go`)
 
 The **family** bytes match `sketchlib-go`'s `wire/asapmsgpack/magic_ids.go` verbatim; `0x0a`+ are new allocations for sketches in [`apis.md`](./apis.md) that Go has not assigned yet.
-Only the HLL variants and Count-Min have designed payloads today; every other row reserves its family byte with variant `0x00` and payload **TBD**.
+The HLL variants, Count-Min, and both KLL variants have designed payloads today; every other row reserves its family byte with variant `0x00` and payload **TBD**.
 Variant sub-ids are not invented ahead of a payload design; a family that later needs several algorithms allocates its variants when it is designed (as HLL did).
 This registry is the master list of algorithms still to design payloads for.
 
@@ -180,8 +186,8 @@ This registry is the master list of algorithms still to design payloads for.
 | `0x03 0x00` | Count-Min-with-heap (CMSHeap) | - | TBD | assigned in Go / payload not designed |
 | `0x04 0x00` | Count Sketch | - | TBD | assigned in Go / payload not designed |
 | `0x05 0x00` | DDSketch | - | TBD | assigned in Go / payload not designed |
-| `0x06 0x00` | KLL | - | TBD | assigned in Go / payload not designed |
-| `0x06 0x01` | KLL dynamic | - | TBD | assigned in Go / payload not designed |
+| `0x06 0x00` | KLL | Compact | Section 3.3 | implemented |
+| `0x06 0x01` | KLL dynamic | Dynamic | Section 3.3 | implemented |
 | `0x07 0x00` | Hydra-KLL | - | TBD | assigned in Go / payload not designed |
 | `0x08 0x00` | SetAggregator | - | TBD | assigned in Go / payload not designed |
 | `0x09 0x00` | DeltaResult | - | TBD | assigned in Go / payload not designed |
@@ -247,10 +253,12 @@ The metadata map is **two groups** of fields, written on the wire in this order:
 | Group | Role | Fields |
 | ----- | ---- | ------ |
 | **Hash spec** | how keys were hashed (check mergeability + re-hash a query key) | `metadata_version`, `hash_profile_id`, `hash_algorithm`, `seed_derivation`, `input_encoding`, `seed_list`, + the seed index(es) it uses |
-| **Structural params** | parameters that shape the payload | `precision` (HLL); `rows`, `cols`, `counter_type`, `mode` (Count-Min) |
+| **Structural params** | parameters that shape the payload | `precision` (HLL); `rows`, `cols`, `counter_type`, `mode` (Count-Min); `k`, `m`, `item_type`, optional `seed` (KLL) |
 
 The two tables below are the field-by-field detail of each group.
 
+> **Non-hashing sketches omit the hash-spec group entirely** (Q-KLL). The hash spec answers "how were keys hashed"; a sketch that does not hash its inputs has no truthful answer. KLL is comparison-based — it orders raw numeric values with `total_cmp` and never invokes a hasher — so its metadata carries **only** structural params (`metadata_version`, `k`, `m`, `item_type`, and an optional `seed`). This keeps the bytes honest (no meaningless `seed_list`) and is the reason KLL's metadata schema is not built from a `HashProfile` the way HLL's / Count-Min's are. Note `seed` is unrelated to the hash `seed_list`: it is the KLL compaction RNG's reproducible seed, and it is the **only optional key** in v1 (present only when the sketch was built with one; the key is omitted otherwise, and `KLLDynamic` never emits it). A consumer that does not use it (including Go) MUST still preserve it verbatim on re-encode so the bytes round-trip identically.
+
 **Hash spec**
 
 | Key | Type | Required | Meaning |
@@ -275,6 +283,10 @@ The two tables below are the field-by-field detail of each group.
 | `cols` | u32 | Count-Min | matrix width (number of columns) |
 | `counter_type` | string | Count-Min | `"i64"` or `"f64"`; element type of `counts` |
 | `mode` | string | Count-Min | `"fast"` or `"regular"`; key-to-column derivation |
+| `k` | u32 | KLL | compactor capacity (accuracy parameter) |
+| `m` | u32 | KLL | minimum level capacity |
+| `item_type` | string | KLL | `"f64"` or `"i64"`; element type of `items` |
+| `seed` | u64 | KLL | **optional**; the reproducible compaction seed, present only when the sketch carries one, else the key is omitted. Compact KLL only (`KLLDynamic` never emits it). The one optional key in v1. |
 
 Count-Min's matrix dimensions are **configuration** (they shape the payload, like HLL's `precision`), so per the config-to-metadata rule they live here.
 Count-Min's canonical structural-param order is `... matrix_seed_index, rows, cols, counter_type, mode`; this is the wire contract and Go must mirror it verbatim.
@@ -361,7 +373,23 @@ Wire counter types are `i64` and `f64` only (`i32` widens to `i64`; `i128` and e
 A counter type other than i64/f64, or non-`Vector2D` storage, must be converted first; see Section 5, "Converting an exotic in-memory sketch".
 Both modes, `FastPath` and `RegularPath`, serialize directly (you'd only "convert" a mode to *change* it, which needs re-inserting the data).
 
-### 3.3 onward: payloads not yet designed
+### 3.3: KLL payload (`0x06 0x00` compact / `0x06 0x01` dynamic)
+
+Both KLL variants (the compact fixed-buffer `KLL` and the growable `KLLDynamic`) share **one payload shape**; they differ only by `kind_id` (like HLL's Classic vs Ertl-MLE), because their in-memory buffers differ but the serialized quantile state is the same. The accuracy params `k` / `m`, the `item_type`, and the optional `seed` live in the metadata (§2), so the payload is just the retained state:
+
+| Pos | Field | Type | Notes |
+| ----- | ------- | ------ | ------- |
+| 0 | `levels` | array | level boundary indices, length `num_levels + 1`; `levels[0] == 0`, `levels[last] == len(items)` |
+| 1 | `items` | array | retained samples in level order; element type = `item_type` |
+| 2 | `coin` | array | compaction RNG state `[state:u64, bit_cache:u64, remaining_bits:u32]` |
+
+`num_levels` is `levels.len() - 1` (derived, so not stored). The `coin` is the randomized-compaction RNG; it is carried so a decoded sketch can keep compacting deterministically (a query-only consumer may ignore it). It is a nested 3-element array mirroring `sketchlib-go`'s `CoinState`.
+
+**Item order (cross-language contract).** `levels` / `items` use the **top-most-level-first** layout, byte-for-byte matching `sketchlib-go`'s `KLLState`: index `i` in `levels` maps to compactor level `num_levels - 1 - i`, and level 0's run is in **input order**. The compact `KLL` grows its buffer leftward and stores level 0 reverse-input, so its encoder reverses level 0 back to input order (and its decoder reverses it in); `KLLDynamic` already stores this layout natively. Within a level, order past the first compaction is not guaranteed byte-identical across the two Rust variants (or across languages), but the retained set and quantiles agree — see the caveat on `KLL::wire_items`.
+
+> Note: this is a distinct wire format from the proto `KLLState` (`proto/kll/kll.proto`), which is the delta-transmission / cross-language *protobuf* path and carries extra machinery (the value-offset fixed-point item encoding). The two share the top-most-first item layout but are separate formats; the ASAPv1 KLL payload above is self-contained and carries no fixed-point form.
+
+### 3.4 onward: payloads not yet designed
 
 The remaining `kind_id`s reserve a family byte with payload TBD (Section 1 registry has their "assigned in Go" status). Likely shape when designed:
 
@@ -370,8 +398,7 @@ The remaining `kind_id`s reserve a family byte with payload TBD (Section 1 regis
 | `0x03 0x00` | Count-Min-with-heap (CMSHeap) | similar to current CMS |
 | `0x04 0x00` | Count Sketch | similar to current CMS |
 | `0x05 0x00` | DDSketch | straightforward bucket |
-| `0x06 0x00` | KLL | next on the list; no CDF serialization |
-| `0x07 0x00` | Hydra-KLL | same challenge as KLL |
+| `0x07 0x00` | Hydra-KLL | wraps KLL payloads (§3.3); nest one per counter |
 | `0x08 0x00` | SetAggregator | aggregation envelope, distinct from a stand-alone sketch (Section 1 mapping notes) |
 | `0x09 0x00` | DeltaResult | delta-result envelope, distinct from a stand-alone sketch (Section 1 mapping notes) |
 | `0x0a 0x00` | Count-Sketch-with-heap (CSHeap) | similar to current CMS |
@@ -483,6 +510,8 @@ Keep it through the transition, retire `portable` once goldens are in place.
 - **Q-SEEDS**: `seed_list` is **inlined** in v1 so the bytes self-describe the hash. Resolving seeds from `hash_profile_id` alone is a v2 space optimization. Each sketch still carries only the seed *index* it uses.
 - **Q-PROFILE**: the hash-spec metadata is **derived from the hasher's `HashProfile`** (`hll_metadata::<H>` / `cms_metadata::<H>`) and never hardcoded, so it is always truthful to the hasher. Custom hash profiles are **supported and self-describing**. Merge compatibility is hash-spec equality, so a custom-profile sketch is not mergeable with a standard one.
 - **Q-CMS**: Count-Min is one `kind_id` (`0x02 0x00`); counter type and mode live in the metadata, so the id stays single.
+- **Q-KLL**: KLL metadata carries **no hash-spec group** — KLL is comparison-based and never hashes, so those fields have no truthful value. Its metadata is structural-only (`metadata_version`, `k`, `m`, `item_type`, optional `seed`) and is *not* `HashProfile`-derived. The two KLL variants (compact `0x06 0x00`, dynamic `0x06 0x01`) share one payload `[levels, items, coin]` and differ only by `kind_id`. `item_type` (`"f64"`/`"i64"`) is a metadata param, not a separate `kind_id` (mirrors Q-CMS's `counter_type`). Retained samples use the top-most-level-first layout that matches `sketchlib-go`'s `KLLState`.
+- **Q-KLL-SEED**: KLL records its reproducible compaction `seed` as an **optional** metadata key. It is construction config (so metadata, not payload, per the config→metadata rule), and it is the first optional key in v1: present only when the sketch carries a seed, omitted otherwise. Rationale: the payload's `coin` already carries the RNG's *current* position (enough to resume compaction), but `seed` is what a later `clear()` re-seeds from — so serializing it lets a decoded sketch keep `clear()`-reproducibility instead of falling back to wall-clock. Cost of omitting it is bounded (only a decoded-then-`clear()`ed sketch loses cross-run byte reproducibility — never correctness), but it is cheap to carry and future-proofs the checkpoint/restore path. `KLLDynamic` has no seed concept and never emits the key; the two variants are deliberately **not** forced to be symmetric here. Go carries and preserves the key without interpreting it.
 - **Q-CMS-DIMS**: Count-Min `rows`/`cols` are **metadata** and the payload omits them. They are configuration that shapes the payload (like HLL's `precision`), so per the config-to-metadata rule they belong in the descriptor. The payload is then just `[counts]`. Canonical structural-param order: `... matrix_seed_index, rows, cols, counter_type, mode`.
 - **Q-VER**: no payload version field. A new incompatible encoding gets a **new `kind_id`**; retired ids are reserved forever and never recycled.
 - **Encoding**: metadata + payload are both msgpack; payload is a positional array. Byte-level rules in Section 4.

src/message_pack_format/native/kll.rs

@@ -4,11 +4,11 @@ use serde::{Deserialize, Serialize};
 
 use crate::common::numerical::NumericalValue;
 use crate::message_pack_format::{Error, MessagePackCodec};
-use crate::sketches::kll::KLL;
+use crate::sketches::kll::{KLL, KllWireItem};
 
 impl<T> MessagePackCodec for KLL<T>
 where
-    T: NumericalValue + Serialize + for<'de> Deserialize<'de>,
+    T: NumericalValue + KllWireItem + Serialize + for<'de> Deserialize<'de>,
 {
     fn to_msgpack(&self) -> Result<Vec<u8>, Error> {
         Ok(self.serialize_to_bytes()?)

src/message_pack_format/native/kll_dynamic.rs

@@ -4,11 +4,12 @@ use serde::{Deserialize, Serialize};
 
 use crate::common::numerical::NumericalValue;
 use crate::message_pack_format::{Error, MessagePackCodec};
+use crate::sketches::kll::KllWireItem;
 use crate::sketches::kll_dynamic::KLLDynamic;
 
 impl<T> MessagePackCodec for KLLDynamic<T>
 where
-    T: NumericalValue + Serialize + for<'de> Deserialize<'de>,
+    T: NumericalValue + KllWireItem + Serialize + for<'de> Deserialize<'de>,
 {
     fn to_msgpack(&self) -> Result<Vec<u8>, Error> {
         Ok(self.serialize_to_bytes()?)