feat(msgpack): add ASAPv1 self-describing envelope to wire format

docs/msgpack-magic-ids.md

@@ -0,0 +1,152 @@
+# ASAPv1 Sketch Wire Format
+
+Every serialised sketch binary produced by this library is wrapped in the
+**ASAPv1 envelope**, a self-describing header that carries the sketch's
+type discriminant plus wrapper-level metadata needed by consumers that update
+or query sketches outside the original process.
+
+```
+┌───────────────┬────────────┬──────────────────┬───────────────────────┬───────────────────┬────────────────────┬──────────────────────┐
+│ b"ASAPv1": 6B │ version: u8 │ kind_id_len: u8  │ kind_id: [kind_id_len] │ metadata_len: u32 │ metadata: msgpack  │ msgpack payload …    │
+└───────────────┴────────────┴──────────────────┴───────────────────────┴───────────────────┴────────────────────┴──────────────────────┘
+```
+
+| Field | Value | Notes |
+|-------|-------|-------|
+| `b"ASAPv1"` | `0x41 0x53 0x41 0x50 0x76 0x31` | 6-byte ASCII sentinel, not a valid msgpack prefix |
+| `version` | `0x02` | Increment only if the envelope layout changes |
+| `kind_id_len` | 1 or 2 | Number of `kind_id` bytes (1 for portable, 2 for native) |
+| `kind_id` | see tables below | Canonical big-endian, no leading zero bytes |
+| `metadata_len` | big-endian `u32` | Number of bytes in the metadata MessagePack block |
+| `metadata` | compact msgpack array | Wrapper metadata; currently records the hash profile |
+| payload | msgpack bytes | Compact (array) or named (map) depending on sketch type |
+
+Current metadata fields, in compact MessagePack array order:
+
+| Index | Field | Standard hash value |
+|-------|-------|---------------------|
+| 0 | `metadata_version` | `1` |
+| 1 | `hash_spec_present` | `true` for portable IDs and registered `DefaultXxHasher` native IDs |
+| 2 | `hash_profile_id` | `projectasap.xxh3.seedlist.v1` |
+| 3 | `hash_algorithm` | `xxh3_64_128` |
+| 4 | `seed_list` | `[0xcafe3553, 0xade3415118, 0x8cc70208, 0x2f024b2b, 0x451a3df5, 0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19, 0xcbbb9d5d, 0x629a292a, 0x9159015a, 0x152fecd8, 0x67332667, 0x8eb44a87, 0xdb0c2e0d]` |
+| 5 | `canonical_seed_index` | `5` |
+| 6 | `matrix_seed_index` | `0` |
+| 7 | `hydra_seed_index` | `6` |
+| 8 | `univmon_bottom_layer_seed_index` | `19` |
+| 9 | `seed_derivation` | `seed_list_index_wrap` |
+| 10 | `input_encoding` | `projectasap.input.v1` |
+
+If a wrapper cannot declare a complete registered hash profile, it still carries
+metadata version `1` with `hash_spec_present = false` and empty/default hash
+fields. Readers validate known hash metadata before exposing the payload.
+
+**Portable** sketches use a 1-byte `kind_id` (`kind_id_len = 1`).
+**Native** Rust sketches use a 2-byte `kind_id` (`kind_id_len = 2`): first byte
+is the type/mode discriminant, second byte is the hasher ID (`HASHER_*`).
+
+The `hasher_id` second byte is `0xFF` (`HASHER_UNKNOWN`) for types without an `H`
+parameter. Custom hashers that do not register an ID also store `0xFF` — the mismatch
+check is skipped on both sides when either value is `0xFF`.
+
+Kind IDs are **stable** — once assigned, a value is never reused or
+reassigned. Adding a new sketch type requires a new constant; removing or
+repurposing an existing constant is a **breaking protocol change**.
+
+The single source of truth in code is
+[`src/message_pack_format/magic_ids.rs`](../src/message_pack_format/magic_ids.rs).
+The Go mirror lives in
+[`sketchlib-go/wire/asapmsgpack/magic_ids.go`](https://github.com/ProjectASAP/sketchlib-go/blob/main/wire/asapmsgpack/magic_ids.go).
+
+---
+
+## Portable IDs (0x01 – 0x09)
+
+These IDs identify the **cross-language wire format** shared with
+`sketchlib-go`. Any byte blob with a portable ID can be decoded by either
+the Rust or Go implementation.
+
+| ID     | Rust type / entry point                          | Go entry point                         |
+|--------|--------------------------------------------------|----------------------------------------|
+| `0x01` | `portable::HllSketch::to_msgpack`                | `HLL.SerializeMsgpack`                 |
+| `0x02` | `portable::CountMinSketch::to_msgpack`           | `CountMinSketch.SerializeMsgpack`      |
+| `0x03` | `portable::CountMinSketchWithHeap::to_msgpack`   | `CountSketch.SerializeMsgpackWithHeap` |
+| `0x04` | `portable::CountSketch::to_msgpack`              | `CountSketch.SerializeMsgpack`         |
+| `0x05` | `portable::DdSketch::to_msgpack`                 | `DDSketch.SerializeMsgpack`            |
+| `0x06` | `portable::KllSketch::to_msgpack`                | _(Rust-only path; no Go equivalent)_   |
+| `0x07` | `portable::HydraKllSketch::to_msgpack`           | _(Rust-only path; no Go equivalent)_   |
+| `0x08` | `portable::SetAggregator::to_msgpack`            | _(Rust-only path; no Go equivalent)_   |
+| `0x09` | `portable::DeltaResult::to_msgpack`              | _(Rust-only path; no Go equivalent)_   |
+
+> **Note on `0x03`:** The Go producer calls this format "CountSketchWithHeap";
+> the Rust consumer knows it as `CountMinSketchWithHeap`. The delta-heap frame
+> (`SerializeMsgpackWithHeapDelta`) shares the same magic ID because the Rust
+> consumer uses the same `from_msgpack` path for both the full and delta shapes.
+
+---
+
+## Native IDs (0x81 – 0x89)
+
+These IDs identify the **Rust-internal** format produced by
+`serialize_to_bytes` / `deserialize_from_bytes` on the generic sketch types
+in `src/sketches/`. Go never reads these bytes directly.
+
+The native wire format uses `rmp_serde`'s **named** (map) encoding
+(`to_vec_named`), whereas the portable format uses **compact** (array)
+encoding. The two are not interchangeable even for logically equivalent types
+(e.g., `sketches::DDSketch` vs `portable::DdSketch`).
+
+| ID     | Rust type / method                              | Notes |
+|--------|--------------------------------------------------|-------|
+| `0x81` | `CountMin<_, RegularPath, _>::serialize_to_bytes` | Named map format |
+| `0x82` | `CountMin<_, FastPath, _>::serialize_to_bytes`   | Named map format |
+| `0x83` | `Count<_, RegularPath, _>::serialize_to_bytes`   | Named map format |
+| `0x84` | `Count<_, FastPath, _>::serialize_to_bytes`      | Named map format |
+| `0x85` | `CountL2HH::serialize_to_bytes`                  | Named map format (CMSHeap / heavy-hitter) |
+| `0x86` | `HyperLogLogImpl<Classic, _, _>::serialize_to_bytes`   | Named map format |
+| `0x87` | `HyperLogLogImpl<ErtlMLE, _, _>::serialize_to_bytes`   | Named map format |
+| `0x88` | `HyperLogLogHIPImpl<_>::serialize_to_bytes`      | Named map; always `HASHER_DEFAULT_XX` as second byte |
+| `0x89` | `sketches::DDSketch::serialize_to_bytes`         | Named map; distinct from portable `0x05` |
+| `0x8a` | `sketches::KLL::serialize_to_bytes`              | Compact array format |
+| `0x8b` | `sketches::KLLDynamic::serialize_to_bytes`       | Compact array format |
+| `0x8c` | `sketches::KMV::serialize_to_bytes`              | Named map format (experimental feature) |
+| `0x8d` | `sketch_framework::Hydra::serialize_to_bytes`    | Named map format |
+| `0x8e` | `sketch_framework::UnivMon::serialize_to_bytes`  | Named map format |
+
+### Relationship between native and portable KLL
+
+`portable::KllSketch::to_msgpack` (`0x06`) embeds the raw KLL cell bytes
+inside a `KllSketchData { k, sketch_bytes }` msgpack struct. Those embedded
+bytes are produced by `KLL::serialize_to_bytes` and therefore carry the native
+`0x8a` prefix. The portable round-trip is:
+
+```
+KllSketch::to_msgpack()    → [ ASAPv1 | v=1 | len=1 | 0x06 | metadata | msgpack([k, [ASAPv1|v=1|len=2|0x8a|0xff|metadata|raw_kll]]) ]
+KllSketch::from_msgpack()  → decode_wrapper → kind_id=[0x06], payload=msgpack struct
+                             → decode struct: k + sketch_bytes
+                             → KLL::deserialize_from_bytes(sketch_bytes)
+                               → decode_wrapper → kind_id=[0x8a, 0xff], payload=raw_kll
+```
+
+The same pattern applies to `HydraKllSketch` (`0x07`), which contains a grid
+of KLL cells.
+
+---
+
+## Adding a new sketch type
+
+1. Add a constant to `src/message_pack_format/magic_ids.rs` (choose the next
+   available value in the appropriate range).
+2. If the type has a **portable** wire format shared with Go, also add the
+   constant to `sketchlib-go/wire/asapmsgpack/magic_ids.go` with the same
+   value, and update `SerializeMsgpack` / `DeserializeMsgpack` in the
+   corresponding Go sketch package to use `asapmsgpack.EncodeWrapper` /
+   `asapmsgpack.DecodeWrapper`.
+3. Implement `MessagePackCodec::to_msgpack` / `from_msgpack` (or
+   `serialize_to_bytes` / `deserialize_from_bytes` for native types) using
+   `magic_ids::encode_wrapper` / `magic_ids::decode_wrapper`.
+   - Portable: `kind_id = &[MAGIC_CONSTANT]` (1 byte)
+   - Native with hasher: `kind_id = &[TYPE_BYTE, H::hasher_magic_id()]` (2 bytes)
+   - Native without hasher: `kind_id = &[TYPE_BYTE, HASHER_UNKNOWN]` (2 bytes)
+4. Add or update round-trip tests.
+5. Update this document.

src/common/hash.rs

@@ -65,6 +65,17 @@ pub trait SketchHasher: Clone + Debug {
         cols: usize,
         key: &DataInput,
     ) -> Self::HashType;
+
+    /// Returns a stable byte discriminant for this hasher, embedded as the
+    /// second byte of every native serialized header so readers can verify
+    /// the hash function matches.
+    ///
+    /// The default is [`crate::message_pack_format::magic_ids::HASHER_UNKNOWN`]
+    /// (`0xff`), which disables the mismatch check on both sides. Override
+    /// this for any hasher that should be verified across process boundaries.
+    fn hasher_magic_id() -> u8 {
+        crate::message_pack_format::magic_ids::HASHER_UNKNOWN
+    }
 }
 
 /// Default hasher using twox_hash (XxHash3). This is the built-in implementation
@@ -75,6 +86,10 @@ pub struct DefaultXxHasher;
 impl SketchHasher for DefaultXxHasher {
     type HashType = MatrixHashType;
 
+    fn hasher_magic_id() -> u8 {
+        crate::message_pack_format::magic_ids::HASHER_DEFAULT_XX
+    }
+
     #[inline(always)]
     fn hash64_seeded(d: usize, key: &DataInput) -> u64 {
         let seed = SEEDLIST[normalized_seed_idx(d)];

src/common/input.rs

@@ -483,7 +483,7 @@ impl HydraCounter {
             (HydraCounter::UNIVERSAL(um), HydraQuery::L1Norm) => Ok(um.calc_l1()),
             (HydraCounter::UNIVERSAL(um), HydraQuery::L2Norm) => Ok(um.calc_l2()),
             (HydraCounter::UNIVERSAL(um), HydraQuery::Entropy) => Ok(um.calc_entropy()),
-            (c, q) => Err(format!("{} does not support {}", c, q)),
+            (c, q) => Err(format!("{c} does not support {q}")),
         }
     }
 

src/common/structure_utils.rs

@@ -365,26 +365,23 @@ mod tests {
             let sort_median = median_three_sort(v);
             assert_eq!(
                 fast_median, sort_median,
-                "median for sort is {sort_median} but fast gives {fast_median}, input is {:?}",
-                v
+                "median for sort is {sort_median} but fast gives {fast_median}, input is {v:?}"
             );
         }
         for v in &mut four_vec {
             let fast_median = compute_median_inline_f64(v);
             let sort_median = median_four_sort(v);
             assert_eq!(
                 fast_median, sort_median,
-                "median for sort is {sort_median} but fast gives {fast_median}, input is {:?}",
-                v
+                "median for sort is {sort_median} but fast gives {fast_median}, input is {v:?}"
             );
         }
         for v in &mut five_vec {
             let fast_median = compute_median_inline_f64(v);
             let sort_median = median_five_sort(v);
             assert_eq!(
                 fast_median, sort_median,
-                "median for sort is {sort_median} but fast gives {fast_median}, input is {:?}",
-                v
+                "median for sort is {sort_median} but fast gives {fast_median}, input is {v:?}"
             );
         }
     }

src/message_pack_format/error.rs

@@ -12,13 +12,26 @@ pub enum Error {
     Encode(rmp_serde::encode::Error),
     /// MessagePack decoding failed.
     Decode(rmp_serde::decode::Error),
+    /// The leading magic-ID byte was missing or did not match the expected value.
+    /// `got` is the byte that was found (or `None` if the buffer was empty).
+    BadMagicId { expected: u8, got: Option<u8> },
 }
 
 impl fmt::Display for Error {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         match self {
             Error::Encode(e) => write!(f, "MessagePack encode failed: {e}"),
             Error::Decode(e) => write!(f, "MessagePack decode failed: {e}"),
+            Error::BadMagicId { expected, got } => match got {
+                Some(b) => write!(
+                    f,
+                    "MessagePack magic-ID mismatch: expected 0x{expected:02x}, got 0x{b:02x}"
+                ),
+                None => write!(
+                    f,
+                    "MessagePack magic-ID missing: expected 0x{expected:02x} but buffer is empty"
+                ),
+            },
         }
     }
 }
@@ -28,6 +41,7 @@ impl StdError for Error {
         match self {
             Error::Encode(e) => Some(e),
             Error::Decode(e) => Some(e),
+            Error::BadMagicId { .. } => None,
         }
     }
 }