ProjectASAP · GordonYuanyc · Jul 19, 2026 · Jul 11, 2026
diff --git a/docs/api/api_countmin.md b/docs/api/api_countmin.md
@@ -53,6 +53,17 @@ fn serialize_to_bytes(&self) -> Result<Vec<u8>, RmpEncodeError>
 fn deserialize_from_bytes(bytes: &[u8]) -> Result<Self, RmpDecodeError>
 ```
 
+These produce/consume the **ASAPv1** wire envelope (kind `0x02 0x00`) — see the
+[ASAPv1 wire format spec](../asapv1_wire_format.md). They are **not** available
+on every `CountMin`: the impl exists only for wire-eligible configs
+`CountMin<Vector2D<T>, Mode, H>` where `T` is `i64` or `f64` (`CmsWireCounter`),
+`Mode` is `FastPath` or `RegularPath` (`CmsWireMode`), and `H: HashProfile`. The
+default storage is `Vector2D<i32>`, which is **not** wire-eligible — an `i32` /
+`i128` / other exotic-counter or non-`Vector2D` sketch must be converted to a
+`Vector2D<i64>` / `Vector2D<f64>` first (only you know if the mapping is
+lossless). `rows`/`cols` are carried in the envelope metadata; the payload is
+just `[counts]`.
+
 ## Examples
 
 ```rust

diff --git a/docs/api/api_hyperloglog.md b/docs/api/api_hyperloglog.md
@@ -78,6 +78,18 @@ fn serialize_to_bytes(&self) -> Result<Vec<u8>, RmpEncodeError>
 fn deserialize_from_bytes(bytes: &[u8]) -> Result<Self, RmpDecodeError>
 ```
 
+These produce/consume the **ASAPv1** wire envelope — see the
+[ASAPv1 wire format spec](../asapv1_wire_format.md). On the generic
+`HyperLogLogImpl<Variant, Registers, H>`, the methods are bounded on
+`Variant: HllWireVariant` and `H: HashProfile`, so every
+(variant × precision × hasher) combination serializes — **HLL is fully
+wire-covered**. Each variant maps to its own `kind_id` (Classic → `0x01 0x01`,
+Ertl-MLE → `0x01 0x02`, HIP → `0x01 0x03`); `precision` and the
+`HashProfile`-derived hash spec live in the envelope metadata, and the payload is
+the register bytes (plus three HIP scalars for the HIP variant). `HyperLogLogHIP`
+is a non-generic struct that hashes through the default functions, so it is
+wire-eligible under the **standard** hash profile only.
+
 ## Examples
 
 ```rust

diff --git a/docs/features.md b/docs/features.md
@@ -100,7 +100,7 @@ Insertion throughput measured on 10,000,000 Zipf-distributed `int64` values (s=1
 
 ### Serialization
 
-MessagePack (`rmp-serde`) support. **serde support** means the type derives `Serialize`/`Deserialize` and can be used with any serde-compatible serializer. **Built-in helpers** (`serialize_to_bytes` / `deserialize_from_bytes`) provide one-call MessagePack round-tripping without requiring users to depend on `rmp-serde` directly.
+MessagePack (`rmp-serde`) support. **serde support** means the type derives `Serialize`/`Deserialize` and can be used with any serde-compatible serializer. **Built-in helpers** (`serialize_to_bytes` / `deserialize_from_bytes`) provide one-call round-tripping without requiring users to depend on `rmp-serde` directly. For HyperLogLog and Count-Min these helpers emit the self-describing **ASAPv1** wire envelope (see the [ASAPv1 wire format spec](./asapv1_wire_format.md)); other sketches are not yet converted.
 
 | Component | serde support | Built-in helpers |
 | --- | --- | --- |

diff --git a/docs/index.md b/docs/index.md
@@ -14,7 +14,8 @@ This page is the docs home for `asap_sketchlib`.
 - [APIs Index](./apis.md)
 - [API Pages Directory](./api/)
 - [Common API](./api/api_common.md)
-- [Message Pack Format](./message_pack_format.md) - wire envelope and wire-format-aligned sketch variants shared with `sketchlib-go`
+- [ASAPv1 Wire Format](./asapv1_wire_format.md) - authoritative byte-level spec for the self-describing sketch envelope mirrored by `sketchlib-go`
+- [Message Pack Format](./message_pack_format.md) - how the serialization code is organized (shared envelope + per-sketch `wire.rs`; `portable`/`native` retiring)
 - [Advanced Use Cases](./advanced_use_cases.md)
 - [Test Coverage Map](./tests.md)
 - [Feature Status](./features.md)

diff --git a/docs/library_map.md b/docs/library_map.md
@@ -12,15 +12,17 @@
 - **`src/sketches/`** - Sketch implementations (status source: [apis.md](./apis.md))
   - `Ready` in API index: `countminsketch.rs`, `countsketch.rs`, `hll.rs`, `kll.rs`, `ddsketch.rs`, `countminsketch_topk.rs`, `countsketch_topk.rs`
   - `Unstable` in API index: `coco.rs`, `elastic.rs`, `uniform.rs`, `kmv.rs`
+  - Sketches converted to the ASAPv1 wire format use a `<sketch>.rs` (algorithm) + `<sketch>/wire.rs` (serialization) split: `hll.rs` + `hll/wire.rs` and `countminsketch.rs` + `countminsketch/wire.rs` today (see [asapv1_wire_format.md](./asapv1_wire_format.md))
 
 - **`src/sketch_framework/`** - Orchestration and serving layers (status source: [apis.md](./apis.md))
   - `Ready` in API index: `hydra.rs`, `hashlayer.rs`, `univmon.rs`, `univmon_optimized.rs`, `nitro.rs`, `eh.rs`, `eh_sketch_list.rs`
   - `Unstable` in API index: `eh_univ_optimized.rs`
   - Infrastructure module: `orchestrator/` (node-level manager used by framework APIs)
 
-- **`src/message_pack_format/`** - MessagePack wire-format definitions, the `MessagePackCodec` trait, and the wire-format-aligned sketch types shared with `sketchlib-go` ([message_pack_format.md](./message_pack_format.md))
-  - `portable/` — wire-format-aligned sketch types (`CountMinSketch`, `CountSketch`, `DdSketch`, `HllSketch`, `KllSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, `SetAggregator`, `DeltaResult`) consumed by the ASAP query engine, byte-compatible with `sketchlib-go`
-  - `native/` — Rust-internal codec shims over `src/sketches/` byte serialization
+- **`src/message_pack_format/`** - Serialization plumbing ([message_pack_format.md](./message_pack_format.md)). The current format is **ASAPv1**, specified in [asapv1_wire_format.md](./asapv1_wire_format.md)
+  - `envelope.rs` — the shared, sketch-agnostic ASAPv1 framing (magic/version/`kind_id` + length prefixes, `encode`/`split`); every per-sketch `wire.rs` calls into it
+  - `portable/` — **deprecated**, being retired. The older per-sketch wire types (`CountMinSketch`, `HllSketch`, …); ASAPv1 (per-sketch `wire.rs`) is now what `sketchlib-go` mirrors, not these
+  - `native/` — **deprecated**, being retired. Older `MessagePackCodec` shims over `src/sketches/` byte serialization
 
 ## Documentation
 

diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md
@@ -1,123 +1,112 @@
 # Message Pack Format
 
-`src/message_pack_format/` is the Rust-side source of truth for the
-MessagePack encode/decode contract. It is split into two sub-modules
-by audience:
-
-- **`portable/`** — cross-language wire format shared with the Go
-  counterpart `sketchlib-go`. Touching anything here is a protocol
-  change and requires the Go side to be kept in lock-step (golden-byte
-  tests catch drift).
-- **`native/`** — thin trait shims over the existing
-  `serialize_to_bytes` / `deserialize_from_bytes` methods on the
-  pure-Rust generic sketch types in [`src/sketches/`](./api/). The byte
-  format is internal to Rust — Go never reads it, and the format is
-  free to evolve without cross-language coordination.
-
-The [`MessagePackCodec`](#core-types) trait and unified `Error` type
-live at the module root so both worlds share the same encode/decode
-contract.
-
-## Core Types
-
-Both live at the top of `src/message_pack_format/` and are re-exported
-through the module root:
-
-- `MessagePackCodec` (in [`codec.rs`](../src/message_pack_format/codec.rs)) —
-  the trait every codec-enabled type implements. Two methods:
-  `to_msgpack`, `from_msgpack`. This is the canonical encode/decode
-  entry point.
-- `Error` (in [`error.rs`](../src/message_pack_format/error.rs)) — the
-  unified encode/decode error type returned by both `native` and
-  `portable` impls.
-
-## `portable/` — Cross-Language Wire Format
-
-One submodule per algorithm, with the filenames mirrored on the Go
-side:
-
-- `countminsketch.rs`, `countminsketch_topk.rs`, `countsketch.rs`,
-  `ddsketch.rs`, `hll.rs`, `kll.rs`, `hydra_kll.rs`,
-  `set_aggregator.rs`, `delta_set_aggregator.rs`
-
-Each submodule owns:
-
-1. The wire-format-aligned runtime type and its delta companion (e.g.
-   `CountMinSketch`, `CountMinSketchDelta`) — these are the types
-   re-exported at the crate root.
-2. The wire DTO struct(s), when the runtime type needs a separate
-   over-the-wire shape (e.g. borrow / owned pairs, byte-compatible
-   field reordering with `sketchlib-go`).
-3. The `MessagePackCodec` impl for the runtime type.
-
-The full list of re-exported wire-format-aligned types:
-
-- `countminsketch.rs` — `CountMinSketch`, `CountMinSketchDelta`
-- `countminsketch_topk.rs` — `CountMinSketchWithHeap`, `CmsHeapItem`
-- `countsketch.rs` — `CountSketch`, `CountSketchDelta`
-- `ddsketch.rs` — `DdSketch`, `DdSketchDelta`
-- `hll.rs` — `HllSketch`, `HllSketchDelta`, `HllVariant`
-- `kll.rs` — `KllSketch`, `KllSketchData`
-- `hydra_kll.rs` — `HydraKllSketch`
-- `set_aggregator.rs` — `SetAggregator`
-- `delta_set_aggregator.rs` — `DeltaResult`
-
-Use these for sketches that must cross a process / language boundary.
-For high-throughput local ingest, custom hashers, and framework
-composition, reach for the generic sketches in
-[`src/sketches/`](./apis.md) instead:
-
-| Need | Use |
-| --- | --- |
-| Local high-throughput ingest, custom hashers, framework composition | [`src/sketches/`](./apis.md) |
-| Cross-process / cross-language transfer matching `sketchlib-go` bytes | `src/message_pack_format/portable/` |
-
-### Types that act as their own DTO
-
-`CountSketch`, `DdSketch`, and `HllSketch` derive `Serialize` /
-`Deserialize` directly because their public field layout already
-matches the wire shape. Their `MessagePackCodec` impls serialize the
-struct verbatim — no separate DTO is required.
-
-### Protocol invariants
-
-- The wire envelope must remain byte-compatible with `sketchlib-go`.
-- Adding, reordering, renaming, or retyping a field counts as a
-  protocol change; bump the format version on both sides and add a
-  golden-byte test before shipping.
-- DTOs that appear as nested fields in another wire type (e.g.
-  `KllSketchData` inside `HydraKllSketchWire`) are part of the same
-  protocol surface — treat them with the same care.
-
-## `native/` — Rust-Internal Codec Shims
-
-One submodule per generic sketch type in [`src/sketches/`](./api/)
-whose serialization is exposed through `MessagePackCodec`:
-
-- `countminsketch.rs`, `countsketch.rs`, `countsketch_topk.rs`,
-  `ddsketch.rs`, `hll.rs`, `kll.rs`, `kll_dynamic.rs`
-- `kmv.rs` (gated behind the `experimental` feature flag)
-
-Each impl forwards `to_msgpack` / `from_msgpack` to the sketch's
-existing `serialize_to_bytes` / `deserialize_from_bytes` methods. The
-byte format is **not** part of the cross-language protocol — it is an
-internal Rust serialization that can evolve freely.
-
-Use the native codecs when you want a single unified trait-based
-encode/decode entry point for the generic in-process sketch types,
-without going through a wire-format-aligned wrapper.
-
-## Choosing Between `portable` and `native`
-
-| Need | Use |
-|------|-----|
-| Send a sketch to Go (`sketchlib-go`) or any non-Rust consumer | `portable` (the wire-format-aligned types re-exported at the crate root) |
-| Persist or transport a sketch within an all-Rust pipeline | `native` (works directly on the generic [`sketches`](./api/) types) |
-| New sketch crossing the wire | Add a `portable/<name>.rs`, mirror the filename in `sketchlib-go`, and add a golden-byte test |
-| New internal-only sketch serialization | Add a `native/<name>.rs` shim and you are done |
+`src/message_pack_format/` holds the Rust-side serialization plumbing. The
+current, self-describing wire format is **ASAPv1**; its byte-level layout is the
+single source of truth in
+[**`asapv1_wire_format.md`**](./asapv1_wire_format.md) — read that for the
+authoritative spec. This page describes how the code is organized.
+
+## The ASAPv1 model
+
+Every serialized sketch is one self-delimiting envelope:
+
+```md
+[ magic:6 | version:u8 | kind_id_len:u8 | kind_id:bytes
+          | metadata_len:u32_be | payload_len:u32_be
+          | metadata:msgpack-map | payload:msgpack-array ]
+```
+
+- **Envelope** — the sketch-agnostic frame (magic `b"ASAPv1"`, version,
+  `kind_id`, and the two length prefixes). It answers *is this ours?*, *how do I
+  parse the frame?*, and *what algorithm?* with zero knowledge of any sketch.
+- **Metadata** — a msgpack **map** (self-describing) carrying the hash spec plus
+  the structural params needed to interpret the payload (HLL `precision`;
+  Count-Min `rows`/`cols`/`counter_type`/`mode`). The hash-spec values are
+  **derived from the hasher's [`HashProfile`](../src/common/hash.rs)** trait, so
+  the bytes truthfully describe how the sketch was hashed and custom hash
+  profiles serialize self-describingly. Each sketch has its own fixed metadata
+  schema with `#[serde(deny_unknown_fields)]` (fail-closed on unknown/missing
+  keys).
+- **Payload** — a positional msgpack **array** of the raw sketch state only
+  (registers, counter matrix). No field names, and nothing the `kind_id` or
+  metadata already determines.
+
+`kind_id` + metadata together fix the payload structure completely. See
+[`asapv1_wire_format.md`](./asapv1_wire_format.md) for the `kind_id` registry,
+the metadata field tables, and the byte-level encoding rules.
+
+## Code organization
+
+### Shared framing — `envelope.rs`
+
+`src/message_pack_format/envelope.rs` is the **one shared, sketch-agnostic**
+module every sketch calls into. It owns the magic sentinel, the layout version,
+and the byte framing (`encode` / `split`). It validates only the magic, version,
+and framing — it does **not** know the `kind_id` registry or any sketch. Rule
+"which `kind_id` do I own?" and all metadata validation happen in each sketch's
+decoder.
+
+### Per-sketch serialization — `src/sketches/<sketch>/wire.rs`
+
+Serialization now lives **with each sketch**, split from the algorithm:
+
+- `src/sketches/<sketch>.rs` — the **algorithm** (struct, marker types, aliases,
+  insert/estimate/merge). It declares `mod wire;`.
+- `src/sketches/<sketch>/wire.rs` — the **serialization** (metadata/payload DTOs,
+  `kind_id` consts, wire-variant/counter/mode marker traits, and the
+  `serialize_to_bytes` / `deserialize_from_bytes` impls). Because `wire` is a
+  child submodule of the sketch, it reads the struct's **private** fields
+  (`self.registers`, `self.counts`) directly — no field is widened for
+  serialization.
+
+Converted today:
+
+- **HLL** (`src/sketches/hll/wire.rs`) — all variants (Classic → `0x01 0x01`,
+  Ertl-MLE → `0x01 0x02`, HIP → `0x01 0x03`) × precisions (P12/P14/P16) ×
+  `H: HashProfile`. HLL is **fully wire-covered**. (HIP is a non-generic struct
+  hashed through the default functions, so it is wire-eligible under the standard
+  profile only.)
+- **Count-Min** (`src/sketches/countminsketch/wire.rs`, kind `0x02 0x00`) —
+  restricted to wire-eligible configs:
+  `CountMin<Vector2D<T>, Mode, H>` where `T` is `i64` or `f64` (`CmsWireCounter`),
+  `Mode` is `FastPath` or `RegularPath` (`CmsWireMode`), and `H: HashProfile`.
+  The default `Vector2D<i32>` CMS is **not** wire-eligible; convert first. `rows`
+  and `cols` live in the metadata; the payload is just `[counts]`.
+
+Other sketches are **not yet converted** to `wire.rs`.
+
+## `portable/` and `native/` are being retired
+
+The `portable/` and `native/` sub-modules (and the `MessagePackCodec` trait /
+unified `Error` at the module root) are the **older** serialization path and are
+being **phased out** in favor of the per-sketch `wire.rs` + the shared
+`envelope.rs`.
+
+- `portable/` was previously described as "the cross-language wire format shared
+  with Go." That is no longer the direction: **ASAPv1 (per-sketch `wire.rs`) is
+  what `sketchlib-go` mirrors.** The custom per-sketch payload replaces the
+  `portable` types.
+- `native/` was a set of thin `MessagePackCodec` shims over the sketches'
+  `serialize_to_bytes` / `deserialize_from_bytes`. Those methods now emit the
+  ASAPv1 envelope directly.
+
+Sequencing note (from the spec): `portable` is not deleted until the golden
+byte-vector fixtures are the drift guard on both sides.
+
+## Cross-language parity
+
+Cross-language parity with `sketchlib-go` is proven by **golden byte-vectors** in
+[`asapv1_golden/`](../asapv1_golden) (exercised by
+[`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs)): both languages must
+decode → re-encode them byte-identically. The `kind_id` registry is mirrored
+verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently
+allocated. These goldens replace the old `portable`-as-oracle round-trip test.
 
 ## Cross-Reference
 
-- Generated rustdoc for the trait and per-algorithm wire DTOs is the
-  most up-to-date reference; build it with
-  `cargo doc --no-deps --all-features --open`.
+- [`asapv1_wire_format.md`](./asapv1_wire_format.md) — the authoritative
+  byte-level spec (envelope, metadata schema, per-sketch payloads, encoding
+  rules, wire coverage).
+- Generated rustdoc: `cargo doc --no-deps --all-features --open`.
+</content>
+</invoke>
diff --git a/docs/project_overview.md b/docs/project_overview.md
@@ -16,7 +16,8 @@ implementations, and orchestration/windowing frameworks in one crate.
 - [APIs Index](./apis.md) - Canonical API entry point, including paper references for each sketch.
 - [Advanced Use Cases](./advanced_use_cases.md) - Hierarchical queries, sketch coordination, and sliding-window frameworks explained separately.
 - [Common Module API](./api/api_common.md) - Shared types, hashing, and structures.
-- [Message Pack Format](./message_pack_format.md) - Wire envelope and wire-format-aligned sketch variants shared with `sketchlib-go`.
+- [ASAPv1 Wire Format](./asapv1_wire_format.md) - Authoritative byte-level spec for the self-describing sketch envelope mirrored by `sketchlib-go`.
+- [Message Pack Format](./message_pack_format.md) - How the serialization code is organized (shared envelope + per-sketch `wire.rs`).
 - [Library Map](./library_map.md) - Source-tree module breakdown.
 - [Feature Status](./features.md) - Implemented, in-progress, and planned work.
 - [Test Coverage Map](./tests.md) - Test organization and coverage notes.