diff --git a/docs/api/api_countmin.md b/docs/api/api_countmin.md index 7467f15..73bd20a 100644 --- a/docs/api/api_countmin.md +++ b/docs/api/api_countmin.md @@ -53,6 +53,17 @@ fn serialize_to_bytes(&self) -> Result, RmpEncodeError> fn deserialize_from_bytes(bytes: &[u8]) -> Result ``` +These produce/consume the **ASAPv1** wire envelope (kind `0x02 0x00`) — see the +[ASAPv1 wire format spec](../asapv1_wire_format.md). They are **not** available +on every `CountMin`: the impl exists only for wire-eligible configs +`CountMin, Mode, H>` where `T` is `i64` or `f64` (`CmsWireCounter`), +`Mode` is `FastPath` or `RegularPath` (`CmsWireMode`), and `H: HashProfile`. The +default storage is `Vector2D`, which is **not** wire-eligible — an `i32` / +`i128` / other exotic-counter or non-`Vector2D` sketch must be converted to a +`Vector2D` / `Vector2D` first (only you know if the mapping is +lossless). `rows`/`cols` are carried in the envelope metadata; the payload is +just `[counts]`. + ## Examples ```rust diff --git a/docs/api/api_hyperloglog.md b/docs/api/api_hyperloglog.md index bc20250..4e2752b 100644 --- a/docs/api/api_hyperloglog.md +++ b/docs/api/api_hyperloglog.md @@ -78,6 +78,18 @@ fn serialize_to_bytes(&self) -> Result, RmpEncodeError> fn deserialize_from_bytes(bytes: &[u8]) -> Result ``` +These produce/consume the **ASAPv1** wire envelope — see the +[ASAPv1 wire format spec](../asapv1_wire_format.md). On the generic +`HyperLogLogImpl`, the methods are bounded on +`Variant: HllWireVariant` and `H: HashProfile`, so every +(variant × precision × hasher) combination serializes — **HLL is fully +wire-covered**. Each variant maps to its own `kind_id` (Classic → `0x01 0x01`, +Ertl-MLE → `0x01 0x02`, HIP → `0x01 0x03`); `precision` and the +`HashProfile`-derived hash spec live in the envelope metadata, and the payload is +the register bytes (plus three HIP scalars for the HIP variant). `HyperLogLogHIP` +is a non-generic struct that hashes through the default functions, so it is +wire-eligible under the **standard** hash profile only. + ## Examples ```rust diff --git a/docs/features.md b/docs/features.md index 7d710cf..2cedd17 100644 --- a/docs/features.md +++ b/docs/features.md @@ -100,7 +100,7 @@ Insertion throughput measured on 10,000,000 Zipf-distributed `int64` values (s=1 ### Serialization -MessagePack (`rmp-serde`) support. **serde support** means the type derives `Serialize`/`Deserialize` and can be used with any serde-compatible serializer. **Built-in helpers** (`serialize_to_bytes` / `deserialize_from_bytes`) provide one-call MessagePack round-tripping without requiring users to depend on `rmp-serde` directly. +MessagePack (`rmp-serde`) support. **serde support** means the type derives `Serialize`/`Deserialize` and can be used with any serde-compatible serializer. **Built-in helpers** (`serialize_to_bytes` / `deserialize_from_bytes`) provide one-call round-tripping without requiring users to depend on `rmp-serde` directly. For HyperLogLog and Count-Min these helpers emit the self-describing **ASAPv1** wire envelope (see the [ASAPv1 wire format spec](./asapv1_wire_format.md)); other sketches are not yet converted. | Component | serde support | Built-in helpers | | --- | --- | --- | diff --git a/docs/index.md b/docs/index.md index e3b6527..390359e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -14,7 +14,8 @@ This page is the docs home for `asap_sketchlib`. - [APIs Index](./apis.md) - [API Pages Directory](./api/) - [Common API](./api/api_common.md) -- [Message Pack Format](./message_pack_format.md) - wire envelope and wire-format-aligned sketch variants shared with `sketchlib-go` +- [ASAPv1 Wire Format](./asapv1_wire_format.md) - authoritative byte-level spec for the self-describing sketch envelope mirrored by `sketchlib-go` +- [Message Pack Format](./message_pack_format.md) - how the serialization code is organized (shared envelope + per-sketch `wire.rs`; `portable`/`native` retiring) - [Advanced Use Cases](./advanced_use_cases.md) - [Test Coverage Map](./tests.md) - [Feature Status](./features.md) diff --git a/docs/library_map.md b/docs/library_map.md index 978c61c..db4377a 100644 --- a/docs/library_map.md +++ b/docs/library_map.md @@ -12,15 +12,17 @@ - **`src/sketches/`** - Sketch implementations (status source: [apis.md](./apis.md)) - `Ready` in API index: `countminsketch.rs`, `countsketch.rs`, `hll.rs`, `kll.rs`, `ddsketch.rs`, `countminsketch_topk.rs`, `countsketch_topk.rs` - `Unstable` in API index: `coco.rs`, `elastic.rs`, `uniform.rs`, `kmv.rs` + - Sketches converted to the ASAPv1 wire format use a `.rs` (algorithm) + `/wire.rs` (serialization) split: `hll.rs` + `hll/wire.rs` and `countminsketch.rs` + `countminsketch/wire.rs` today (see [asapv1_wire_format.md](./asapv1_wire_format.md)) - **`src/sketch_framework/`** - Orchestration and serving layers (status source: [apis.md](./apis.md)) - `Ready` in API index: `hydra.rs`, `hashlayer.rs`, `univmon.rs`, `univmon_optimized.rs`, `nitro.rs`, `eh.rs`, `eh_sketch_list.rs` - `Unstable` in API index: `eh_univ_optimized.rs` - Infrastructure module: `orchestrator/` (node-level manager used by framework APIs) -- **`src/message_pack_format/`** - MessagePack wire-format definitions, the `MessagePackCodec` trait, and the wire-format-aligned sketch types shared with `sketchlib-go` ([message_pack_format.md](./message_pack_format.md)) - - `portable/` — wire-format-aligned sketch types (`CountMinSketch`, `CountSketch`, `DdSketch`, `HllSketch`, `KllSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, `SetAggregator`, `DeltaResult`) consumed by the ASAP query engine, byte-compatible with `sketchlib-go` - - `native/` — Rust-internal codec shims over `src/sketches/` byte serialization +- **`src/message_pack_format/`** - Serialization plumbing ([message_pack_format.md](./message_pack_format.md)). The current format is **ASAPv1**, specified in [asapv1_wire_format.md](./asapv1_wire_format.md) + - `envelope.rs` — the shared, sketch-agnostic ASAPv1 framing (magic/version/`kind_id` + length prefixes, `encode`/`split`); every per-sketch `wire.rs` calls into it + - `portable/` — **deprecated**, being retired. The older per-sketch wire types (`CountMinSketch`, `HllSketch`, …); ASAPv1 (per-sketch `wire.rs`) is now what `sketchlib-go` mirrors, not these + - `native/` — **deprecated**, being retired. Older `MessagePackCodec` shims over `src/sketches/` byte serialization ## Documentation diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index a2017a9..adb3209 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -1,123 +1,112 @@ # Message Pack Format -`src/message_pack_format/` is the Rust-side source of truth for the -MessagePack encode/decode contract. It is split into two sub-modules -by audience: - -- **`portable/`** — cross-language wire format shared with the Go - counterpart `sketchlib-go`. Touching anything here is a protocol - change and requires the Go side to be kept in lock-step (golden-byte - tests catch drift). -- **`native/`** — thin trait shims over the existing - `serialize_to_bytes` / `deserialize_from_bytes` methods on the - pure-Rust generic sketch types in [`src/sketches/`](./api/). The byte - format is internal to Rust — Go never reads it, and the format is - free to evolve without cross-language coordination. - -The [`MessagePackCodec`](#core-types) trait and unified `Error` type -live at the module root so both worlds share the same encode/decode -contract. - -## Core Types - -Both live at the top of `src/message_pack_format/` and are re-exported -through the module root: - -- `MessagePackCodec` (in [`codec.rs`](../src/message_pack_format/codec.rs)) — - the trait every codec-enabled type implements. Two methods: - `to_msgpack`, `from_msgpack`. This is the canonical encode/decode - entry point. -- `Error` (in [`error.rs`](../src/message_pack_format/error.rs)) — the - unified encode/decode error type returned by both `native` and - `portable` impls. - -## `portable/` — Cross-Language Wire Format - -One submodule per algorithm, with the filenames mirrored on the Go -side: - -- `countminsketch.rs`, `countminsketch_topk.rs`, `countsketch.rs`, - `ddsketch.rs`, `hll.rs`, `kll.rs`, `hydra_kll.rs`, - `set_aggregator.rs`, `delta_set_aggregator.rs` - -Each submodule owns: - -1. The wire-format-aligned runtime type and its delta companion (e.g. - `CountMinSketch`, `CountMinSketchDelta`) — these are the types - re-exported at the crate root. -2. The wire DTO struct(s), when the runtime type needs a separate - over-the-wire shape (e.g. borrow / owned pairs, byte-compatible - field reordering with `sketchlib-go`). -3. The `MessagePackCodec` impl for the runtime type. - -The full list of re-exported wire-format-aligned types: - -- `countminsketch.rs` — `CountMinSketch`, `CountMinSketchDelta` -- `countminsketch_topk.rs` — `CountMinSketchWithHeap`, `CmsHeapItem` -- `countsketch.rs` — `CountSketch`, `CountSketchDelta` -- `ddsketch.rs` — `DdSketch`, `DdSketchDelta` -- `hll.rs` — `HllSketch`, `HllSketchDelta`, `HllVariant` -- `kll.rs` — `KllSketch`, `KllSketchData` -- `hydra_kll.rs` — `HydraKllSketch` -- `set_aggregator.rs` — `SetAggregator` -- `delta_set_aggregator.rs` — `DeltaResult` - -Use these for sketches that must cross a process / language boundary. -For high-throughput local ingest, custom hashers, and framework -composition, reach for the generic sketches in -[`src/sketches/`](./apis.md) instead: - -| Need | Use | -| --- | --- | -| Local high-throughput ingest, custom hashers, framework composition | [`src/sketches/`](./apis.md) | -| Cross-process / cross-language transfer matching `sketchlib-go` bytes | `src/message_pack_format/portable/` | - -### Types that act as their own DTO - -`CountSketch`, `DdSketch`, and `HllSketch` derive `Serialize` / -`Deserialize` directly because their public field layout already -matches the wire shape. Their `MessagePackCodec` impls serialize the -struct verbatim — no separate DTO is required. - -### Protocol invariants - -- The wire envelope must remain byte-compatible with `sketchlib-go`. -- Adding, reordering, renaming, or retyping a field counts as a - protocol change; bump the format version on both sides and add a - golden-byte test before shipping. -- DTOs that appear as nested fields in another wire type (e.g. - `KllSketchData` inside `HydraKllSketchWire`) are part of the same - protocol surface — treat them with the same care. - -## `native/` — Rust-Internal Codec Shims - -One submodule per generic sketch type in [`src/sketches/`](./api/) -whose serialization is exposed through `MessagePackCodec`: - -- `countminsketch.rs`, `countsketch.rs`, `countsketch_topk.rs`, - `ddsketch.rs`, `hll.rs`, `kll.rs`, `kll_dynamic.rs` -- `kmv.rs` (gated behind the `experimental` feature flag) - -Each impl forwards `to_msgpack` / `from_msgpack` to the sketch's -existing `serialize_to_bytes` / `deserialize_from_bytes` methods. The -byte format is **not** part of the cross-language protocol — it is an -internal Rust serialization that can evolve freely. - -Use the native codecs when you want a single unified trait-based -encode/decode entry point for the generic in-process sketch types, -without going through a wire-format-aligned wrapper. - -## Choosing Between `portable` and `native` - -| Need | Use | -|------|-----| -| Send a sketch to Go (`sketchlib-go`) or any non-Rust consumer | `portable` (the wire-format-aligned types re-exported at the crate root) | -| Persist or transport a sketch within an all-Rust pipeline | `native` (works directly on the generic [`sketches`](./api/) types) | -| New sketch crossing the wire | Add a `portable/.rs`, mirror the filename in `sketchlib-go`, and add a golden-byte test | -| New internal-only sketch serialization | Add a `native/.rs` shim and you are done | +`src/message_pack_format/` holds the Rust-side serialization plumbing. The +current, self-describing wire format is **ASAPv1**; its byte-level layout is the +single source of truth in +[**`asapv1_wire_format.md`**](./asapv1_wire_format.md) — read that for the +authoritative spec. This page describes how the code is organized. + +## The ASAPv1 model + +Every serialized sketch is one self-delimiting envelope: + +```md +[ magic:6 | version:u8 | kind_id_len:u8 | kind_id:bytes + | metadata_len:u32_be | payload_len:u32_be + | metadata:msgpack-map | payload:msgpack-array ] +``` + +- **Envelope** — the sketch-agnostic frame (magic `b"ASAPv1"`, version, + `kind_id`, and the two length prefixes). It answers *is this ours?*, *how do I + parse the frame?*, and *what algorithm?* with zero knowledge of any sketch. +- **Metadata** — a msgpack **map** (self-describing) carrying the hash spec plus + the structural params needed to interpret the payload (HLL `precision`; + Count-Min `rows`/`cols`/`counter_type`/`mode`). The hash-spec values are + **derived from the hasher's [`HashProfile`](../src/common/hash.rs)** trait, so + the bytes truthfully describe how the sketch was hashed and custom hash + profiles serialize self-describingly. Each sketch has its own fixed metadata + schema with `#[serde(deny_unknown_fields)]` (fail-closed on unknown/missing + keys). +- **Payload** — a positional msgpack **array** of the raw sketch state only + (registers, counter matrix). No field names, and nothing the `kind_id` or + metadata already determines. + +`kind_id` + metadata together fix the payload structure completely. See +[`asapv1_wire_format.md`](./asapv1_wire_format.md) for the `kind_id` registry, +the metadata field tables, and the byte-level encoding rules. + +## Code organization + +### Shared framing — `envelope.rs` + +`src/message_pack_format/envelope.rs` is the **one shared, sketch-agnostic** +module every sketch calls into. It owns the magic sentinel, the layout version, +and the byte framing (`encode` / `split`). It validates only the magic, version, +and framing — it does **not** know the `kind_id` registry or any sketch. Rule +"which `kind_id` do I own?" and all metadata validation happen in each sketch's +decoder. + +### Per-sketch serialization — `src/sketches//wire.rs` + +Serialization now lives **with each sketch**, split from the algorithm: + +- `src/sketches/.rs` — the **algorithm** (struct, marker types, aliases, + insert/estimate/merge). It declares `mod wire;`. +- `src/sketches//wire.rs` — the **serialization** (metadata/payload DTOs, + `kind_id` consts, wire-variant/counter/mode marker traits, and the + `serialize_to_bytes` / `deserialize_from_bytes` impls). Because `wire` is a + child submodule of the sketch, it reads the struct's **private** fields + (`self.registers`, `self.counts`) directly — no field is widened for + serialization. + +Converted today: + +- **HLL** (`src/sketches/hll/wire.rs`) — all variants (Classic → `0x01 0x01`, + Ertl-MLE → `0x01 0x02`, HIP → `0x01 0x03`) × precisions (P12/P14/P16) × + `H: HashProfile`. HLL is **fully wire-covered**. (HIP is a non-generic struct + hashed through the default functions, so it is wire-eligible under the standard + profile only.) +- **Count-Min** (`src/sketches/countminsketch/wire.rs`, kind `0x02 0x00`) — + restricted to wire-eligible configs: + `CountMin, Mode, H>` where `T` is `i64` or `f64` (`CmsWireCounter`), + `Mode` is `FastPath` or `RegularPath` (`CmsWireMode`), and `H: HashProfile`. + The default `Vector2D` CMS is **not** wire-eligible; convert first. `rows` + and `cols` live in the metadata; the payload is just `[counts]`. + +Other sketches are **not yet converted** to `wire.rs`. + +## `portable/` and `native/` are being retired + +The `portable/` and `native/` sub-modules (and the `MessagePackCodec` trait / +unified `Error` at the module root) are the **older** serialization path and are +being **phased out** in favor of the per-sketch `wire.rs` + the shared +`envelope.rs`. + +- `portable/` was previously described as "the cross-language wire format shared + with Go." That is no longer the direction: **ASAPv1 (per-sketch `wire.rs`) is + what `sketchlib-go` mirrors.** The custom per-sketch payload replaces the + `portable` types. +- `native/` was a set of thin `MessagePackCodec` shims over the sketches' + `serialize_to_bytes` / `deserialize_from_bytes`. Those methods now emit the + ASAPv1 envelope directly. + +Sequencing note (from the spec): `portable` is not deleted until the golden +byte-vector fixtures are the drift guard on both sides. + +## Cross-language parity + +Cross-language parity with `sketchlib-go` is proven by **golden byte-vectors** in +[`asapv1_golden/`](../asapv1_golden) (exercised by +[`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs)): both languages must +decode → re-encode them byte-identically. The `kind_id` registry is mirrored +verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently +allocated. These goldens replace the old `portable`-as-oracle round-trip test. ## Cross-Reference -- Generated rustdoc for the trait and per-algorithm wire DTOs is the - most up-to-date reference; build it with - `cargo doc --no-deps --all-features --open`. +- [`asapv1_wire_format.md`](./asapv1_wire_format.md) — the authoritative + byte-level spec (envelope, metadata schema, per-sketch payloads, encoding + rules, wire coverage). +- Generated rustdoc: `cargo doc --no-deps --all-features --open`. + + diff --git a/docs/project_overview.md b/docs/project_overview.md index c8ee56a..4cb6537 100644 --- a/docs/project_overview.md +++ b/docs/project_overview.md @@ -16,7 +16,8 @@ implementations, and orchestration/windowing frameworks in one crate. - [APIs Index](./apis.md) - Canonical API entry point, including paper references for each sketch. - [Advanced Use Cases](./advanced_use_cases.md) - Hierarchical queries, sketch coordination, and sliding-window frameworks explained separately. - [Common Module API](./api/api_common.md) - Shared types, hashing, and structures. -- [Message Pack Format](./message_pack_format.md) - Wire envelope and wire-format-aligned sketch variants shared with `sketchlib-go`. +- [ASAPv1 Wire Format](./asapv1_wire_format.md) - Authoritative byte-level spec for the self-describing sketch envelope mirrored by `sketchlib-go`. +- [Message Pack Format](./message_pack_format.md) - How the serialization code is organized (shared envelope + per-sketch `wire.rs`). - [Library Map](./library_map.md) - Source-tree module breakdown. - [Feature Status](./features.md) - Implemented, in-progress, and planned work. - [Test Coverage Map](./tests.md) - Test organization and coverage notes.