From 8caa3bc72e6e11289deb8f059316dc1488db6bfb Mon Sep 17 00:00:00 2001 From: sdf-jkl Date: Mon, 6 Jul 2026 22:33:41 -0400 Subject: [PATCH 1/2] 0.1 draft --- content/en/blog/features/alp_encoding.md | 157 +++++++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 content/en/blog/features/alp_encoding.md diff --git a/content/en/blog/features/alp_encoding.md b/content/en/blog/features/alp_encoding.md new file mode 100644 index 00000000..55f795af --- /dev/null +++ b/content/en/blog/features/alp_encoding.md @@ -0,0 +1,157 @@ +--- +title: "ALP lightweight floating-point encoding in Apache Parquet" +date: 2026-06-23 +description: "The consensus building machine strikes again" +author: "[Konstantin Tarasov](https://github.com/sdf-jkl)" +categories: ["features"] +--- + +## What is ALP? + +ALP is a new lightweight floating-point encoding that shows better compression ratio than heavy-weight encodings like zstd **and** faster [de]compression speed than other lightweight floating-point encodings. + +It was developed by the folks at [CWI](https://www.cwi.nl/en/research/database-architectures/) and uses smart tricks to represent floating-point data as integers. + +ALP shines at data like: +- Monetary values (exchange rates, public funds, stocks, prices, etc.) +- Geographic coordinates (longtitute/latitude) +- Scientific measures (temperature, pressure, speed, degrees, energy, etc.) +- Timestamps stored as floating-point + +## Why ALP? + + + +Two reasons to care: +1) Less storage/faster read/write +2) Better partition pruning support + +### Better physical data representation + +Better, faster, stronger? But how much? + + + +[De]compression speed improvement depends on the Parquet reader/writer implementation, but users can expect performance simialar to one provided in the paper. + + + +### The most efficient way to read is to not read at all + +One of the gripes users have with parquet is opimization for random access[1]. The minimum amount of data you need to decode and decompress in Parquet is usually the entire page. + +ALP encoding avoids that by splitting the page into vectors (usually 1024-len, for better SIMD and being able to store in L1). Each vector stores the necessary metadata to decode any row inside. More in the [Technical overview](#technical-overview). + +This way, instead of: + +- Load the whole page to retrieve the value + +Becomes: + +- Load just the page metadata -> Get the vector offset -> only decode that vector + +This basically introduces lower grain than a Parquet page and can significantly speed up single row decode/random access. + +## Technical overview + + + +ALP takes inspiration from [PDE (Pseudodecimal encoding)](https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf) and havily improves on it. + +The way that PDE works is by finding the smallest precision integer to represent the floating-point value. + +For example: +``` +8.0605 that can't be physically represented in [IEEE 754](https://ieeexplore.ieee.org/document/8766229) as is, is instead approximated as 8.06049999999999933209. + +PDE takes a value and brute-force searches over the exponent, where - significand = round(value × 10^e) + +For this value it should get to 80605 with exponent 4. +``` + +The catch is PDE keeps an exponent per each value. + +The key ideas behind ALP are: +- the majority of floating-point values share similar decimal precision within 1024-len vectors +- there is a huge room and benefit in vectorizing the encoding + +The first idea leads to an improvement over PDE. Instead of storing an exponent per value, we store it per vector. + +This leads to smaller metadata storage footprint. + +The second idea uses SIMD intrinsics to make the code as parallelizable as possible to achieve greater [de]compression speeds. + +### ALP Encoding pipeline in Paruqet + +The way that ALP encodes data is: + + Input: float/double array + | + v + +----------------------------------------------------------+ + | 1. CHOOSE PARAMETERS | + | Select (exponent, factor) pair for this vector | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 2. DECIMAL ENCODING | + | encoded[i] = fast_round(value[i] * 10^e * 10^(-f)) | + | Detect exceptions where decode(encode(v)) != v | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 3. FRAME OF REFERENCE (FOR) | + | min_val = min(encoded[]) | + | delta[i] = encoded[i] - min_val | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 4. BIT PACKING | + | bit_width = ceil(log2(max_delta + 1)) | + | Pack each delta into bit_width bits | + +----------------------------------------------------------+ + | + v + Output: Serialized vector bytes + + + + +ALP paper also introduces an ALPrd enocding. It's used when ALP can't compress enought values in a row group. Apache Parquet already implements similar mechanism via BYTE_STREAM_SPLIT + ZSTD, therefore this encoding was not included. + +## Ecosystem adoption + + + +The paper was published in 2024 and now has a sibling G-ALP https://ir.cwi.nl/pub/35205/35205.pdf for optimized for GPUs. + +Since then ALP has been adopted by the new file formats like Vortex [https://github.com/vortex-data/vortex], F3 [https://github.com/future-file-format/f3] and Fastlanes (created by the same folks as ALP) [https://github.com/cwida/fastlanes]. + +These formats aspire to dethrone Parquet, some(Vortex) say - "Parquet is for the floor" + +. + +This comment may sound mean, but what it really means is that Parquet stands as a baseline and a benchmark for the new contenders. + +The new file formats are created and improving rapidly because they don't have to worry about backwards compatibility for a giant userbase. Parquet, however, does not stand in place - it's a moving target. It takes more time, but it learns from the new formats and their ideas and implements them. + +## Real-World Examples + + + +## Conclusion + + + +ALP encoding will bridge the gap between Parquet and parallelizible + random access for floating-point data. One step closer to meeting the needs of the rapidly scaling environment. + +## Resources + +- [ALP paper (CWI)](https://ir.cwi.nl/pub/33334/33334.pdf) +- [Column Storage for the AI era](https://sympathetic.ink/2025/12/11/Column-Storage-for-the-AI-era.html) +- [Column Storage for the AI era - file formats encoding parity slide](https://docs.google.com/presentation/d/19F-XvNJ8sgIpIeIduA3PhbsWp4pC-P632J2eJV1cLG8/edit?slide=id.g3b660485fe2_0_9#slide=id.g3b660485fe2_0_9) +- [ALP Parquet encoding PR](https://github.com/apache/parquet-format/pull/557) +- [BtrBlocks paper](https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf) \ No newline at end of file From abe8b1f350f062f09c85ff67a31745590906caab Mon Sep 17 00:00:00 2001 From: sdf-jkl Date: Wed, 8 Jul 2026 13:40:30 -0400 Subject: [PATCH 2/2] second pass --- content/en/blog/features/alp_encoding.md | 101 ++++++++++++----------- 1 file changed, 51 insertions(+), 50 deletions(-) diff --git a/content/en/blog/features/alp_encoding.md b/content/en/blog/features/alp_encoding.md index 55f795af..77136d66 100644 --- a/content/en/blog/features/alp_encoding.md +++ b/content/en/blog/features/alp_encoding.md @@ -1,6 +1,6 @@ --- title: "ALP lightweight floating-point encoding in Apache Parquet" -date: 2026-06-23 +date: 2026-07-08 description: "The consensus building machine strikes again" author: "[Konstantin Tarasov](https://github.com/sdf-jkl)" categories: ["features"] @@ -8,80 +8,69 @@ categories: ["features"] ## What is ALP? -ALP is a new lightweight floating-point encoding that shows better compression ratio than heavy-weight encodings like zstd **and** faster [de]compression speed than other lightweight floating-point encodings. +ALP (Adaptive Lossless floating-Point) is a new lightweight floating-point encoding that shows a better compression ratio than heavyweight compressors like zstd **and** faster [de]compression speed than other lightweight floating-point encodings. It was developed by the folks at [CWI](https://www.cwi.nl/en/research/database-architectures/) and uses smart tricks to represent floating-point data as integers. ALP shines at data like: - Monetary values (exchange rates, public funds, stocks, prices, etc.) -- Geographic coordinates (longtitute/latitude) +- Geographic coordinates (longitude/latitude) - Scientific measures (temperature, pressure, speed, degrees, energy, etc.) - Timestamps stored as floating-point ## Why ALP? - - -Two reasons to care: -1) Less storage/faster read/write -2) Better partition pruning support - ### Better physical data representation Better, faster, stronger? But how much? -[De]compression speed improvement depends on the Parquet reader/writer implementation, but users can expect performance simialar to one provided in the paper. +[De]compression speed improvement depends on the Parquet reader/writer implementation, but users can expect performance similar to what the paper reports. ### The most efficient way to read is to not read at all -One of the gripes users have with parquet is opimization for random access[1]. The minimum amount of data you need to decode and decompress in Parquet is usually the entire page. +One of the [gripes users have with Parquet](https://sympathetic.ink/2025/12/11/Column-Storage-for-the-AI-era.html) is that it is not optimized for random access. The minimum amount of data you need to decompress and decode in Parquet is usually an entire page. -ALP encoding avoids that by splitting the page into vectors (usually 1024-len, for better SIMD and being able to store in L1). Each vector stores the necessary metadata to decode any row inside. More in the [Technical overview](#technical-overview). +ALP encoding avoids that by splitting the page into vectors (usually 1024 values each, sized for SIMD and for fitting in L1 cache). Each vector stores the necessary metadata to decode any row inside. -This way, instead of: +**Before**, reading a single value: -- Load the whole page to retrieve the value +> Load the whole page → decompress and decode all of it → retrieve the value -Becomes: +**After**, with ALP: -- Load just the page metadata -> Get the vector offset -> only decode that vector +> Load just the page metadata → get the vector offset → decode only that vector -This basically introduces lower grain than a Parquet page and can significantly speed up single row decode/random access. +This introduces a finer read granularity than the Parquet page and can significantly speed up single-row decodes and random access. ## Technical overview - +ALP takes inspiration from [PDE (Pseudodecimal encoding)](https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf) and heavily improves on it. -ALP takes inspiration from [PDE (Pseudodecimal encoding)](https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf) and havily improves on it. - -The way that PDE works is by finding the smallest precision integer to represent the floating-point value. +PDE works by finding the smallest integer, paired with a decimal exponent, that represents the floating-point value exactly. For example: -``` -8.0605 that can't be physically represented in [IEEE 754](https://ieeexplore.ieee.org/document/8766229) as is, is instead approximated as 8.06049999999999933209. - -PDE takes a value and brute-force searches over the exponent, where - significand = round(value × 10^e) -For this value it should get to 80605 with exponent 4. -``` +> 8.0605 can't be physically represented in [IEEE 754](https://ieeexplore.ieee.org/document/8766229) as is, and is instead approximated as 8.06049999999999933209. +> +> PDE takes the value and brute-force searches over exponents, computing `significand = round(value × 10^e)` for each. +> +> For this value it arrives at significand 80605 with exponent 4. -The catch is PDE keeps an exponent per each value. +The catch is that PDE stores an exponent for every single value. The key ideas behind ALP are: -- the majority of floating-point values share similar decimal precision within 1024-len vectors -- there is a huge room and benefit in vectorizing the encoding +- the majority of floating-point values share similar decimal precision within vectors +- there is a huge benefit in vectorizing the encoding -The first idea leads to an improvement over PDE. Instead of storing an exponent per value, we store it per vector. - -This leads to smaller metadata storage footprint. +The first idea leads to an improvement over PDE. Instead of storing an exponent per value, we store it per vector. This way we get a much smaller metadata footprint. The second idea uses SIMD intrinsics to make the code as parallelizable as possible to achieve greater [de]compression speeds. -### ALP Encoding pipeline in Paruqet +### ALP Encoding pipeline in Parquet The way that ALP encodes data is: @@ -117,41 +106,53 @@ The way that ALP encodes data is: v Output: Serialized vector bytes +**1. Choose parameters** - +First, sample multiple vectors within a row group to find the overall best combinations of exponent and factor. -ALP paper also introduces an ALPrd enocding. It's used when ALP can't compress enought values in a row group. Apache Parquet already implements similar mechanism via BYTE_STREAM_SPLIT + ZSTD, therefore this encoding was not included. +Then re-evaluate those candidate combinations on each vector to pick the pair that compresses it best. -## Ecosystem adoption +**2. Decimal encoding** - +Encode each value with the formula above, then round-trip it (decode it back) to make sure it matches the original. -The paper was published in 2024 and now has a sibling G-ALP https://ir.cwi.nl/pub/35205/35205.pdf for optimized for GPUs. +If the round-trip fails, the value is added to the vector's exceptions list. -Since then ALP has been adopted by the new file formats like Vortex [https://github.com/vortex-data/vortex], F3 [https://github.com/future-file-format/f3] and Fastlanes (created by the same folks as ALP) [https://github.com/cwida/fastlanes]. +**3. Frame of reference (FOR)** -These formats aspire to dethrone Parquet, some(Vortex) say - "Parquet is for the floor" +Once the values are encoded as integers, we pick the smallest one as the frame of reference. Every value is then stored as its delta from that minimum. -. +**4. Bit packing** -This comment may sound mean, but what it really means is that Parquet stands as a baseline and a benchmark for the new contenders. +Finally, we take the bit width of the largest delta and pack every delta into that many bits. -The new file formats are created and improving rapidly because they don't have to worry about backwards compatibility for a giant userbase. Parquet, however, does not stand in place - it's a moving target. It takes more time, but it learns from the new formats and their ideas and implements them. -## Real-World Examples +### ALPrd - +The ALP paper also introduces an ALPrd encoding, used when ALP can't compress enough of the values in a row group. Apache Parquet already provides a similar mechanism via BYTE_STREAM_SPLIT + ZSTD, so ALPrd was not included. -## Conclusion +## Ecosystem adoption - +The paper was published in 2024 and now has a sibling [G-ALP](https://ir.cwi.nl/pub/35205/35205.pdf) optimized for GPUs. + +Since then ALP has been adopted as the default floating-point encoding by [DuckDB](https://duckdb.org/2024/02/13/announcing-duckdb-0100#adaptive-lossless-floating-point-compression-alp) and [CedarDB](https://cedardb.com/blog/release_notes/early_2026/#better-compression-less-storage-floats-and-text), and implemented as a codec by [ClickHouse](https://github.com/ClickHouse/ClickHouse/pull/91362). + +It has also been adopted by the new file formats like [Vortex](https://github.com/vortex-data/vortex), [F3](https://github.com/future-file-format/f3) and [FastLanes](https://github.com/cwida/fastlanes) (created by the same folks as ALP). + +These formats aspire to dethrone Parquet, some (Vortex) say - ["Parquet is for the floor"](). + +This comment may sound mean, but what it really means is that Parquet stands as a baseline and a benchmark for the new contenders. + +The new file formats improve rapidly because they don't have to worry about backwards compatibility for a giant userbase. Parquet, however, does not stand still: it's a moving target. It takes more time, but it learns from the new formats and adopts their best ideas. + +## Conclusion -ALP encoding will bridge the gap between Parquet and parallelizible + random access for floating-point data. One step closer to meeting the needs of the rapidly scaling environment. +ALP brings fast, parallelizable decoding and practical random access to floating-point data in Parquet. It's one more step toward closing the gap with the newest file formats. ## Resources - [ALP paper (CWI)](https://ir.cwi.nl/pub/33334/33334.pdf) - [Column Storage for the AI era](https://sympathetic.ink/2025/12/11/Column-Storage-for-the-AI-era.html) -- [Column Storage for the AI era - file formats encoding parity slide](https://docs.google.com/presentation/d/19F-XvNJ8sgIpIeIduA3PhbsWp4pC-P632J2eJV1cLG8/edit?slide=id.g3b660485fe2_0_9#slide=id.g3b660485fe2_0_9) +- [Column Storage for the AI era slides - file formats encoding parity slide](https://docs.google.com/presentation/d/19F-XvNJ8sgIpIeIduA3PhbsWp4pC-P632J2eJV1cLG8/edit?slide=id.g3b660485fe2_0_9#slide=id.g3b660485fe2_0_9) - [ALP Parquet encoding PR](https://github.com/apache/parquet-format/pull/557) - [BtrBlocks paper](https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf) \ No newline at end of file