[WIP][POC] Pfor encoding - #50088

Draft

prtkgaur wants to merge 41 commits into

apache:mainfrom

prtkgaur:pfor-encoding

Commits on Jul 15, 2026

Add PFOR core algorithm and tests
sfc-gh-pgaur
committed
Integrate PFOR encoding into parquet encoder/decoder
sfc-gh-pgaur
committed
Add PFOR encoding benchmark
sfc-gh-pgaur
committed
Use signed integer types consistently per Arrow style guide
sfc-gh-pgaur
committed
Use arrow::util::span for buffer parameters in Store/Load/Decode/Serialize
sfc-gh-pgaur
committed
Return Result<T>/Status on decode paths instead of ARROW_DCHECK
sfc-gh-pgaur
committed
Add static_assert(ARROW_LITTLE_ENDIAN) and replace reinterpret_cast with SafeCopy
sfc-gh-pgaur
committed
Use single-call unpack() instead of manual batch loop + BitReader remainder
sfc-gh-pgaur
committed
Add pragma GCC unroll/ivdep to decode loops for better vectorization
sfc-gh-pgaur
committed
Add PforEncodedVectorView for zero-copy decode path
sfc-gh-pgaur
committed
Make vector_size configurable on encode path with default kPforVectorSize
sfc-gh-pgaur
committed
Fix pfor_test.cc: unwrap Result<> from PforVectorInfo::Load()
sfc-gh-pgaur
committed
Convert PforWrapper::LoadHeader to return Result<PforHeader>
sfc-gh-pgaur
committed
Use SafeLoadAs/SafeStore for header and offset array in PforWrapper
sfc-gh-pgaur
committed
Validate header fields in PforWrapper::LoadHeader
sfc-gh-pgaur
committed
Use int64_t and global ::arrow:: prefix at PFOR encoder/decoder call sites
sfc-gh-pgaur
committed
Use uint8_t* consistently in PforWrapper API instead of char*
sfc-gh-pgaur
committed
Convert PforVectorInfo to class with SafeLoadAs/SafeStore and bit_width validation
sfc-gh-pgaur
committed
Convert PforEncodedVector and PforEncodedVectorView to classes
sfc-gh-pgaur
committed
Tighten PforVectorInfo encapsulation: validate num_exceptions in Load, use ctor in EncodeVector
sfc-gh-pgaur
committed
make
sfc-gh-pgaur
committed
Use bit_util::IsPowerOf2/CeilDiv and add incremental encode/decode TODOs (PFOR)
sfc-gh-pgaur
committed
Drop Snowflake attribution from PFOR header comments
sfc-gh-pgaur
committed
Fix IsPowerOf2(int32_t) ambiguity in pfor_wrapper.cc
sfc-gh-pgaur
committed
Add FastLanes auto-vectorized bit-packing library
sfc-gh-pgaur
committed
Add FastLanes-FOR to parquet-pfor-comparison-benchmark
sfc-gh-pgaur
committed
FastLanes-FOR: add DecodeFlat (flat output) and benchmark it
sfc-gh-pgaur
committed
FastLanes: add toTransposed32; correct fromTransposed32 docstring
sfc-gh-pgaur
committed
PFOR: add PackingMode (BitPack | FastLanes) per-vector flag
sfc-gh-pgaur
committed
PFOR: add OutputOrder { Flat, Transposed } decode option
sfc-gh-pgaur
committed
Local: keep -O3 from CMake Release defaults (drop -O2 downgrade)
sfc-gh-pgaur
committed

Commits on Jul 24, 2026