Skip to content

Optimize collect_list for fixed-width and varchar types - #731

Open
taiyang-li wants to merge 6 commits into
bytedance:mainfrom
taiyang-li:collect-list-top1-typed-oss
Open

Optimize collect_list for fixed-width and varchar types#731
taiyang-li wants to merge 6 commits into
bytedance:mainfrom
taiyang-li:collect-list-top1-typed-oss

Conversation

@taiyang-li

@taiyang-li taiyang-li commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #xxx

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ✨ New feature (non-breaking change which adds functionality)
  • 🚀 Performance improvement (optimization)
  • ⚠️ Breaking change (fix or feature that would cause existing functionality to change)
  • 🔨 Refactoring (no logic changes)
  • 🔧 Build/CI or Infrastructure changes
  • 📝 Documentation only

Description

This PR optimizes Spark-compatible collect_list aggregation in two paths.

For fixed-width primitive inputs, such as BIGINT, INT, DOUBLE, etc., collect_list now uses a typed accumulator backed by contiguous typed storage instead of the generic ValueList + ContainerRowSerde path. This avoids per-value generic row serialization/deserialization and reduces accumulator overhead for fixed-width values.

For VARCHAR / VARBINARY inputs, this PR keeps the existing ValueList accumulator layout, but adds a string-specific fast path for appending and reading string values. The serialized layout remains length + bytes, while avoiding the broader ContainerRowSerde dispatch path for string collect_list accumulation and extraction.

Together, these changes improve both fixed-width and string collect_list workloads, including Spark-style partial aggregation -> final aggregation plans.

Performance Impact

Benchmark target:

./_build/Release/bolt/functions/sparksql/benchmarks/bolt_collect_list_aggregate_bm

Before:

============================================================================
[...]rks/CollectListAggregateBenchmark.cpp     relative  time/iter   iters/s
============================================================================
collect_list_bigint_partial_final_plan                     18.83ms     53.11
collect_list_string_partial_final_plan                     26.87ms     37.21

After:

============================================================================
[...]rks/CollectListAggregateBenchmark.cpp     relative  time/iter   iters/s
============================================================================
collect_list_bigint_partial_final_plan                      3.51ms    285.12
collect_list_string_partial_final_plan                     12.09ms     82.70

Summary:

Benchmark Before After Speedup
collect_list_bigint_partial_final_plan 18.83ms 3.51ms 5.36x
collect_list_string_partial_final_plan 26.87ms 12.09ms 2.22x

This shows a significant improvement for both primitive and string collect_list in a plan shape that includes both partial and final aggregation.

Checklist (For Author)

  • I have added/updated unit tests (ctest).
  • I have verified the code with local build (Release/Debug).
  • I have run clang-format / linters.
  • (Optional) I have run Sanitizers (ASAN/TSAN) locally for complex C++ changes.
  • No need to test or manual test.

Breaking Changes

  • No

  • Yes (Description: ...)

    Click to view Breaking Changes
    Breaking Changes:
    - Description of the breaking change.
    - Possible solutions or workarounds.
    - Any other relevant information.
    

@taiyang-li
taiyang-li force-pushed the collect-list-top1-typed-oss branch from b26fd5b to b05c2e0 Compare July 12, 2026 06:58
@taiyang-li
taiyang-li marked this pull request as ready for review July 12, 2026 08:27
@taiyang-li taiyang-li changed the title [wip]Optimize collect_list for fixed-width types Optimize collect_list for fixed-width and varchar types Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant