Skip to content

GH-35815: [C++][Compute] Support dictionary inputs in UTF-8 trim - #50501

Open
LarryHu0217 wants to merge 4 commits into
apache:mainfrom
LarryHu0217:codex/arrow-dictionary-utf8-trim-35815
Open

GH-35815: [C++][Compute] Support dictionary inputs in UTF-8 trim#50501
LarryHu0217 wants to merge 4 commits into
apache:mainfrom
LarryHu0217:codex/arrow-dictionary-utf8-trim-35815

Conversation

@LarryHu0217

@LarryHu0217 LarryHu0217 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Pure unary scalar functions produce the same result for repeated input values. For dictionary-encoded input, running the function against every logical row is therefore unnecessary: the function can run once against the dictionary values and the original indices can expand the result.

What changes are included in this PR?

  • Add a generic dictionary adapter for fixed-arity, pure unary ScalarFunctions.
  • Dispatch the underlying function against the dictionary value type, execute it on the dictionary values, then expand the transformed values with the original indices.
  • Only advertise the adapter when the dictionary value type has a matching underlying kernel; impure unary functions remain unsupported.
  • Add generic coverage for type-changing output, unique-value execution, unsupported value types, and impure functions.
  • Keep issue-specific regression coverage for utf8_trim, utf8_ltrim, and utf8_rtrim with dictionary-encoded string and large-string inputs, including nulls.

Are these changes tested?

  • arrow-compute-internals-test --gtest_brief=1 (73 passed)
  • arrow-compute-scalar-type-test --gtest_brief=1 (344 passed, 1 pre-existing disabled test)
  • C++ Debug build with warnings treated as errors
  • clang-format --dry-run --Werror on all touched C++ files

Are there any user-facing changes?

Yes. Pure unary scalar functions can now accept dictionary-encoded inputs whenever their dictionary value type has a matching kernel. Results retain the function's existing decoded output type, so no public API signature changes are introduced.

AI-assisted development

OpenAI Codex assisted with reproducing the issue, implementing the patch, and drafting the regression tests. The change was checked against Arrow's scalar execution and dictionary dispatch paths and validated with the commands above.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #35815 has been automatically assigned in GitHub to PR creator.

@github-actions github-actions Bot added awaiting review Awaiting review Component: C++ and removed awaiting review Awaiting review labels Jul 13, 2026
@LarryHu0217
LarryHu0217 marked this pull request as ready for review July 16, 2026 01:15
@LarryHu0217
LarryHu0217 requested a review from pitrou as a code owner July 16, 2026 01:15
@github-actions github-actions Bot added the awaiting review Awaiting review label Jul 16, 2026
@pitrou

pitrou commented Jul 22, 2026

Copy link
Copy Markdown
Member
  • Decode dictionary input types during dispatch for utf8_trim, utf8_ltrim, and utf8_rtrim.

This is wasteful, isn't it? The scalar function could just be applied to the dictionary values.

Also I don't understand why we would only tackle these three specific functions, while this could apply to all scalar unary functions.

cc @zanmato1984 for opinions.

@LarryHu0217

Copy link
Copy Markdown
Contributor Author

Thanks, addressed both points in d2f3763. The trim-specific dispatch subclass is gone. The new adapter applies generically to fixed-arity pure unary ScalarFunctions when the dictionary value type has a matching kernel, executes the function against the dictionary values, and expands the result with the original indices while retaining the existing decoded output type. I added generic tests for type-changing output, unique-value execution, unsupported value types, and impure functions, plus the trim regressions. Local results: 73/73 compute-internals tests and 344/344 scalar-type tests passed.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #35815 has been automatically assigned in GitHub to PR creator.

@pitrou

pitrou commented Jul 27, 2026

Copy link
Copy Markdown
Member

@zanmato1984 What do you think about this? I don't know if this should be handled at the function execution level like in this PR, or somewhere else.

@LarryHu0217

Copy link
Copy Markdown
Contributor Author

The failed C++ jobs exposed that registering a synthetic dictionary kernel changed the public dispatch contract for every pure unary ScalarFunction, which also affected cast and expression behavior. I moved the generic support to the eager ScalarFunction::Execute path in fdfa405.

Existing dictionary kernels still take precedence. When no dictionary kernel matches, a fixed-arity pure unary function is applied to the dictionary values and expanded through the original indices. The fallback covers array, scalar, chunked-array, and empty-chunked inputs while leaving DispatchBest/DispatchExact unchanged.

This restores the previously failing TestUnaryArithmetic.DispatchBest, TestUnaryRound.DispatchBestRound, and Cast.ExtensionDictToExtension tests. Local validation passed: compute internals 73/73, scalar cast 108/108, scalar type/string 255/255, scalar math 526 passed with 26 existing half-float skips, and expression 47/47.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants