GH-37476: [C++][Python] Preserve unsigned dictionary index types when building from values by mkzung · Pull Request #50475 · apache/arrow

mkzung · 2026-07-11T15:17:27Z

Rationale for this change

An unsigned dictionary index type is silently replaced by the signed one of the same width:

>>> pa.array(["a", "b"], type=pa.dictionary(pa.uint32(), pa.string())).type
dictionary<values=string, indices=int32, ordered=0>

It also makes pa.chunked_array(values, dict_type) fail with Array chunks must all be same type, since the chunk built internally comes back a different type than requested.

DictionaryBuilderCase::CreateFor() reduces the requested index type to its byte width and hands that to AdaptiveIntBuilder, so everything but the width is dropped and the type is rebuilt from what the indices builder reports, which is always signed.

@jorisvandenbossche suggested AdaptiveUIntBuilder for unsigned types. That returns a different builder class, which util/converter.h, json/from_string.cc and the R binding all cast to DictionaryBuilder<T>, and TestDictionaryUnifier.ChunkedArrayNestedDict hits a checked_pointer_cast DCHECK. So this keeps one builder class and just reports the requested signedness. Indices are non-negative and same-width signed and unsigned types have the same layout, so it is value-preserving and free, and the width stays adaptive.

What changes are included in this PR?

DictionaryBuilderBase and its NullType specialization record the requested signedness and map the signed index type to the unsigned one of the same width in type(), FinishInternal() and FinishDelta(); CreateFor() passes it through. The mapping helper lives in builder.cc, not the header. util/converter.h, python_to_arrow.cc and r_to_arrow.cpp are untouched, so R is fixed rather than broken.

uint64 indices also convert to pandas now, mapped to int64 codes. That is safe because the indices are bounds-checked below the dictionary length, so they never reach the int64 limit; WriteIndicesUniform handles uint64 like it already handles uint32.

Are these changes tested?

Six tests in array_dict_test.cc cover the distinct paths: type and finish preservation across signed and unsigned, the width adapting while staying unsigned, the NullType builder, FinishDelta, the supplied-dictionary constructor and the exact-index builder. Three parametrized tests in test_array.py cover the eight index types, the width adapting and the uint64-to-pandas conversion, and test_dictionary_with_pandas now checks that conversion instead of the old error.

A uint8 grows to uint16 once more than 128 distinct values are added and stays unsigned; the underlying builder is signed, so it widens after 128 rather than the 256 a real uint8 could hold. arrow-array-test, arrow-ipc-read-write-test, arrow-compute-scalar-cast-test, arrow-c-bridge-test and the pyarrow test_array and test_pandas suites pass.

Are there any user-facing changes?

pa.array and pa.chunked_array return the requested unsigned index type. dictionary(uint64(), ...) produces real uint64 indices, and converting such an array to pandas now works where it used to raise. take, filter, cast, dictionary_decode, IPC and scalar access all keep the index type.

GitHub Issue: [Python] pyarrow.array silently rewrites dictionary types to use signed integers for indexes #37476

…s when building from values

mkzung · 2026-07-21T02:16:46Z

Friendly ping on this one. The workflow runs look like they're still waiting on
approval, so none of the C++ or Python jobs have actually executed here yet.

@jorisvandenbossche you diagnosed this back in 2023 and suggested letting it pick
AdaptiveIntBuilder vs AdaptiveUIntBuilder depending on the signedness of the original
index type, so you may want to see why I didn't go that way. It changes which class
MakeDictionaryBuilder returns, and util/converter.h, json/from_string.cc and the R
binding all cast that result to DictionaryBuilder.
TestDictionaryUnifier.ChunkedArrayNestedDict trips a checked_pointer_cast DCHECK
straight away. This keeps one builder class instead and preserves the requested
signedness where the index type is reported.

Happy to rebase if it's gone stale.

pitrou · 2026-07-21T15:10:08Z

+  }
+}
+
+TEST(TestDictionaryBuilderIndexType, TypePreservesSignedIndexType) {


You could fold this test into the above, no need for duplicate code here IMHO.

pitrou · 2026-07-21T15:11:29Z

+    std::shared_ptr<Array> result;
+    ASSERT_OK(builder.Finish(&result));


Can use the Result-returning Finish version:

Suggested change

std::shared_ptr<Array> result;

ASSERT_OK(builder.Finish(&result));

ASSERT_OK_AND_ASSIGN(auto result, builder.Finish());

pitrou · 2026-07-21T15:12:29Z

+  auto& builder = checked_cast<DictionaryBuilder<StringType>&>(*boxed_builder);
+
+  // More distinct values than a uint8 index start width accommodates.
+  for (int i = 0; i < 200; ++i) {


Hmm, is that right? A uint8 index should be able to represent 256 distinct dictionary entries (or 255 depending on how it's coded internally :-)).

pitrou · 2026-07-21T15:14:38Z

+TEST(TestDictionaryBuilderIndexType, SuppliedDictionaryPreservesUnsignedIndexType) {
+  // The supplied-dictionary constructor starts the adaptive indices builder at its
+  // default width rather than at the requested one, so the requested *width* is not
+  // honoured on this path. That predates this change: a requested int32 reports int8


"this change" will not be understandable to a future reader of this comment, can you reword it to be less context-dependent?

Also, if we already have a test for this on the signed index side, I wonder what it brings to add this new test.

AIs tend to produce very verbose testing, which gives the illusion of better quality, but also makes the code less maintainable in the long term (why all those very similar test functions)?

pitrou · 2026-07-21T15:16:30Z

+                  *checked_cast<const DictionaryType&>(*unsigned_type).index_type());
+
+  ASSERT_OK_AND_ASSIGN(auto signed_builder,
+                       MakeDictionaryBuilder(dictionary(int32(), utf8()), dict_values));


So we're also testing for a signed index type here? Wasn't it already tested? If not, at least the test name and comment should be fixed.

pitrou · 2026-07-21T15:20:44Z

+/// width have identical memory layout and reporting one as the other is
+/// value-preserving (GH-37476). The width itself stays adaptive, exactly as it is
+/// for signed index types; only the signedness of the requested type is preserved.
+inline std::shared_ptr<DataType> MaybeUnsignedIndexType(


I don't think this has to be inlined in this header file, as this function shouldn't be performance-critical.

pitrou · 2026-07-21T15:21:20Z

                        MemoryPool* pool = default_memory_pool(),
-                        int64_t alignment = kDefaultBufferAlignment, bool ordered = false)
+                        int64_t alignment = kDefaultBufferAlignment, bool ordered = false,
+                        bool unsigned_index = false)


Can we call this use_unsigned_index?

pitrou · 2026-07-21T15:22:11Z

+
+@pytest.mark.parametrize("index_type", [pa.int8(), pa.int16(),
+                                        pa.int32(), pa.int64()])
+def test_dictionary_array_signed_index_type_unchanged(index_type):


Can you fold this into the above test?

pitrou · 2026-07-21T15:22:40Z

+def test_dictionary_array_index_width_still_adapts():
+    # The index width remains adaptive, as it is for signed index types; only
+    # the signedness of the requested index type is preserved.


Same comment wrt comments being too much context-dependent ("still adapts", "remains adaptive").

Also, can we check both signed and unsigned here?

pitrou · 2026-07-21T15:24:57Z

+    # silently downgraded to int64. Arrow deliberately does not support converting
+    # them to pandas (pandas categorical codes are signed), so that pre-existing
+    # limitation is now reachable, where the silent downgrade used to mask it.


Hmm, that sounds like a potential regression for third-party code. Can we support converting uint64 indices to Pandas?

You can take a look at CategoricalWriter in arrow_to_pandas.cc. It seems that WriteIndicesUniform is the main place that needs to be changed.

Consolidate the dictionary index-type tests down to the distinct code paths and reword the context-dependent comments. Move MaybeUnsignedIndexType out of the header into builder.cc and rename the flag to use_unsigned_index. Support converting uint64 dictionary indices to pandas: they map to int64 categorical codes, which is safe because the indices are bounds-checked below the dictionary length. This removes the previous TypeError.

mkzung · 2026-07-24T15:10:58Z

Thanks for the review. I folded the index-type tests down to the distinct code paths - type and finish preservation now run in one loop over signed and unsigned, and I dropped the ordered and fixed-size-binary ones since they don't exercise a separate path. Switched to ASSERT_OK_AND_ASSIGN and reworded the context-dependent comments. Moved MaybeUnsignedIndexType into builder.cc and renamed the flag to use_unsigned_index.

On uint64 to pandas: I added support for it instead of keeping the error. uint64 indices map to int64 codes, which is safe because the indices are bounds-checked below the dictionary length, so they never reach the int64 limit. WriteIndicesUniform handles it the same way as uint32, and I updated test_dictionary_with_pandas to check the conversion instead of the raise.

On the missing top bit: you're right, the adaptive builder is signed so a uint8 index widens after 128 rather than 256. I left that as-is and noted it in the builder.cc comment - happy to change the builder to track the unsigned range if you'd prefer.

pitrou

Thanks for the update, a couple minor additional suggestions

pitrou · 2026-07-27T15:23:19Z

+    case Type::INT64:
+      return ::arrow::uint64();
+    default:
+      return index_type;


Can we use arrow::Unreachable here? Or can the index_type argument actually be unsigned?

- MaybeUnsignedIndexType: the default switch case is unreachable (the adaptive index builder only produces signed int8/16/32/64), so mark it with arrow::Unreachable instead of returning the argument - SuppliedDictionary/ExactIndexBuilder index-type tests: build the array and assert the finished array's index type, not just the builder's reported type - rename WidthStillAdaptsWhenUnsigned -> WidthAdaptsWhenUnsigned (drop the context-dependent "Still")

mkzung · 2026-07-28T15:37:23Z

Thanks, done:

MaybeUnsignedIndexType: the default case is unreachable (the adaptive builder only produces
signed int8/16/32/64), so it's arrow::Unreachable now.
Both tests now build the array and check the finished array's index type, not just the
builder's.
Renamed to WidthAdaptsWhenUnsigned.

The uint8-widens-at-128 bit is still a documented limitation; happy to do the unsigned-range
follow-up separately if you want it. The red R/Windows checks look like infra, not this change
(bioconductor was down for R, MinGW timed out in the deps).

pitrou · 2026-07-28T16:11:23Z

The uint8-widens-at-128 bit is still a documented limitation; happy to do the unsigned-range
follow-up separately if you want it.

It's probably desirable though not critical, so feel free to work on it if you'd like to.

The uint8-widens-at-128 bit is still a documented limitation; happy to do the unsigned-range follow-up separately if you want it. The red R/Windows checks look like infra, not this change (bioconductor was down for R, MinGW timed out in the deps).

Probably, let's wait for the updated CI results.

pitrou · 2026-07-28T17:01:36Z

The remaining CI failures are unrelated, I'll merge.

conbench-apache-arrow · 2026-07-30T09:56:53Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit bda98c4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 20 possible false positives for unstable benchmarks that are known to sometimes produce them.

apacheGH-37476: [C++][Python] Preserve unsigned dictionary index type…

25e49e7

…s when building from values

mkzung requested review from AlenkaF, pitrou, raulcd and rok as code owners July 11, 2026 15:17

github-actions Bot added Component: C++ Component: Python awaiting review Awaiting review labels Jul 11, 2026

mkzung mentioned this pull request Jul 12, 2026

[Bug]: constructing a Categorical column from Python values raises on the PyArrow backend narwhals-dev/narwhals#3775

Open

1 task

pitrou reviewed Jul 21, 2026

View reviewed changes

pitrou reviewed Jul 27, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jul 27, 2026

pitrou approved these changes Jul 28, 2026

View reviewed changes

pitrou merged commit bda98c4 into apache:main Jul 28, 2026
59 of 64 checks passed

pitrou removed the awaiting committer review Awaiting committer review label Jul 28, 2026

pitrou mentioned this pull request Jul 28, 2026

[Python] pyarrow.array silently rewrites dictionary types to use signed integers for indexes #37476

Closed

This was referenced Jul 28, 2026

[C++] Unsigned dictionary index types should use the full unsigned range #50695

Open

GH-50695: [C++] Use the full unsigned range for unsigned dictionary index types #50696

Open

		std::shared_ptr<Array> result;
		ASSERT_OK(builder.Finish(&result));

	std::shared_ptr<Array> result;
	ASSERT_OK(builder.Finish(&result));
	ASSERT_OK_AND_ASSIGN(auto result, builder.Finish());

Uh oh!

Conversation

mkzung commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

mkzung commented Jul 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkzung commented Jul 24, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mkzung commented Jul 28, 2026

Uh oh!

pitrou commented Jul 28, 2026

Uh oh!

pitrou commented Jul 28, 2026

Uh oh!

Uh oh!

conbench-apache-arrow Bot commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mkzung commented Jul 11, 2026 •

edited

Loading