GH-50429: [Python] Convert maps to dicts without per-element Scalars in to_pylist

[viirya]

Rationale for this change

Follow-up of #50326, building on the merged #50327.

GH-50327 converts arrays to Python objects without per-element Scalars, but the maps_as_pydicts option still routes to the Scalar-based path: every map row allocates a MapScalar, converts its keys via per-element as_py, and builds the dict in Python. Map→dict is the natural consumption pattern for engines whose map values are Python dicts (e.g. Spark's Arrow-serialized Python UDFs currently receive association lists and rebuild a dict per row in pure Python — one of the dominant remaining costs in that path).

What changes are included in this PR?

Thread maps_as_pydicts through the scalar-free _getitem_py mechanism:

• MapArray._getitem_py converts the row's keys first (as MapScalar.as_py does via keys()), then builds the dict with a per-key loop that checks each key immediately before converting its value — so the 'lossy' warnings, the 'strict' KeyError, and the TypeError for unhashable keys are raised at exactly the same points as MapScalar.as_py (including messages and warning-per-duplicate behavior). StructArray._getitem_py also reproduces StructScalar.as_py's translation of nested KeyErrors into the duplicate-field-names ValueError.
• Invalid option values raise the same ValueError when a map value is converted — including null map rows, matching the Scalar path — while non-map arrays keep ignoring the option.
• The option propagates through nested types (list/struct children, map values) as before, and unspecialized types keep the exact Scalar fallback, which now receives the option.

Benchmark (macOS arm64, M4 Max; 1M rows of 2-entry map<string,int64>, 10% nulls):

conversion	before	after	speedup
to_pylist(maps_as_pydicts='lossy')	2.45 s	0.13 s	~19x

Notably the dict form is now also faster than the default association-list form (0.49 s), which allocates a 2-tuple per entry.

Are these changes tested?

New test_to_pylist_maps_as_pydicts compares against the per-scalar conversion for flat maps, list<map>, map<string, map<...>> and struct<map> (plain and sliced) in both modes, and asserts the duplicate-key semantics ('lossy' warns and keeps the last value; 'strict' raises KeyError), the exact warning sequence when a raising value follows a duplicate key, the unhashable-key TypeError parity (struct-keyed maps), the nested KeyError→ValueError translation inside structs, the invalid-value ValueError (including on null map rows), and that non-map arrays ignore the option. Randomized differential tests against the Scalar path (exact type equality) and pytest test_array.py test_scalars.py test_convert_builtin.py test_table.py (1210 passed) also pass.

Are there any user-facing changes?

No behavior changes, only performance.

• GitHub Issue: [Python] Convert maps to dicts without per-element Scalars in to_pylist(maps_as_pydicts=...) #50429

This pull request and its description were written by Isaac.

[github-actions]

⚠️ GitHub issue #50429 has been automatically assigned in GitHub to PR creator.

[Copilot]

Pull request overview

This PR extends the scalar-free Array.to_pylist() conversion path to support maps_as_pydicts for MapArray, enabling direct Map→dict materialization without per-element Scalar allocation while preserving existing MapScalar.as_py semantics.

Changes:

• Add a cdef _getitem_py(i, maps_as_pydicts) mechanism and update Array.to_pylist() to use it for scalar-free conversions.
• Implement _getitem_py specializations for common leaf and nested array types (numeric, boolean, string/binary, list, struct, map), with caching of wrapped child arrays.
• Add tests comparing to_pylist() results against per-scalar conversion, including maps_as_pydicts behaviors and nested compositions.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
python/pyarrow/array.pxi	Implements _getitem_py and adds scalar-free to_pylist specializations, including MapArray dict conversion for maps_as_pydicts.
python/pyarrow/lib.pxd	Declares new Array Cython API surface (_children_cache, _getitem_py).
python/pyarrow/tests/test_array.py	Adds regression/differential tests for scalar-free to_pylist and maps_as_pydicts semantics.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[rok]

Nice work — the parity coverage here is careful (the poison-value-after-duplicate-key ordering test in particular).

One small test gap: LargeListArray._getitem_py and FixedSizeListArray._getitem_py now thread maps_as_pydicts through to their values, but the parity matrix in test_to_pylist_maps_as_pydicts only covers list_(map). Two more entries would exercise those overrides:

     arrays = [
         flat,
         flat.slice(1),
         pa.array([[[('k', 1)], None], None], type=pa.list_(map_type)),
+        pa.array([[[('k', 1)], None], None], type=pa.large_list(map_type)),
+        pa.array([[[('k', 1)], None], None], type=pa.list_(map_type, 2)),
         pa.array([[("o", [("i", 5)])]],
                  type=pa.map_(pa.string(), map_type)),
         pa.array([{"m": [("k", 1)]}, None],
                  type=pa.struct([("m", map_type)])),
     ]

The same data works for all three (each non-null outer element has exactly 2 children, so it's valid for the fixed-size list too), and it mirrors how test_to_pylist_bulk_paths covers large_list/list_(..., 2) alongside list_. The duplicate-key/strict/lossy sections don't need changes — those semantics live in MapArray._getitem_py, which is already covered; this just pins that the two list variants keep propagating the option.

[viirya]

Added in a650d5a — both variants with the same data and literal expected rows. Thanks for the careful review!

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[rok]

@viirya can you mark comments that were addressed as resolved? Especially the copilot ones are noisy to read.

@pitrou I've done a first pass here, could you take a look too?

[viirya]

Thank you for review @rok. I marked the addressed comments as resolved.

[rok]

@pitrou could you take a look at this please?

[pitrou]

@pitrou I've done a first pass here, could you take a look too?

You or your agent?

[rok]

@pitrou I've done a first pass here, could you take a look too?

You or your agent?

both! :)

[raulcd]

@viirya , is this the second blocker for the possible patch release that was mentioned on the Arrow 25.0.0 ML thread? I haven't been following the development of the issues causing spark performance regressions but I haven't seen any follow up on the ML. What would be your current expectations?

[pitrou]

@raulcd I don't think this is blocking anything.

[pitrou]

Why not factor out this check?

[viirya]

Done in 93b8043 — factored into _check_maps_as_pydicts(), now shared by MapScalar.as_py and MapArray._getitem_py.

[pitrou]

This looks a bit tedious. I think you could reuse the "association list" above, build a dict from it and check that the dict length is equal to the list length.

[viirya]

I simplified it to a single interleaved loop in 93b8043 (no more separate keys pass), which I think addresses the tedium. I didn't go through the association list though: its per-entry 2-tuples are exactly what the direct dict build avoids, and it measures accordingly on 1M rows of 2-entry maps — 0.12 s for the direct dict vs 0.48 s for the association list (≈0.56 s with a dict built on top). The duplicate handling (warn/raise) still needs the explicit loop either way.

[pitrou]

Why are we testing for this? I don't think this is part of the contract.

[viirya]

It pins that duplicate detection happens before converting that entry's value, matching MapScalar.as_py — the first implementation got this wrong (values converted before duplicate detection) and review caught it, so the test guards the regression. The exact-sequence form was requested in an earlier review pass; if you don't consider the ordering part of the contract I'm happy to reduce it to a plain pytest.warns.

[pitrou]

Why test for this?

[viirya]

It pins that the rework doesn't newly reject previously-accepted calls: the Scalar path only validates the option when a map value is actually converted, so to_pylist(maps_as_pydicts="bogus") on a non-map array succeeds today, and eager validation would be a (small) behavior change. Can drop it if you don't think it's worth pinning.

[pitrou]

@rok I'd appreciate if you did actual human code reviews instead of letting your agent post for you.

[viirya]

@raulcd To confirm what Antoine said: this is not a blocker. #50327 (already merged) is the fix Spark needs — it covers the native to_pylist() fast paths for all nested types, including maps' association-list form. This PR speeds up the optional maps_as_pydicts dict form, which Spark doesn't currently use, so it can ride 26.0.0 if it doesn't make the patch release. From Spark's perspective, a 25.0.1 containing #50327 is everything we need.

[Copilot]

🟡 Not ready to approve

Map dict-mode key conversion order in MapArray._getitem_py can still differ from MapScalar.as_py (keys converted lazily vs all upfront), potentially changing exception/warning ordering in key-conversion-failure cases.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (1)

python/pyarrow/array.pxi:3647

• MapArray._getitem_py (dict modes) converts keys lazily inside the per-entry loop, whereas MapScalar.as_py converts all keys first via keys() before doing any duplicate detection / warnings. This can change exception ordering if a later key conversion raises (e.g. invalid UTF-8 in a string key): MapScalar.as_py would raise during key conversion before any duplicate-key KeyError/warning, but the array path can raise/warn on an earlier duplicate key first. To match MapScalar.as_py more closely, pre-convert the full key list before the duplicate-check/value-conversion loop.

        for j in range(start, end):
            key = keys._getitem_py(j, None)
            if key in result:
                if maps_as_pydicts == "strict":
                    raise KeyError(

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[viirya]

Note on the Copilot assessment above: the ordering difference it describes (lazy vs upfront key conversion) is real but deliberate — the keys-first pass existed precisely to mirror MapScalar.as_py's ordering and was removed in 93b8043 following the review discussion that the exact exception/warning interleaving isn't part of the contract. It's only observable when a later key's own conversion raises (e.g. invalid UTF-8 in a string key). Happy to restore the keys-first pass if we do want that ordering pinned.