GH-50591: [Python] Fix reference in ConvertToSequenceAndInferSize when an iterator raises on array conversion

[raulcd]

Rationale for this change

We currently leak the list and items in the case of an iterator raising an Exception and size is present.

What changes are included in this PR?

Use OwnedRef instead of relying on manual Py_DECREF in order to avoid exception scenario returning early in RETURN_IF_PYERROR.

Are these changes tested?

Yes, I've created a new test that fails previously to the fix and passes after the fix

Are there any user-facing changes?

No

• GitHub Issue: [Python] ConvertToSequenceAndInferSize leaks a reference to a list and the items when iterator raises an Exception #50591

[github-actions]

⚠️ GitHub issue #50591 has been automatically assigned in GitHub to PR creator.

[raulcd]

@github-actions crossbow submit -g python

[github-actions]

Revision: afc8c10

Submitted crossbow builds: ursacomputing/crossbow @ actions-1dcc69b782

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hdfs-2.9.2
test-conda-python-3.11-hdfs-3.2.1
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-2.0.3-numpy-1.23.2
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-latest-numpy-1.26.2
test-conda-python-3.13-pandas-latest-numpy-latest
test-conda-python-3.14
test-conda-python-3.14-cpython-debug
test-conda-python-3.14-pandas-nightly-numpy-nightly
test-conda-python-3.14-pandas-upstream_devel-numpy-nightly
test-conda-python-emscripten
test-debian-13-python-3-amd64
test-debian-13-python-3-i386
test-fedora-42-python-3
test-ubuntu-24.04-python-3

[Copilot]

Pull request overview

Fixes a Python reference leak in ConvertToSequenceAndInferSize when converting a sized iterator that raises during iteration (GH-50591). The change improves exception-safety in the C++/Python bridge by using RAII (OwnedRef) so early returns on Python exceptions don’t skip required decrefs.

Changes:

• Wrap the intermediate PyList_New(n) result in an OwnedRef and detach() it only on success to avoid leaking the list/items on exception paths.
• Remove manual Py_DECREF(lst) in the slice-shrinking error path since RAII now handles cleanup.
• Add a regression test that asserts refcounts don’t change when an iterator raises during pa.array(..., size=...).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
python/pyarrow/src/arrow/python/python_to_arrow.cc	Make sized-iterator conversion exception-safe by managing the temporary list with OwnedRef and detaching only on success.
python/pyarrow/tests/test_convert_builtin.py	Add a regression test ensuring no reference leak when a raising iterator is converted with size specified.

[raulcd]

@pitrou I think this is another reference leak

[raulcd]

btw the leak was here

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 826ac37.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.