Skip to content

feat(pyarrow): describe conversions in PyO3 introspection data - #10492

Open
jonasdedden wants to merge 1 commit into
apache:mainfrom
jonasdedden:pyarrow-introspection-type-hints
Open

feat(pyarrow): describe conversions in PyO3 introspection data#10492
jonasdedden wants to merge 1 commit into
apache:mainfrom
jonasdedden:pyarrow-introspection-type-hints

Conversation

@jonasdedden

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

PyO3's experimental-inspect feature records the Python type of every conversion into the built binary, so that tools like maturin generate-stubs can emit a real .pyi. The type comes from the INPUT_TYPE / OUTPUT_TYPE associated constants on FromPyObject / IntoPyObject.

PyArrowType implements both traits but leaves those constants at their default, _typeshed.Incomplete. Because PyArrowType is how bindings exchange every Arrow value with Python, that one default erases the entire public API of any such binding. A function like

#[pyfunction]
fn cast_record_batch(
    record_batch: PyArrowType<RecordBatch>,
    schema: PyArrowType<Schema>,
) -> PyResult<PyArrowType<RecordBatch>> { ... }

generates

def cast_record_batch(record_batch: Incomplete, schema: Incomplete) -> Incomplete: ...

What changes are included in this PR?

  • A new experimental-inspect feature on arrow-pyarrow, which just turns on PyO3's. arrow gets a matching feature that implies pyarrow.
  • FromPyArrow::INPUT_TYPE, ToPyArrow::OUTPUT_TYPE and IntoPyArrow::OUTPUT_TYPE, defaulting to _typeshed.Incomplete so that out-of-tree implementors are unaffected.
  • Implementations for every type the crate converts: DataType, Field, Schema, ArrayData (→ pyarrow.Array), RecordBatch, Vec<T>, ArrowArrayStreamReader, Box<dyn RecordBatchReader + Send> and Table.
  • PyArrowType's FromPyObject / IntoPyObject impls forward them.
  • Unit tests pinning the rendered hint for each type, and a note in the crate docs.

Input and output are separate constants because they genuinely differ: Vec<T> is built from anything iterable but handed back as a list, so it is collections.abc.Iterable[pyarrow.RecordBatch] in and list[pyarrow.RecordBatch] out.

Are these changes tested?

Yes. arrow-pyarrow gains unit tests asserting the rendered hint for every conversion, run with cargo test -p arrow-pyarrow --features experimental-inspect.

Beyond that, the change was verified end to end against a real binding: with arrow patched to this branch and the binding's own code left on plain PyArrowType, maturin generate-stubs produces the signatures and the from pyarrow import ... line shown above.

Are there any user-facing changes?

No behaviour changes, and nothing is compiled unless experimental-inspect is enabled. The traits gain associated constants, but they are defaulted and feature-gated, so existing implementations keep compiling untouched.

PyO3's `experimental-inspect` feature records the Python type of every
conversion so that tools like `maturin generate-stubs` can write a real `.pyi`.
`PyArrowType` implements `FromPyObject` and `IntoPyObject` but leaves their
`INPUT_TYPE`/`OUTPUT_TYPE` constants at the default, `_typeshed.Incomplete`.

Since `PyArrowType` is how bindings exchange every Arrow value with Python, that
default erases their whole public API. A binding whose functions take and return
Arrow data generates stubs in which every signature is `Incomplete`, which is
`Any`, so the stubs verify nothing.

Add the hints to `FromPyArrow`, `ToPyArrow` and `IntoPyArrow` and forward them
through `PyArrowType`, behind a new `experimental-inspect` feature that just
turns on PyO3's. Nothing is compiled unless it is enabled, and the default stays
`Incomplete` so an out-of-tree implementor of these traits is unaffected.

Input and output hints are separate because they genuinely differ: `Vec<T>` is
built from anything iterable but handed back as a `list`. Input hints name the
pyarrow class only. Every conversion here also accepts any object implementing
the relevant PyCapsule interface method, but that is duck-typed and neither
pyarrow nor typeshed defines a protocol to point at, so naming one would emit an
import that does not resolve. This is documented on the trait; a binding that
wants to advertise the wider protocol can declare its own `Protocol` and carry
it on a newtype.

`arrow` gets a matching feature that implies `pyarrow`.
@github-actions github-actions Bot added the arrow Changes to the arrow crate label Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arrow-pyarrow offers no way to generate stubs through PyO3's experimental-inspect

1 participant