feat(pyarrow): describe conversions in PyO3 introspection data - #10492
Open
jonasdedden wants to merge 1 commit into
Open
feat(pyarrow): describe conversions in PyO3 introspection data#10492jonasdedden wants to merge 1 commit into
jonasdedden wants to merge 1 commit into
Conversation
PyO3's `experimental-inspect` feature records the Python type of every conversion so that tools like `maturin generate-stubs` can write a real `.pyi`. `PyArrowType` implements `FromPyObject` and `IntoPyObject` but leaves their `INPUT_TYPE`/`OUTPUT_TYPE` constants at the default, `_typeshed.Incomplete`. Since `PyArrowType` is how bindings exchange every Arrow value with Python, that default erases their whole public API. A binding whose functions take and return Arrow data generates stubs in which every signature is `Incomplete`, which is `Any`, so the stubs verify nothing. Add the hints to `FromPyArrow`, `ToPyArrow` and `IntoPyArrow` and forward them through `PyArrowType`, behind a new `experimental-inspect` feature that just turns on PyO3's. Nothing is compiled unless it is enabled, and the default stays `Incomplete` so an out-of-tree implementor of these traits is unaffected. Input and output hints are separate because they genuinely differ: `Vec<T>` is built from anything iterable but handed back as a `list`. Input hints name the pyarrow class only. Every conversion here also accepts any object implementing the relevant PyCapsule interface method, but that is duck-typed and neither pyarrow nor typeshed defines a protocol to point at, so naming one would emit an import that does not resolve. This is documented on the trait; a binding that wants to advertise the wider protocol can declare its own `Protocol` and carry it on a newtype. `arrow` gets a matching feature that implies `pyarrow`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
arrow-pyarrowoffers no way to generate stubs through PyO3'sexperimental-inspect#10491.Rationale for this change
PyO3's
experimental-inspectfeature records the Python type of every conversion into the built binary, so that tools likematurin generate-stubscan emit a real.pyi. The type comes from theINPUT_TYPE/OUTPUT_TYPEassociated constants onFromPyObject/IntoPyObject.PyArrowTypeimplements both traits but leaves those constants at their default,_typeshed.Incomplete. BecausePyArrowTypeis how bindings exchange every Arrow value with Python, that one default erases the entire public API of any such binding. A function likegenerates
What changes are included in this PR?
experimental-inspectfeature onarrow-pyarrow, which just turns on PyO3's.arrowgets a matching feature that impliespyarrow.FromPyArrow::INPUT_TYPE,ToPyArrow::OUTPUT_TYPEandIntoPyArrow::OUTPUT_TYPE, defaulting to_typeshed.Incompleteso that out-of-tree implementors are unaffected.DataType,Field,Schema,ArrayData(→pyarrow.Array),RecordBatch,Vec<T>,ArrowArrayStreamReader,Box<dyn RecordBatchReader + Send>andTable.PyArrowType'sFromPyObject/IntoPyObjectimpls forward them.Input and output are separate constants because they genuinely differ:
Vec<T>is built from anything iterable but handed back as alist, so it iscollections.abc.Iterable[pyarrow.RecordBatch]in andlist[pyarrow.RecordBatch]out.Are these changes tested?
Yes.
arrow-pyarrowgains unit tests asserting the rendered hint for every conversion, run withcargo test -p arrow-pyarrow --features experimental-inspect.Beyond that, the change was verified end to end against a real binding: with
arrowpatched to this branch and the binding's own code left on plainPyArrowType,maturin generate-stubsproduces the signatures and thefrom pyarrow import ...line shown above.Are there any user-facing changes?
No behaviour changes, and nothing is compiled unless
experimental-inspectis enabled. The traits gain associated constants, but they are defaulted and feature-gated, so existing implementations keep compiling untouched.