GH-49103: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) by rok · Pull Request #48622 · apache/arrow

rok · 2025-12-22T19:53:04Z

Rationale for this change

This is the second in series of PRs adding type annotations to pyarrow and resolving #32609. It builds on top of and should be merged after #48618.

What changes are included in this PR?

This adds:

_types.pyi - Core type definitions including
_stubs_typing.pyi - Internal typing protocols and helpers used across stub files
error.pyi - Exception classes (ArrowException, ArrowInvalid, ArrowIOError, etc.)
Minimal placeholder stubs - lib.pyi, io.pyi, scalar.pyi - using __getattr__ to allow imports to resolve while deferring to subsequent PRs

Are these changes tested?

Via CI type checks established in #48618.

Are there any user-facing changes?

Users will start seeing some minimal annotated types.

GitHub Issue: [Python] Type checking support #32609
GitHub Issue: [Python][Annotations] Add internal type system stubs (_types, error, _stubs_typing) #49103

rok · 2026-01-26T13:31:15Z

I've rebased this on the annotation infra check PR (#48618) to make sure we're on the right track.

github-actions · 2026-01-31T15:25:45Z

⚠️ GitHub issue #49103 has been automatically assigned in GitHub to PR creator.

Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

rok · 2026-06-14T19:17:46Z

Finally took some time to work on this again. @dangotbanned I accepted you proposals and added some other changes. Could you please take another look and maybe we can merge?

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

rok · 2026-07-16T09:58:21Z

@paddyroddy 👋

paddyroddy

Had a scan thought. A lot of the stubs went over my head as I don't know the code base well enough, and hard to see them without the context of the function there too. Have some thoughts anyway.

Nice to see this coming along!

paddyroddy · 2026-07-23T13:35:57Z

 # specific language governing permissions and limitations
 # under the License.

-"""Type stubs for PyArrow.


Just noting this file was added in #48618

paddyroddy · 2026-07-23T13:38:19Z

@@ -136,11 +136,22 @@ repos:
      - id: flake8


Out of scope, but would suggest we move to ruff at some point. Faster, more frequent updates, and nicer maintainer.

paddyroddy · 2026-07-23T13:40:40Z

        # handle fsspec-compatible filesystems
        try:
-            import fsspec
+            import fsspec  # type: ignore[import-not-found,import-untyped]


Looks like #48618 introduced mypy, pyright, and ty. What about https://github.com/facebook/pyrefly?

I haven't checked, do type: ignore statement codes work across these different type checkers. I've recently switched from mypy to ty and noticed some codes were different. But I have been using ty: ignore rather than type: ignore statements, which might explain things.

paddyroddy · 2026-07-23T13:47:24Z

+IntoArray = TypeAliasType(
+    "IntoArray",


Is this "int to array" or "into array"?

paddyroddy · 2026-07-23T13:52:56Z

+from typing import Any, Generic, Literal, Protocol, TypeAlias, overload
+
+import numpy as np
+import pandas as pd  # type: ignore[import-not-found,import-untyped]  # pyright: ignore[reportMissingImports,reportMissingTypeStubs]  # noqa: E501


Ah, so this is what I asked about elsewhere. So we're having to do different ignores per library? I've noticed that ty works with both type: ignore and ty: ignore.

paddyroddy · 2026-07-23T14:05:02Z

+def fixed_shape_tensor(
+    value_type: _ValueT,
+    shape: Sequence[int],
+    dim_names: Sequence[str] | None = None,


Went to some talks at EuroPython about the new sentinel built-in. At some point we might want to consider, e.g. sentinel("MISSING"), see this Reddit example.

paddyroddy · 2026-07-23T14:06:05Z

+def type_for_alias(name: str) -> DataType: ...
+def schema(
+    fields: SupportsArrowSchema | _SchemaFieldsInput,
+    metadata: _KeyValueMetadataInput | KeyValueMetadata = None,


Only just spotted this now, but should we also have | None?

Suggested change

metadata: _KeyValueMetadataInput | KeyValueMetadata = None,

metadata: _KeyValueMetadataInput | KeyValueMetadata | None = None,

paddyroddy · 2026-07-23T14:06:55Z

+) -> Schema: ...
+def from_numpy_dtype(dtype: np.dtype[Any] | type | str) -> DataType: ...
+
+__all__ = [


These are usually sorted and are done automatically for me. Looks like that is coming from RUF022.

github-actions Bot added Component: Python Component: Documentation awaiting committer review Awaiting committer review labels Dec 22, 2025

rok mentioned this pull request Dec 22, 2025

GH-32609: [Python] Add type annotations to PyArrow #47609

Draft

dangotbanned reviewed Dec 22, 2025

View reviewed changes

Comment thread python/pyarrow-stubs/pyarrow/_stubs_typing.pyi Outdated

rok force-pushed the pyarrow-stubs-pr2-core-types branch from d3c5740 to 27d1c65 Compare January 26, 2026 12:57

rok force-pushed the pyarrow-stubs-pr2-core-types branch 2 times, most recently from 3f9ed3b to 0ac95b0 Compare January 26, 2026 19:11

rok mentioned this pull request Jan 27, 2026

GH-49102: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

Merged

rok changed the title ~~GH-32609: [Python] Add internal type system stubs (_types, error, _stubs_typing)~~ GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) Jan 31, 2026

rok force-pushed the pyarrow-stubs-pr2-core-types branch 3 times, most recently from 7873930 to 43e7cc6 Compare February 9, 2026 19:11

github-actions Bot removed the Component: Documentation label Feb 9, 2026

rok marked this pull request as ready for review February 9, 2026 20:09

rok requested review from AlenkaF, assignUser, jonkeane, kou and raulcd as code owners February 9, 2026 20:09

rok force-pushed the pyarrow-stubs-pr2-core-types branch from 51a7a4e to 7bc0a98 Compare February 13, 2026 15:24

rok force-pushed the pyarrow-stubs-pr2-core-types branch 5 times, most recently from 0d15871 to 8f3796d Compare March 9, 2026 23:51

rok requested a review from dangotbanned March 10, 2026 00:01

rok force-pushed the pyarrow-stubs-pr2-core-types branch from 8f3796d to 72571d2 Compare March 10, 2026 13:39

rok and others added 12 commits June 14, 2026 10:06

Apply suggestions from code review

8e55077

Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>

review feedback

eae20d1

Decimals should be covariant

789934a

fix editable install collisions

7729137

review feedback

ddd48c0

move placeholder

6f2a03a

post rebase

ed2c72b

Apply suggestions from code review

f0272fe

Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>

Update python/pyarrow-stubs/pyarrow/_types.pyi

50077b6

Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>

update

94fcdf4

lint

9701c2f

more covariant types

8982d1e

Copilot AI review requested due to automatic review settings June 14, 2026 17:45

rok force-pushed the pyarrow-stubs-pr2-core-types branch from 99f5809 to 8982d1e Compare June 14, 2026 17:45

Copilot started reviewing on behalf of rok June 14, 2026 17:45 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread python/pyarrow/fs.py Outdated

Comment thread python/pyarrow-stubs/pyarrow/_types.pyi Outdated

Comment thread python/pyarrow-stubs/pyarrow/io.pyi

Comment thread python/pyarrow-stubs/pyarrow/scalar.pyi

Apply suggestions from code review

604b09b

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 14, 2026 19:15

Copilot started reviewing on behalf of rok June 14, 2026 19:15 View session

rok requested a review from dangotbanned June 14, 2026 19:15

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread python/pyarrow-stubs/pyarrow/error.pyi

Comment thread python/pyarrow-stubs/pyarrow/_types.pyi

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jun 14, 2026

lint

3e9e864

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 14, 2026

rok removed the request for review from dangotbanned July 16, 2026 09:57

paddyroddy reviewed Jul 23, 2026

View reviewed changes

	metadata: _KeyValueMetadataInput \| KeyValueMetadata = None,
	metadata: _KeyValueMetadataInput \| KeyValueMetadata \| None = None,

Uh oh!

Conversation

rok commented Dec 22, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

rok commented Jan 26, 2026

Uh oh!

github-actions Bot commented Jan 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rok commented Jun 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

rok commented Jul 16, 2026

Uh oh!

paddyroddy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rok commented Dec 22, 2025 •

edited by github-actions Bot

Loading