Background
In #517, pd.read_pickle(file) requires cast(Any, file) to suppress a type error because the current FileLike Protocol defined in gokart/utils.py is not structurally compatible with pandas' ReadPickleBuffer Protocol.
# gokart/utils.py
return pd.read_pickle(cast(Any, file))
Current FileLike
class FileLike(Protocol):
def read(self, n: int) -> bytes: ...
def readline(self) -> bytes: ...
def seek(self, offset: int) -> None: ...
def seekable(self) -> bool: ...
pandas ReadPickleBuffer requirements
mode: str # property
read(n: int = ..., /) -> bytes
readline() -> bytes
seek(offset, whence=..., /) -> int
seekable() -> bool
tell() -> int
Proposal
Refine FileLike so that it is a structural subtype of ReadPickleBuffer. This would let us drop the cast(Any, file) workaround.
Required changes:
- Update
FileLike signatures in gokart/utils.py:
- Add
mode property
read(n: int = ..., /) -> bytes (default + positional-only)
seek(offset: int, whence: int = ..., /) -> int (return int, accept whence)
- Add
tell() -> int
- Update
_ChunkedLargeFileReader in gokart/file_processor/base.py to match the new signatures and explicitly expose mode / tell (currently they only work at runtime via __getattr__).
- Simplify
FileLike | BytesIO to just FileLike since BytesIO already satisfies ReadPickleBuffer.
Notes
Background
In #517,
pd.read_pickle(file)requirescast(Any, file)to suppress a type error because the currentFileLikeProtocol defined ingokart/utils.pyis not structurally compatible with pandas'ReadPickleBufferProtocol.Current
FileLikepandas
ReadPickleBufferrequirementsProposal
Refine
FileLikeso that it is a structural subtype ofReadPickleBuffer. This would let us drop thecast(Any, file)workaround.Required changes:
FileLikesignatures ingokart/utils.py:modepropertyread(n: int = ..., /) -> bytes(default + positional-only)seek(offset: int, whence: int = ..., /) -> int(returnint, acceptwhence)tell() -> int_ChunkedLargeFileReaderingokart/file_processor/base.pyto match the new signatures and explicitly exposemode/tell(currently they only work at runtime via__getattr__).FileLike | BytesIOto justFileLikesinceBytesIOalready satisfiesReadPickleBuffer.Notes
_ChunkedLargeFileReaderwrapping S3/GCS streams) actually exposemode/tell.