Skip to content

Resolve DataFrame type through intermediate generic classes#525

Merged
kitagry merged 2 commits into
m3dev:masterfrom
yusuke14tanaka:fix/dataframe-type-typevar-resolution
Jun 10, 2026
Merged

Resolve DataFrame type through intermediate generic classes#525
kitagry merged 2 commits into
m3dev:masterfrom
yusuke14tanaka:fix/dataframe-type-typevar-resolution

Conversation

@yusuke14tanaka

Copy link
Copy Markdown
Contributor

Problem

get_dataframe_type_from_task infers the DataFrame backend (pandas / polars / polars-lazy) from the TaskOnKart[T] type parameter to choose the matching file processor. It inspects the type argument of a TaskOnKart[...] base directly, so when the concrete type is bound through an intermediate generic class, the TaskOnKart base still carries an unbound TypeVar and detection silently falls back to pandas.

This shows up with a common pattern where a library defines a generic base task and the concrete type is bound on a subclass:

T = TypeVar('T')

class MyBase(TaskOnKart[T], Generic[T]):
    ...

class MyTask(MyBase[pl.DataFrame]):  # detected as pandas before this change
    ...

A polars task detected as pandas means its cache is read/written with the pandas processor.

What this changes

Walk the MRO to collect TypeVar → type-argument bindings, then resolve the TaskOnKart type argument through that map (following TypeVarTypeVar chains across multiple generic levels). Direct and nested cases are unaffected, since a non-TypeVar argument resolves to itself.

Added tests cover the intermediate-generic case for both pl.DataFrame and pl.LazyFrame.

get_dataframe_type_from_task auto-detects the DataFrame backend (pandas/polars) from the TaskOnKart[T] type parameter to pick the right file processor. When the type was bound through an intermediate generic class (e.g. Base(TaskOnKart[T]) with Concrete(Base[pl.DataFrame])), the TypeVar was left unresolved and detection fell back to pandas, so polars caches were silently read/written with the pandas processor.

Walk the MRO to collect TypeVar bindings and resolve the TaskOnKart type argument so the intermediate-generic case is detected correctly. Downstream projects that worked around this by overriding output() can then drop that workaround.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@kitagry kitagry left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

luigi's Register metaclass globally registers every Task subclass by task_family name at class-definition time. The two new tests both defined a `_GenericTask`, producing an ambiguous registry entry that raised TaskClassAmbigiousException when PandasTypeConfigMap later resolved task names, intermittently failing unrelated worker/tree tests under parallel/random test ordering.

Give each test's intermediate generic base a unique name (matching the existing per-test unique-naming convention) and align the polars skip decorator with the rest of the file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@hirosassa hirosassa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kitagry kitagry merged commit 7320a62 into m3dev:master Jun 10, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants