Resolve DataFrame type through intermediate generic classes#525
Merged
kitagry merged 2 commits intoJun 10, 2026
Merged
Conversation
get_dataframe_type_from_task auto-detects the DataFrame backend (pandas/polars) from the TaskOnKart[T] type parameter to pick the right file processor. When the type was bound through an intermediate generic class (e.g. Base(TaskOnKart[T]) with Concrete(Base[pl.DataFrame])), the TypeVar was left unresolved and detection fell back to pandas, so polars caches were silently read/written with the pandas processor. Walk the MRO to collect TypeVar bindings and resolve the TaskOnKart type argument so the intermediate-generic case is detected correctly. Downstream projects that worked around this by overriding output() can then drop that workaround. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
luigi's Register metaclass globally registers every Task subclass by task_family name at class-definition time. The two new tests both defined a `_GenericTask`, producing an ambiguous registry entry that raised TaskClassAmbigiousException when PandasTypeConfigMap later resolved task names, intermittently failing unrelated worker/tree tests under parallel/random test ordering. Give each test's intermediate generic base a unique name (matching the existing per-test unique-naming convention) and align the polars skip decorator with the rest of the file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
get_dataframe_type_from_taskinfers the DataFrame backend (pandas / polars / polars-lazy) from theTaskOnKart[T]type parameter to choose the matching file processor. It inspects the type argument of aTaskOnKart[...]base directly, so when the concrete type is bound through an intermediate generic class, theTaskOnKartbase still carries an unboundTypeVarand detection silently falls back to pandas.This shows up with a common pattern where a library defines a generic base task and the concrete type is bound on a subclass:
A polars task detected as pandas means its cache is read/written with the pandas processor.
What this changes
Walk the MRO to collect
TypeVar→ type-argument bindings, then resolve theTaskOnKarttype argument through that map (followingTypeVar→TypeVarchains across multiple generic levels). Direct and nested cases are unaffected, since a non-TypeVarargument resolves to itself.Added tests cover the intermediate-generic case for both
pl.DataFrameandpl.LazyFrame.