Skip to content

SparkDistributedBackend._is_support_stage_scheduling always prints warning #59

Description

@philipmat

The else condition in SparkDistributedBackend._is_support_stage_scheduling seems to always print a "Spark version does not support stage-level scheduling." warning whether the context can be parallelize or not.

    def _is_support_stage_scheduling(self):
        spark_master = self._spark_context.master
        is_spark_local_mode = spark_master == "local" or spark_master.startswith("local[")
        if is_spark_local_mode:
            support_stage_scheduling = False
            warnings.warn("Spark local mode doesn't support stage-level scheduling.")
        else:
            support_stage_scheduling = hasattr(
                self._spark_context.parallelize([1]), "withResources"
            )
            warnings.warn("Spark version does not support stage-level scheduling.")
        return support_stage_scheduling

If seems like maybe it should only print the warning if support_state_scheduling was false?

        else:
            support_stage_scheduling = hasattr(
                self._spark_context.parallelize([1]), "withResources"
            )
            if not support_stage_scheduling:
                 warnings.warn("Spark version does not support stage-level scheduling.")

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions