feat(dagster): Update dagster (minor) by plural-copilot[bot] · Pull Request #508 · pluralsh/containers

plural-copilot · 2025-02-25T14:25:39Z

This PR contains the following updates:

Package	Type	Update	Change
dagster-azure (source)		minor	`==0.23.12` -> `==0.27.2`
dagster-gcp (source)		minor	`==0.23.12` -> `==0.27.2`
docker.io/dagster/user-code-example	final	minor	`1.7.12` -> `1.11.2`

Release Notes

dagster-io/dagster (dagster-azure)

`v0.27.2`

New

The dagster package now supports Python 3.13.
[dagster-tableau] Tableau assets can now be subsetted and materialized individually. #31078
[dagster-snowflake-polars] The new dagster-snowflake-polars package adds a SnowflakePolarsIOManager that can be used to read and write Snowflake tables using Polars.

Bugfixes

[ui] Fixed some cases where strings would be incorrectly middle-truncated.

Documentation

[dbt] Fixed incorrect reference to dbt_profiles.yml in the dagster-dbt tutorial (thanks @EFox2413!).
[dbt] Added documentation for the new dbt Cloud integration.

Dagster Plus

[ui][observe] You can now bulk add/remove assets to/from your favorites.

`v0.27.1`

New

dagster definitions validate will now raise an exception if there are invalid partition mappings between any assets in your asset graph (for example, an upstream and downstream asset with time-based partitions definitions using different timezones).
Performance improvements for run dequeuing when there are many queued runs using pools.
[ui] For times in the last two days, Dagster UI now shows e.g. "47 hours ago" instead of "2 days ago."
[ui] Asset checks now show whether they are blocking.
[dagster-tableau] Tableau workbooks fetched in Dagster can now be filtered and selected using the WorkbookSelectorFn.

Bugfixes

@graph now correctly allows omitting inputs when the destinations of an input mapping have a default value.
@record classes no longer create problematic namespace conflicts with the symbol "check."
[ui] Filtering by partition on the Asset Events view now works as expected.
[ui] Assets without definitions can now be properly wiped in the Asset Catalog.

Documentation

Added clearer setup instructions for Windows and Linux users to the Contributing documentation, thanks @oohwooh!
Fixed broken links in the Contributing documentation, thanks @emmanuel-ferdman!

Dagster Plus

[ui] Fixed an issue that prevented status filtering from working within the selection syntax.

dg & Components

Fixed an issue where dg scaffold github-actions would invoke the dg CLI with outdated parameters for serverless orgs.
[dagster-dlt] Fixed an issue where the default scaffolded dlt load led to an invalid asset key.

`v0.27.0`

`v0.26.21`

New

[dagster-tableau] The tableau_assets decorator is now available to create the asset definitions of all materializable assets in a given Tableau workspace. These assets can be subsetted and materialized using the TableauCloudWorkspace.refresh_and_poll method.
[dagster-tableau] The deprecated get_*_asset_key methods on the DagsterTableauTranslator class have been removed.
[ui] Show tags for a Job on the sidebar of the Job page. #30728

Bugfixes

Fixed a bug where "falsey" defualt config values were not showing up in the launchpad. This has been fixed for all cases except the empty dictionary.
Fixed an issue with the new "re-execute from asset failure" functionality that could cause additional steps to be included if the job was previously re-executed from step failure.
Fixed an issue where the staleStatusByPartition, staleCausesByPartition, and dataVersionByPartition fields on the graphql AssetNode would fail when called on an asset with dynamic partitions.
[dagster-dbt] Fixed an issue where creating a DagsterDbtTranslator that didn't call the parent class's constructor in its __init__ method would raise an Exception during execution.
[dagster-sling] Removed upper-bound pin on the sling dependency.
[dagster-sling] Fixed an issue with the SlingResource that could cause values specified with EnvVars to provide the env var name instead of the env var value to the sling replication configuration.
[dagster-fivetran] Introduced a dagster-fivetran snapshot command, allowing Fivetran workspaces to be captured to a file for faster subsequent loading.

Dagster Plus

[ui] The integrations marketplace (currently behind a feature flag) now allows you to create, edit, and delete private packages.

dg & Components

Added FunctionComponent, PythonScriptComponent, and UvRunComponent to make it easier to define arbitrary computations that execute assets or asset checks when invoked.
Weekly and arbitrary time-window partitions can now be provided to the partitions_def asset customization in YAML.
A clean and informative error message is now printed when an invalid set of parameters is passed to dg scaffold defs ....
Component asset specs now allow specifying partition definitions through the partitions_def key.
The dg cache is no longer operative, since dg now operates in the same python environment as the projects it manipulates. Config options for the cache have been removed.
The load_defs entrypoint has been deprecated in favor of load_from_defs_folder, which takes a single path for a folder to load definitions from rather than a module object.
The asset_post_processors field on SlingReplicationCollectionComponent and AirflowInstanceComponent is no longer supported, and has been replaced with the top-level post_processors field.
Fixed an issue where dg projects using autoload_defs=true could not be deployed to Dagster+.
Removed @preview from Component.
ComponentLoadContext is now frozen and no longer in preview.
Resolved subclasses now support Enum fields.

`v0.26.20`

New

The @asset decorator now supports a hooks argument to allow for functions to be executed on asset success / failure (thanks @brunobbaraujo)!
The log message produced when an asset check is evaluated now includes its pass / fail state.
The @multi_asset_check decorator now supports the pool argument.
[dagster-dbt] The DagsterDbtTranslator class now has a get_asset_check_spec method which can be overridden to customize the AssetCheckSpecs that are produced for each individual dbt test.

Bugfixes

Fixed an issue where setting the non-public blocking attribute on an AssetCheckSpec would halt execution of the step as soon as any asset check failure was emitted, even if the step still had asset materializations or check evaluations to emit that were not downstream of the failed asset check. Now that this issue has been fixed, the blocking attribute on AssetCheckSpec has been made a public attribute. If you were making use of the blocking attribute on AssetCheckSpec before it was public and relying on the previous behavior, you should exit from your asset evaluation function after emitting an AssetCheckFailure from within your multi-asset in order to halt further execution of the step.
Fixed a bug where GraphDefinition.to_job() would not work if an op had a custom IO manager key.
Fixed an issue that would cause .allow() and .ignore() applications to not propagate through .since() automation conditions.
Fixed a bug where assets with cross-location dependencies could sometimes be incorrectly reported as "Unsynced".
Fixed an issue where dagster dev was sometimes failing to load code locations with a "Deadline Exceeded" error unless the --use-legacy-code-server-behavior flag was set.
The backfill daemon can now be configured to use a threadpool executor via helm (thanks @hynekblaha)!
[dagster-gcp] Added a google-cloud-bigquery>=1.28.3 pin to correctly reflect the lowest compatible version.

Breaking Changes

[ui] Moved legacy Auto-materialize (global AMP) tab from Overview to Automations.

Dagster Plus

Fixed an issue where certain rare network conditions could cause steps to hang while uploading compute logs after a step finished.
[ui] For users with the new Observe UIs enabled, the Asset Health and Resources tabs are no longer shown on the Timeline page.

dg & Components (Preview)

Fix a bug with dg check yaml where valid component type names were rejected if they were not registered (i.e. visible from dg check components).
dg create-dagster now warns when scaffolding a project or workspace if it is not the latest version.
The project.registry_modules configuration can now accept wildcards (e.g. foo_bar.components.*). This will register any module matching the pattern with dg.
The env YAML function now errors if the specified env var is unset. Default values can be provided as an additional argument: {{ env('MY_ENV_VAR', 'default') }}
defs.yaml files can now specify a component in the module where it is defined, as opposed to just the module where it is exposed in the dg registry.
The PipesSubprocessScriptCollectionComponent has been removed.
Running dg commands such as dg check defs and dg dev in a project folder that is part of the workspace will now only apply to that project, instead of every project in the workspace.
Scaffolded projects no longer contain a "components" directory or a Python dagster_dg_cli.plugin entry point.
Scaffolded components can now be placed anywhere within a project module hierarchy.
The entry point group used by shared libraries exposing custom components to dg has been renamed from dagster_dg_cli.plugin to dagster_dg_cli.registry_modules (projects no longer need to define an entry point group at all).
dg list plugin-modules has been renamed to dg list registry-modules.
dg list defs now supports configuring output columns with the --columns/-c option.
[dagster-airbyte] Introduced a AirbyteCloudWorkspaceComponent which can be used to pull in Airbyte Cloud connections into Dagster

`v0.26.19`

New

The database table used by the DbIOManager is now configurable via "table" output (asset) metadata key #30310
Changed default settings for backfill daemon to use_threads=True, num_workers=4. Thanks @HynekBlaha!
A new function build_asset_check_context can be used to build asset check contexts for direct invocation.
Changed Definitions.get_all_asset_specs to only return assets directly passed in as AssetSpecs or AssetsDefinitions.
Removed selection argument from Definitions.map_asset_specs. If selection is needed use the new Definitions.map_resolved_asset_specs.
Definitions.get_job_def now warns when it finds an unresolved job or no job.
Changed Definitions.get_assets_def to return an AssetsDefinition without resolving if it was passed in directly.
[dagster-dbt] build_schedule_from_dbt_selection now supports a selector argument, allowing you to use yaml-based selectors.
[dagster-k8s] Pods created by the Kubernetes run launcher and executor from Dagster Plus now include the dagster/deployment-name label.
[dagster-pipes] Pipes execution errors are no longer treated as framework errors, meaning they properly invoke RetryPolicies.
[helm] Backfill daemon configuration now supported. Thanks @HynekBlaha!
[ui] Show relative start time on runs in run timeline hover lists. #30327

Bugfixes

[ui] Fixed live updating of asset materialization statuses in asset graph.

dg & Components (Preview)

Running dg commands like dg check defs and dg dev in a project folder that is part of the workspace will now only apply to that project, instead of every project in the workspace.
dg list defs now supports the --assets/-a option, to show only assets matching the provided selection.
dg list defs now supports a --path argument to subset the defs files shown.
The create-dagster workspace command now accepts the same required path argument as the create-dagster project command, instead of defaulting to a dagster-workspace subfolder of the current working directory.
The entry point group used by shared libraries exposing custom components to dg has been renamed from dagster_dg_cli.plugin to dagster_dg_cli.registry_modules (projects no longer need to define an entry point group at all).
dg list plugin-modules has been renamed to dg list registry-modules.
Newly scaffolded projects no longer contain a "components" directory or a Python dagster_dg_cli.plugin entry point.
Newly scaffolded components can now be placed anywhere within a project module hierarchy.
Resolvable subclasses can now use bare dict and list as field types.
Resolving a Resolvable subclass will no longer change empty strings to None.
Users can define multiple @definitions-decorated functions in a single module in the defs hierarchy and they are automatically merged and incorporated into the project.
Added @component_instance to replace @component. This allows multiple component instances in a python file.
Fixed an issue where dg commands would sometimes output extra dagster_telemetry_logger lines to stdout at the end of commands.
Added @template_var as an alternative approach for defining variables in a templating context.

`v0.26.18`

New

BackfillPolicy is now marked as generally available (GA).
Optimized the order of the @asset decorator overloads to make custom wrappers around the asset decorator easier. Thanks @jonathanunderwood!
[dagster-slack] Added get_client() to SlackResource.

Bugfixes

Definitions and AssetDefinition will now warn if they get different AssetSpecs with the same key. This will become an exception in 1.11.
Functions that load all definitions from modules like load_definitions_from_modules now handle duplicate AssetSpecs.
Fixed typo in logging. Thanks @eli-b!
[dagster-dbt] An issue occurring when using dbt selection arguments with a dbt project using saved queries and semantic models has been fixed.
[dagster-fivetran] Fixed an issue with load_assets_from_fivetran_instance where assets whose asset keys have been customized using a Fivetran translator would lead to an exception.

Documentation

Fixed grammar issues in GCP docs. Thanks @D1n0!
Fixed missing docs for required_resource_keys in @sensor. Thanks @seyf97!

Breaking Changes

Definitions and AssetDefinition will now warn if they get different AssetSpecs with the same key. This will become an exception in 1.11.

Dagster Plus

[dagster-cloud-cli] Added a --wait to the dagster-cloud job launch command that makes it wait until the launched run copmletes. Thanks @stevenayers!
[fix][dagster-cloud-cli] Fixed an issue where the dagster-cloud cli failed to deploy PEX projects on python 3.12 in certain environments without setuptools already installed.

dg & Components (Preview)

The dg CLI is now installed in each project's Python environment instead of as a global tool. If you previously had dg installed globally and are upgrading, first uninstall the global dg tool (uv tool uninstall dagster-dg or pip uninstall dagster-dg) and add dagster-dg-cli as a dependency to each of your projects. Newly scaffolded projects will automatically include the dg CLI going forward.
A new create-dagster CLI has been added for scaffolding projects instead of workspaces. uvx create-dagster project has replaced dg scaffold project, and uvx create-dagster workspace has replaced dg scaffold workspace. The commands take identical arguments.
Definitions and component instances are now scaffolded with dg scaffold defs <scaffolder> instead of dg scaffold <scaffolder>.
The component.yaml file to specify a component instance is now called "defs.yaml". "component.yaml" will remain supported for several weeks, but is deprecated.
The lib folder in newly scaffolded projects has been renamed to "components".
dg scaffold component-type has been renamed to dg scaffold component.
dg list plugins has been renamed to dg list plugin-modules and now outputs only plugin module names, not plugin objects.
dg list component now lists component types instead of instances.
Exports from dagster.components are now available in the top-level dagster module.
Added @component_instance to replace @component, which allows multiple component instances in a python file.
If you type a partial component name for dg scaffold defs <component>, you will now be prompted with possible matches.
When component classnames are unique, you can now use just the classname as an alias for the fully qualified name when running dg scaffold defs <component>.
dg launch now supports passing config files through --config/-c.
Added Component.from_attributes_dict and Component.from_yaml_path to help with testing.
Added dagster.components.testing.component_defs utility to help with testing components.
Scaffolded schedules and sensors in dg are now loadable by default.
dg scaffold defs inline-component can now be used to create inline components and corresponding instances.
dg list defs now outputs resources.
[fix] The dagster-dbt cli flag --components flag now correctly finds DbtProjectComponent again.
[dagster-fivetran] Added a FivetranAccountComponent which can be used to pull in Fivetran connections into Dagster.
[dagster-dlt] The DltLoadCollectionComponent scaffolder no longer attempts to automatically construct loads for the given source and destination type.
Fixed an issue where components failed to load when using load_assets_from_airbyte_instance or other APIs that return a CacheableAssetsDefinition.

`v0.26.17`

Bugfixes

Fixed an issue where an error was displayed in the UI while viewing run logs
[dagster-dbt] Fixed an issue occurring when using dbt selection arguments with a dbt project using semantic models.

`v0.26.16`

New

typing_extensions is now pinned to >=4.11.0 instead of >=4.10.0.
[ui] Viewing an automation condition evaluation now automatically expands the set of applicable sub-conditions.
[ui] Added the ability to navigate from an automation condition evaluation to upstream automation condition evaluations.
[ui] Added an asset graph node facet for viewing automation conditions and the most recent evaluation. This can be enabled in the user settings via a feature flag (Enable faceted asset nodes).
[ui] A new experimental integrations marketplace tab is now available and can be enabled in your user settings via a feature flag (Display integrations marketplace). It provides easy access to the gallery of dagster-supported plugins.

Bugfixes

Fixed an issue with the ExternalNotebookDataRequest GRPC call that would allow it to access files outside of the current directory.
Fixed an issue that would cause op_tags set on @observable_source_assets to be dropped from the underlying step context object when executed.
Fixed an issue where dagster dev would sometimes raise a gRPC error when loading several code locations at once.
Fixed an issue where setting an environment variable to the string "false", "0" or "None" for a dagster config field using a BoolSource would evaluate to True.
Fixed an issue where specifying executable_path in a workspace.yaml file to run code locations in a different virtual environment would not correctly inherit the PATH of that virtual environment in the code location.
[dagster-dbt] Fixed an issue causing dbt CLI invocation to fail when materializing assets when OpExecutionContext was used as the type hint for the context.
[dagster-deltalake] Corrected the timeout property data type in ClientConfig to be str instead of int (thanks, @edsoncezar16!)

Documentation

Added additional config fields to the K8sRunLauncher example (thanks, @nishan-soni!)
Corrected broken links on the automation landing page (thanks, @briandailey!)

Dagster Plus

[ui] Alert policy event tags no longer appear red and yellow outside of the policy notification history.

dg & Components (Preview)

Added suggestions to component model error messages when using built-in models for common classes such as AssetKey and AssetSpec.
dg list env now displays whether env vars are configured in each Dagster Plus scope.
Introduced Resolver.passthrough() to avoid processing fields on a component model.
ResolvedAssetKey is now exported from dagster.components.
dg init has been removed. dg scaffold project and dg scaffold workspace should be used instead.
Fixed an issue where dg dev failed with a temporarily file permissions error when running on Windows. Thanks @polivbr!

`v0.26.15`

New

Added a config section to dagster.yaml to enable submitting backfill runs in a threadpool.
Expanded definition time validation for partition mappings to avoid runtime errors querying asset status.
[ui][beta] You can now re-execute a run that targeted a multi-asset from the point of asset failure instead of step failure, meaning only assets that failed or were skipped will be re-executed. To enable this option, turn on the Enable retries from asset failure feature flag in your user settings.
[ui] Made it easier to select and copy image names for code locations.
[ui] Added asset lineage navigation within the automation condition evaluation tree.
[ui] Viewing an evaluation tick now auto-expands the set of applicable automation conditions.
[ui] Added an asset graph node facet for viewing automation conditions and the most recent evaluation in the global asset graph.
[dagster-fivetran] The FivetranWorkspace resource is now marked as generally available (GA).

Bugfixes

Changed asset wipes to also wipe associated asset check evaluations.
[dagster-fivetran] Fixed an issue causing the Fivetran integration to fail when the schema config does not exist for a connector.

Documentation

Fixed a broken link in the airflow migration docs. Thanks @jjyeo!
Updated example snippet to include a missing type hint. Thanks @agrueneberg!

Deprecations

[dagster-fivetran] The FivetranResource resource is now deprecated. Use the new FivetranWorkspace resource instead.

dg & Components (Preview)

Changed Scaffolder.scaffold to have the params object as an attribute of the ScaffoldRequest object instead of a dictionary. This is a breaking change for those who have implemented a custom scaffolder.
Added support for scaffolding resources via dg scaffold dagster.resources path/to/resources.py.
Added support for the usage of @definitions in the defs hierarchy.
Dagster components now include code references by default. When viewing an asset emitted by a component in the asset catalog, this will allow you to jump to the backing component.yaml file in your editor.
[dagster-dbt] DbtProjectComponent fields now properly evaluate templates.
[dagster-sling] Updated the SlingReplicationCollectionComponent from using the asset_attributes parameter to translation, in order to match our other integration components.
Fixed an issue where dg dev failed with a temporary file permissions error when running on Windows. Thanks @polivbr!

`v0.26.14`

New

[dagster-tableau] Refined Tableau integration for API 3.25 or greater.
[dagster-tableau] Data sources with extracts can now be materialized in Tableau assets created with build_tableau_materializable_assets_definition.
[ui] Added kinds tag for treasuredata.
[ui] Add Supabase kind icon.

Bugfixes

Fixed a bug which could cause an error when calling MultiPartitionsDefinition.has_partition_key() on invalid keys.
Fixed a bug where the default multiprocess executor would fail runs where the child process for a step crashed, even if a retry policy resulted in a successful retry of that crashed step.
Fixed a bug with AutomationCondition.initial_evaluation which could cause it to return False for an asset that went from having a condition, to having no condition at all, back to having the original condition again.
[ui] Fixed an issue which could cause the "Target" field of AutomationConditionSensorDefinitions to render incorrectly when exactly one asset check was defined in a code location.
[dagster-dbt] Fix DagsterDbtTranslatorSettings.enable_source_tests_as_checks returning duplicate asset checks.

Documentation

Added a sample Dagster+ ECS CloudFormation template which incorporates private subnets.
Fixed incorrect storage values in the Fargate task section of the AWS deployment guide, thanks @alexpotv!
Updated log stream docs, thanks @jjyeo!
Fixed broken code in the configurable resources guide, thanks @nightscape!

Deprecations

dagster.InitResourceContext.dagster_run has been deprecated in favor of InitResourceContext.run.

dg & Components (Preview)

[dagster-k8s] PipesK8sComponent has been added.
Dagster components no longer change the working directory while they are being loaded. This allows components to store relative paths and ensure that they will still work when accessed outside of the component loading codepath. This change may affect user-defined components that depend on Path.cwd() or os.getcwd(). Instead, you should use a path relative to the current source file when loading paths in a component, using the context.resolve_source_relative_path method (see resolve_dbt_project in DbtProjectComponent for an example).
Added dagster.job scaffolder.
[dagster-dbt] The DbtProjectComponent now has a translation_settings argument for adjusting DagsterDbtTranslatorSettings.
[dagster-dbt][fix] DbtProjectComponent fields now properly evaluate templates.
Fixed docstring for load_defs entrypoint, thanks @mattgiles!

`v0.26.13`

New

If an unselected asset check is executed during a run, the system will now warn instead of throwing a hard error.
When evaluating AutomationCondition.any_deps_match or AutomationCondition.all_dep_match with an allow / ignore specified, an error will no longer be produced if the provided asset selection references an asset key that does not exist.
Added the ability to restrict the list of ports that dagster dev is allowed to use to open subprocesses when running on Windows, by setting the DAGSTER_PORT_RANGE env var to a string of the form <start>=<end> - for example "20000-30000".
[dagster-aws] The S3 sensor's get_objects now returns an empty list if no new files can be found since the since_last_modified parameter. Thanks @bartcode!
[dagster-dbt] @dbt_assets and build_dbt_manifest_asset_selection now support a selector argument, allowing you to use yaml-based selectors.
[dagster-k8s] improved run monitoring when running with increased backoff limits. Thanks @adam-bloom!

Bugfixes

Fixed a bug with AutomationCondition.initial_evaluation which could cause it to return False for an asset that went from having a condition, to having no condition at all, back to having the original condition again.
Fixed a bug that would cause the AutomationCondition.any_deps_updated() condition to evaluate to False when evaluated on a self-dependency.
Fixed a bug in the quickstart_aws example. Thanks @Thenkei!
[ui] Fixed navigation between asset tabs by no longer preserving query parameters from one tab to the next.
[ui] Fixed an issue where the asset graph looked like it was still loading when it wasn't.

Documentation

Added Scala Spark / Dagster Pipes guide.

dg & Components (Preview)

[dg] Error message when an invalid configuration file is detected is now shorter and more clear.
[components] Descriptions and examples have been restored for core models such as ResolvedAssetSpec. description and examples arguments have been added to Resolver for documenting fields on non-pydantic model based Resolvable classes.

`v0.26.12`

New

[ui] Removed the By partition grouping view for recent events for assets that do not have a definition in the workspace.
[ui] The asset graph now displays asset health information when the new Observe UI feature flag is enabled.
[ui] A new feature flag allows you to customize the appearance of assets on the asset graph by enabling and disabling individual facets.
[ui] Fixed a bug that prevented filtering asset events by type.
[dagster-k8s] K8sPipeClient failures will now include the last 100 lines of the logs of the pod that failed, instead of the full log output.

Bugfixes

Fixed an issue which caused multi assets that were automatically broken apart in some contexts to remain separated even in cases where this was not necessary to maintain execution dependencies.
Fixed a bug that would cause multi-assets defined with can_subset=True to error when using dagster-pipes if not all outputs were emitted.
[ui] Fixed an issue where the asset graph looked like it was still loading when it wasn't.

Documentation

[docs] New integration pages for Polars and Patito.

dg & Components (Preview)

dg will now fail with an error message if it's version is below the minimum supported version for the version of dagster in your environment.
dg now checks for new versions and prompts the user to update. The check runs automatically at dg startup once per day.
dg will now emit a warning prompting the user to reinstall the package if it detects an entry point in project metadata that does not show up when running in dg list plugins.
dg list defs now collapses long system stacktraces #29507
Added dagster.multi_asset scaffolder
Added dagster.asset_check scaffolder
Fixed a bug where dg list defs would crash if anything was written to stdout while loading a project's definitions.
The default location for the dg user config file on Unix has been moved from ~/.dg.toml to ~/.config/dg.toml. ~/.config can be overridden by setting $XDG_CONFIG_HOME.
Cache deserialization errors are now ignored. Previously, when the dg contents of the cache were in an outdated format, dg could crash. Now it will just rebuild the cache.
The Components ETL Pipeline Tutorial now supports users of both pip and uv.
The dg CLI will now emit a warning if you are using "active" mode for your project python environment, there is a virtual environment at <project_root>/.venv, and the activated venv is not <project_root>/.venv
A new dg setting cli.suppress_warnings is available. This takes a list of warning types to suppress.
Added a warning message to inform users they need to install their project package when skipping automatic environment setup.
Changed configuration of project Python environments. The tool.dg.project.python_environment previously accepted a string, "active" or "persistent_uv". Now it accepts a table with one of two keys:
- {active = true}: equivalent of previous "active"
- {uv_managed = true}: equivalent of previous "persistent_uv"
Changed the default python environment for newly scaffolded projects to tool.dg.project.python_environment to {active = true}. This means by default, no virtual environment or uv.lock will be created when scaffolding a new project (via dg init or dg scaffold project). You can pass --python-environment uv_managed for the old behavior.
Removed the --skip-venv flag on dg scaffold project and dg init.
The dagster_components package has been merged into dagster and use of the dagster-components package has been deprecated. dagster-components will remain as a stub package for the next few weeks, but code should be updated to import from dagster.components instead of dagster_components.
[dagster-dbt] the dagster-dbt project prepare-and-package cli now supports --components for handling DbtProjectComponent
[dagster-dbt] DbtProjectComponent has been reworked, changing both the python api and the yaml schema. dbt has been replaced with project with a slightly different schema, and asset_attributes with translation .

`v0.26.11`

New

[ui] Runs launched from the Dagster UI get a dagster/from_ui = true tag, making it easy to filter for them.
[ui] The run page now shows the number of log levels selected as well as the number of log levels available.
Added AirflowFilter API for use with dagster-airlift, allows you to filter down the set of dags retrieved up front for perf improvements.

Bugfixes

[ui] Fixed a bug that prevented filtering asset events by type.
Fixed a bug that would cause multi-assets defined with can_subset=True to error when using dagster-pipes if not all outputs were emitted.
[dagster-dbt] the state_path argument to DbtCliResource now resolves relative to the project directory as documented.
[dagster-k8s] Made reliability improvements to PipesK8sClient log streaming when transient networking errors occur. The default behavior of the PipesK8sClient is now to reconnect to the stream of logs every hour (this value can be overridden by setting the DAGSTER_PIPES_K8S_CONSUME_POD_LOGS_REQUEST_TIMEOUT environment variable) and to retry up to 5 times if an error occurs while streaming logs from the launched Kubernetes pod (this value can be overridden by setting the DAGSTER_PIPES_K8S_CONSUME_POD_LOGS_RETRIES environment variable.)
[dagster-dbt] Fixed a bug where dbt jobs would fail due to unparseable logs causing errors in DbtCliInvocation.stream_raw_events. (Thanks @ross-whatnot!)

Dagster Plus

[ui] It is now possible to roll back a code location to a version that had previously been in an errored state.

dg & Components (Preview)

The dg CLI will now emit a warning if you are using "active" mode for your project python environment, there is a virtual environment at <project_root>/.venv, and the activated venv is not <project_root>/.venv
A new dg setting cli.suppress_warnings is now available. This takes a list of warning types to suppress.
Changed configuration of project Python environments. The tool.dg.project.python_environment previously accepted a string, "active" or "persistent_uv". Now it accepts a table with one of three keys: - {active = true}: equivalent of previous "active" - {uv_managed = true}: equivalent of previous "persistent_uv"
Changed the default python environment for newly scaffolded projects to tool.dg.project.python_environment to {active = true}. This means by default, no virtual environment or uv.lock will be created when scaffolding a new project (via dg init or dg scaffold project). You can pass --python-environment uv_managed for the old behavior.
Removed the --skip-venv flag on dg scaffold project and dg init.
Fixed a bug where new projects scaffolded with dg scaffold project were lacking dagster as a dependency.
The "Creating a library of components" guide has been replaced by a new and more general "Creating a dg plugin" guide.
The arguments/options of the dg init command have changed. You may pass . as an argument to initialize a project/workspace in the CWD. See dg init --help for more details.
The dagster-components package (which was converted to a stub package in the last release) is no longer being published. All dagster-components functionality is now part of dagster.
Projects should now expose custom component types under the dagster_dg.plugin entry point group instead of dagster_dg.library. dagster_dg.library support is being kept for now for backcompatibility, but will be dropped in a few weeks.
Fixed formatting of line added to project/lib/__init__.py when scaffolding a component type.
The dg env list command is now dg list env
The dg plus env pull command is now dg plus pull env.
The dg list component-type command has been removed. There is a new dg list plugins with output that is a superset of dg list component-type.
The dg user config file on Unix is now looked for at ~/.dg.toml instead of ~/dg.toml.
[dagster-dbt] DbtProjectComponent has been reworked, changing both the python api and the yaml schema. dbt has been replaced with project with a slightly different schema, and asset_attributes with translation.

`v0.26.10`

New

A new blocking parameter has been added to build_last_update_freshness_checks and build_time_partition_freshness_checks.
The default byte size limit for gRPC requests and responses is now 100MB instead of 50MB. This value can be adjusted by setting the DAGSTER_GRPC_MAX_RX_BYTES and DAGSTER_GRPC_MAX_SEND_BYTES environment variables on the gRPC client and server processes.
Added a new Definitions.map_asset_specs method, which allows for the transformation of properties on any AssetSpec or AssetsDefinition objects in the Definitions object which match a given asset selection.
Definitions.validate_loadable and dagster definitions validate will now raise an error on assets with invalid partition mappings, like a TimeWindowPartitionMapping between two time-based partitions definitions with different timezones. Previously, these invalid partition mappings would not raise an error until they were used to launch a run.
[dagster-k8s] Reliability improvements to PipesK8sClient log streaming when transient networking errors occur. The default behavior of the PipesK8sClient is now to reconnect to the stream of logs every 3600 seconds (this value can be overridden by setting the DAGSTER_PIPES_K8S_CONSUME_POD_LOGS_REQUEST_TIMEOUT environment variable) and to retry up to 5 times if an error occurs while streaming logs from the launched Kubernetes pod (this value can be overridden by setting the DAGSTER_PIPES_K8S_CONSUME_POD_LOGS_RETRIES environment variable.)

Bugfixes

Fixed an issue where run monitoring sometimes didn't fail runs that were stuck in a NOT_STARTED status instead of a STARTING status.
Fixed an issue where Dagster run metrics produced large amounts of error lines when running in containers without a CPU limit set on the container.
Fixed an issue where using partial resources in Pydantic >= 2.5.0 could result in an unexpected keyword argument TypeError. (Thanks @HynekBlaha!)
Fixed an issue with new AutomationCondition.executed_with_tags() that would cause the tag_keys argument to not be respected. Updated the display name of AutomationCondition.executed_with_tags() to contain the relevant tag_keys and tag_values.
[ui] Fixed a bug preventing filtering on the asset events page.
[ui] Fix custom time datepicker filter selection for users with custom Dagster timezone settings.
[dagster-fivetran] Fixed a bug causing the Fivetran integration to fetch only 100 connectors per destination.
[dagster-fivetran] Paused connectors no longer raise an exception in FivetranWorkspace.sync_and_poll(...). Instead, they skip and a warning message is logged.
[dagster-fivetran] Fixed an issue where new runs of code locations using Fivetran assets would sometimes raise a "Failure condition: No metadata found for CacheableAssetsDefinition" error if the run was started immediately after a new version of the code location was deployed.
[dagster-cloud] Reliability improvements to the dagster-cloud job launch command when launching runs targeting large numbers of assets or asset checks.

Dagster Plus

Fixed an issue with the identification of the base repository URL the DAGSTER_CLOUD_GIT_URL. The raw URL is now exposed as DAGSTER_CLOUD_RAW_GIT_URL.

dg & Components (Preview)

The dagster_components package has been merged into dagster. All symbols exported from dagster_components are now exported by dagster.
Projects should now expose custom component types under the dagster_dg.plugin entry point group instead of dagster_dg.library. dagster_dg.library support is being kept for now for backcompatibility, but will be dropped in a few weeks.
Component.get_schema has been renamed to Component.get_model_cls. Override that instead to customize the frontend of your component. Component.get_schema will continue to work for the time being but will be removed at some point in the future.
Add support for dg.toml files. dg settings in pyproject.toml are set under tool.dg, but in dg.toml they are set at the top level. A dg.toml may be used in place of pyproject.toml at either the project or workspace level.
dg-scaffolded workspaces now include a dg.toml instead of pyproject.toml file.
Projects scaffolded using dg init or dg scaffold project now follow modern Python packaging conventions, placing the root module in a top-level src directory and use hatchling as build backend rather than setuptools.
Scaffolding a component type defaults to inheriting from dagster.components.Model instead of getting decorated with @dataclasses.dataclass.
Assets scaffolded using dg scaffold dagster.asset will no longer be commented out.
The dg list component-type command has been removed. There is a new dg list plugins with output that is a superset of dg list component-type.
dg list defs now includes infomation about asset checks.
Fix formatting of line added to project/lib/__init__.py when scaffolding a component type.
Fixed bug where schedules were displayed in the sensors section of dg list defs.
Fixed a bug where dg would crash when working with Python packages with an src-based layout.

`v0.26.9`

Bugfixes

[ui] Fix custom time datepicker filter selection for users with custom Dagster timezone settings.

dg & Components (Preview)

Add support for dg.toml files. dg settings in pyproject.toml are set under tool.dg, but in dg.toml they are set at the top level. A dg.toml may be used in place of pyproject.toml at either the project or workspace level.
dg-scaffolded workspaces now include a dg.toml instead of pyproject.toml file.
dg-scaffolded projects now place the root package in an src/ directory and use hatchling as the build backend rather than setuptools.
Fixed a bug where dg would crash when working with Python packages with an src-based layout.
dg check yaml now properly validates component files in nested subdirectories of the defs/ folder.

`v0.26.8`

New

[ui] The Dagster UI now allows you to specify extra tags when re-executing runs from failure from the runs feed re-execute dialog, or by holding shift when clicking Re-execute menu items throughout the app.
[ui] Performance improvements for loading the partitions page for multi-partitioned assets.
[ui] Fix link in toast messages that appear when launching backfills.
[ui] Dagster's UI now allows you to copy run tags as a YAML block from the Tags and Configuration modals.
[ui] The Dagster Run UI now allows you to view the execution plan of a queued run.

Bugfixes

The AutomationCondition.initial_evaluation condition has been updated to become true for all partitions of an asset whenever the PartitionsDefinition of that asset changes, rather than whenever the structure of the condition changes.
[dagster-fivetran] Fixed an issue where new runs of code locations using fivetran assets would sometimes raise a "Failure condition: No metadata found for CacheableAssetsDefinition" error if the run was started immediately after a new version of the code location was deployed.
[dagster-fivetran] Fixed an issue where including multiple sets of assets from build_fivetran_assets_definitions in a single Definitions object would result in "duplicate node" errors when launching a run.
[ui] Fixed line charts for colorblind themes.
[ui] Fixed an issue with querystring parsing that can arise when selecting a large number of items in the selection syntax input.
[ui] Fixed tag filtering on automations list.
[ui] Fixed hover state on focused inputs.
[ui] Fixed an issue with the Run step selection input autocomplete where it would suggest key:"*substring*" instead of name:"*substring*".
[ui] Fixed the "View run" link shown when launching runs

Documentation

Fix a bug in example code for pyspark.

dg & Components (Preview)

Added the ability to scaffold Python components.
The DefsModuleComponent has been renamed to DefsFolderComponent.
When scaffolding a component, the command is now dg scaffold my_project.ComponentType instead of dg scaffold component my_project.ComponentType.
[dagster-dg] dagster list defs will now read environment variables from a local .env file if present when constructing the definitions.
dagster-components has been merged into dagster and use of the dagster-components package has been deprecated.
dagster-components will remain as a stub package for the next few weeks, but code should be updated to import from dagster.components instead of dagster_components.
The DbtProjectComponent has been relocated to the dagster-dbt package,
importable as dagster_dbt.DbtProjectComponent.
The SlingReplicationCollectionComponent has been relocated to the dagster-sling package,
importable as dagster_sling.SlingReplicationCollectionComponent.

`v0.26.7`

New

Applying changes from sensor test results now also applies changes from dynamic partition requests.
When merging assets from multiple code locations, autogenerated specs are now prioritized lower than customized external asset specs.
[ui] Allowed using command-click to view a run from the toast message that appears when starting a materialization of an asset.
[ui] Asset graph can now zoom out a lot more.
[ui] Added a kind tag for dbt Cloud.
[dagster-dlt] Added backfill policy to dlt_assets, defaulting to single-run. (Thanks @neuromantik33!)

Bugfixes

Updated AutomationCondition.initial_evaluation condition to become true for all partitions of an asset whenever the PartitionsDefinition of that asset changes, rather than whenever the structure of the condition changes.
Fixed a bug with several integrations that caused data fetched from external APIs not to be properly cached during code server initialization, leading to unnecessary API calls in run and step worker processes. This affected dagster-airbyte, dagster-dlift, dagster-dbt, dagster-fivetran, dagster-looker, dagster-powerbi, dagster-sigma, and dagster-tableau.
[ui] Fixed an issue with the Run step selection input autocomplete where it would suggest key:"*substring*" instead of name:"*substring*".
[ui] Fixed the "View run" link shown when launching runs.
[ui] Fixed an issue where updating a catalog view caused an infinite loading state.
Fixed an issue which could cause asset check evaluations emitted from the body of the op to not impact the check status of an asset in the UI.
Fixed an issue that could cause an asset backfill created by re-executing another backfill from the point of failure to error on the first tick in rare cases.
Fixed an issue that could cause automation condition evaluations to fail to render in the UI in rare cases.
[ui] Fixed a regression in the "Cancel Backfill" option for job backfills that have finished queuing runs.
[ui] Fixed overflow of long runs feed table on backfill page.
[dagster-dbt] Replaced @validator with @field_validator in dagster_dbt/core/resource.py to prevent Pydantic deprecation warnings. (Thanks @tintamarre!)

Documentation

Updated the "Asset versioning and caching" guide to reflect the current Dagster UI and "Unsynced" labels.
Removed a few repeated lines in documentation on customizing automation conditions. (Thanks @zero-stroke!)
Fixed example in TableRecord documentation to use the new input format.

Configuration

[dagster-gcp] Updated Dataproc configuration to the latest version. If necessary, consider pinning your dagster-gcp version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/b4d562e04fe571e40037a344b7a9937d
[dagster-aws][dagster-spark] Updated Spark configuration to the latest version (3.5.5). If necessary, consider pinning your dagster-aws and/or dagster-spark version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/f358f5a70fea28d5f164aca8da3dee04

Dagster Plus

[ui] Fixed filtering for multiple tags on list view pages, including Automations.
[ui] Fixed an issue where the urls generated by catalog filtering would remove all filters if loaded directly.
[ui] Added a warning on the sign-in page indicating that the sign-in and signup flows will be changing soon.
[ui] Require confirmation when rolling back a code location to a previous version.

dg & Components (Preview)

Virtual environment detection settings for projects have changed. Previously, the global settings use_dg_managed_environment and require_local_venv controlled the environment used when launching project subprocesses. This is now configured at the project level. The tool.dg.project.python_environment setting takes a value of either persistent_uv or active. persistent_uv will be used by default in new scaffolded projects and uses a uv-managed .venv in the project root. active is the default if no tool.dg.project.python_environment is set, and just uses the active python environment and opts out of dg management of the environment.
A new base class, Resolvable, has been added. This can be used to simplify the process of defining a yaml schema for your components. Instead of manually defining a manual ResolvedFrom[...] and ResolvableModel, the framework will automatically derive the model schema for you based off of the annotations of your class.
Python files with Pythonic Components (i.e. defined with @component) can now contain relative imports.
The dg init command now accepts optional --workspace-name and --project-name options to allow scaffolding an initial workspace and project via CLI options instead of prompts.
Added a new dagster_components.dagster.DefsFolderComponent that can be used at any level of your defs/ folder to apply asset attributes to the definitions at or below that level. This was previously named dagster_components.dagster.DefsModuleComponent.

`v0.26.6`

New

Added a new AutomationCondition.executed_with_tags() condition that makes it possible to filter for updates from runs with particular tags.
AssetCheckEvaluation can now be yielded from Dagster ops to log an evaluation of an asset check outside of an asset context.
Added the kinds argument to dagster.AssetOut, allowing kinds to be specified in @multi_asset.
[dagster-dbt] AssetCheckEvaluations are now yielded from ops leveraging DbtCliResource.cli(...) when asset checks are included in the dbt asset lineage.
[dagster-sling] The SlingResource.replicate() method now takes an option stream parameter, which allows events to be streamed as the command executes, instead of waiting until it completes (thanks, @natpatchara-w!).
[dagster-graphql] The DagsterGraphQLClient now supports an auth keyword argument, which is passed to the underlying RequestsHTTPTransport constructor.
[ui] The asset selection syntax input now allows slashes "/" in the freeform search.
[ui] The backfill pages now show summary information on all tabs for easier backfill monitoring.

Bugfixes

Fixed issue with AutomationCondition.newly_requested() which could cause it to fail when nested within AutomationCondition.any_deps_match() or AutomationCondition.all_deps_match().
Fixed a bug with AutomationCondition.replace() that would cause it to not effect AutomationCondition.since() conditions.
Fixed a bug with several integrations that caused data fetched from external APIs not to be properly cached during code server initialization, leading to unnecessary API calls in run and step worker processes. This affected: dagster-airbyte, dagster-dlift, dagster-dbt, dagster-fivetran, dagster-looker, dagster-powerbi, dagster-sigma, and dagster-tableau.
Fixed a bug that could cause invalid circular dependency errors when using asset checks with additional dependencies.
[dagster-fivetran] Loading assets for a Fivetran workspace containing incomplete and broken connectors now no longer raises an exception.
[ui] Fixed the colorblind (no red/green) theme behavior when in dark mode.
[ui] The Asset > Partitions page no longer displays an error in some cases when creating dynamic partitions.
[ui] The Launch and Report Events buttons no longer error if you click it immediately after creating a new dynamic partition.

dg & Components (Preview)

__pycache__ files are no longer included in the output of dg list component. (Thanks @stevenayers!)
When resolving the deps of an AssetSpec from yaml, multi-part asset keys are now correctly parsed. (Thanks @stevenayers!)
The entrypoint group for dg projects has been renamed from dagster.components to dagster_dg.library.
dg check yaml is now run by default before dg dev and dg check defs.

`v0.26.5`

New

async def yield_for_execution is now supported on ConfigurableResource. An event_loop argument has been added to context builders to support direct execution.
dagster dev deduplicates stacktraces when code locations fail to load, and will by default truncate them t

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

@@ -1,4 +1,4 @@
-FROM docker.io/dagster/user-code-example:1.7.12
+FROM docker.io/dagster/user-code-example:1.11.2


@@ -1,4 +1,4 @@
-FROM docker.io/dagster/user-code-example:1.7.12
+FROM docker.io/dagster/user-code-example:1.11.2


Signed-off-by: plural-copilot[bot] <172153387+plural-copilot[bot]@users.noreply.github.com>

plural-copilot Bot added the dependencies label Feb 25, 2025

plural-copilot Bot enabled auto-merge (squash) February 25, 2025 14:25

github-advanced-security AI found potential problems Feb 25, 2025

View reviewed changes