Changelog#

0.9.0 “Laundry Service”#

Breaking Changes

  • The dagit key is no longer part of the instance configuration schema and must be removed from dagster.yaml files before they can be used.
  • -d can no longer be used as a command-line argument to specify a mode. Use --mode instead.
  • Use --preset instead of --preset-name to specify a preset to the pipeline launch command.
  • We have removed the config argument to the ConfigMapping, @composite_solid, @solid, SolidDefinition, @executor, ExecutorDefinition, @logger, LoggerDefinition, @resource, and ResourceDefinition APIs, which we deprecated in 0.8.0. Use config_schema instead.

New

  • Python 3.8 is now fully supported.
  • -d or --working-directory can be used to specify a working directory in any command that takes in a -f or --python_file argument.
  • Removed the deprecation of create_dagster_pandas_dataframe_type. This is the currently supported API for custom pandas data frame type creation.
  • Removed gevent dependency from dagster
  • New configured API for predefining configuration for various definitions: https://legacy-docs.dagster.io/overview/configuration/#configured
  • Added hooks to enable success and failure handling policies on pipelines. This enables users to set up policies on all solids within a pipeline or on a per solid basis. Example usage can be found here
  • New instance level view of Scheduler and running schedules
  • dagster-graphql is now only required in dagit images.

0.8.11#

Breaking Changes

  • AssetMaterializations no longer accepts a dagster_type argument. This reverts the change billed as "AssetMaterializations can now have type information attached as metadata." in the previous release.

0.8.10#

New

  • Added new GCS and Azure file manager resources
  • AssetMaterializations can now have type information attached as metadata. See the materializations tutorial for more
  • Added verification for resource arguments (previously only validated at runtime)

Bugfixes

  • Fixed bug with order-dependent python module resolution seen with some packages (e.g. numpy)
  • Fixed bug where Airflow's context['ts'] was not passed properly
  • Fixed a bug in celery-k8s when using task_acks_late: true that resulted in a 409 Conflict error from Kubernetes. The creation of a Kubernetes Job will now be aborted if another Job with the same name exists
  • Fixed a bug with composite solid output results when solids are skipped
  • Hide the re-execution button in Dagit when the pipeline is not re-executable in the currently loaded repository

Docs

  • Fixed code example in the advanced scheduling doc (Thanks @wingyplus!)
  • Various other improvements

0.8.9#

New

  • CeleryK8sRunLauncher supports termination of pipeline runs. This can be accessed via the “Terminate” button in Dagit’s Pipeline Run view or via “Cancel” in Dagit’s All Runs page. This will terminate the run master K8s Job along with all running step job K8s Jobs; steps that are still in the Celery queue will not create K8s Jobs. The pipeline and all impacted steps will be marked as failed. We recommend implementing resources as context managers and we will execute the finally block upon termination.
  • K8sRunLauncher supports termination of pipeline runs.
  • AssetMaterialization events display the asset key in the Runs view.
  • Added a new "Actions" button in Dagit to allow to cancel or delete mulitple runs.

Bugfixes

  • Fixed an issue where DagsterInstance was leaving database connections open due to not being garbage collected.
  • Fixed an issue with fan-in inputs skipping when upstream solids have skipped.
  • Fixed an issue with getting results from composites with skippable outputs in python API.
  • Fixed an issue where using Enum in resource config schemas resulted in an error.

0.8.8#

New

  • The new configured API makes it easy to create configured versions of resources.
  • Deprecated the Materialization event type in favor of the new AssetMaterialization event type, which requires the asset_key parameter. Solids yielding Materialization events will continue to work as before, though the Materialization event will be removed in a future release.
  • We are starting to deprecate "system storages" - instead of pipelines having a system storage definition which creates an intermediate storage, pipelines now directly have an intermediate storage definition.
    • We have added an intermediate_storage_defs argument to ModeDefinition, which accepts a list of IntermediateStorageDefinitions, e.g. s3_plus_default_intermediate_storage_defs. As before, the default includes an in-memory intermediate and a local filesystem intermediate storage.
    • We have deprecated system_storage_defs argument to ModeDefinition in favor of intermediate_storage_defs. system_storage_defs will be removed in 0.10.0 at the earliest.
    • We have added an @intermediate_storage decorator, which makes it easy to define intermediate storages.
    • We have added s3_file_manager and local_file_manager resources to replace the file managers that previously lived inside system storages. The airline demo has been updated to include an example of how to do this: https://github.com/dagster-io/dagster/blob/0.8.8/examples/airline_demo/airline_demo/solids.py#L171.
  • The help panel in the dagit config editor can now be resized and toggled open or closed, to enable easier editing on smaller screens.

Bugfixes

  • Opening new Dagit browser windows maintains your current repository selection. #2722
  • Pipelines with the same name in different repositories no longer incorrectly share playground state. #2720
  • Setting default_value config on a field now works as expected. #2725
  • Fixed rendering bug in the dagit run reviewer where yet-to-be executed execution steps were rendered on left-hand side instead of the right.