Converting an existing project

info

Dagster Components is currently in Release Candidate status. APIs are stable, with broader integration coverage and full feature parity in active development and coming soon. Check it out and let us know what you think in the #dg-components channel in the Dagster Community Slack!

In this guide, we walk through converting an existing Dagster project that defines a Python package with a single Dagster asset, which is exposed in a top-level Definitions object in my_existing_project/definitions.py.

note

We'll cover both the case of uv with pyproject.toml, and pip with setup.py.

uv

If you used uv to set up your Dagster project, you will have a pyproject.toml project configuration file:

tree

.
├── my_existing_project
│   ├── __init__.py
│   ├── assets.py
│   ├── definitions.py
│   └── py.typed
├── pyproject.toml
└── uv.lock

2 directories, 6 files

pip

If you used pip to set up your Dagster project, you will have a setup.py project configuration file:

tree

.
├── my_existing_project
│   ├── __init__.py
│   ├── assets.py
│   ├── definitions.py
│   └── py.typed
└── setup.py

2 directories, 5 files

Step 1: Create and activate a virtual environment

First, you will need to make sure you have an activated and up-to-date virtual environment. We recommend creating the virtual environment in the project root (particularly when using uv), although it is not required.

uv

If you don't have a virtual environment yet, run:

uv sync

Then activate it:

source .venv/bin/activate

pip

If you don't have a virtual environment yet, run:

python -m venv .venv

Next, activate it:

source .venv/bin/activate

Finally, install the project package as an editable install:

pip install --editable .

Step 2: Install dependencies

Next, you will need to install the dg command line tool into your project virtual environment.

uv

uv add dagster-dg-cli

pip

pip install dagster-dg-cli

Step 3: Update project structure

Step 3.1: Add dg configuration

The dg command recognizes Dagster projects through the presence of TOML configuration. This may be either a pyproject.toml file with tool.dg and tool.dg.project sections, or a dg.toml file.

uv

Since the example project already has a pyproject.toml file, you just need to add tool.dg and tool.dg.project sections:

pyproject.toml

...
[tool.dg]
directory_type = "project"

[tool.dg.project]
root_module = "my_existing_project"
code_location_target_module = "my_existing_project.definitions"

pip

Since the example project has a setup.py file and no pyproject.toml file, you will need to create a dg.toml file:

dg.toml

directory_type = "project"

[project]
root_module = "my_existing_project"
code_location_target_module = "my_existing_project.definitions"

Configuration reference

• directory_type = "project": This is how dg identifies your package as a Dagster project. This setting is required.
• root_module = "my_existing_project": This points to the root module of your project. This setting is required.
• code_location_target_module = "my_existing_project.definitions": This tells dg where to find the top-level Definitions object in your project. Since this setting defaults to NAME_OF_ROOT_MODULE.definitions, it is not strictly necessary to set it for this example. However, if your project has the top-level Definitions object defined in a different module, this setting is required.

Step 3.2: Check configuration

Now that the project configuration file has been updated, you can interact with your project using dg. To check the configuration and see the one asset in this project, run dg list defs:

dg list defs

┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                         ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key      ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ my_asset │ default │      │       │             │ │
│         │ └──────────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────┘

Step 3.3: Add an entry point to the project for custom component types

dg uses the Python entry point API to expose custom component types and other scaffoldable objects from user projects. In the project configuration file, the entry point specifies the location of a Python submodule where the project exposes registry modules.

note

By convention, this submodule is named <root_module>.lib, but in the case of a Dagster project, it is <root_module>.components.

Step 3.3.1: Create submodule directory

First, create the submodule directory and __init__.py file:

mkdir my_existing_project/components && touch my_existing_project/components/__init__.py

Step 3.3.2: Add entry point to project configuration file

Next, add a dagster_dg_cli.registry_modules entry point to your project configuration file, then reinstall the project package into your virtual environment.

uv

Since the package metadata is in pyproject.toml, you will need to add the entry point declaration there:

pyproject.toml

...
[project.entry-points]
"dagster_dg_cli.registry_modules" = { my_existing_project = "my_existing_project.components"}
...

pip

With pip, the package metadata is in setup.py. While it is possible to add entry point declarations to setup.py directly, you will want to be able to read the entry point declaration from dg, and there is no reliable way to read setup.py (since it is arbitrary Python code). Instead you will need to add the entry point to a new setup.cfg file, which can be used alongside setup.py.

Create setup.cfg with the following contents:

setup.cfg

[options.entry_points]
dagster_dg_cli.registry_modules =
    my_existing_project = my_existing_project.components

note

If your package has existing entry points declared in setup.py, you will need to move their definitions to setup.cfg as well.

Step 3.3.3: Reinstall the project package

uv

Reinstall the package. Note that uv sync will not reinstall your package, so you will need to use uv pip install instead:

uv pip install --editable .

pip

Reinstall the package:

pip install --editable .

warning

The reinstallation step is crucial. Python entry points are registered at package installation time, so if you simply add a new entry point to an existing editable-installed package, it won't be picked up.

Step 3.4: Check configuration

If you've done everything correctly, you should now be able to run dg list registry-modules and see the module my_existing_project.components, which you have registered as an entry point, listed in the output:

dg list registry-modules

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Module                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster                        │
│ my_existing_project.components │
└────────────────────────────────┘

You can now scaffold a new component in your project, and it will be available to dg commands. First create the component:

dg scaffold component Foo

Scaffolded Dagster component at /.../my-existing-project/my_existing_project/components/foo.py.

Then run dg list components to confirm that the new component is available:

dg list components

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Key                                ┃ Summary                                                                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster.DefinitionsComponent       │ An arbitrary set of Dagster definitions.                                        │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.DefsFolderComponent        │ A component that represents a directory containing multiple Dagster definition  │
│                                    │ modules.                                                                        │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.FunctionComponent          │ Represents a Python function, alongside the set of assets or asset checks that  │
│                                    │ it is responsible for executing.                                                │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.PythonScriptComponent      │ Represents a Python script, alongside the set of assets and asset checks that   │
│                                    │ it is responsible for executing.                                                │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.TemplatedSqlComponent      │ A component which executes templated SQL from a string or file.                 │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.UvRunComponent             │ Represents a Python script, alongside the set of assets or asset checks that it │
│                                    │ is responsible for executing.                                                   │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ my_existing_project.components.Foo │ COMPONENT SUMMARY HERE.                                                         │
└────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────┘

You should see the my_project.lib.MyComponentType listed in the output.

Step 3.5: Create a defs submodule

Part of the dg experience is autoloading definitions. This means automatically picking up any definitions that exist in a particular module. In this step, you will create a new submodule named my_existing_project.defs (defs is the conventional name of the module for where definitions live in dg) from which you will autoload definitions:

mkdir my_existing_project/defs

Step 4: Modify top-level definitions

Step 4.1: Modify definitions.py file

Autoloading is provided by a function that returns a Definitions object. Because there are already some other definitions in the example project, you will need to combine those with the autoloaded ones from my_existing_project.defs.

To do so, you'll need to modify your definitions.py file, or whichever file contains your top-level Definitions object.

You will autoload definitions using load_from_defs_folder, then merge them with your existing definitions using Definitions.merge. You pass load_from_defs_folder the defs submodule you just created:

Before

import dagster as dg
from my_existing_project.assets import my_asset

defs = dg.Definitions(
    assets=[my_asset],
)

After

from pathlib import Path

from my_existing_project.assets import my_asset

import dagster as dg

defs = dg.Definitions.merge(
    dg.Definitions(assets=[my_asset]),
    dg.load_from_defs_folder(project_root=Path(__file__).parent.parent),
)

Step 4.2: Add an asset to new defs submodule

Next, add an asset to the new defs submodule. Create a file called my_existing_project/defs/autoloaded_asset.py with the following contents:

import dagster as dg


@dg.asset
def autoloaded_asset(): ...

Step 4.3: Confirm the new asset is autoloaded

Finally, confirm the new asset is being autoloaded. If you run dg list defs again, you should see both the new autoloaded_asset and old my_asset:

dg list defs

┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                                 ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key              ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ autoloaded_asset │ default │      │       │             │ │
│         │ ├──────────────────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ my_asset         │ default │      │       │             │ │
│         │ └──────────────────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────────────┘

Now the project is fully compatible with dg!

Next steps

• Restructure existing definitions
• Add a new definition to your project