Create a New Project#
This section will show you how to a create a new Dagster project and organize your files as you build larger and larger pipelines. Dagster comes with a convenient CLI command for generating a project skeleton, but you can also choose to organize your files differently as your project evolves.
If you're completely new to Dagster, we recommend that you visit our Tutorial to learn all the basic concepts of Dagster.
Generating a Project Skeleton#
If you're just starting a new Dagster project, the CLI command dagster new-project will generate a project skeleton with boilerplate code for development and testing. If you have dagster installed in your Python environment, then you can run the following shell command to generate a Dagster project called PROJECT_NAME:
dagster new-project PROJECT_NAME
cd PROJECT_NAME
The newly generated PROJECT_NAME directory is in fact a fully functioning Python package and can be installed with pip. This Python package is considered a repository location in Dagster. A single Python file can also be a repository location, like in the Hello World example. For more details about different types of repository locations, visit the Repositories overview.
Here's a breakdown of the files and directories that are generated:
| File/Directory | Description |
|---|---|
PROJECT_NAME/ | A Python package that contains code for your new Dagster repository |
PROJECT_NAME_tests/ | A Python package that contains tests for PROJECT_NAME |
workspace.yaml | A file that specifies the location of the user code for Dagit and the Dagster CLI. Visit the Workspaces overview for more details. |
README.md | A description and guide for your new code repository |
setup.py | A build script with Python package dependencies for your new code repository |
Inside of the directory PROJECT_NAME/, the following files and directories are generated:
| File/Directory | Description |
|---|---|
PROJECT_NAME/solids/ | A Python package that contains SolidDefinitions, which represent individual units of computation |
PROJECT_NAME/pipelines/ | A Python package that contains PipelineDefinitions, which are built up from solids |
PROJECT_NAME/schedules/ | A Python package that contains ScheduleDefinitions, to trigger recurring pipeline runs based on time |
PROJECT_NAME/sensors/ | A Python package that contains SensorDefinitions, to trigger pipeline runs based on external state |
PROJECT_NAME/repository.py | A Python module that contains a RepositoryDefinition, to specify which pipelines, schedules, and sensors are available in your repository |
This file structure is a good starting point and suitable for most Dagster projects. As you build more and more pipelines, you may eventually find your own way of structuring your code that works best for you.
Local Development#
- Install your repository as a Python package. By using the
--editableflag,pipwill install your repository in "editable mode" so that as you develop, local code changes will automatically apply.
pip install --editable .
- Start the Dagit process. This will start a Dagit web server that, by default, is served on http://localhost:3000.
dagit
The Dagit process automatically uses the file workspace.yaml to find your repository location(s), from which Dagster will load your pipelines, schedules, and sensors. To see how you can customize the Dagit process, run dagit --help.
- (Optional) If you want to enable Dagster Schedules or Sensors for your pipelines, start the Dagster Daemon process in a different shell or terminal:
dagster-daemon run
Once your Dagster Daemon process is running, you should be able to enable schedules and sensors for your Dagster pipelines in your repository.
Local Testing#
Once you have created a new Dagster repository with the CLI command dagster new-project, you can find tests in PROJECT_NAME_tests, where PROJECT_NAME is the name of your project. You can run all of your tests with the following command:
pytest PROJECT_NAME_tests
As you create Dagster solids and pipelines, add tests in PROJECT_NAME_tests/ to check that your code behaves as desired and does not break over time.
For hints on how to write tests for solids and pipelines in Dagster, see our documentation tutorial on Testing.
Deployment#
Once your Dagster project is ready, visit the Deployment Guides to learn how to run Dagster in production environments, such as Docker, Kubernetes, AWS EC2, etc.