Dynamic Mapping & Collect (Experimental)¶

These APIs provide the means for a simple kind of dynamic orchestration — where the work to be orchestrated is determined not at pipeline definition time but at runtime, dependent on data that’s observed as part of pipeline execution.

class dagster.experimental.DynamicOutputDefinition(dagster_type=None, name=None, description=None, is_required=None, io_manager_key=None, metadata=None, asset_key=None, asset_partitions=None)[source]¶

(Experimental) Variant of OutputDefinition for an output that will dynamically alter the graph at runtime.

When using in a composition function such as @pipeline, dynamic outputs must be used with either

• map - clone downstream solids for each separate DynamicOutput

• collect - gather across all DynamicOutput in to a list

Uses the same constructor as OutputDefinition

@solid(
    config_schema={
        "path": Field(str, default_value=file_relative_path(__file__, "sample"))
    },
    output_defs=[DynamicOutputDefinition(str)],
)
def files_in_directory(context):
    path = context.solid_config["path"]
    dirname, _, filenames = next(os.walk(path))
    for file in filenames:
        yield DynamicOutput(os.path.join(dirname, file), mapping_key=_clean(file))

@pipeline
def process_directory():
    files = files_in_directory()

    # use map to invoke a solid on each dynamic output
    file_results = files.map(process_file)

    # use collect to gather the results in to a list
    summarize_directory(file_results.collect())

class dagster.experimental.DynamicOutput(value, mapping_key, output_name='result', metadata_entries=None, metadata=None)[source]¶

(Experimental) Variant of Output used to support dynamic mapping & collect. Each DynamicOutput produced by a solid represents one item in a set that can be processed individually with map or gathered with collect.

Each DynamicOutput must have a unique mapping_key to distinguish it with it’s set.

Parameters

• value (Any) – The value returned by the compute function.

• mapping_key (str) – The key that uniquely identifies this dynamic value relative to its peers. This key will be used to identify the downstream solids when mapped, ie mapped_solid[example_mapping_key]

• output_name (Optional[str]) – Name of the corresponding DynamicOutputDefinition defined on the solid. (default: “result”)

• metadata_entries (Optional[Union[EventMetadataEntry, PartitionMetadataEntry]]) – (Experimental) A set of metadata entries to attach to events related to this output.

• metadata (Optional[Dict[str, Union[str, float, int, Dict, EventMetadata]]]) – Arbitrary metadata about the failure. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a EventMetadata static method.