Pipeline Subset Executor

A utility that enables selective execution of portions of a data pipeline based on specific input and output datasets.

Capabilities

Extract pipeline nodes by input datasets

• Given a pipeline and a list of input dataset names, returns a new pipeline containing only the nodes that depend on at least one of the specified inputs, along with all their downstream dependencies @test
• Returns an empty pipeline when the specified input datasets are not consumed by any nodes in the original pipeline @test

Extract pipeline nodes by output datasets

• Given a pipeline and a list of output dataset names, returns a new pipeline containing only the nodes required to produce the specified outputs, along with all their upstream dependencies @test
• Returns an empty pipeline when the specified output datasets are not produced by any nodes in the original pipeline @test

Implementation

@generates

API

def extract_pipeline_from_inputs(pipeline, input_names):
    """
    Extract a subset of the pipeline starting from specified input datasets.

    Args:
        pipeline: A Kedro Pipeline object
        input_names: A list of dataset names to start from

    Returns:
        A new Pipeline containing nodes that depend on the specified inputs
        and all their downstream dependencies
    """
    pass


def extract_pipeline_to_outputs(pipeline, output_names):
    """
    Extract a subset of the pipeline required to produce specified outputs.

    Args:
        pipeline: A Kedro Pipeline object
        output_names: A list of dataset names to produce

    Returns:
        A new Pipeline containing nodes required to produce the specified
        outputs and all their upstream dependencies
    """
    pass

Dependencies { .dependencies }

kedro { .dependency }

Provides pipeline construction and filtering capabilities.

@satisfied-by