Pipeline Subset Extractor

A utility for extracting specific subsets of a data pipeline based on starting points, ending points, and data dependencies.

Capabilities

Extract downstream pipeline from specific nodes

Given a data pipeline with multiple nodes, extract all nodes that are downstream (dependent on) a specified set of starting nodes, including the starting nodes themselves.

• Given a pipeline with nodes A→B→C→D and E→F, calling the extraction with starting node "A" returns nodes A, B, C, D @test
• Given a pipeline with nodes A→B→C and D→E→C→F, calling the extraction with starting node "D" returns nodes D, E, C, F @test

Extract upstream pipeline to specific nodes

Given a data pipeline with multiple nodes, extract all nodes that are upstream (prerequisites for) a specified set of target nodes, including the target nodes themselves.

• Given a pipeline with nodes A→B→C→D, calling the extraction with target node "D" returns nodes A, B, C, D @test
• Given a pipeline with nodes A→B→D, A→C→D, and E→F, calling the extraction with target node "D" returns nodes A, B, C, D @test

Extract pipeline based on input datasets

Given a data pipeline with nodes consuming various datasets, extract all nodes that depend on specific input datasets (directly or transitively).

• Given a pipeline where node A produces "data1", node B consumes "data1" and produces "data2", and node C consumes "data2", extracting based on input "data1" returns nodes B and C @test

Extract pipeline based on output datasets

Given a data pipeline with nodes producing various datasets, extract all nodes required to produce specific output datasets.

• Given a pipeline where node A produces "intermediate", node B consumes "intermediate" and produces "final", extracting to output "final" returns nodes A and B @test

Implementation

@generates

API

def extract_downstream_from_nodes(pipeline, node_names):
    """
    Extract a pipeline subset containing specified nodes and all their downstream dependencies.

    Args:
        pipeline: A Kedro Pipeline object
        node_names: A list of node names to use as starting points

    Returns:
        A new Pipeline object containing only the selected nodes
    """
    pass


def extract_upstream_to_nodes(pipeline, node_names):
    """
    Extract a pipeline subset containing specified nodes and all their upstream dependencies.

    Args:
        pipeline: A Kedro Pipeline object
        node_names: A list of node names to use as target points

    Returns:
        A new Pipeline object containing only the selected nodes
    """
    pass


def extract_from_inputs(pipeline, input_names):
    """
    Extract a pipeline subset containing all nodes that depend on specified input datasets.

    Args:
        pipeline: A Kedro Pipeline object
        input_names: A list of dataset names to use as starting inputs

    Returns:
        A new Pipeline object containing only the selected nodes
    """
    pass


def extract_to_outputs(pipeline, output_names):
    """
    Extract a pipeline subset containing all nodes required to produce specified outputs.

    Args:
        pipeline: A Kedro Pipeline object
        output_names: A list of dataset names to use as target outputs

    Returns:
        A new Pipeline object containing only the selected nodes
    """
    pass

Dependencies { .dependencies }

kedro { .dependency }

Provides pipeline construction and manipulation capabilities.

@satisfied-by