Pipeline Filter Utility

A utility module that provides flexible pipeline filtering capabilities for data processing workflows.

Capabilities

Basic Pipeline Filtering

• Given a pipeline with 5 nodes (A, B, C, D, E) where node C has tags ["transform", "critical"] and node D has tags ["transform"], filtering by tag "transform" returns a pipeline containing only nodes C and D @test
• Given a pipeline where node B depends on node A and node C depends on node B, filtering from node B returns a pipeline containing nodes B and C but not A @test
• Given a pipeline where node C produces output "processed_data", filtering to outputs "processed_data" returns a pipeline containing all nodes required to produce that output @test

Combined Filtering

• Given a pipeline with nodes tagged variously, filtering with both tags=["transform"] and from_nodes=["node_b"] returns only nodes that satisfy both conditions @test

Implementation

@generates

API

def filter_pipeline(pipeline, tags=None, from_nodes=None, to_nodes=None, from_inputs=None, to_outputs=None):
    """
    Filter a pipeline using multiple criteria combined with AND logic.

    Args:
        pipeline: The Kedro pipeline to filter
        tags: List of tags - include nodes with any of these tags
        from_nodes: List of node names - include these nodes and all downstream
        to_nodes: List of node names - include these nodes and all upstream
        from_inputs: List of input names - include nodes depending on these inputs
        to_outputs: List of output names - include nodes required for these outputs

    Returns:
        A filtered pipeline containing only nodes matching all specified criteria
    """
    pass

Dependencies { .dependencies }

kedro { .dependency }

Provides pipeline construction and filtering capabilities.