Data Pipeline Executor

Build a data processing application that executes a multi-step data transformation pipeline in sequential order.

Requirements

Your application should implement a data processing pipeline with the following steps:

1. Load raw data: Read data from a source
2. Clean data: Remove invalid entries and normalize values
3. Transform data: Apply business logic transformations
4. Aggregate results: Compute summary statistics
5. Save output: Write the final results

The pipeline should:

• Execute all steps in the correct dependency order
• Handle data passing between steps automatically
• Provide a way to run the entire pipeline with a single command
• Support both in-memory data storage and persistent storage

Test Cases

• The pipeline executes all nodes in the correct topological order based on their dependencies @test
• Data flows correctly from one step to the next through the data catalog @test
• The pipeline can be run with a sequential execution strategy @test
• Intermediate results are accessible through the catalog after pipeline execution @test

Implementation

@generates

API

def create_pipeline():
    """
    Create and return a data processing pipeline.

    Returns:
        A pipeline object containing all processing nodes
    """
    pass

def run_pipeline(pipeline, catalog, runner):
    """
    Execute the pipeline using the provided runner and catalog.

    Args:
        pipeline: The pipeline to execute
        catalog: Data catalog for loading and saving datasets
        runner: The runner to use for execution

    Returns:
        Execution results or output data
    """
    pass

def create_catalog():
    """
    Create and return a data catalog with all required datasets.

    Returns:
        A catalog object containing dataset definitions
    """
    pass

Dependencies { .dependencies }

kedro { .dependency }

Provides data pipeline framework and execution capabilities.