Resource-Aware ML Pipeline Scheduler

A serverless pipeline scheduler that executes machine learning workloads with appropriate compute resources based on task requirements.

Problem Statement

Build a pipeline scheduler that can run different types of ML tasks (data preprocessing, model training, and inference) on cloud infrastructure. Each task type requires different compute resources:

• Data preprocessing: CPU-intensive, requires 2 CPU cores and 2048 MiB memory
• Model training: GPU-accelerated, requires 1 GPU, 4 CPU cores, and 16384 MiB memory
• Inference: Lightweight, requires 0.5 CPU cores and 1024 MiB memory

The scheduler should configure appropriate compute resources for each task type and execute a simple pipeline that runs all three tasks in sequence.

Requirements

Resource Configuration

Your implementation must:

1. Create three separate serverless functions, one for each task type (preprocessing, training, inference)
2. Configure each function with the appropriate CPU and memory resources as specified above
3. Configure the training function with GPU resources
4. Ensure that each function prints a message indicating which task it's running and what resources it was configured with

Pipeline Execution

1. Create a main entry point that orchestrates the pipeline by calling the three functions in sequence:
   • First run data preprocessing
   • Then run model training
   • Finally run inference
2. Each function should accept a task_name parameter and return a completion message

Test Cases

• When preprocessing task is called with name "data_prep", it prints that it's running preprocessing and returns "data_prep completed" @test
• When training task is called with name "train_model", it prints that it's running training and returns "train_model completed" @test
• When inference task is called with name "run_inference", it prints that it's running inference and returns "run_inference completed" @test
• The main pipeline orchestration runs all three tasks in the correct sequence @test

Implementation

@generates

API

def preprocess_data(task_name: str) -> str:
    """
    Runs data preprocessing task with CPU-optimized resources.

    Args:
        task_name: Name of the preprocessing task

    Returns:
        Completion message
    """
    pass

def train_model(task_name: str) -> str:
    """
    Runs model training task with GPU resources.

    Args:
        task_name: Name of the training task

    Returns:
        Completion message
    """
    pass

def run_inference(task_name: str) -> str:
    """
    Runs inference task with lightweight resources.

    Args:
        task_name: Name of the inference task

    Returns:
        Completion message
    """
    pass

def run_pipeline() -> None:
    """
    Orchestrates the complete ML pipeline by running preprocessing,
    training, and inference tasks in sequence.
    """
    pass

Dependencies { .dependencies }

modal { .dependency }

Provides serverless cloud compute with configurable resources.