Scheduler-Aware Pipeline Runner

Build a command-line utility that runs a tiny two-stage data pipeline using the package's execution entrypoints. The pipeline has an upstream task that materializes a small dataset and a downstream task that consumes that dataset. The runner must support both embedded execution and coordination with a central scheduler.

Capabilities

Local execution with dependency resolution

• Running the runner with a flag that keeps scheduling local (for example, --local-only) executes the upstream task before the downstream task and exits with code 0 once both succeed. @test

Remote scheduler selection and failure handling

• Invoking the runner with scheduler host and port arguments attempts to coordinate with a remote scheduler; when the scheduler is unreachable the process aborts before tasks run, prints an unreachable notice, and exits non-zero. @test

Worker limit and status summary

• Providing a worker-count option limits concurrent task execution to that number and emits a completion summary that includes counts of completed, failed, and pending tasks. @test

Exit codes reflect task failures

• When the upstream task raises an exception, the runner reports the failed task in its summary and exits non-zero. @test

Implementation

@generates

API

def main(argv: list[str] | None = None) -> int:
    """Entry point for invoking the pipeline runner from CLI; returns process exit code."""

Dependencies { .dependencies }

luigi { .dependency }

Provides task definitions, dependency resolution, scheduler coordination, and CLI/build entrypoints.