Atomic Target Pipeline

A small workflow task that uppercases lines from a newline-delimited text file while relying on the workflow package's target abstractions for atomic writes and format-aware I/O.

Capabilities

Transparent target reads

• When input_path ends with .gz, the task reads it without manual decompression and writes processed output identical to handling a plain text input. @test

Atomic processed output

• If fail_midway=True, the run intentionally stops after handling the first line to simulate a crash and leaves no processed or backup files on disk. @test
• With fail_midway=False, the task writes the processed file atomically so the file only appears after processing completes. @test

Backup target with format

• After a successful run, the task creates a compressed backup target whose decompressed contents exactly match the processed file. @test

Output declarations

• The task's output() returns a mapping with keys processed and backup, each pointing to target handles for output_path and backup_path respectively. @test

Implementation

@generates

API

class ProcessAndBackupTask:
    def __init__(self, input_path: str, output_path: str, backup_path: str, fail_midway: bool = False): ...
    def requires(self) -> object: ...
    def output(self) -> dict: ...
    def run(self) -> None: ...

Dependencies { .dependencies }

luigi { .dependency }

Workflow orchestration library providing target abstractions, atomic read/write helpers, and optional gzip formats.