Tuned Sqoop Import Job

Create a helper that runs a relational-to-HDFS import focused on mapper parallelism, split tuning, and per-run JVM/system properties. The helper should keep configuration minimal while surfacing the tuning controls.

Capabilities

Mapper parallelism and split control

• When given a mapper count above 1 and a split column, the helper runs the import with those values and returns a command preview that includes both. @test
• If a split column is not provided, the helper forces a single mapper even if a higher count was requested, preventing unbalanced shards. @test

JVM property injection

• Accepts a dictionary of JVM/system properties and applies each as -Dkey=value for the import run; the command preview lists each injected property. @test

Result reporting

• Returns structured output containing the chosen target directory and the redacted command used for the run so logs avoid exposing credentials. @test

Implementation

@generates

API

from typing import Any, Dict, Optional, TypedDict


class ImportRequest(TypedDict):
    conn_id: str
    table: str
    target_dir: str
    num_mappers: Optional[int]
    split_by: Optional[str]
    properties: Optional[Dict[str, str]]


class ImportResult(TypedDict):
    target_dir: str
    command_preview: str


def run_tuned_import(request: ImportRequest) -> ImportResult:
    """Runs an import using the Sqoop provider with mapper tuning and JVM property injection."""

Dependencies { .dependencies }

apache-airflow-backport-providers-apache-sqoop { .dependency }

Provides Sqoop import functionality with mapper tuning and JVM property flags.