Direct Import Overrides for Sqoop

Design a data ingestion helper that leverages the Sqoop integration to import a relational table into HDFS while exposing toggles for direct connectors, JDBC driver overrides, and passthrough options. The helper should favor concise configuration while ensuring each toggle is honored in the executed import.

Capabilities

Direct connector toggle

• When a caller enables direct mode, the import runs via the direct connector; when disabled, it falls back to the standard path while still completing the import to the target directory. @test

JDBC driver override

• If a custom JDBC driver class name is provided, the import uses it instead of any default driver; when omitted, the import proceeds with defaults. @test

Passthrough Sqoop options

• Extra Sqoop CLI options provided as key-value pairs are forwarded unchanged into the import invocation, allowing flags such as compression or delimiter tweaks to take effect. @test

Combined configuration

• When direct mode, a custom driver, and extra options are all supplied, the resulting import invocation retains every setting and still writes the table into the requested target directory. @test

Implementation

@generates

API

from typing import Mapping

def run_direct_table_import(
    connection_uri: str,
    table: str,
    target_dir: str,
    *,
    use_direct: bool = False,
    jdbc_driver: str | None = None,
    extra_options: Mapping[str, str] | None = None,
) -> None:
    """
    Launches a Sqoop table import into the given target directory using the package's Sqoop facilities.
    - Enables or disables the direct connector based on use_direct.
    - Overrides the JDBC driver when jdbc_driver is provided.
    - Forwards extra Sqoop CLI options exactly as supplied.
    """

Dependencies { .dependencies }

apache-airflow-providers-apache-sqoop { .dependency }

Provides Sqoop import and execution utilities integrated with Airflow connections.