or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

data-sources.mddata-types.mdindex.mdscheduling.mdservice-clients.mdtransfer-configs.mdtransfer-runs.md
tile.json

tessl/pypi-google-cloud-bigquery-datatransfer

Python client library for managing BigQuery Data Transfer Service operations and scheduling data transfers from partner SaaS applications.

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/google-cloud-bigquery-datatransfer@3.19.x

To install, run

npx @tessl/cli install tessl/pypi-google-cloud-bigquery-datatransfer@3.19.0

index.mddocs/

Google Cloud BigQuery Data Transfer

A Python client library for Google Cloud BigQuery Data Transfer Service, enabling developers to programmatically manage scheduled data transfers from partner SaaS applications to Google BigQuery. The library provides comprehensive API access for creating, managing, and monitoring data transfer configurations with support for various data sources and automated scheduling capabilities.

Package Information

  • Package Name: google-cloud-bigquery-datatransfer
  • Language: Python
  • Installation: pip install google-cloud-bigquery-datatransfer

Core Imports

from google.cloud import bigquery_datatransfer

For direct access to versioned API:

from google.cloud import bigquery_datatransfer_v1

Import specific components:

from google.cloud.bigquery_datatransfer import (
    DataTransferServiceClient,
    DataTransferServiceAsyncClient,
    TransferConfig,
    TransferRun,
    DataSource,
)

For working with pager results:

from google.cloud.bigquery_datatransfer_v1.services.data_transfer_service import pagers

Basic Usage

from google.cloud import bigquery_datatransfer

# Initialize the client
client = bigquery_datatransfer.DataTransferServiceClient()

# List available data sources
parent = f"projects/{project_id}/locations/{location}"
data_sources = client.list_data_sources(parent=parent)

for data_source in data_sources:
    print(f"Data Source: {data_source.display_name}")
    print(f"ID: {data_source.data_source_id}")

# Create a transfer configuration
transfer_config = {
    "display_name": "My Transfer Config",
    "data_source_id": "scheduled_query",
    "destination_dataset_id": "my_dataset",
    "params": {
        "query": "SELECT * FROM source_table",
        "destination_table_name_template": "destination_table_{run_date}",
    },
}

response = client.create_transfer_config(
    parent=parent,
    transfer_config=transfer_config
)
print(f"Created transfer config: {response.name}")

Architecture

The library follows Google Cloud client library patterns with these key components:

  • Service Clients: Main interfaces for API operations (sync and async versions)
  • Request/Response Types: Structured data for API communication
  • Resource Types: Core entities like TransferConfig, TransferRun, DataSource
  • Authentication: Integrated with Google Cloud authentication and authorization
  • Transport Layer: Support for both gRPC and REST protocols with retry logic

Capabilities

Service Clients

Primary client interfaces for interacting with BigQuery Data Transfer Service, including both synchronous and asynchronous versions with complete CRUD operations for transfer configurations and monitoring capabilities.

class DataTransferServiceClient:
    def __init__(self, **kwargs): ...

class DataTransferServiceAsyncClient:
    def __init__(self, **kwargs): ...

Service Clients

Data Source Management

Operations for discovering and managing available data sources, including listing supported sources, retrieving source configurations, and validating credentials for data source connections.

def get_data_source(request: GetDataSourceRequest) -> DataSource: ...
def list_data_sources(request: ListDataSourcesRequest) -> pagers.ListDataSourcesPager: ...
def check_valid_creds(request: CheckValidCredsRequest) -> CheckValidCredsResponse: ...

Data Source Management

Transfer Configuration Management

Complete lifecycle management of data transfer configurations, including creation, updates, deletion, and listing operations with support for various scheduling options and data source parameters.

def create_transfer_config(request: CreateTransferConfigRequest) -> TransferConfig: ...
def update_transfer_config(request: UpdateTransferConfigRequest) -> TransferConfig: ...
def delete_transfer_config(request: DeleteTransferConfigRequest) -> None: ...
def get_transfer_config(request: GetTransferConfigRequest) -> TransferConfig: ...
def list_transfer_configs(request: ListTransferConfigsRequest) -> pagers.ListTransferConfigsPager: ...

Transfer Configuration Management

Transfer Run Management

Management and monitoring of individual transfer executions, including manual execution, status monitoring, log retrieval, and run lifecycle operations.

def get_transfer_run(request: GetTransferRunRequest) -> TransferRun: ...
def delete_transfer_run(request: DeleteTransferRunRequest) -> None: ...
def list_transfer_runs(request: ListTransferRunsRequest) -> pagers.ListTransferRunsPager: ...
def start_manual_transfer_runs(request: StartManualTransferRunsRequest) -> StartManualTransferRunsResponse: ...
def schedule_transfer_runs(request: ScheduleTransferRunsRequest) -> ScheduleTransferRunsResponse: ...
def list_transfer_logs(request: ListTransferLogsRequest) -> pagers.ListTransferLogsPager: ...

Transfer Run Management

Core Data Types

Essential data structures representing transfer configurations, runs, data sources, and related metadata used throughout the API for defining and managing data transfer operations.

class TransferConfig: ...
class TransferRun: ...
class DataSource: ...
class DataSourceParameter: ...
class TransferMessage: ...

Core Data Types

Scheduling and Configuration

Comprehensive scheduling options and configuration types for defining when and how data transfers execute, including time-based, manual, and event-driven scheduling patterns.

class ScheduleOptions: ...
class ScheduleOptionsV2: ...
class TimeBasedSchedule: ...
class ManualSchedule: ...
class EventDrivenSchedule: ...

Scheduling and Configuration

Types

Core Enums

class TransferState(Enum):
    """Transfer run state enumeration."""
    TRANSFER_STATE_UNSPECIFIED = 0
    PENDING = 2
    RUNNING = 3
    SUCCEEDED = 4
    FAILED = 5
    CANCELLED = 6

class TransferType(Enum):
    """Deprecated transfer type enumeration."""
    TRANSFER_TYPE_UNSPECIFIED = 0
    BATCH = 1
    STREAMING = 2

Resource Path Helpers

@staticmethod
def data_source_path(project: str, location: str, data_source: str) -> str: ...

@staticmethod
def parse_data_source_path(path: str) -> Dict[str, str]: ...

@staticmethod
def transfer_config_path(project: str, location: str, transfer_config: str) -> str: ...

@staticmethod
def parse_transfer_config_path(path: str) -> Dict[str, str]: ...

@staticmethod
def run_path(project: str, location: str, transfer_config: str, run: str) -> str: ...

@staticmethod
def parse_run_path(path: str) -> Dict[str, str]: ...