0
# Google Cloud BigQuery Data Transfer
1

2
A Python client library for Google Cloud BigQuery Data Transfer Service, enabling developers to programmatically manage scheduled data transfers from partner SaaS applications to Google BigQuery. The library provides comprehensive API access for creating, managing, and monitoring data transfer configurations with support for various data sources and automated scheduling capabilities.
3

4
## Package Information
5

6
- **Package Name**: google-cloud-bigquery-datatransfer
7
- **Language**: Python
8
- **Installation**: `pip install google-cloud-bigquery-datatransfer`
9

10
## Core Imports
11

12
```python
13
from google.cloud import bigquery_datatransfer
14
```
15

16
For direct access to versioned API:
17

18
```python
19
from google.cloud import bigquery_datatransfer_v1
20
```
21

22
Import specific components:
23

24
```python
25
from google.cloud.bigquery_datatransfer import (
26
    DataTransferServiceClient,
27
    DataTransferServiceAsyncClient,
28
    TransferConfig,
29
    TransferRun,
30
    DataSource,
31
)
32
```
33

34
For working with pager results:
35

36
```python
37
from google.cloud.bigquery_datatransfer_v1.services.data_transfer_service import pagers
38
```
39

40
## Basic Usage
41

42
```python
43
from google.cloud import bigquery_datatransfer
44

45
# Initialize the client
46
client = bigquery_datatransfer.DataTransferServiceClient()
47

48
# List available data sources
49
parent = f"projects/{project_id}/locations/{location}"
50
data_sources = client.list_data_sources(parent=parent)
51

52
for data_source in data_sources:
53
    print(f"Data Source: {data_source.display_name}")
54
    print(f"ID: {data_source.data_source_id}")
55

56
# Create a transfer configuration
57
transfer_config = {
58
    "display_name": "My Transfer Config",
59
    "data_source_id": "scheduled_query",
60
    "destination_dataset_id": "my_dataset",
61
    "params": {
62
        "query": "SELECT * FROM source_table",
63
        "destination_table_name_template": "destination_table_{run_date}",
64
    },
65
}
66

67
response = client.create_transfer_config(
68
    parent=parent,
69
    transfer_config=transfer_config
70
)
71
print(f"Created transfer config: {response.name}")
72
```
73

74
## Architecture
75

76
The library follows Google Cloud client library patterns with these key components:
77

78
- **Service Clients**: Main interfaces for API operations (sync and async versions)
79
- **Request/Response Types**: Structured data for API communication
80
- **Resource Types**: Core entities like TransferConfig, TransferRun, DataSource
81
- **Authentication**: Integrated with Google Cloud authentication and authorization
82
- **Transport Layer**: Support for both gRPC and REST protocols with retry logic
83

84
## Capabilities
85

86
### Service Clients
87

88
Primary client interfaces for interacting with BigQuery Data Transfer Service, including both synchronous and asynchronous versions with complete CRUD operations for transfer configurations and monitoring capabilities.
89

90
```python { .api }
91
class DataTransferServiceClient:
92
    def __init__(self, **kwargs): ...
93

94
class DataTransferServiceAsyncClient:
95
    def __init__(self, **kwargs): ...
96
```
97

98
[Service Clients](./service-clients.md)
99

100
### Data Source Management
101

102
Operations for discovering and managing available data sources, including listing supported sources, retrieving source configurations, and validating credentials for data source connections.
103

104
```python { .api }
105
def get_data_source(request: GetDataSourceRequest) -> DataSource: ...
106
def list_data_sources(request: ListDataSourcesRequest) -> pagers.ListDataSourcesPager: ...
107
def check_valid_creds(request: CheckValidCredsRequest) -> CheckValidCredsResponse: ...
108
```
109

110
[Data Source Management](./data-sources.md)
111

112
### Transfer Configuration Management
113

114
Complete lifecycle management of data transfer configurations, including creation, updates, deletion, and listing operations with support for various scheduling options and data source parameters.
115

116
```python { .api }
117
def create_transfer_config(request: CreateTransferConfigRequest) -> TransferConfig: ...
118
def update_transfer_config(request: UpdateTransferConfigRequest) -> TransferConfig: ...
119
def delete_transfer_config(request: DeleteTransferConfigRequest) -> None: ...
120
def get_transfer_config(request: GetTransferConfigRequest) -> TransferConfig: ...
121
def list_transfer_configs(request: ListTransferConfigsRequest) -> pagers.ListTransferConfigsPager: ...
122
```
123

124
[Transfer Configuration Management](./transfer-configs.md)
125

126
### Transfer Run Management
127

128
Management and monitoring of individual transfer executions, including manual execution, status monitoring, log retrieval, and run lifecycle operations.
129

130
```python { .api }
131
def get_transfer_run(request: GetTransferRunRequest) -> TransferRun: ...
132
def delete_transfer_run(request: DeleteTransferRunRequest) -> None: ...
133
def list_transfer_runs(request: ListTransferRunsRequest) -> pagers.ListTransferRunsPager: ...
134
def start_manual_transfer_runs(request: StartManualTransferRunsRequest) -> StartManualTransferRunsResponse: ...
135
def schedule_transfer_runs(request: ScheduleTransferRunsRequest) -> ScheduleTransferRunsResponse: ...
136
def list_transfer_logs(request: ListTransferLogsRequest) -> pagers.ListTransferLogsPager: ...
137
```
138

139
[Transfer Run Management](./transfer-runs.md)
140

141
### Core Data Types
142

143
Essential data structures representing transfer configurations, runs, data sources, and related metadata used throughout the API for defining and managing data transfer operations.
144

145
```python { .api }
146
class TransferConfig: ...
147
class TransferRun: ...
148
class DataSource: ...
149
class DataSourceParameter: ...
150
class TransferMessage: ...
151
```
152

153
[Core Data Types](./data-types.md)
154

155
### Scheduling and Configuration
156

157
Comprehensive scheduling options and configuration types for defining when and how data transfers execute, including time-based, manual, and event-driven scheduling patterns.
158

159
```python { .api }
160
class ScheduleOptions: ...
161
class ScheduleOptionsV2: ...
162
class TimeBasedSchedule: ...
163
class ManualSchedule: ...
164
class EventDrivenSchedule: ...
165
```
166

167
[Scheduling and Configuration](./scheduling.md)
168

169
## Types
170

171
### Core Enums
172

173
```python { .api }
174
class TransferState(Enum):
175
    """Transfer run state enumeration."""
176
    TRANSFER_STATE_UNSPECIFIED = 0
177
    PENDING = 2
178
    RUNNING = 3
179
    SUCCEEDED = 4
180
    FAILED = 5
181
    CANCELLED = 6
182

183
class TransferType(Enum):
184
    """Deprecated transfer type enumeration."""
185
    TRANSFER_TYPE_UNSPECIFIED = 0
186
    BATCH = 1
187
    STREAMING = 2
188
```
189

190
### Resource Path Helpers
191

192
```python { .api }
193
@staticmethod
194
def data_source_path(project: str, location: str, data_source: str) -> str: ...
195

196
@staticmethod
197
def parse_data_source_path(path: str) -> Dict[str, str]: ...
198

199
@staticmethod
200
def transfer_config_path(project: str, location: str, transfer_config: str) -> str: ...
201

202
@staticmethod
203
def parse_transfer_config_path(path: str) -> Dict[str, str]: ...
204

205
@staticmethod
206
def run_path(project: str, location: str, transfer_config: str, run: str) -> str: ...
207

208
@staticmethod
209
def parse_run_path(path: str) -> Dict[str, str]: ...
210
```