0
# Core Data Types
1

2
Essential data structures representing transfer configurations, runs, data sources, and related metadata used throughout the API for defining and managing data transfer operations.
3

4
## Capabilities
5

6
### TransferConfig
7

8
Represents a data transfer configuration including destination, schedule, parameters, and authorization details.
9

10
```python { .api }
11
class TransferConfig:
12
    """
13
    Represents a data transfer configuration.
14
    
15
    Attributes:
16
        name (str): Output only. Identifier for the transfer configuration resource.
17
        display_name (str): The user specified display name for the transfer config.
18
        destination_dataset_id (str): The BigQuery target dataset id.
19
        data_source_id (str): Data source id.
20
        params (Struct): Data transfer specific parameters.
21
        schedule (str): Data transfer schedule in unix-cron format.
22
        schedule_options (ScheduleOptions): Options customizing the data transfer schedule.
23
        data_refresh_window_days (int): The number of days to look back to automatically refresh data.
24
        disabled (bool): Is this config disabled.
25
        update_time (Timestamp): Output only. Data transfer modification time.
26
        next_run_time (Timestamp): Output only. Next time when data transfer will run.
27
        state (TransferState): Output only. State of the most recently updated transfer run.
28
        user_id (int): Deprecated. Unique ID of the user on whose behalf transfer is done.
29
        dataset_region (str): Output only. Region in which BigQuery dataset is located.
30
        notification_pubsub_topic (str): Pub/Sub topic where notifications will be sent.
31
        email_preferences (EmailPreferences): Email notification preferences.
32
        owner_info (UserInfo): Output only. Information about the user for transfer run.
33
        encryption_configuration (EncryptionConfiguration): The encryption configuration part.
34
    """
35
    name: str
36
    display_name: str
37
    destination_dataset_id: str
38
    data_source_id: str
39
    params: struct_pb2.Struct
40
    schedule: str
41
    schedule_options: ScheduleOptions
42
    data_refresh_window_days: int
43
    disabled: bool
44
    update_time: timestamp_pb2.Timestamp
45
    next_run_time: timestamp_pb2.Timestamp
46
    state: TransferState
47
    user_id: int
48
    dataset_region: str
49
    notification_pubsub_topic: str
50
    email_preferences: EmailPreferences
51
    owner_info: UserInfo
52
    encryption_configuration: EncryptionConfiguration
53
```
54

55
### TransferRun
56

57
Represents a data transfer run with execution status, timing, and error information.
58

59
```python { .api }
60
class TransferRun:
61
    """
62
    Represents a data transfer run.
63
    
64
    Attributes:
65
        name (str): Identifier for the transfer run resource.
66
        schedule_time (Timestamp): Minimum time after which a transfer run can be started.
67
        run_time (Timestamp): For batch transfer runs, specifies the date and time of the data.
68
        error_status (Status): Status of the transfer run.
69
        start_time (Timestamp): Output only. Time when transfer run was started.
70
        end_time (Timestamp): Output only. Time when transfer run ended.
71
        update_time (Timestamp): Output only. Last time the data transfer run state was updated.
72
        params (Struct): Output only. Data transfer specific parameters.
73
        destination_dataset_id (str): Output only. The BigQuery target dataset id.
74
        data_source_id (str): Output only. Data source id.
75
        state (TransferState): Data transfer run state.
76
        user_id (int): Deprecated. Unique ID of the user on whose behalf transfer is done.
77
        schedule (str): Output only. Describes the schedule of this transfer run.
78
        notification_pubsub_topic (str): Output only. Pub/Sub topic where notifications will be sent.
79
        email_preferences (EmailPreferences): Output only. Email notification preferences.
80
    """
81
    name: str
82
    schedule_time: timestamp_pb2.Timestamp
83
    run_time: timestamp_pb2.Timestamp
84
    error_status: status_pb2.Status
85
    start_time: timestamp_pb2.Timestamp
86
    end_time: timestamp_pb2.Timestamp
87
    update_time: timestamp_pb2.Timestamp
88
    params: struct_pb2.Struct
89
    destination_dataset_id: str
90
    data_source_id: str
91
    state: TransferState
92
    user_id: int
93
    schedule: str
94
    notification_pubsub_topic: str
95
    email_preferences: EmailPreferences
96
```
97

98
### DataSource
99

100
Represents a data source that can be used in BigQuery Data Transfer.
101

102
```python { .api }
103
class DataSource:
104
    """
105
    Represents a data source.
106
    
107
    Attributes:
108
        name (str): Output only. Data source resource name.
109
        data_source_id (str): Data source id.
110
        display_name (str): User friendly display name of the data source.
111
        description (str): User friendly data source description string.
112
        client_id (str): Data source client id.
113
        scopes (Sequence[str]): Api auth scopes for which refresh token needs to be obtained.
114
        transfer_type (TransferType): Deprecated. This field has no effect.
115
        supports_multiple_transfers (bool): Indicates whether the data source supports multiple transfers.
116
        update_deadline_seconds (int): The number of seconds to wait for a status update.
117
        default_schedule (str): Default data transfer schedule.
118
        supports_custom_schedule (bool): Specifies whether the data source supports a user defined schedule.
119
        parameters (Sequence[DataSourceParameter]): Data source parameters.
120
        help_url (str): Url for the help document for this data source.
121
        authorization_type (AuthorizationType): Indicates the type of authorization.
122
        data_refresh_type (DataRefreshType): Specifies whether the data source supports automatic data refresh.
123
        default_data_refresh_window_days (int): Default data refresh window on days.
124
        manual_runs_disabled (bool): Disables backfill and manual run functionality.
125
        minimum_schedule_interval (Duration): The minimum interval for scheduler to schedule runs.
126
    """
127
    name: str
128
    data_source_id: str
129
    display_name: str
130
    description: str
131
    client_id: str
132
    scopes: Sequence[str]
133
    transfer_type: TransferType
134
    supports_multiple_transfers: bool
135
    update_deadline_seconds: int
136
    default_schedule: str
137
    supports_custom_schedule: bool
138
    parameters: Sequence[DataSourceParameter]
139
    help_url: str
140
    authorization_type: AuthorizationType
141
    data_refresh_type: DataRefreshType
142
    default_data_refresh_window_days: int
143
    manual_runs_disabled: bool
144
    minimum_schedule_interval: duration_pb2.Duration
145

146
    class AuthorizationType(proto.Enum):
147
        """The type of authorization needed for this data source."""
148
        AUTHORIZATION_TYPE_UNSPECIFIED = 0
149
        AUTHORIZATION_CODE = 1
150
        GOOGLE_PLUS_AUTHORIZATION_CODE = 2
151
        FIRST_PARTY_OAUTH = 3
152

153
    class DataRefreshType(proto.Enum):
154
        """Represents how the data source supports data auto refresh."""
155
        DATA_REFRESH_TYPE_UNSPECIFIED = 0
156
        SLIDING_WINDOW = 1
157
        CUSTOM_SLIDING_WINDOW = 2
158
```
159

160
### DataSourceParameter
161

162
Represents a parameter used to define custom fields in a data source definition.
163

164
```python { .api }
165
class DataSourceParameter:
166
    """
167
    A parameter used to define custom fields in a data source definition.
168
    
169
    Attributes:
170
        param_id (str): Parameter identifier.
171
        display_name (str): Parameter display name in the user interface.
172
        description (str): Parameter description.
173
        type_ (Type): Parameter type.
174
        required (bool): Is parameter required.
175
        repeated (bool): Deprecated. This field has no effect.
176
        validation_regex (str): Regular expression which can be used for parameter validation.
177
        allowed_values (Sequence[str]): All possible values for the parameter.
178
        min_value (DoubleValue): For integer and double values specifies minimum allowed value.
179
        max_value (DoubleValue): For integer and double values specifies maximum allowed value.
180
        fields (Sequence[DataSourceParameter]): Deprecated. This field has no effect.
181
        validation_description (str): Description of the requirements for this field.
182
        validation_help_url (str): URL to a help document to further explain the naming requirements.
183
        immutable (bool): Cannot be changed after initial creation.
184
        recurse (bool): Deprecated. This field has no effect.
185
    """
186
    param_id: str
187
    display_name: str
188
    description: str
189
    type_: Type
190
    required: bool
191
    repeated: bool
192
    validation_regex: str
193
    allowed_values: Sequence[str]
194
    min_value: wrappers_pb2.DoubleValue
195
    max_value: wrappers_pb2.DoubleValue
196
    fields: Sequence[DataSourceParameter]
197
    validation_description: str
198
    validation_help_url: str
199
    immutable: bool
200
    recurse: bool
201

202
    class Type(proto.Enum):
203
        """Parameter types."""
204
        TYPE_UNSPECIFIED = 0
205
        STRING = 1
206
        INTEGER = 2
207
        DOUBLE = 3
208
        BOOLEAN = 4
209
        RECORD = 5
210
        PLUS_PAGE = 6
211
        LIST = 7
212
```
213

214
### TransferMessage
215

216
Represents a user-facing message for a particular data transfer run.
217

218
```python { .api }
219
class TransferMessage:
220
    """
221
    Represents a user-facing message for a particular data transfer run.
222
    
223
    Attributes:
224
        message_time (Timestamp): Time when message was logged.
225
        severity (MessageSeverity): Message severity.
226
        message_text (str): Message text.
227
    """
228
    message_time: timestamp_pb2.Timestamp
229
    severity: MessageSeverity
230
    message_text: str
231

232
    class MessageSeverity(proto.Enum):
233
        """Message severity levels."""
234
        MESSAGE_SEVERITY_UNSPECIFIED = 0
235
        INFO = 1
236
        WARNING = 2
237
        ERROR = 3
238
```
239

240
### UserInfo
241

242
Information about the user for whom the transfer config was created.
243

244
```python { .api }
245
class UserInfo:
246
    """
247
    Information about the user for whom the transfer config was created.
248
    
249
    Attributes:
250
        email (str): E-mail address of the user.
251
    """
252
    email: str
253
```
254

255
## Usage Examples
256

257
### Working with TransferConfig
258

259
```python
260
from google.cloud import bigquery_datatransfer
261
from google.protobuf import struct_pb2
262

263
client = bigquery_datatransfer.DataTransferServiceClient()
264

265
# Create parameters for scheduled query
266
params = struct_pb2.Struct()
267
params.update({
268
    "query": "SELECT * FROM `project.dataset.table` WHERE date = @run_date",
269
    "destination_table_name_template": "results_{run_date}",
270
    "use_legacy_sql": False
271
})
272

273
# Create transfer config
274
transfer_config = bigquery_datatransfer.TransferConfig(
275
    display_name="My Scheduled Query",
276
    data_source_id="scheduled_query",
277
    destination_dataset_id="my_dataset",
278
    schedule="every day 08:00",
279
    params=params,
280
    disabled=False,
281
    email_preferences=bigquery_datatransfer.EmailPreferences(
282
        enable_failure_email=True
283
    )
284
)
285

286
# Print config details
287
print(f"Config Name: {transfer_config.display_name}")
288
print(f"Data Source: {transfer_config.data_source_id}")
289
print(f"Schedule: {transfer_config.schedule}")
290
print(f"Disabled: {transfer_config.disabled}")
291
```
292

293
### Working with TransferRun
294

295
```python
296
from google.cloud import bigquery_datatransfer
297

298
client = bigquery_datatransfer.DataTransferServiceClient()
299

300
# Get a transfer run
301
run_name = f"projects/{project_id}/locations/{location}/transferConfigs/{config_id}/runs/{run_id}"
302
run = client.get_transfer_run(name=run_name)
303

304
# Access run properties
305
print(f"Run Name: {run.name}")
306
print(f"State: {run.state}")
307
print(f"Data Source: {run.data_source_id}")
308
print(f"Schedule Time: {run.schedule_time}")
309
print(f"Start Time: {run.start_time}")
310
print(f"End Time: {run.end_time}")
311

312
# Check for errors
313
if run.error_status and run.error_status.code != 0:
314
    print(f"Error Code: {run.error_status.code}")
315
    print(f"Error Message: {run.error_status.message}")
316

317
# Access run parameters
318
print("Parameters:")
319
for key, value in run.params.items():
320
    print(f"  {key}: {value}")
321
```
322

323
### Working with DataSource
324

325
```python
326
from google.cloud import bigquery_datatransfer
327

328
client = bigquery_datatransfer.DataTransferServiceClient()
329

330
# Get data source details
331
data_source_name = f"projects/{project_id}/locations/{location}/dataSources/scheduled_query"
332
data_source = client.get_data_source(name=data_source_name)
333

334
print(f"Data Source: {data_source.display_name}")
335
print(f"ID: {data_source.data_source_id}")
336
print(f"Description: {data_source.description}")
337
print(f"Supports Custom Schedule: {data_source.supports_custom_schedule}")
338
print(f"Default Schedule: {data_source.default_schedule}")
339

340
# List parameters
341
print("Parameters:")
342
for param in data_source.parameters:
343
    print(f"  {param.param_id}: {param.display_name}")
344
    print(f"    Type: {param.type_}")
345
    print(f"    Required: {param.required}")
346
    if param.description:
347
        print(f"    Description: {param.description}")
348
    if param.allowed_values:
349
        print(f"    Allowed Values: {list(param.allowed_values)}")
350
```
351

352
### Working with DataSourceParameter
353

354
```python
355
from google.cloud import bigquery_datatransfer
356

357
# Examine parameter details
358
def describe_parameter(param):
359
    print(f"Parameter: {param.param_id}")
360
    print(f"  Display Name: {param.display_name}")
361
    print(f"  Type: {param.type_}")
362
    print(f"  Required: {param.required}")
363
    
364
    if param.description:
365
        print(f"  Description: {param.description}")
366
    
367
    if param.validation_regex:
368
        print(f"  Validation Regex: {param.validation_regex}")
369
    
370
    if param.allowed_values:
371
        print(f"  Allowed Values: {list(param.allowed_values)}")
372
    
373
    if param.min_value:
374
        print(f"  Min Value: {param.min_value.value}")
375
    
376
    if param.max_value:
377
        print(f"  Max Value: {param.max_value.value}")
378
    
379
    if param.validation_description:
380
        print(f"  Validation Help: {param.validation_description}")
381

382
# Example usage with a data source
383
client = bigquery_datatransfer.DataTransferServiceClient()
384
data_source_name = f"projects/{project_id}/locations/{location}/dataSources/google_ads"
385
data_source = client.get_data_source(name=data_source_name)
386

387
for param in data_source.parameters:
388
    describe_parameter(param)
389
    print()
390
```
391

392
## Pager Types
393

394
Pager classes used for iterating through paginated list responses.
395

396
### ListDataSourcesPager
397

398
```python { .api }
399
class ListDataSourcesPager:
400
    """
401
    A pager for iterating through list_data_sources requests.
402
    
403
    This class provides an iterator interface for paginated results.
404
    """
405
    def __iter__(self) -> Iterator[DataSource]:
406
        """Iterate over data sources in the response."""
407
```
408

409
### ListTransferConfigsPager
410

411
```python { .api }
412
class ListTransferConfigsPager:
413
    """
414
    A pager for iterating through list_transfer_configs requests.
415
    
416
    This class provides an iterator interface for paginated results.
417
    """
418
    def __iter__(self) -> Iterator[TransferConfig]:
419
        """Iterate over transfer configurations in the response."""
420
```
421

422
### ListTransferRunsPager
423

424
```python { .api }
425
class ListTransferRunsPager:
426
    """
427
    A pager for iterating through list_transfer_runs requests.
428
    
429
    This class provides an iterator interface for paginated results.
430
    """
431
    def __iter__(self) -> Iterator[TransferRun]:
432
        """Iterate over transfer runs in the response."""
433
```
434

435
### ListTransferLogsPager
436

437
```python { .api }
438
class ListTransferLogsPager:
439
    """
440
    A pager for iterating through list_transfer_logs requests.
441
    
442
    This class provides an iterator interface for paginated results.
443
    """
444
    def __iter__(self) -> Iterator[TransferMessage]:
445
        """Iterate over transfer log messages in the response."""
446
```