0
# Core Source Operations
1
2
Main source functionality providing the primary entry points and core operations for the Airbyte S3 connector. These functions handle connector launching, configuration management, and specification generation.
3
4
## Capabilities
5
6
### Entry Points
7
8
Main entry functions that launch the S3 connector and handle command-line execution.
9
10
```python { .api }
11
def run() -> None:
12
"""
13
Main entry function that launches the SourceS3 connector.
14
Called when running the connector as a standalone application.
15
16
This function calls SourceS3.launch() to start the connector.
17
"""
18
```
19
20
### SourceS3 Class
21
22
Main source implementation inheriting from FileBasedSource, providing the core connector functionality.
23
24
```python { .api }
25
class SourceS3(FileBasedSource):
26
"""
27
Main S3 source implementation using the file-based CDK.
28
29
Attributes:
30
_concurrency_level: Set to DEFAULT_CONCURRENCY (10)
31
"""
32
33
@classmethod
34
def read_config(cls, config_path: str) -> Mapping[str, Any]:
35
"""
36
Reads and transforms legacy V3 configurations to V4 format.
37
38
Args:
39
config_path: Path to the configuration file
40
41
Returns:
42
Configuration dictionary with V4 format
43
"""
44
45
@classmethod
46
def spec(cls, *args, **kwargs) -> ConnectorSpecification:
47
"""
48
Generates the connector specification schema.
49
50
Returns:
51
ConnectorSpecification object defining the connector's configuration schema
52
"""
53
54
@classmethod
55
def launch(cls, args: list[str] | None = None) -> None:
56
"""
57
Launches the source connector with optional command-line arguments.
58
59
Args:
60
args: Optional list of command-line arguments
61
"""
62
63
@classmethod
64
def create(cls, *, configured_catalog_path: Path | str | None = None) -> SourceS3:
65
"""
66
Creates a new SourceS3 instance.
67
68
Args:
69
configured_catalog_path: Optional path to the configured catalog
70
71
Returns:
72
New SourceS3 instance
73
"""
74
```
75
76
### Legacy Configuration Support
77
78
Support for backward compatibility with V3 configurations through the SourceS3Spec class.
79
80
```python { .api }
81
class SourceS3Spec(SourceFilesAbstractSpec, BaseModel):
82
"""
83
Legacy V3 configuration specification for backward compatibility.
84
Contains nested S3Provider class for provider-specific settings.
85
"""
86
87
class S3Provider:
88
"""Provider-specific configuration fields for S3 access"""
89
bucket: str
90
aws_access_key_id: Optional[str]
91
aws_secret_access_key: Optional[str]
92
role_arn: Optional[str]
93
path_prefix: str
94
endpoint: str
95
region_name: Optional[str]
96
start_date: Optional[str]
97
```
98
99
## Usage Examples
100
101
### Basic Connector Launch
102
103
```python
104
from source_s3.run import run
105
106
# Launch the connector with default configuration
107
run()
108
```
109
110
### Programmatic Source Creation
111
112
```python
113
from source_s3.v4 import SourceS3
114
from source_s3.source import SourceS3Spec
115
from pathlib import Path
116
117
# Create source instance
118
source = SourceS3.create(configured_catalog_path="path/to/catalog.json")
119
120
# Launch with custom arguments
121
SourceS3.launch(args=["--config", "config.json", "--catalog", "catalog.json"])
122
```
123
124
### Configuration Reading
125
126
```python
127
from source_s3.v4 import SourceS3
128
129
# Read and transform configuration
130
config = SourceS3.read_config("config.json")
131
print(config) # V4 format configuration
132
```