or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

Files

docs

configuration.mdcore-source.mdfile-formats.mdindex.mdstream-operations.mdutilities.mdzip-support.md

core-source.mddocs/

0

# Core Source Operations

1

2

Main source functionality providing the primary entry points and core operations for the Airbyte S3 connector. These functions handle connector launching, configuration management, and specification generation.

3

4

## Capabilities

5

6

### Entry Points

7

8

Main entry functions that launch the S3 connector and handle command-line execution.

9

10

```python { .api }

11

def run() -> None:

12

"""

13

Main entry function that launches the SourceS3 connector.

14

Called when running the connector as a standalone application.

15

16

This function calls SourceS3.launch() to start the connector.

17

"""

18

```

19

20

### SourceS3 Class

21

22

Main source implementation inheriting from FileBasedSource, providing the core connector functionality.

23

24

```python { .api }

25

class SourceS3(FileBasedSource):

26

"""

27

Main S3 source implementation using the file-based CDK.

28

29

Attributes:

30

_concurrency_level: Set to DEFAULT_CONCURRENCY (10)

31

"""

32

33

@classmethod

34

def read_config(cls, config_path: str) -> Mapping[str, Any]:

35

"""

36

Reads and transforms legacy V3 configurations to V4 format.

37

38

Args:

39

config_path: Path to the configuration file

40

41

Returns:

42

Configuration dictionary with V4 format

43

"""

44

45

@classmethod

46

def spec(cls, *args, **kwargs) -> ConnectorSpecification:

47

"""

48

Generates the connector specification schema.

49

50

Returns:

51

ConnectorSpecification object defining the connector's configuration schema

52

"""

53

54

@classmethod

55

def launch(cls, args: list[str] | None = None) -> None:

56

"""

57

Launches the source connector with optional command-line arguments.

58

59

Args:

60

args: Optional list of command-line arguments

61

"""

62

63

@classmethod

64

def create(cls, *, configured_catalog_path: Path | str | None = None) -> SourceS3:

65

"""

66

Creates a new SourceS3 instance.

67

68

Args:

69

configured_catalog_path: Optional path to the configured catalog

70

71

Returns:

72

New SourceS3 instance

73

"""

74

```

75

76

### Legacy Configuration Support

77

78

Support for backward compatibility with V3 configurations through the SourceS3Spec class.

79

80

```python { .api }

81

class SourceS3Spec(SourceFilesAbstractSpec, BaseModel):

82

"""

83

Legacy V3 configuration specification for backward compatibility.

84

Contains nested S3Provider class for provider-specific settings.

85

"""

86

87

class S3Provider:

88

"""Provider-specific configuration fields for S3 access"""

89

bucket: str

90

aws_access_key_id: Optional[str]

91

aws_secret_access_key: Optional[str]

92

role_arn: Optional[str]

93

path_prefix: str

94

endpoint: str

95

region_name: Optional[str]

96

start_date: Optional[str]

97

```

98

99

## Usage Examples

100

101

### Basic Connector Launch

102

103

```python

104

from source_s3.run import run

105

106

# Launch the connector with default configuration

107

run()

108

```

109

110

### Programmatic Source Creation

111

112

```python

113

from source_s3.v4 import SourceS3

114

from source_s3.source import SourceS3Spec

115

from pathlib import Path

116

117

# Create source instance

118

source = SourceS3.create(configured_catalog_path="path/to/catalog.json")

119

120

# Launch with custom arguments

121

SourceS3.launch(args=["--config", "config.json", "--catalog", "catalog.json"])

122

```

123

124

### Configuration Reading

125

126

```python

127

from source_s3.v4 import SourceS3

128

129

# Read and transform configuration

130

config = SourceS3.read_config("config.json")

131

print(config) # V4 format configuration

132

```