0
# AWS Common Runtime (CRT) Support
1
2
High-performance transfer manager implementation using the AWS Common Runtime (CRT) for significantly improved throughput and efficiency. Provides a drop-in replacement for the regular TransferManager with automatic throughput optimization and advanced performance features.
3
4
## Capabilities
5
6
### CRTTransferManager
7
8
Drop-in replacement for TransferManager using AWS Common Runtime for high-performance S3 operations with automatic throughput optimization.
9
10
```python { .api }
11
class CRTTransferManager:
12
"""
13
High-performance transfer manager using AWS Common Runtime.
14
15
Args:
16
crt_s3_client: AWS CRT S3 client instance
17
crt_request_serializer: Request serializer for botocore compatibility
18
osutil: OSUtils instance for file operations (default: OSUtils())
19
"""
20
def __init__(self, crt_s3_client, crt_request_serializer, osutil=None): ...
21
22
def upload(self, fileobj, bucket, key, extra_args=None, subscribers=None):
23
"""
24
Upload a file-like object to S3 using CRT.
25
26
Args:
27
fileobj: File-like object to upload (must support read())
28
bucket (str): S3 bucket name
29
key (str): S3 object key/name
30
extra_args (dict, optional): Additional S3 operation arguments
31
subscribers (list, optional): List of subscriber objects for events
32
33
Returns:
34
CRTTransferFuture: Future object for tracking transfer progress
35
"""
36
37
def download(self, bucket, key, fileobj, extra_args=None, subscribers=None):
38
"""
39
Download an S3 object to a file-like object using CRT.
40
41
Args:
42
bucket (str): S3 bucket name
43
key (str): S3 object key/name
44
fileobj: File-like object to write to (must support write())
45
extra_args (dict, optional): Additional S3 operation arguments
46
subscribers (list, optional): List of subscriber objects for events
47
48
Returns:
49
CRTTransferFuture: Future object for tracking transfer progress
50
"""
51
52
def delete(self, bucket, key, extra_args=None, subscribers=None):
53
"""
54
Delete an S3 object using CRT.
55
56
Args:
57
bucket (str): S3 bucket name
58
key (str): S3 object key/name
59
extra_args (dict, optional): Additional S3 operation arguments
60
subscribers (list, optional): List of subscriber objects for events
61
62
Returns:
63
CRTTransferFuture: Future object for tracking deletion progress
64
"""
65
66
def shutdown(self, cancel=False):
67
"""
68
Shutdown the CRT transfer manager.
69
70
Args:
71
cancel (bool): Whether to cancel ongoing transfers (default: False)
72
"""
73
74
def __enter__(self):
75
"""Context manager entry."""
76
77
def __exit__(self, exc_type, exc_val, exc_tb):
78
"""Context manager exit with automatic shutdown."""
79
```
80
81
### CRTTransferFuture
82
83
Future object representing a CRT transfer operation with methods for monitoring progress and retrieving results.
84
85
```python { .api }
86
class CRTTransferFuture:
87
"""
88
Future representing a CRT transfer request.
89
"""
90
def done(self) -> bool:
91
"""
92
Check if the transfer is complete.
93
94
Returns:
95
bool: True if transfer is complete (success or failure), False otherwise
96
"""
97
98
def result(self, timeout=None):
99
"""
100
Get the transfer result, blocking until complete.
101
102
Args:
103
timeout (float, optional): Maximum time to wait for completion
104
105
Returns:
106
None: Returns None on successful completion
107
108
Raises:
109
Exception: Any exception that occurred during transfer
110
TimeoutError: If timeout is reached before completion
111
"""
112
113
def cancel(self):
114
"""
115
Cancel the transfer if possible.
116
117
Returns:
118
bool: True if cancellation was successful, False otherwise
119
"""
120
121
@property
122
def meta(self):
123
"""
124
Transfer metadata object containing call arguments and status information.
125
126
Returns:
127
CRTTransferMeta: Metadata object for this transfer
128
"""
129
```
130
131
### BotocoreCRTRequestSerializer
132
133
Request serializer that provides compatibility between botocore and AWS CRT S3 client.
134
135
```python { .api }
136
class BotocoreCRTRequestSerializer:
137
"""
138
Serializes HTTP requests using botocore logic for CRT compatibility.
139
140
Args:
141
session: Botocore session instance
142
region_name (str): AWS region name
143
signature_version (str): Signature version (default: 's3v4')
144
"""
145
def __init__(self, session, region_name, signature_version='s3v4'): ...
146
147
def serialize_http_request(self, request_dict):
148
"""
149
Serialize a request dictionary to HTTP request format.
150
151
Args:
152
request_dict (dict): Request parameters dictionary
153
154
Returns:
155
dict: Serialized HTTP request
156
"""
157
```
158
159
### CRT Client Creation
160
161
Utility functions for creating and managing CRT S3 clients with optimized performance settings.
162
163
```python { .api }
164
def create_s3_crt_client(
165
region_name,
166
num_threads=None,
167
target_throughput=None,
168
part_size=8388608,
169
use_ssl=True,
170
verify=None
171
):
172
"""
173
Create an optimized CRT S3 client.
174
175
Args:
176
region_name (str): AWS region name
177
num_threads (int, optional): Number of worker threads (defaults to CPU count)
178
target_throughput (int, optional): Target throughput in bytes/second (auto-detected)
179
part_size (int): Part size for multipart operations (default: 8MB)
180
use_ssl (bool): Enable SSL/TLS (default: True)
181
verify (bool/str, optional): Certificate verification settings
182
183
Returns:
184
S3Client: Configured CRT S3 client
185
"""
186
187
def acquire_crt_s3_process_lock():
188
"""
189
Acquire a process-level lock for CRT S3 client usage.
190
191
Prevents multiple CRT S3 clients from running simultaneously in the same process,
192
which can cause resource conflicts.
193
194
Returns:
195
Lock: Process lock object
196
"""
197
```
198
199
## Installation and Requirements
200
201
### Installing CRT Support
202
203
CRT support is an optional feature requiring additional dependencies:
204
205
```bash
206
# Install s3transfer with CRT support
207
pip install s3transfer[crt]
208
209
# Or install with specific botocore CRT support
210
pip install botocore[crt] s3transfer
211
```
212
213
### System Requirements
214
215
- Python >= 3.9
216
- AWS Common Runtime libraries (installed with `awscrt` package)
217
- Multi-core system recommended for optimal performance
218
219
## Usage Examples
220
221
### Basic CRT Usage
222
223
```python
224
import boto3
225
from s3transfer.crt import CRTTransferManager, BotocoreCRTRequestSerializer, create_s3_crt_client
226
227
# Create CRT S3 client with performance optimizations
228
crt_client = create_s3_crt_client(
229
region_name='us-west-2',
230
target_throughput=10 * 1024 * 1024 * 1024, # 10 Gbps
231
num_threads=16
232
)
233
234
# Create request serializer
235
session = boto3.Session()
236
serializer = BotocoreCRTRequestSerializer(session, 'us-west-2')
237
238
# Create CRT transfer manager
239
transfer_manager = CRTTransferManager(crt_client, serializer)
240
241
try:
242
# Upload a file with high performance
243
with open('/path/to/large-file.zip', 'rb') as f:
244
future = transfer_manager.upload(f, 'my-bucket', 'large-file.zip')
245
future.result() # Wait for completion
246
247
print("Upload completed with CRT acceleration")
248
249
finally:
250
transfer_manager.shutdown()
251
```
252
253
### Context Manager Usage
254
255
```python
256
from s3transfer.crt import CRTTransferManager, BotocoreCRTRequestSerializer, create_s3_crt_client
257
258
# Setup CRT client and serializer
259
crt_client = create_s3_crt_client('us-west-2')
260
serializer = BotocoreCRTRequestSerializer(boto3.Session(), 'us-west-2')
261
262
# Use context manager for automatic cleanup
263
with CRTTransferManager(crt_client, serializer) as transfer_manager:
264
# Download with CRT acceleration
265
with open('/tmp/downloaded-file.zip', 'wb') as f:
266
future = transfer_manager.download('my-bucket', 'large-file.zip', f)
267
future.result()
268
269
print("Download completed")
270
# Transfer manager automatically shut down
271
```
272
273
### Process Lock Usage
274
275
```python
276
from s3transfer.crt import acquire_crt_s3_process_lock, create_s3_crt_client
277
278
# Acquire process lock to prevent conflicts
279
with acquire_crt_s3_process_lock():
280
crt_client = create_s3_crt_client('us-west-2')
281
# Use CRT client safely
282
```
283
284
## Performance Features
285
286
### Automatic Throughput Optimization
287
288
- **Auto-Detection**: Automatically detects optimal throughput settings
289
- **Target Throughput**: Configurable target throughput (defaults to 10 Gbps or system limit)
290
- **Adaptive Scaling**: Dynamically adjusts based on network conditions
291
292
### Advanced Capabilities
293
294
- **S3 Express One Zone**: Built-in support for S3 Express buckets
295
- **Multi-Region Access Points**: Automatic handling of MRAP buckets
296
- **Checksum Validation**: Built-in checksum validation and trailing checksums
297
- **Process Coordination**: Cross-process locking to prevent resource conflicts
298
299
### Performance Comparison
300
301
**CRT vs Regular TransferManager:**
302
303
| Feature | Regular TransferManager | CRTTransferManager |
304
|---------|------------------------|-------------------|
305
| Throughput | Good | Excellent (10+ Gbps) |
306
| CPU Usage | Higher | Lower |
307
| Memory Usage | Moderate | Optimized |
308
| Setup Complexity | Simple | Moderate |
309
| Dependencies | Minimal | CRT libraries required |
310
311
## When to Use CRT
312
313
### Recommended For:
314
- High-throughput applications
315
- Large file transfers (>100MB)
316
- Applications transferring large volumes of data
317
- Performance-critical workloads
318
- Multi-core systems with high-bandwidth network connections
319
320
### Consider Regular TransferManager For:
321
- Simple applications with modest transfer needs
322
- Environments where installing native dependencies is problematic
323
- Small file transfers (<10MB)
324
- Applications prioritizing minimal dependencies over performance