0
# Configuration Management
1

2
Comprehensive configuration classes for controlling S3 transfer behavior including multipart thresholds, concurrency limits, retry settings, bandwidth limits, and memory management.
3

4
## Capabilities
5

6
### Modern TransferConfig
7

8
The enhanced configuration class for TransferManager with comprehensive options for fine-tuning transfer behavior.
9

10
```python { .api }
11
class TransferConfig:
12
    """
13
    Configuration for TransferManager operations with comprehensive options.
14
    
15
    Args:
16
        multipart_threshold (int): Size threshold for multipart transfers (default: 8MB)
17
        multipart_chunksize (int): Size of multipart chunks (default: 8MB)
18
        max_request_concurrency (int): Max concurrent S3 API requests (default: 10)
19
        max_submission_concurrency (int): Max concurrent task submissions (default: 5)
20
        max_request_queue_size (int): Max request queue size (default: 1024)
21
        max_submission_queue_size (int): Max submission queue size (default: 1024)
22
        max_io_queue_size (int): Max IO operation queue size (default: 1024)
23
        io_chunksize (int): IO chunk size for reading/writing (default: 256KB)
24
        num_download_attempts (int): Number of download retry attempts (default: 5)
25
        max_in_memory_upload_chunks (int): Max upload chunks held in memory (default: 10)
26
        max_in_memory_download_chunks (int): Max download chunks held in memory (default: 10)
27
        max_bandwidth (int, optional): Maximum bandwidth in bytes/second (default: None)
28
    """
29
    def __init__(
30
        self,
31
        multipart_threshold=8 * 1024 * 1024,
32
        multipart_chunksize=8 * 1024 * 1024,
33
        max_request_concurrency=10,
34
        max_submission_concurrency=5,
35
        max_request_queue_size=1024,
36
        max_submission_queue_size=1024,
37
        max_io_queue_size=1024,
38
        io_chunksize=256 * 1024,
39
        num_download_attempts=5,
40
        max_in_memory_upload_chunks=10,
41
        max_in_memory_download_chunks=10,
42
        max_bandwidth=None
43
    ): ...
44
    
45
    multipart_threshold: int
46
    multipart_chunksize: int
47
    max_request_concurrency: int
48
    max_submission_concurrency: int
49
    max_request_queue_size: int
50
    max_submission_queue_size: int
51
    max_io_queue_size: int
52
    io_chunksize: int
53
    num_download_attempts: int
54
    max_in_memory_upload_chunks: int
55
    max_in_memory_download_chunks: int
56
    max_bandwidth: Optional[int]
57
```
58

59
### Legacy TransferConfig
60

61
The original configuration class for S3Transfer with basic options for multipart operations and concurrency.
62

63
```python { .api }
64
class TransferConfig:
65
    """
66
    Legacy configuration for S3Transfer operations.
67
    
68
    Args:
69
        multipart_threshold (int): Size threshold for multipart uploads (default: 8MB)
70
        max_concurrency (int): Maximum number of concurrent transfers (default: 10)
71
        multipart_chunksize (int): Size of multipart chunks (default: 8MB)
72
        num_download_attempts (int): Number of download retry attempts (default: 5)
73
        max_io_queue (int): Maximum size of IO queue (default: 100)
74
    """
75
    def __init__(
76
        self,
77
        multipart_threshold=8 * 1024 * 1024,
78
        max_concurrency=10,
79
        multipart_chunksize=8 * 1024 * 1024,
80
        num_download_attempts=5,
81
        max_io_queue=100
82
    ): ...
83
    
84
    multipart_threshold: int
85
    max_concurrency: int
86
    multipart_chunksize: int
87
    num_download_attempts: int
88
    max_io_queue: int
89
```
90

91
## Configuration Examples
92

93
### High-Performance Configuration
94

95
Optimized for large files and high-bandwidth connections:
96

97
```python
98
from s3transfer.manager import TransferConfig
99

100
# High-performance configuration for large files
101
high_perf_config = TransferConfig(
102
    multipart_threshold=64 * 1024 * 1024,      # 64MB - larger threshold
103
    multipart_chunksize=64 * 1024 * 1024,      # 64MB chunks
104
    max_request_concurrency=50,                 # High concurrency
105
    max_submission_concurrency=20,              # More task submissions
106
    max_request_queue_size=2048,                # Larger request queue
107
    max_submission_queue_size=2048,             # Larger submission queue
108
    max_io_queue_size=2048,                     # Larger IO queue
109
    io_chunksize=1024 * 1024,                   # 1MB IO chunks
110
    num_download_attempts=3,                    # Fewer retries (good connection)
111
    max_in_memory_upload_chunks=20,             # More memory usage
112
    max_in_memory_download_chunks=20,           # More memory usage
113
    max_bandwidth=1000 * 1024 * 1024            # 1GB/s limit
114
)
115
```
116

117
### Memory-Constrained Configuration
118

119
Optimized for environments with limited memory:
120

121
```python
122
# Low-memory configuration
123
low_memory_config = TransferConfig(
124
    multipart_threshold=16 * 1024 * 1024,       # 16MB threshold
125
    multipart_chunksize=8 * 1024 * 1024,        # 8MB chunks (smaller)
126
    max_request_concurrency=5,                   # Lower concurrency
127
    max_submission_concurrency=2,                # Fewer submissions
128
    max_request_queue_size=256,                  # Smaller queues
129
    max_submission_queue_size=256,
130
    max_io_queue_size=256,
131
    io_chunksize=64 * 1024,                      # 64KB IO chunks
132
    num_download_attempts=10,                    # More retries
133
    max_in_memory_upload_chunks=2,               # Minimal memory usage
134
    max_in_memory_download_chunks=2,
135
    max_bandwidth=10 * 1024 * 1024               # 10MB/s limit
136
)
137
```
138

139
### Bandwidth-Limited Configuration
140

141
Optimized for slow or metered connections:
142

143
```python
144
# Bandwidth-limited configuration
145
bandwidth_limited_config = TransferConfig(
146
    multipart_threshold=32 * 1024 * 1024,       # 32MB threshold
147
    multipart_chunksize=16 * 1024 * 1024,       # 16MB chunks
148
    max_request_concurrency=3,                   # Low concurrency
149
    max_submission_concurrency=1,                # Sequential submissions
150
    max_request_queue_size=128,                  # Small queues
151
    max_submission_queue_size=64,
152
    max_io_queue_size=128,
153
    io_chunksize=128 * 1024,                     # 128KB IO chunks
154
    num_download_attempts=15,                    # Many retries
155
    max_in_memory_upload_chunks=3,               # Conservative memory
156
    max_in_memory_download_chunks=3,
157
    max_bandwidth=1 * 1024 * 1024                # 1MB/s strict limit
158
)
159
```
160

161
### Reliable Network Configuration
162

163
Optimized for unreliable networks with good bandwidth:
164

165
```python
166
# Reliable network configuration
167
reliable_config = TransferConfig(
168
    multipart_threshold=32 * 1024 * 1024,       # 32MB threshold
169
    multipart_chunksize=16 * 1024 * 1024,       # 16MB chunks (smaller for retries)
170
    max_request_concurrency=15,                  # Good concurrency
171
    max_submission_concurrency=8,                # Balanced submissions
172
    max_request_queue_size=1024,                 # Standard queues
173
    max_submission_queue_size=512,
174
    max_io_queue_size=1024,
175
    io_chunksize=512 * 1024,                     # 512KB IO chunks
176
    num_download_attempts=20,                    # Many retries for reliability
177
    max_in_memory_upload_chunks=8,               # Balanced memory usage
178
    max_in_memory_download_chunks=8,
179
    max_bandwidth=None                           # No bandwidth limit
180
)
181
```
182

183
## Configuration Parameters Explained
184

185
### Multipart Settings
186

187
```python
188
# Multipart threshold - when to switch to multipart uploads/downloads
189
multipart_threshold=8 * 1024 * 1024  # Files >= 8MB use multipart
190

191
# Chunk size for multipart operations
192
multipart_chunksize=8 * 1024 * 1024  # Each part is 8MB
193
```
194

195
**Guidelines:**
196
- Larger thresholds reduce overhead for medium files
197
- Smaller chunks enable better progress tracking and retry granularity
198
- S3 minimum chunk size is 5MB (except last part)
199
- S3 maximum parts per upload is 10,000
200

201
### Concurrency Settings
202

203
```python
204
# Maximum concurrent S3 API requests (upload_part, get_object, etc.)
205
max_request_concurrency=10
206

207
# Maximum concurrent task submissions to executor
208
max_submission_concurrency=5
209
```
210

211
**Guidelines:**
212
- Higher request concurrency improves throughput but uses more resources
213
- Balance with S3 request rate limits and local resource constraints
214
- Submission concurrency should be lower than request concurrency
215

216
### Queue Settings
217

218
```python
219
# Maximum pending requests in various queues
220
max_request_queue_size=1024      # S3 API requests
221
max_submission_queue_size=1024   # Task submissions
222
max_io_queue_size=1024          # IO operations
223
```
224

225
**Guidelines:**
226
- Larger queues provide more buffering but use more memory
227
- Size based on expected peak load and available memory
228
- IO queue affects download performance most significantly
229

230
### Memory Management
231

232
```python
233
# Maximum chunks held in memory simultaneously
234
max_in_memory_upload_chunks=10   # Upload chunks buffered
235
max_in_memory_download_chunks=10 # Download chunks buffered
236

237
# IO chunk size for reading/writing operations
238
io_chunksize=256 * 1024         # 256KB per IO operation
239
```
240

241
**Guidelines:**
242
- More in-memory chunks improve performance but increase memory usage
243
- Memory usage ≈ chunks × chunk_size × concurrency
244
- Balance based on available RAM and performance needs
245

246
### Retry and Reliability
247

248
```python
249
# Number of download retry attempts
250
num_download_attempts=5
251
```
252

253
**Guidelines:**
254
- More attempts improve reliability but increase latency on failures
255
- Consider network reliability and timeout settings
256
- Upload retries are handled by boto3/botocore
257

258
### Bandwidth Management
259

260
```python
261
# Maximum bandwidth in bytes per second
262
max_bandwidth=100 * 1024 * 1024  # 100MB/s limit
263
```
264

265
**Guidelines:**
266
- Use for rate limiting in shared environments
267
- Set to None for no bandwidth limits
268
- Applies across all concurrent transfers
269

270
## Configuration Validation
271

272
s3transfer performs automatic validation of configuration parameters:
273

274
```python
275
# These will raise ValueError if invalid
276
config = TransferConfig(
277
    multipart_chunksize=1024,          # Too small (< 5MB)
278
    max_request_concurrency=0,         # Must be > 0
279
    num_download_attempts=0,           # Must be > 0
280
    max_bandwidth=-1                   # Must be None or > 0
281
)
282
```
283

284
## Performance Tuning Guidelines
285

286
### For Small Files (< 100MB)
287

288
```python
289
config = TransferConfig(
290
    multipart_threshold=100 * 1024 * 1024,  # Higher threshold
291
    max_request_concurrency=20,              # Higher concurrency
292
    max_in_memory_upload_chunks=5,           # Lower memory usage
293
    max_in_memory_download_chunks=5
294
)
295
```
296

297
### For Large Files (> 1GB)
298

299
```python
300
config = TransferConfig(
301
    multipart_threshold=16 * 1024 * 1024,   # Lower threshold
302
    multipart_chunksize=64 * 1024 * 1024,   # Larger chunks
303
    max_request_concurrency=30,              # High concurrency
304
    max_in_memory_upload_chunks=15,          # More buffering
305
    max_in_memory_download_chunks=15
306
)
307
```
308

309
### For Many Small Files
310

311
```python
312
config = TransferConfig(
313
    multipart_threshold=64 * 1024 * 1024,   # Avoid multipart
314
    max_request_concurrency=50,              # Very high concurrency
315
    max_submission_concurrency=25,           # High submissions
316
    max_request_queue_size=4096,             # Large queues
317
    max_in_memory_upload_chunks=3,           # Low memory per transfer
318
    max_in_memory_download_chunks=3
319
)
320
```
321

322
## Legacy vs Modern Configuration
323

324
### Migration Mapping
325

326
| Legacy Parameter | Modern Equivalent | Notes |
327
|------------------|-------------------|-------|
328
| `max_concurrency` | `max_request_concurrency` | Similar purpose |
329
| `max_io_queue` | `max_io_queue_size` | Renamed |
330
| N/A | `max_submission_concurrency` | New parameter |
331
| N/A | `max_request_queue_size` | New parameter |
332
| N/A | `max_submission_queue_size` | New parameter |
333
| N/A | `io_chunksize` | New parameter |
334
| N/A | `max_in_memory_*_chunks` | New parameters |
335
| N/A | `max_bandwidth` | New parameter |
336
337
### Migration Example
338

339
```python
340
# Legacy configuration
341
from s3transfer import TransferConfig as LegacyConfig
342
legacy_config = LegacyConfig(
343
    multipart_threshold=16 * 1024 * 1024,
344
    max_concurrency=15,
345
    multipart_chunksize=16 * 1024 * 1024,
346
    num_download_attempts=10,
347
    max_io_queue=200
348
)
349

350
# Equivalent modern configuration
351
from s3transfer.manager import TransferConfig
352
modern_config = TransferConfig(
353
    multipart_threshold=16 * 1024 * 1024,
354
    max_request_concurrency=15,           # Was max_concurrency
355
    multipart_chunksize=16 * 1024 * 1024,
356
    num_download_attempts=10,
357
    max_io_queue_size=200,                # Was max_io_queue
358
    max_submission_concurrency=8,         # New - set appropriately
359
    max_bandwidth=None                    # New - no limit
360
)
361
```