0
# Sources and Integration
1
2
Support for multiple configuration data sources including YAML files, environment variables, and command-line arguments with automatic type conversion and priority handling. Configuration sources enable flexible data input from various locations with seamless merging and override capabilities.
3
4
## Capabilities
5
6
### Configuration Sources
7
8
Base classes for different types of configuration data sources that can be layered together with priority ordering.
9
10
```python { .api }
11
class ConfigSource(dict):
12
def __init__(self, value, filename=None, default=False, base_for_paths=False):
13
"""
14
Create a configuration source from a dictionary.
15
16
Parameters:
17
- value (dict): Configuration data dictionary
18
- filename (str, optional): Source file path for this configuration data
19
- default (bool): Whether this source provides application defaults
20
- base_for_paths (bool): Use source file's directory for relative path resolution
21
"""
22
23
@classmethod
24
def of(cls, value):
25
"""
26
Create ConfigSource from dictionary or existing ConfigSource.
27
28
Parameters:
29
- value: Dictionary or ConfigSource object
30
31
Returns:
32
ConfigSource: New or existing ConfigSource object
33
"""
34
```
35
36
### YAML File Sources
37
38
Configuration sources that read from YAML files with customizable loading and error handling.
39
40
```python { .api }
41
class YamlSource(ConfigSource):
42
def __init__(self, filename=None, default=False, base_for_paths=False,
43
optional=False, loader=yaml_util.Loader):
44
"""
45
Create YAML configuration source by reading from file.
46
47
Parameters:
48
- filename (str): Path to YAML configuration file
49
- default (bool): Whether this is a default configuration source
50
- base_for_paths (bool): Use file's directory as base for relative paths
51
- optional (bool): Don't raise error if file doesn't exist
52
- loader: PyYAML Loader class for parsing YAML
53
54
Raises:
55
- ConfigReadError: If file cannot be read (unless optional=True)
56
"""
57
58
def load(self):
59
"""
60
Load YAML data from the source's filename.
61
62
Raises:
63
- ConfigReadError: If file cannot be read or parsed
64
"""
65
```
66
67
### Environment Variable Sources
68
69
Configuration sources that read from environment variables with flexible naming and type conversion.
70
71
```python { .api }
72
class EnvSource(ConfigSource):
73
def __init__(self, prefix, sep='__', lower=True, handle_lists=True,
74
parse_yaml_docs=False, loader=yaml_util.Loader):
75
"""
76
Create configuration source from environment variables.
77
78
Parameters:
79
- prefix (str): Environment variable prefix for identification
80
- sep (str): Separator within variable names for nested keys
81
- lower (bool): Convert variable names to lowercase after prefix matching
82
- handle_lists (bool): Convert sequential integer keys to lists
83
- parse_yaml_docs (bool): Parse values as full YAML documents vs scalars
84
- loader: PyYAML Loader class for parsing values
85
"""
86
87
def load(self):
88
"""
89
Load configuration data from environment variables.
90
"""
91
92
@classmethod
93
def _convert_dict_lists(cls, obj):
94
"""
95
Convert dicts with sequential integer keys (0, 1, 2...) to lists.
96
97
Parameters:
98
- obj: Dictionary potentially containing list-like structures
99
100
Returns:
101
Converted object with dicts converted to lists where appropriate
102
"""
103
```
104
105
### YAML Processing Utilities
106
107
Custom YAML loading and dumping classes with enhanced features for configuration management.
108
109
```python { .api }
110
class Loader(yaml.SafeLoader):
111
"""
112
Custom YAML loader with Unicode strings and OrderedDict support.
113
Features:
114
- All strings as Unicode objects
115
- All maps as OrderedDicts
116
- Strings can begin with % without quotation
117
"""
118
119
@staticmethod
120
def add_constructors(loader):
121
"""
122
Add custom constructors to a PyYAML Loader class.
123
124
Parameters:
125
- loader: PyYAML Loader class to modify
126
"""
127
128
class Dumper(yaml.SafeDumper):
129
"""
130
Custom YAML dumper with OrderedDict and formatting enhancements.
131
Features:
132
- OrderedDicts as ordinary mappings
133
- Short lists in inline style
134
- Boolean values as 'yes'/'no'
135
- None values as empty strings
136
"""
137
138
def load_yaml(filename, loader=Loader):
139
"""
140
Read YAML document from file with error handling.
141
142
Parameters:
143
- filename (str): Path to YAML file
144
- loader: PyYAML Loader class
145
146
Returns:
147
Parsed YAML data
148
149
Raises:
150
- ConfigReadError: If file cannot be read or parsed
151
"""
152
153
def load_yaml_string(yaml_string, name, loader=Loader):
154
"""
155
Parse YAML document from string with error handling.
156
157
Parameters:
158
- yaml_string (str): YAML content as string
159
- name (str): Name for error messages
160
- loader: PyYAML Loader class
161
162
Returns:
163
Parsed YAML data
164
165
Raises:
166
- ConfigReadError: If string cannot be parsed
167
"""
168
169
def parse_as_scalar(value, loader=Loader):
170
"""
171
Parse value as YAML scalar for consistent type conversion.
172
173
Parameters:
174
- value (str): String value to parse
175
- loader: PyYAML Loader class
176
177
Returns:
178
Type-converted value (int, float, bool, None, or original string)
179
"""
180
181
def restore_yaml_comments(data, default_data):
182
"""
183
Restore comments from default YAML to generated data.
184
185
Parameters:
186
- data (str): Generated YAML data string
187
- default_data (str): Default YAML data with comments
188
189
Returns:
190
str: YAML data with comments restored from default_data
191
"""
192
```
193
194
### Platform Utilities
195
196
Cross-platform utilities for configuration directory discovery and path handling.
197
198
```python { .api }
199
def config_dirs():
200
"""
201
Get platform-specific configuration directory candidates.
202
203
Returns:
204
list: Configuration directory paths in priority order
205
"""
206
207
def xdg_config_dirs():
208
"""
209
Get list of paths from XDG_CONFIG_DIRS and XDG_CONFIG_HOME environment variables.
210
211
Returns:
212
list: XDG configuration directory paths
213
"""
214
215
def iter_first(sequence):
216
"""
217
Get the first element from an iterable or raise ValueError if empty.
218
219
Parameters:
220
- sequence: Iterable to get first element from
221
222
Returns:
223
First element from sequence
224
225
Raises:
226
ValueError: If sequence is empty
227
"""
228
229
def namespace_to_dict(obj):
230
"""
231
Convert argparse.Namespace or optparse.Values to dict representation.
232
233
Parameters:
234
- obj: Namespace or Values object to convert
235
236
Returns:
237
dict: Dictionary representation or original object if not convertible
238
"""
239
240
def find_package_path(name):
241
"""
242
Find the filesystem path for a Python package.
243
244
Parameters:
245
- name (str): Package name
246
247
Returns:
248
str or None: Package directory path, or None if not found
249
"""
250
251
def build_dict(obj, sep='', keep_none=False):
252
"""
253
Recursively build nested dictionary from namespace/dict with key splitting.
254
255
Parameters:
256
- obj: Namespace, Values, dict, or other object to convert
257
- sep (str): Separator for splitting keys into nested structure
258
- keep_none (bool): Whether to keep keys with None values
259
260
Returns:
261
dict: Nested dictionary structure
262
"""
263
```
264
265
### Platform Constants
266
267
```python { .api }
268
UNIX_DIR_FALLBACK = '~/.config' # Default Unix configuration directory
269
WINDOWS_DIR_VAR = 'APPDATA' # Windows environment variable name
270
WINDOWS_DIR_FALLBACK = '~\\AppData\\Roaming' # Windows config directory fallback
271
MAC_DIR = '~/Library/Application Support' # macOS configuration directory
272
```
273
274
## Usage Examples
275
276
### YAML File Sources
277
278
```python
279
import confuse
280
281
config = confuse.Configuration('myapp', read=False)
282
283
# Add specific YAML file
284
config.set_file('/etc/myapp/config.yaml')
285
286
# Add optional YAML file (no error if missing)
287
yaml_source = confuse.YamlSource('/home/user/myapp.yaml', optional=True)
288
config.add(yaml_source)
289
290
# YAML file with path resolution relative to file location
291
config_source = confuse.YamlSource('config.yaml', base_for_paths=True)
292
config.add(config_source)
293
```
294
295
### Environment Variable Sources
296
297
```python
298
import confuse
299
import os
300
301
config = confuse.Configuration('myapp', read=False)
302
303
# Basic environment source with MYAPP_ prefix
304
config.set_env('MYAPP_')
305
306
# Environment variables:
307
# MYAPP_DATABASE_HOST=localhost -> config['database']['host'] = 'localhost'
308
# MYAPP_DATABASE_PORT=5432 -> config['database']['port'] = 5432
309
310
# Custom environment source with different settings
311
env_source = confuse.EnvSource(
312
prefix='MYAPP_',
313
sep='__', # Use __ as separator instead of _
314
lower=True, # Convert to lowercase
315
handle_lists=True, # Convert sequential keys to lists
316
parse_yaml_docs=False # Parse as scalars only
317
)
318
config.add(env_source)
319
320
# Example environment variables:
321
# MYAPP_SERVERS__0__HOST=server1.example.com
322
# MYAPP_SERVERS__0__PORT=8080
323
# MYAPP_SERVERS__1__HOST=server2.example.com
324
# MYAPP_SERVERS__1__PORT=8081
325
# Results in: config['servers'] = [{'host': 'server1.example.com', 'port': 8080}, ...]
326
```
327
328
### Environment Variable List Handling
329
330
```python
331
import confuse
332
import os
333
334
# Set up environment variables for list conversion
335
os.environ['MYAPP_ITEMS__0'] = 'first'
336
os.environ['MYAPP_ITEMS__1'] = 'second'
337
os.environ['MYAPP_ITEMS__2'] = 'third'
338
339
config = confuse.Configuration('myapp', read=False)
340
config.set_env('MYAPP_')
341
342
# Automatically converted to list
343
items = config['items'].get(list) # ['first', 'second', 'third']
344
```
345
346
### YAML Document Parsing from Environment
347
348
```python
349
import confuse
350
import os
351
352
# Environment variable with YAML content
353
os.environ['MYAPP_COMPLEX'] = '''
354
database:
355
host: localhost
356
port: 5432
357
features: [auth, logging, metrics]
358
'''
359
360
config = confuse.Configuration('myapp', read=False)
361
362
# Parse environment values as full YAML documents
363
env_source = confuse.EnvSource('MYAPP_', parse_yaml_docs=True)
364
config.add(env_source)
365
366
# Access parsed YAML structure
367
db_host = config['complex']['database']['host'].as_str() # 'localhost'
368
features = config['complex']['features'].as_str_seq() # ['auth', 'logging', 'metrics']
369
```
370
371
### Command-Line Integration
372
373
```python
374
import confuse
375
import argparse
376
377
config = confuse.Configuration('myapp')
378
379
# Set up argument parser
380
parser = argparse.ArgumentParser()
381
parser.add_argument('--host', default='localhost')
382
parser.add_argument('--port', type=int, default=8080)
383
parser.add_argument('--database-url', dest='database.url')
384
parser.add_argument('--enable-feature', action='append', dest='features')
385
parser.add_argument('--verbose', action='store_true')
386
387
args = parser.parse_args()
388
389
# Overlay arguments onto configuration
390
config.set_args(args, dots=True)
391
392
# Command line overrides configuration files
393
host = config['host'].as_str() # From --host or config file
394
port = config['port'].as_number() # From --port or config file
395
db_url = config['database']['url'].as_str() # From --database-url
396
features = config['features'].as_str_seq() # From --enable-feature (multiple)
397
```
398
399
### Custom YAML Loading
400
401
```python
402
import confuse
403
import yaml
404
405
# Create custom loader with additional constructors
406
class CustomLoader(confuse.yaml_util.Loader):
407
pass
408
409
def construct_include(loader, node):
410
"""Custom constructor for !include directive"""
411
filename = loader.construct_scalar(node)
412
with open(filename, 'r') as f:
413
return yaml.load(f, Loader=CustomLoader)
414
415
CustomLoader.add_constructor('!include', construct_include)
416
417
# Use custom loader
418
config = confuse.Configuration('myapp', read=False, loader=CustomLoader)
419
config.set_file('config.yaml') # Can now use !include directive
420
```
421
422
### Source Priority and Layering
423
424
```python
425
import confuse
426
427
config = confuse.Configuration('myapp', read=False)
428
429
# Add sources in reverse priority order (lowest to highest)
430
config.add({'timeout': 30, 'debug': False}) # 1. Defaults (lowest)
431
config.set_file('/etc/myapp/system.yaml') # 2. System config
432
config.read(user=True, defaults=False) # 3. User config
433
config.set_env('MYAPP_') # 4. Environment variables
434
# Command-line arguments would be added with set_args() # 5. CLI args (highest)
435
436
# Runtime overrides (highest priority)
437
config.set({'debug': True}) # 6. Runtime (highest)
438
439
# Values are resolved in priority order
440
debug_mode = config['debug'].get(bool) # True (from runtime override)
441
timeout = config['timeout'].get(int) # 30 (from defaults, unless overridden)
442
```
443
444
### Platform-Specific Configuration
445
446
```python
447
import confuse
448
449
config = confuse.Configuration('myapp')
450
451
# Configuration files searched in platform-specific locations:
452
# Linux: ~/.config/myapp/config.yaml, /etc/xdg/myapp/config.yaml, etc.
453
# macOS: ~/Library/Application Support/myapp/config.yaml, ~/.config/myapp/config.yaml, etc.
454
# Windows: %APPDATA%\\myapp\\config.yaml
455
456
print(f"Config directory: {config.config_dir()}")
457
print(f"User config file: {config.user_config_path()}")
458
459
# Get all platform-specific directories
460
config_directories = confuse.config_dirs()
461
print("Config search paths:", config_directories)
462
```
463
464
### Error Handling
465
466
```python
467
import confuse
468
469
try:
470
# This may raise ConfigReadError if file is malformed
471
yaml_source = confuse.YamlSource('bad_config.yaml')
472
except confuse.ConfigReadError as e:
473
print(f"Configuration file error: {e}")
474
475
try:
476
# Load YAML string with error handling
477
config_data = confuse.load_yaml_string('''
478
invalid: [unclosed list
479
''', 'config string')
480
except confuse.ConfigReadError as e:
481
print(f"YAML parsing error: {e}")
482
483
# Optional file source (no error if missing)
484
optional_source = confuse.YamlSource('optional_config.yaml', optional=True)
485
config.add(optional_source)
486
```
487
488
### Configuration Dumping
489
490
```python
491
import confuse
492
493
config = confuse.Configuration('myapp')
494
config.set({'database': {'host': 'localhost', 'port': 5432}})
495
496
# Generate YAML representation
497
yaml_output = config.dump(full=True, redact=False)
498
print("Full configuration:")
499
print(yaml_output)
500
501
# Generate with sensitive values redacted
502
safe_output = config.dump(full=True, redact=True)
503
print("Safe configuration:")
504
print(safe_output)
505
506
# Custom YAML dumping with Confuse's Dumper
507
import confuse.yaml_util
508
509
data = {'servers': ['web1', 'web2'], 'enabled': True}
510
yaml_str = confuse.yaml_util.yaml.dump(data, Dumper=confuse.yaml_util.Dumper)
511
print(yaml_str) # Uses Confuse's formatting preferences
512
```