0
# Configuration Management
1
2
Kedro's configuration system provides flexible loading and management of project settings, parameters, and environment-specific configurations. It supports multiple file formats and enables parameter injection into pipelines.
3
4
## Capabilities
5
6
### Configuration Loaders
7
8
Base abstractions and implementations for loading configuration from various sources with environment support and parameter resolution.
9
10
```python { .api }
11
class AbstractConfigLoader:
12
"""Abstract base class for configuration loaders."""
13
14
def load_and_merge_dir_config(self, config_path, env=None, **kwargs):
15
"""
16
Load and merge configuration from directory.
17
18
Args:
19
config_path (str): Path to configuration directory
20
env (str, optional): Environment name for env-specific configs
21
**kwargs: Additional loader-specific arguments
22
23
Returns:
24
dict: Merged configuration dictionary
25
"""
26
27
def get(self, *patterns, **kwargs):
28
"""
29
Get configuration matching specified patterns.
30
31
Args:
32
*patterns: Configuration key patterns to match
33
**kwargs: Additional filtering arguments
34
35
Returns:
36
dict: Configuration values matching patterns
37
"""
38
39
class OmegaConfigLoader(AbstractConfigLoader):
40
"""Configuration loader using OmegaConf for YAML/JSON configs."""
41
42
def __init__(self, conf_source, base_env="base", default_run_env="local", **kwargs):
43
"""
44
Initialize OmegaConf-based configuration loader.
45
46
Args:
47
conf_source (str): Path to configuration source directory
48
base_env (str): Base environment name (default: "base")
49
default_run_env (str): Default runtime environment (default: "local")
50
**kwargs: Additional OmegaConf configuration options
51
"""
52
```
53
54
### Configuration Exceptions
55
56
Exception classes for configuration-related errors and validation failures.
57
58
```python { .api }
59
class BadConfigException(Exception):
60
"""Raised when configuration is malformed or invalid."""
61
62
class MissingConfigException(Exception):
63
"""Raised when required configuration is missing."""
64
```
65
66
## Usage Examples
67
68
### Basic Configuration Loading
69
70
```python
71
from kedro.config import OmegaConfigLoader
72
73
# Initialize configuration loader
74
config_loader = OmegaConfigLoader(
75
conf_source="conf",
76
base_env="base",
77
default_run_env="local"
78
)
79
80
# Load configuration for specific environment
81
config = config_loader.load_and_merge_dir_config(
82
config_path="conf",
83
env="production"
84
)
85
86
# Access configuration values
87
database_config = config_loader.get("database.*")
88
logging_config = config_loader.get("logging")
89
```
90
91
### Environment-Specific Configuration
92
93
```python
94
from kedro.config import OmegaConfigLoader
95
96
# Configuration structure:
97
# conf/
98
# base/
99
# catalog.yml
100
# parameters.yml
101
# production/
102
# catalog.yml
103
# parameters.yml
104
105
config_loader = OmegaConfigLoader(conf_source="conf", default_run_env="production")
106
107
# Loads base config merged with production overrides
108
catalog_config = config_loader.get("catalog")
109
parameters = config_loader.get("parameters")
110
```
111
112
### Custom Configuration Loader
113
114
```python
115
from kedro.config import AbstractConfigLoader
116
import yaml
117
from pathlib import Path
118
119
class CustomConfigLoader(AbstractConfigLoader):
120
"""Custom configuration loader with additional features."""
121
122
def load_and_merge_dir_config(self, config_path, env=None, **kwargs):
123
"""Load configuration with custom logic."""
124
base_path = Path(config_path) / "base"
125
env_path = Path(config_path) / env if env else None
126
127
# Load base configuration
128
config = {}
129
for yaml_file in base_path.glob("*.yml"):
130
with open(yaml_file) as f:
131
config.update(yaml.safe_load(f))
132
133
# Merge environment-specific config
134
if env_path and env_path.exists():
135
for yaml_file in env_path.glob("*.yml"):
136
with open(yaml_file) as f:
137
config.update(yaml.safe_load(f))
138
139
return config
140
141
# Usage
142
custom_loader = CustomConfigLoader()
143
config = custom_loader.load_and_merge_dir_config("conf", env="staging")
144
```
145
146
## Types
147
148
```python { .api }
149
from typing import Dict, Any, List, Optional, Union
150
151
ConfigDict = Dict[str, Any]
152
ConfigPath = Union[str, Path]
153
EnvName = Optional[str]
154
ConfigPatterns = List[str]
155
```