0
# Secrets Backend
1

2
Automatic retrieval of Airflow connections, variables, and configurations from HashiCorp Vault. The VaultBackend integrates seamlessly with Airflow's secrets management system, enabling transparent access to secrets stored in Vault without code changes.
3

4
## Capabilities
5

6
### VaultBackend Class
7

8
Secrets backend implementation that retrieves Airflow secrets from HashiCorp Vault with configurable path mapping and authentication.
9

10
```python { .api }
11
class VaultBackend(BaseSecretsBackend, LoggingMixin):
12
    def __init__(
13
        self,
14
        connections_path: str | None = "connections",
15
        variables_path: str | None = "variables",
16
        config_path: str | None = "config",
17
        url: str | None = None,
18
        auth_type: str = "token",
19
        auth_mount_point: str | None = None,
20
        mount_point: str | None = "secret",
21
        kv_engine_version: int = 2,
22
        token: str | None = None,
23
        token_path: str | None = None,
24
        username: str | None = None,
25
        password: str | None = None,
26
        key_id: str | None = None,
27
        secret_id: str | None = None,
28
        role_id: str | None = None,
29
        assume_role_kwargs: dict | None = None,
30
        region: str | None = None,
31
        kubernetes_role: str | None = None,
32
        kubernetes_jwt_path: str = "/var/run/secrets/kubernetes.io/serviceaccount/token",
33
        gcp_key_path: str | None = None,
34
        gcp_keyfile_dict: dict | None = None,
35
        gcp_scopes: str | None = None,
36
        azure_tenant_id: str | None = None,
37
        azure_resource: str | None = None,
38
        radius_host: str | None = None,
39
        radius_secret: str | None = None,
40
        radius_port: int | None = None,
41
        **kwargs
42
    ):
43
        """
44
        Initialize VaultBackend with path and authentication configuration.
45

46
        Parameters:
47
        - connections_path: Vault path for Airflow connections (default: "connections")
48
        - variables_path: Vault path for Airflow variables (default: "variables")
49
        - config_path: Vault path for Airflow configurations (default: "config")
50
        - url: Base URL for Vault instance
51
        - auth_type: Authentication method (default: "token")
52
        - auth_mount_point: Mount point for authentication method
53
        - mount_point: Secret engine mount point (default: "secret")
54
        - kv_engine_version: KV engine version (1 or 2, default: 2)
55
        - token: Authentication token (for token/github auth)
56
        - token_path: Path to token file (for token/github auth)
57
        - username: Username (for ldap/userpass auth)
58
        - password: Password (for ldap/userpass auth)
59
        - key_id: Key ID (for aws_iam/azure auth)
60
        - secret_id: Secret ID (for approle/aws_iam/azure auth)
61
        - role_id: Role ID (for approle/aws_iam auth)
62
        - assume_role_kwargs: AWS assume role parameters
63
        - region: AWS region for STS API calls
64
        - kubernetes_role: Kubernetes authentication role
65
        - kubernetes_jwt_path: Path to Kubernetes JWT token
66
        - gcp_key_path: Path to GCP service account key file
67
        - gcp_keyfile_dict: GCP keyfile parameters as dictionary
68
        - gcp_scopes: OAuth2 scopes for GCP authentication
69
        - azure_tenant_id: Azure AD tenant ID
70
        - azure_resource: Azure application URL
71
        - radius_host: RADIUS server host
72
        - radius_secret: RADIUS shared secret
73
        - radius_port: RADIUS server port
74
        """
75
```
76

77
### Connection Retrieval
78

79
Retrieve Airflow connection objects from Vault with automatic deserialization.
80

81
```python { .api }
82
def get_connection(self, conn_id: str) -> Connection | None:
83
    """
84
    Retrieve Airflow connection from Vault.
85

86
    Parameters:
87
    - conn_id: Connection identifier
88

89
    Returns:
90
    Connection | None: Airflow Connection object or None if not found
91
    """
92

93
def get_response(self, conn_id: str) -> dict | None:
94
    """
95
    Get raw response dictionary for connection from Vault.
96

97
    Parameters:
98
    - conn_id: Connection identifier
99

100
    Returns:
101
    dict | None: Raw connection data or None if not found
102
    """
103
```
104

105
### Variable Management
106

107
Access Airflow variables stored in Vault as key-value pairs.
108

109
```python { .api }
110
def get_variable(self, key: str) -> str | None:
111
    """
112
    Retrieve Airflow variable from Vault.
113

114
    Parameters:
115
    - key: Variable key name
116

117
    Returns:
118
    str | None: Variable value as string or None if not found
119
    """
120
```
121

122
### Configuration Access
123

124
Retrieve Airflow configuration values from Vault for dynamic configuration management.
125

126
```python { .api }
127
def get_config(self, key: str) -> str | None:
128
    """
129
    Retrieve Airflow configuration value from Vault.
130

131
    Parameters:
132
    - key: Configuration key name
133

134
    Returns:
135
    str | None: Configuration value as string or None if not found
136
    """
137
```
138

139
## Configuration Examples
140

141
### Airflow Configuration
142

143
Configure VaultBackend in `airflow.cfg`:
144

145
```ini
146
[secrets]
147
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
148
backend_kwargs = {
149
    "connections_path": "connections",
150
    "variables_path": "variables", 
151
    "url": "http://127.0.0.1:8200",
152
    "mount_point": "secret",
153
    "auth_type": "token",
154
    "token": "your-vault-token"
155
}
156
```
157

158
### Advanced Authentication
159

160
```ini
161
[secrets]
162
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
163
backend_kwargs = {
164
    "connections_path": "airflow/connections",
165
    "variables_path": "airflow/variables",
166
    "config_path": "airflow/config",
167
    "url": "https://vault.company.com:8200",
168
    "mount_point": "kv",
169
    "kv_engine_version": 2,
170
    "auth_type": "kubernetes",
171
    "kubernetes_role": "airflow-prod",
172
    "kubernetes_jwt_path": "/var/run/secrets/kubernetes.io/serviceaccount/token"
173
}
174
```
175

176
### Custom Path Configuration
177

178
```ini
179
backend_kwargs = {
180
    "connections_path": "prod/airflow/connections",
181
    "variables_path": "prod/airflow/variables",
182
    "config_path": "prod/airflow/config",
183
    "url": "https://vault.example.com",
184
    "mount_point": "secret-v2",
185
    "auth_type": "approle",
186
    "role_id": "12345678-1234-1234-1234-123456789012",
187
    "secret_id": "abcdef12-3456-7890-abcd-ef1234567890"
188
}
189
```
190

191
## Vault Secret Structure
192

193
### Connection Secrets
194

195
Store Airflow connections in Vault with standard connection attributes:
196

197
```json
198
{
199
  "conn_type": "postgres",
200
  "host": "db.example.com",
201
  "login": "airflow_user",
202
  "password": "secure_password",
203
  "schema": "airflow_db",
204
  "port": 5432,
205
  "extra": "{\"sslmode\": \"require\"}"
206
}
207
```
208

209
Vault path: `{mount_point}/{connections_path}/{conn_id}`
210

211
### Variable Secrets
212

213
Store variables as simple key-value pairs:
214

215
```json
216
{
217
  "value": "production_api_key_12345"
218
}
219
```
220

221
Vault path: `{mount_point}/{variables_path}/{variable_key}`
222

223
### Configuration Secrets
224

225
Store configuration values for dynamic Airflow configuration:
226

227
```json
228
{
229
  "value": "INFO"
230
}
231
```
232

233
Vault path: `{mount_point}/{config_path}/{config_key}`
234

235
## Usage in DAGs
236

237
Once configured, secrets are automatically retrieved without code changes:
238

239
```python
240
from airflow import DAG
241
from airflow.providers.postgres.operators.postgres import PostgresOperator
242
from airflow.models import Variable
243

244
# Connection automatically retrieved from Vault
245
postgres_task = PostgresOperator(
246
    task_id='run_query',
247
    postgres_conn_id='postgres_default',  # Retrieved from Vault
248
    sql='SELECT * FROM users;'
249
)
250

251
# Variable automatically retrieved from Vault  
252
api_key = Variable.get('api_key')  # Retrieved from Vault
253
```
254

255
## Path Resolution
256

257
The backend constructs Vault paths using the pattern:
258
`{mount_point}/{path_type}/{identifier}`
259

260
Examples:
261
- Connection `postgres_default`: `secret/connections/postgres_default`
262
- Variable `api_key`: `secret/variables/api_key`  
263
- Config `logging_level`: `secret/config/logging_level`
264

265
Set `mount_point` to `None` to disable the mount point prefix and use full paths directly.
266

267
## Types
268

269
```python { .api }
270
# External types used in API signatures
271
from airflow.models.connection import Connection
272
from airflow.secrets import BaseSecretsBackend
273
from airflow.utils.log.logging_mixin import LoggingMixin
274
```