0
# Secrets Backend
1
2
Control Service-based secrets storage and retrieval system for data jobs. The secrets backend enables secure storage of sensitive information like API keys, passwords, and certificates remotely through the Control Service Secrets API, with encrypted storage and secure access patterns.
3
4
## Types
5
6
```python { .api }
7
from typing import Dict
8
from vdk.api.plugin.plugin_input import ISecretsServiceClient
9
from vdk.internal.builtin_plugins.run.job_context import JobContext
10
```
11
12
## Capabilities
13
14
### Secrets Service Client
15
16
Implementation of ISecretsServiceClient that connects to VDK Control Service Secrets API with secure handling.
17
18
```python { .api }
19
class ControlServiceSecretsServiceClient(ISecretsServiceClient):
20
def __init__(self, rest_api_url: str):
21
"""
22
Initialize Secrets client for Control Service.
23
24
Parameters:
25
- rest_api_url: str - Base URL for Control Service REST API
26
"""
27
28
@ConstrolServiceApiErrorDecorator()
29
def read_secrets(self, job_name: str, team_name: str):
30
"""
31
Read secrets for a data job from Control Service.
32
33
Parameters:
34
- job_name: str - Name of the data job
35
- team_name: str - Name of the team owning the job
36
37
Returns:
38
dict: Secrets data retrieved from Control Service (values may be masked)
39
"""
40
41
@ConstrolServiceApiErrorDecorator()
42
def write_secrets(self, job_name: str, team_name: str, secrets: Dict) -> Dict:
43
"""
44
Write secrets for a data job to Control Service.
45
46
Parameters:
47
- job_name: str - Name of the data job
48
- team_name: str - Name of the team owning the job
49
- secrets: Dict - Secrets data to store securely
50
51
Returns:
52
Dict: The secrets that were written (values may be masked)
53
"""
54
```
55
56
### Secrets Plugin Initialization
57
58
Hook implementation that registers the Control Service secrets backend with vdk-core.
59
60
```python { .api }
61
@hookimpl
62
def initialize_job(context: JobContext) -> None:
63
"""
64
Initialize Control Service Secrets client implementation.
65
66
Parameters:
67
- context: JobContext - Job execution context
68
69
Sets up secrets factory methods for:
70
- "default": Default secrets backend
71
- "control-service": Explicit Control Service backend
72
"""
73
```
74
75
## Usage Patterns
76
77
### CLI Usage
78
79
Access secrets through vdk CLI commands:
80
81
```bash
82
# Set a secret
83
vdk secrets --set 'api-key' 'sk-abc123xyz789'
84
85
# Set multiple secrets
86
vdk secrets --set 'db-password' 'secretpass' --set 'api-token' 'token123'
87
88
# Get all secrets (values may be masked for security)
89
vdk secrets --get-all
90
91
# Get specific secret
92
vdk secrets --get 'api-key'
93
```
94
95
### JobInput API Usage
96
97
Access secrets in data job code through the JobInput interface:
98
99
```python
100
from vdk.api.job_input import IJobInput
101
import requests
102
103
def run(job_input: IJobInput):
104
# Get a specific secret
105
api_key = job_input.get_secret('api-key')
106
107
# Get all secrets
108
all_secrets = job_input.get_all_secrets()
109
110
# Use secret in API call
111
headers = {'Authorization': f'Bearer {api_key}'}
112
response = requests.get('https://api.example.com/data', headers=headers)
113
114
# Set secrets (typically done during job setup)
115
job_input.set_secret('new-token', 'generated-token-value')
116
```
117
118
### Programmatic Usage
119
120
Direct usage of the secrets client:
121
122
```python
123
from vdk.plugin.control_cli_plugin.control_service_secrets_client import (
124
ControlServiceSecretsServiceClient
125
)
126
127
# Initialize client
128
client = ControlServiceSecretsServiceClient("https://api.example.com")
129
130
# Read secrets
131
secrets = client.read_secrets("my-job", "my-team")
132
api_key = secrets.get('api-key')
133
134
# Write secrets
135
new_secrets = {
136
"database-password": "super-secret-password",
137
"encryption-key": "base64-encoded-key"
138
}
139
client.write_secrets("my-job", "my-team", new_secrets)
140
```
141
142
## Security Features
143
144
### Secure Storage
145
146
- **Encryption at rest**: Secrets are encrypted when stored in Control Service
147
- **Encryption in transit**: API calls use HTTPS/TLS encryption
148
- **Access control**: Secrets are scoped to specific jobs and teams
149
- **Audit logging**: Access and modifications are logged for security auditing
150
151
### Access Patterns
152
153
- **Job-scoped access**: Secrets are only accessible to the owning job and team
154
- **Runtime access**: Secrets are retrieved at job execution time, not stored locally
155
- **Masked responses**: Secret values may be masked in API responses for security
156
- **Secure deletion**: Secrets can be securely removed when no longer needed
157
158
## Integration Details
159
160
### Backend Registration
161
162
The secrets plugin automatically registers with vdk-core during job initialization:
163
164
1. Checks if `CONTROL_SERVICE_REST_API_URL` is configured
165
2. If configured, registers `ControlServiceSecretsServiceClient` as both "default" and "control-service" backends
166
3. If not configured, logs warning and skips registration
167
168
### Factory Method Setup
169
170
```python
171
# Registered factory methods
172
context.secrets.set_secrets_factory_method(
173
"default",
174
lambda: ControlServiceSecretsServiceClient(url)
175
)
176
context.secrets.set_secrets_factory_method(
177
"control-service",
178
lambda: ControlServiceSecretsServiceClient(url)
179
)
180
```
181
182
### Error Handling
183
184
All API calls are decorated with `@ConstrolServiceApiErrorDecorator()` which provides:
185
186
- HTTP error handling with user-friendly messages
187
- Authentication error detection and secure error responses
188
- Retry logic for transient failures
189
- Proper error categorization without exposing sensitive information
190
- Security-focused error messages that don't leak secret details
191
192
## Configuration Requirements
193
194
The secrets backend requires these configuration values:
195
196
```python
197
# Required
198
CONTROL_SERVICE_REST_API_URL = "https://api.example.com"
199
200
# Authentication (required for secrets access)
201
API_TOKEN = "your-api-token"
202
API_TOKEN_AUTHORIZATION_URL = "https://auth.example.com/oauth/token"
203
204
# Security settings
205
CONTROL_HTTP_VERIFY_SSL = True # Strongly recommended for secrets
206
207
# Optional HTTP settings
208
CONTROL_HTTP_TOTAL_RETRIES = 3
209
CONTROL_HTTP_READ_TIMEOUT_SECONDS = 30
210
```
211
212
## Best Practices
213
214
### Secret Management
215
216
```python
217
def run(job_input: IJobInput):
218
# Good: Load secrets at runtime
219
api_key = job_input.get_secret('api-key')
220
221
# Good: Use secrets immediately, don't store
222
response = make_api_call(api_key)
223
224
# Avoid: Don't log secret values
225
# log.info(f"Using key: {api_key}") # BAD!
226
log.info("API call completed successfully") # Good
227
228
# Good: Clear sensitive data from memory
229
api_key = None
230
```
231
232
### Error Handling
233
234
```python
235
def run(job_input: IJobInput):
236
try:
237
secret = job_input.get_secret('required-secret')
238
if not secret:
239
raise ValueError("Required secret not configured")
240
except Exception as e:
241
# Good: Log errors without exposing secrets
242
log.error("Failed to retrieve required secret")
243
raise
244
```
245
246
## Data Persistence
247
248
Secrets are stored remotely in the Control Service with:
249
250
- **Encrypted storage**: Secrets are encrypted at rest and in transit
251
- **Job-scoped isolation**: Secrets are isolated per job and team
252
- **Persistent storage**: Secrets survive job executions and deployments
253
- **Secure API access**: Available through authenticated REST API
254
- **Deployment integration**: Uses deployment ID "TODO" (placeholder for future enhancement)
255
- **Audit trail**: Access and modifications are logged for compliance