CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/pypi-vdk-plugin-control-cli

Versatile Data Kit SDK plugin exposing CLI commands for managing the lifecycle of a Data Jobs.

Overview
Eval results
Files

secrets-backend.mddocs/

Secrets Backend

Control Service-based secrets storage and retrieval system for data jobs. The secrets backend enables secure storage of sensitive information like API keys, passwords, and certificates remotely through the Control Service Secrets API, with encrypted storage and secure access patterns.

Types

from typing import Dict
from vdk.api.plugin.plugin_input import ISecretsServiceClient
from vdk.internal.builtin_plugins.run.job_context import JobContext

Capabilities

Secrets Service Client

Implementation of ISecretsServiceClient that connects to VDK Control Service Secrets API with secure handling.

class ControlServiceSecretsServiceClient(ISecretsServiceClient):
    def __init__(self, rest_api_url: str):
        """
        Initialize Secrets client for Control Service.
        
        Parameters:
        - rest_api_url: str - Base URL for Control Service REST API
        """
        
    @ConstrolServiceApiErrorDecorator()
    def read_secrets(self, job_name: str, team_name: str):
        """
        Read secrets for a data job from Control Service.
        
        Parameters:
        - job_name: str - Name of the data job
        - team_name: str - Name of the team owning the job
        
        Returns:
        dict: Secrets data retrieved from Control Service (values may be masked)
        """
        
    @ConstrolServiceApiErrorDecorator()
    def write_secrets(self, job_name: str, team_name: str, secrets: Dict) -> Dict:
        """
        Write secrets for a data job to Control Service.
        
        Parameters:
        - job_name: str - Name of the data job
        - team_name: str - Name of the team owning the job
        - secrets: Dict - Secrets data to store securely
        
        Returns:
        Dict: The secrets that were written (values may be masked)
        """

Secrets Plugin Initialization

Hook implementation that registers the Control Service secrets backend with vdk-core.

@hookimpl
def initialize_job(context: JobContext) -> None:
    """
    Initialize Control Service Secrets client implementation.
    
    Parameters:
    - context: JobContext - Job execution context
    
    Sets up secrets factory methods for:
    - "default": Default secrets backend
    - "control-service": Explicit Control Service backend
    """

Usage Patterns

CLI Usage

Access secrets through vdk CLI commands:

# Set a secret
vdk secrets --set 'api-key' 'sk-abc123xyz789'

# Set multiple secrets
vdk secrets --set 'db-password' 'secretpass' --set 'api-token' 'token123'

# Get all secrets (values may be masked for security)
vdk secrets --get-all

# Get specific secret
vdk secrets --get 'api-key'

JobInput API Usage

Access secrets in data job code through the JobInput interface:

from vdk.api.job_input import IJobInput
import requests

def run(job_input: IJobInput):
    # Get a specific secret
    api_key = job_input.get_secret('api-key')
    
    # Get all secrets
    all_secrets = job_input.get_all_secrets()
    
    # Use secret in API call
    headers = {'Authorization': f'Bearer {api_key}'}
    response = requests.get('https://api.example.com/data', headers=headers)
    
    # Set secrets (typically done during job setup)
    job_input.set_secret('new-token', 'generated-token-value')

Programmatic Usage

Direct usage of the secrets client:

from vdk.plugin.control_cli_plugin.control_service_secrets_client import (
    ControlServiceSecretsServiceClient
)

# Initialize client
client = ControlServiceSecretsServiceClient("https://api.example.com")

# Read secrets
secrets = client.read_secrets("my-job", "my-team")
api_key = secrets.get('api-key')

# Write secrets
new_secrets = {
    "database-password": "super-secret-password",
    "encryption-key": "base64-encoded-key"
}
client.write_secrets("my-job", "my-team", new_secrets)

Security Features

Secure Storage

  • Encryption at rest: Secrets are encrypted when stored in Control Service
  • Encryption in transit: API calls use HTTPS/TLS encryption
  • Access control: Secrets are scoped to specific jobs and teams
  • Audit logging: Access and modifications are logged for security auditing

Access Patterns

  • Job-scoped access: Secrets are only accessible to the owning job and team
  • Runtime access: Secrets are retrieved at job execution time, not stored locally
  • Masked responses: Secret values may be masked in API responses for security
  • Secure deletion: Secrets can be securely removed when no longer needed

Integration Details

Backend Registration

The secrets plugin automatically registers with vdk-core during job initialization:

  1. Checks if CONTROL_SERVICE_REST_API_URL is configured
  2. If configured, registers ControlServiceSecretsServiceClient as both "default" and "control-service" backends
  3. If not configured, logs warning and skips registration

Factory Method Setup

# Registered factory methods
context.secrets.set_secrets_factory_method(
    "default", 
    lambda: ControlServiceSecretsServiceClient(url)
)
context.secrets.set_secrets_factory_method(
    "control-service", 
    lambda: ControlServiceSecretsServiceClient(url)
)

Error Handling

All API calls are decorated with @ConstrolServiceApiErrorDecorator() which provides:

  • HTTP error handling with user-friendly messages
  • Authentication error detection and secure error responses
  • Retry logic for transient failures
  • Proper error categorization without exposing sensitive information
  • Security-focused error messages that don't leak secret details

Configuration Requirements

The secrets backend requires these configuration values:

# Required
CONTROL_SERVICE_REST_API_URL = "https://api.example.com"

# Authentication (required for secrets access)
API_TOKEN = "your-api-token"
API_TOKEN_AUTHORIZATION_URL = "https://auth.example.com/oauth/token"

# Security settings
CONTROL_HTTP_VERIFY_SSL = True  # Strongly recommended for secrets

# Optional HTTP settings
CONTROL_HTTP_TOTAL_RETRIES = 3
CONTROL_HTTP_READ_TIMEOUT_SECONDS = 30

Best Practices

Secret Management

def run(job_input: IJobInput):
    # Good: Load secrets at runtime
    api_key = job_input.get_secret('api-key')
    
    # Good: Use secrets immediately, don't store
    response = make_api_call(api_key)
    
    # Avoid: Don't log secret values
    # log.info(f"Using key: {api_key}")  # BAD!
    log.info("API call completed successfully")  # Good
    
    # Good: Clear sensitive data from memory
    api_key = None

Error Handling

def run(job_input: IJobInput):
    try:
        secret = job_input.get_secret('required-secret')
        if not secret:
            raise ValueError("Required secret not configured")
    except Exception as e:
        # Good: Log errors without exposing secrets
        log.error("Failed to retrieve required secret")
        raise

Data Persistence

Secrets are stored remotely in the Control Service with:

  • Encrypted storage: Secrets are encrypted at rest and in transit
  • Job-scoped isolation: Secrets are isolated per job and team
  • Persistent storage: Secrets survive job executions and deployments
  • Secure API access: Available through authenticated REST API
  • Deployment integration: Uses deployment ID "TODO" (placeholder for future enhancement)
  • Audit trail: Access and modifications are logged for compliance

Install with Tessl CLI

npx tessl i tessl/pypi-vdk-plugin-control-cli

docs

cli-commands.md

configuration.md

error-handling.md

execution-control.md

index.md

properties-backend.md

secrets-backend.md

tile.json