tessl/pypi-vdk-plugin-control-cli

Versatile Data Kit SDK plugin exposing CLI commands for managing the lifecycle of a Data Jobs.

Overview

Eval results

Files

Properties Backend

Name: tessl/pypi-vdk-plugin-control-cli
Author: tessl

Control Service-based properties storage and retrieval system for data jobs. The properties backend enables storing configuration, state, and other key-value data remotely through the Control Service Properties API, accessible through both CLI commands and the JobInput API in data job code.

Types

from typing import Dict
from vdk.api.plugin.plugin_input import IPropertiesServiceClient
from vdk.internal.builtin_plugins.run.job_context import JobContext

Capabilities

Properties Service Client

Implementation of IPropertiesServiceClient that connects to VDK Control Service Properties API.

class ControlServicePropertiesServiceClient(IPropertiesServiceClient):
    def __init__(self, rest_api_url: str):
        """
        Initialize Properties client for Control Service.
        
        Parameters:
        - rest_api_url: str - Base URL for Control Service REST API
        """
        
    @ConstrolServiceApiErrorDecorator()
    def read_properties(self, job_name: str, team_name: str):
        """
        Read properties for a data job from Control Service.
        
        Parameters:
        - job_name: str - Name of the data job
        - team_name: str - Name of the team owning the job
        
        Returns:
        dict: Properties data retrieved from Control Service
        """
        
    @ConstrolServiceApiErrorDecorator()
    def write_properties(self, job_name: str, team_name: str, properties: Dict) -> Dict:
        """
        Write properties for a data job to Control Service.
        
        Parameters:
        - job_name: str - Name of the data job
        - team_name: str - Name of the team owning the job
        - properties: Dict - Properties data to store
        
        Returns:
        Dict: The properties that were written
        """

Properties Plugin Initialization

Hook implementation that registers the Control Service properties backend with vdk-core.

@hookimpl
def initialize_job(context: JobContext) -> None:
    """
    Initialize Control Service Properties client implementation.
    
    Parameters:
    - context: JobContext - Job execution context
    
    Sets up properties factory methods for:
    - "default": Default properties backend
    - "control-service": Explicit Control Service backend
    """

Usage Patterns

CLI Usage

Access properties through vdk CLI commands:

# Set a property
vdk properties --set 'database-url' 'postgresql://localhost:5432/mydb'

# Set multiple properties
vdk properties --set 'api-key' 'abc123' --set 'timeout' '30'

# Get all properties
vdk properties --get-all

# Get specific property
vdk properties --get 'database-url'

JobInput API Usage

Access properties in data job code through the JobInput interface:

from vdk.api.job_input import IJobInput

def run(job_input: IJobInput):
    # Get a specific property
    api_url = job_input.get_property('api-url')
    
    # Get all properties
    all_props = job_input.get_all_properties()
    
    # Set properties
    job_input.set_property('last-run', '2023-01-15')
    
    # Set multiple properties
    new_props = {
        'status': 'completed',
        'record-count': '1000'
    }
    job_input.set_all_properties(new_props)

Programmatic Usage

Direct usage of the properties client:

from vdk.plugin.control_cli_plugin.control_service_properties_client import (
    ControlServicePropertiesServiceClient
)

# Initialize client
client = ControlServicePropertiesServiceClient("https://api.example.com")

# Read properties
properties = client.read_properties("my-job", "my-team")
print(f"Database URL: {properties.get('database-url')}")

# Write properties
new_properties = {
    "api-endpoint": "https://api.newservice.com",
    "timeout-seconds": "60"
}
client.write_properties("my-job", "my-team", new_properties)

Integration Details

Backend Registration

The properties plugin automatically registers with vdk-core during job initialization:

Checks if CONTROL_SERVICE_REST_API_URL is configured
If configured, registers ControlServicePropertiesServiceClient as both "default" and "control-service" backends
If not configured, logs warning and skips registration

Factory Method Setup

# Registered factory methods
context.properties.set_properties_factory_method(
    "default", 
    lambda: ControlServicePropertiesServiceClient(url)
)
context.properties.set_properties_factory_method(
    "control-service", 
    lambda: ControlServicePropertiesServiceClient(url)
)

Error Handling

All API calls are decorated with @ConstrolServiceApiErrorDecorator() which provides:

HTTP error handling with user-friendly messages
Authentication error detection and guidance
Retry logic for transient failures
Proper error categorization (user vs. platform errors)

Configuration Requirements

The properties backend requires these configuration values:

# Required
CONTROL_SERVICE_REST_API_URL = "https://api.example.com"

# Authentication (if required)
API_TOKEN = "your-api-token"
API_TOKEN_AUTHORIZATION_URL = "https://auth.example.com/oauth/token"

# Optional HTTP settings
CONTROL_HTTP_VERIFY_SSL = True
CONTROL_HTTP_TOTAL_RETRIES = 3
CONTROL_HTTP_READ_TIMEOUT_SECONDS = 30

Data Persistence

Properties are stored remotely in the Control Service with:

Job-scoped storage: Properties are isolated per job and team
Persistent storage: Data survives job executions and deployments
API-based access: Available through REST API and client libraries
Deployment integration: Uses deployment ID "TODO" (placeholder for future enhancement)

Install with Tessl CLI

npx tessl i tessl/pypi-vdk-plugin-control-cli

docs

properties-backend.md

secrets-backend.md

tile.json

tessl/pypi-vdk-plugin-control-cli