CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/pypi-google-cloud-bigquery-connection

Google Cloud BigQuery Connection API client library for managing external data source connections and credentials

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Overview
Eval results
Files

resource-path-helpers.mddocs/

Resource Path Helpers

Utility methods for constructing and parsing Google Cloud resource paths for BigQuery Connection resources. These helper methods simplify working with Google Cloud resource names and ensure proper formatting.

Capabilities

Connection Resource Paths

Methods for working with BigQuery Connection resource names.

@staticmethod
def connection_path(project: str, location: str, connection: str) -> str:
    """
    Returns a fully-qualified connection string.
    
    Parameters:
    - project: Project ID or project number
    - location: Location/region identifier
    - connection: Connection identifier
    
    Returns:
    str: Connection resource path in format 'projects/{project}/locations/{location}/connections/{connection}'
    """

@staticmethod
def parse_connection_path(path: str) -> Dict[str, str]:
    """
    Parses a connection path into its component parts.
    
    Parameters:
    - path: Connection resource path
    
    Returns:
    Dict[str, str]: Dictionary with keys 'project', 'location', 'connection'
    
    Raises:
    ValueError: If the path is not a valid connection path
    """

Usage Examples:

from google.cloud.bigquery_connection import ConnectionServiceClient

client = ConnectionServiceClient()

# Construct connection path
connection_path = client.connection_path(
    project="my-project",
    location="us-central1", 
    connection="analytics-db"
)
print(connection_path)
# Output: projects/my-project/locations/us-central1/connections/analytics-db

# Parse connection path
path_components = client.parse_connection_path(connection_path)
print(path_components)
# Output: {'project': 'my-project', 'location': 'us-central1', 'connection': 'analytics-db'}

# Use parsed components
project_id = path_components['project']
location = path_components['location']
connection_id = path_components['connection']

# Get the connection using parsed values
connection = client.get_connection(name=connection_path)

Cluster Resource Paths

Methods for working with cluster resource names (used with Spark connections).

@staticmethod
def cluster_path(project: str, region: str, cluster: str) -> str:
    """
    Returns a fully-qualified cluster string.
    
    Parameters:
    - project: Project ID or project number
    - region: Region identifier
    - cluster: Cluster identifier
    
    Returns:
    str: Cluster resource path in format 'projects/{project}/regions/{region}/clusters/{cluster}'
    """

@staticmethod
def parse_cluster_path(path: str) -> Dict[str, str]:
    """
    Parses a cluster path into its component parts.
    
    Parameters:
    - path: Cluster resource path
    
    Returns:
    Dict[str, str]: Dictionary with keys 'project', 'region', 'cluster'
    
    Raises:
    ValueError: If the path is not a valid cluster path
    """

Usage Examples:

# Construct cluster path for Spark History Server
cluster_path = client.cluster_path(
    project="my-project",
    region="us-central1",
    cluster="spark-history-cluster"
)
print(cluster_path)
# Output: projects/my-project/regions/us-central1/clusters/spark-history-cluster

# Use in Spark connection configuration
from google.cloud.bigquery_connection import (
    Connection,
    SparkProperties,
    SparkHistoryServerConfig
)

connection = Connection()
connection.spark = SparkProperties()
connection.spark.spark_history_server_config = SparkHistoryServerConfig()
connection.spark.spark_history_server_config.dataproc_cluster = cluster_path

# Parse cluster path
cluster_components = client.parse_cluster_path(cluster_path)
print(f"Cluster: {cluster_components['cluster']} in {cluster_components['region']}")

Service Resource Paths

Methods for working with service resource names (used with Spark Metastore connections).

@staticmethod
def service_path(project: str, location: str, service: str) -> str:
    """
    Returns a fully-qualified service string.
    
    Parameters:
    - project: Project ID or project number
    - location: Location/region identifier  
    - service: Service identifier
    
    Returns:
    str: Service resource path in format 'projects/{project}/locations/{location}/services/{service}'
    """

@staticmethod
def parse_service_path(path: str) -> Dict[str, str]:
    """
    Parses a service path into its component parts.
    
    Parameters:
    - path: Service resource path
    
    Returns:
    Dict[str, str]: Dictionary with keys 'project', 'location', 'service'
    
    Raises:  
    ValueError: If the path is not a valid service path
    """

Usage Examples:

# Construct service path for Dataproc Metastore
service_path = client.service_path(
    project="my-project",
    location="us-central1",
    service="analytics-metastore"
)
print(service_path)
# Output: projects/my-project/locations/us-central1/services/analytics-metastore

# Use in Spark connection configuration
from google.cloud.bigquery_connection import MetastoreServiceConfig

connection = Connection()
connection.spark = SparkProperties()
connection.spark.metastore_service_config = MetastoreServiceConfig()
connection.spark.metastore_service_config.metastore_service = service_path

# Parse service path
service_components = client.parse_service_path(service_path)
print(f"Service: {service_components['service']} in {service_components['location']}")

Common Resource Paths

Methods for working with common Google Cloud resource paths.

@staticmethod
def common_project_path(project: str) -> str:
    """
    Returns a fully-qualified project string.
    
    Parameters:
    - project: Project ID or project number
    
    Returns:
    str: Project resource path in format 'projects/{project}'
    """

@staticmethod
def parse_common_project_path(path: str) -> Dict[str, str]:
    """
    Parses a project path into its component parts.
    
    Parameters:
    - path: Project resource path
    
    Returns:
    Dict[str, str]: Dictionary with key 'project'
    """

@staticmethod
def common_location_path(project: str, location: str) -> str:
    """
    Returns a fully-qualified location string.
    
    Parameters:
    - project: Project ID or project number
    - location: Location/region identifier
    
    Returns:
    str: Location resource path in format 'projects/{project}/locations/{location}'
    """

@staticmethod
def parse_common_location_path(path: str) -> Dict[str, str]:
    """
    Parses a location path into its component parts.
    
    Parameters:
    - path: Location resource path
    
    Returns:
    Dict[str, str]: Dictionary with keys 'project', 'location'
    """

Usage Examples:

# Project path
project_path = client.common_project_path("my-project")
print(project_path)  # Output: projects/my-project

project_info = client.parse_common_project_path(project_path)
print(project_info)  # Output: {'project': 'my-project'}

# Location path (parent for connection operations)
location_path = client.common_location_path("my-project", "us-central1")
print(location_path)  # Output: projects/my-project/locations/us-central1

location_info = client.parse_common_location_path(location_path)
print(location_info)  # Output: {'project': 'my-project', 'location': 'us-central1'}

# Use location path as parent for listing connections
connections = client.list_connections(parent=location_path)

Billing Account Paths

Methods for working with billing account resource paths.

@staticmethod
def common_billing_account_path(billing_account: str) -> str:
    """
    Returns a fully-qualified billing account string.
    
    Parameters:
    - billing_account: Billing account ID
    
    Returns:
    str: Billing account resource path in format 'billingAccounts/{billing_account}'
    """

@staticmethod
def parse_common_billing_account_path(path: str) -> Dict[str, str]:
    """
    Parses a billing account path into its component parts.
    
    Parameters:
    - path: Billing account resource path
    
    Returns:
    Dict[str, str]: Dictionary with key 'billing_account'
    """

Folder and Organization Paths

Methods for working with resource hierarchy paths.

@staticmethod
def common_folder_path(folder: str) -> str:
    """
    Returns a fully-qualified folder string.
    
    Parameters:
    - folder: Folder ID
    
    Returns:
    str: Folder resource path in format 'folders/{folder}'
    """

@staticmethod
def parse_common_folder_path(path: str) -> Dict[str, str]:
    """
    Parses a folder path into its component parts.
    
    Parameters:
    - path: Folder resource path
    
    Returns:
    Dict[str, str]: Dictionary with key 'folder'
    """

@staticmethod
def common_organization_path(organization: str) -> str:
    """
    Returns a fully-qualified organization string.
    
    Parameters:
    - organization: Organization ID
    
    Returns:
    str: Organization resource path in format 'organizations/{organization}'
    """

@staticmethod
def parse_common_organization_path(path: str) -> Dict[str, str]:
    """
    Parses an organization path into its component parts.
    
    Parameters:
    - path: Organization resource path
    
    Returns:
    Dict[str, str]: Dictionary with key 'organization'
    """

Common Usage Patterns

Building Resource Names Dynamically

def create_connection_for_environment(env: str, region: str, conn_type: str):
    """Create connections with consistent naming patterns."""
    client = ConnectionServiceClient()
    
    # Build paths dynamically
    project_id = f"analytics-{env}"
    connection_id = f"{conn_type}-{env}-connection"
    
    parent = client.common_location_path(project_id, region)
    connection_name = client.connection_path(project_id, region, connection_id)
    
    print(f"Parent: {parent}")
    print(f"Connection: {connection_name}")
    
    # Create connection
    connection = Connection()
    connection.friendly_name = f"{conn_type.title()} {env.title()} Connection"
    
    result = client.create_connection(
        parent=parent,
        connection=connection,
        connection_id=connection_id
    )
    
    return result

# Usage
dev_connection = create_connection_for_environment("dev", "us-central1", "cloudsql")
prod_connection = create_connection_for_environment("prod", "us-central1", "cloudsql")

Parsing Resource Names from Configurations

def analyze_connection_config(config: dict):
    """Analyze connection configuration and extract resource details."""
    client = ConnectionServiceClient()
    
    connections = config.get("connections", [])
    
    for conn_config in connections:
        connection_name = conn_config["name"]
        
        # Parse the connection path
        try:
            components = client.parse_connection_path(connection_name)
            print(f"Connection: {components['connection']}")
            print(f"  Project: {components['project']}")
            print(f"  Location: {components['location']}")
            
            # Check if connection exists
            try:
                connection = client.get_connection(name=connection_name)
                print(f"  Status: Exists ({connection.friendly_name})")
            except Exception:
                print(f"  Status: Not found")
                
        except ValueError as e:
            print(f"Invalid connection path: {connection_name} - {e}")

# Example configuration
config = {
    "connections": [
        {"name": "projects/my-project/locations/us-central1/connections/prod-db"},
        {"name": "projects/my-project/locations/eu-west1/connections/analytics-db"},
    ]
}

analyze_connection_config(config)

Resource Path Validation

def validate_resource_paths(paths: dict) -> dict:
    """Validate and normalize resource paths."""
    client = ConnectionServiceClient()
    results = {}
    
    for name, path in paths.items():
        try:
            if "/connections/" in path:
                components = client.parse_connection_path(path)
                normalized = client.connection_path(**components)
                results[name] = {"valid": True, "normalized": normalized, "type": "connection"}
                
            elif "/clusters/" in path:
                components = client.parse_cluster_path(path)
                normalized = client.cluster_path(**components)
                results[name] = {"valid": True, "normalized": normalized, "type": "cluster"}
                
            elif "/services/" in path and "/locations/" in path:
                components = client.parse_service_path(path)
                normalized = client.service_path(**components)
                results[name] = {"valid": True, "normalized": normalized, "type": "service"}
                
            else:
                results[name] = {"valid": False, "error": "Unknown resource type"}
                
        except ValueError as e:
            results[name] = {"valid": False, "error": str(e)}
    
    return results

# Example usage
paths_to_validate = {
    "main_connection": "projects/my-project/locations/us-central1/connections/analytics",
    "spark_cluster": "projects/my-project/regions/us-central1/clusters/spark-cluster",
    "metastore": "projects/my-project/locations/us-central1/services/metastore",
    "invalid_path": "invalid/path/format"
}

validation_results = validate_resource_paths(paths_to_validate)
for name, result in validation_results.items():
    if result["valid"]:
        print(f"✓ {name}: {result['type']} - {result['normalized']}")
    else:
        print(f"✗ {name}: {result['error']}")

Types

Helper Method Return Types

from typing import Dict

# All parse methods return string dictionaries
PathComponents = Dict[str, str]

# Example path component structures:
ConnectionComponents = Dict[str, str]  # Keys: 'project', 'location', 'connection'
ClusterComponents = Dict[str, str]     # Keys: 'project', 'region', 'cluster'  
ServiceComponents = Dict[str, str]     # Keys: 'project', 'location', 'service'
ProjectComponents = Dict[str, str]     # Keys: 'project'
LocationComponents = Dict[str, str]    # Keys: 'project', 'location'
BillingComponents = Dict[str, str]     # Keys: 'billing_account'
FolderComponents = Dict[str, str]      # Keys: 'folder'
OrganizationComponents = Dict[str, str]  # Keys: 'organization'

Resource Name Formats

# Connection resource names
CONNECTION_FORMAT = "projects/{project}/locations/{location}/connections/{connection}"

# Cluster resource names (for Spark)
CLUSTER_FORMAT = "projects/{project}/regions/{region}/clusters/{cluster}"

# Service resource names (for Metastore)
SERVICE_FORMAT = "projects/{project}/locations/{location}/services/{service}"

# Common resource names
PROJECT_FORMAT = "projects/{project}"
LOCATION_FORMAT = "projects/{project}/locations/{location}"
BILLING_ACCOUNT_FORMAT = "billingAccounts/{billing_account}"
FOLDER_FORMAT = "folders/{folder}"
ORGANIZATION_FORMAT = "organizations/{organization}"

Error Handling

# All parse methods may raise ValueError for invalid paths
try:
    components = client.parse_connection_path(invalid_path)
except ValueError as e:
    print(f"Invalid connection path: {e}")

# Common validation patterns
def is_valid_connection_path(path: str) -> bool:
    """Check if a string is a valid connection path."""
    try:
        client.parse_connection_path(path)
        return True
    except ValueError:
        return False

Install with Tessl CLI

npx tessl i tessl/pypi-google-cloud-bigquery-connection

docs

client-configuration.md

connection-management.md

connection-types.md

iam-policy-management.md

index.md

resource-path-helpers.md

tile.json