tessl/pypi-grafana-backup

A Python-based application to backup Grafana settings using the Grafana API

—

Pending

Overview

Eval results

Files

Cloud Storage Integration

Name: tessl/pypi-grafana-backup
Author: tessl

Comprehensive cloud storage integration supporting Amazon S3, Azure Storage, and Google Cloud Storage for automated backup archiving and retrieval. The cloud storage system enables offsite backup storage with automatic upload after backup completion and download for restore operations.

Capabilities

AWS S3 Integration

Complete integration with Amazon S3 and S3-compatible services for backup storage and retrieval.

S3 Upload

def main(args, settings):
    """
    Upload backup archive to Amazon S3 bucket
    
    Module: grafana_backup.s3_upload
    Features: Multi-part upload, encryption support, custom endpoints
    Requirements: AWS credentials and bucket configuration
    """

S3 Download

def main(args, settings):
    """
    Download backup archive from Amazon S3 bucket
    
    Module: grafana_backup.s3_download
    Returns: BytesIO stream of compressed backup data
    Features: Streaming download, encryption support, custom endpoints
    """

S3 Common Utilities

# S3 utility functions
def get_s3_client(settings): ...  # Create configured S3 client
def get_s3_object_key(settings): ...  # Generate S3 object key with timestamp

Azure Storage Integration

Integration with Azure Blob Storage for backup archiving and retrieval.

Azure Upload

def main(args, settings):
    """
    Upload backup archive to Azure Blob Storage
    
    Module: grafana_backup.azure_storage_upload
    Features: Block blob upload, container management, connection string auth
    Requirements: Azure storage connection string and container configuration
    """

Azure Download

def main(args, settings):
    """
    Download backup archive from Azure Blob Storage
    
    Module: grafana_backup.azure_storage_download
    Returns: BytesIO stream of compressed backup data
    Features: Streaming download, container management
    """

Google Cloud Storage Integration

Integration with Google Cloud Storage for backup archiving and retrieval.

GCS Upload

def main(args, settings):
    """
    Upload backup archive to Google Cloud Storage bucket
    
    Module: grafana_backup.gcs_upload
    Features: Resumable upload, service account authentication, bucket management
    Requirements: GCS service account credentials and bucket configuration
    """

GCS Download

def main(args, settings):
    """
    Download backup archive from Google Cloud Storage bucket
    
    Module: grafana_backup.gcs_download
    Returns: BytesIO stream of compressed backup data
    Features: Streaming download, service account authentication
    """

Configuration Requirements

AWS S3 Configuration

# Required AWS S3 settings
AWS_S3_BUCKET_NAME: str  # S3 bucket name for backup storage
AWS_S3_BUCKET_KEY: str  # Object key prefix for organizing backups
AWS_DEFAULT_REGION: str  # AWS region for S3 bucket
AWS_ACCESS_KEY_ID: str  # AWS access key for authentication
AWS_SECRET_ACCESS_KEY: str  # AWS secret key for authentication
AWS_ENDPOINT_URL: str  # Custom endpoint URL for S3-compatible services (optional)

Azure Storage Configuration

# Required Azure storage settings
AZURE_STORAGE_CONTAINER_NAME: str  # Azure container name for backup storage
AZURE_STORAGE_CONNECTION_STRING: str  # Azure storage account connection string

Google Cloud Storage Configuration

# Required GCS settings
GCS_BUCKET_NAME: str  # GCS bucket name for backup storage
GOOGLE_APPLICATION_CREDENTIALS: str  # Path to service account JSON key file

Upload Workflow Integration

Automatic Upload After Backup

Cloud storage upload is automatically triggered after successful backup completion:

# Backup workflow with automatic cloud upload
from grafana_backup.save import main as save_backup

# Backup process automatically handles cloud upload
save_backup(args, settings)
# 1. Performs backup of selected components
# 2. Creates local archive (unless --no-archive specified)
# 3. Uploads to configured cloud storage providers
# 4. Reports upload status

Upload Priority Order

When multiple cloud storage providers are configured, uploads occur in this order:

AWS S3: If AWS_S3_BUCKET_NAME is configured
Azure Storage: If AZURE_STORAGE_CONTAINER_NAME is configured
Google Cloud Storage: If GCS_BUCKET_NAME is configured

Download Workflow Integration

Automatic Download During Restore

Cloud storage download is automatically triggered during restore operations when cloud storage is configured and no local file is specified:

# Restore workflow with automatic cloud download
from grafana_backup.restore import main as restore_backup

# Configure cloud storage
settings['AWS_S3_BUCKET_NAME'] = 'my-backups'

# Restore process automatically downloads from cloud
restore_backup(args, settings)
# 1. Detects cloud storage configuration
# 2. Downloads specified archive from cloud storage
# 3. Extracts and processes backup components
# 4. Restores to Grafana instance

Usage Examples

AWS S3 Backup and Restore

from grafana_backup.save import main as save_backup
from grafana_backup.restore import main as restore_backup
from grafana_backup.grafanaSettings import main as load_config

# Load configuration with S3 settings
settings = load_config('/path/to/config.json')
settings.update({
    'AWS_S3_BUCKET_NAME': 'my-grafana-backups',
    'AWS_S3_BUCKET_KEY': 'production',
    'AWS_DEFAULT_REGION': 'us-east-1',
    'AWS_ACCESS_KEY_ID': 'AKIAIOSFODNN7EXAMPLE',
    'AWS_SECRET_ACCESS_KEY': 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
})

# Backup with automatic S3 upload
save_args = {
    'save': True,
    '--components': None,
    '--no-archive': False,
    '--config': None
}
save_backup(save_args, settings)

# Restore from S3
restore_args = {
    'restore': True,
    '<archive_file>': 'backup_202501011200.tar.gz',
    '--components': None,
    '--config': None
}
restore_backup(restore_args, settings)

Azure Storage Backup and Restore

# Configure Azure Storage
settings.update({
    'AZURE_STORAGE_CONTAINER_NAME': 'grafana-backups',
    'AZURE_STORAGE_CONNECTION_STRING': 'DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=...'
})

# Backup with automatic Azure upload
save_backup(save_args, settings)

# Restore from Azure Storage
restore_backup(restore_args, settings)

Google Cloud Storage Backup and Restore

import os

# Configure GCS with service account
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/path/to/service-account.json'
settings.update({
    'GCS_BUCKET_NAME': 'my-grafana-backups'
})

# Backup with automatic GCS upload
save_backup(save_args, settings)

# Restore from GCS
restore_backup(restore_args, settings)

Multi-Cloud Configuration

# Configure multiple cloud providers (all will be used)
settings.update({
    # AWS S3
    'AWS_S3_BUCKET_NAME': 'primary-backups',
    'AWS_DEFAULT_REGION': 'us-east-1',
    
    # Azure Storage  
    'AZURE_STORAGE_CONTAINER_NAME': 'secondary-backups',
    'AZURE_STORAGE_CONNECTION_STRING': 'DefaultEndpointsProtocol=https;...',
    
    # Google Cloud Storage
    'GCS_BUCKET_NAME': 'tertiary-backups'
})

# Backup will upload to all configured providers
save_backup(save_args, settings)

Storage Organization

Backup File Naming

Cloud storage objects use consistent naming with timestamps:

Format: backup_{timestamp}.tar.gz
Timestamp: Based on BACKUP_FILE_FORMAT setting (default: %Y%m%d%H%M)
Example: backup_202501011200.tar.gz

S3 Object Keys

S3 objects are organized using the configured key prefix:

Pattern: {AWS_S3_BUCKET_KEY}/backup_{timestamp}.tar.gz
Example: production/backup_202501011200.tar.gz

Azure Blob Names

Azure blobs use simple naming without additional prefixes:

Pattern: backup_{timestamp}.tar.gz
Container: Specified by AZURE_STORAGE_CONTAINER_NAME

GCS Object Names

GCS objects use simple naming without additional prefixes:

Pattern: backup_{timestamp}.tar.gz
Bucket: Specified by GCS_BUCKET_NAME

Error Handling

Upload Error Handling

Network failures: Retry logic with exponential backoff
Authentication errors: Clear error messages for credential issues
Permission errors: Detailed bucket/container permission guidance
Storage quota: Appropriate error messages for storage limits

Download Error Handling

Missing objects: Clear messages when backup files don't exist
Network failures: Retry logic for temporary connectivity issues
Authentication errors: Credential validation before download attempts
Corruption detection: Integrity checks on downloaded data

Security Considerations

Credential Management

Environment variables: Use environment variables for production credentials
IAM roles: Prefer IAM roles over access keys for EC2/container deployments
Service accounts: Use service accounts with minimal required permissions
Connection strings: Protect Azure connection strings as sensitive data

Encryption

S3: Supports server-side encryption (SSE-S3, SSE-KMS)
Azure: Automatic encryption at rest for Blob Storage
GCS: Automatic encryption at rest with optional customer-managed keys

Access Control

S3: Configure bucket policies and IAM permissions appropriately
Azure: Use Azure RBAC for container access control
GCS: Configure IAM permissions for bucket and object access

Performance Optimization

Upload Performance

Multi-part uploads: Automatic for large archives
Streaming uploads: Memory-efficient upload process
Parallel uploads: Multiple cloud providers upload simultaneously

Download Performance

Streaming downloads: Memory-efficient download process
Compression: Archives are compressed to minimize transfer time
Resume capability: Support for resuming interrupted downloads where available

The cloud storage integration provides robust, production-ready backup storage with comprehensive error handling and security features.

Install with Tessl CLI