tessl/pypi-grafana-backup

A Python-based application to backup Grafana settings using the Grafana API

—

Pending

Overview

Eval results

Files

Monitoring Integration

Name: tessl/pypi-grafana-backup
Author: tessl

InfluxDB integration for backup operation monitoring and metrics collection, enabling operational visibility into backup processes. The monitoring system tracks backup operations, performance metrics, and operational health indicators.

Capabilities

InfluxDB Metrics Collection

Comprehensive metrics collection and reporting to InfluxDB for backup operation monitoring.

def main(args, settings):
    """
    Send backup operation metrics to InfluxDB
    
    Module: grafana_backup.influx
    Args:
        args (dict): Command line arguments from backup operation
        settings (dict): Configuration settings including InfluxDB connection details
        
    Features: Operation metrics, timing data, success/failure tracking
    Metrics: Backup duration, component counts, operation status
    """

Configuration Requirements

InfluxDB Connection Settings

# InfluxDB configuration settings
INFLUXDB_MEASUREMENT: str  # InfluxDB measurement name (default: "grafana_backup")
INFLUXDB_HOST: str         # InfluxDB server hostname
INFLUXDB_PORT: int         # InfluxDB server port (default: 8086)
INFLUXDB_USERNAME: str     # InfluxDB username for authentication
INFLUXDB_PASSWORD: str     # InfluxDB password for authentication
INFLUXDB_DATABASE: str     # InfluxDB database name for metrics storage

Configuration File Example

{
  "influxdb": {
    "measurement": "grafana_backup",
    "host": "localhost",
    "port": 8086,
    "username": "monitoring",
    "password": "monitoring_password",
    "database": "grafana_metrics"
  }
}

Environment Variable Configuration

export INFLUXDB_MEASUREMENT="grafana_backup"
export INFLUXDB_HOST="influxdb.example.com"
export INFLUXDB_PORT=8086
export INFLUXDB_USERNAME="monitoring"
export INFLUXDB_PASSWORD="secure_password"
export INFLUXDB_DATABASE="grafana_metrics"

Metrics Collection

Automatic Metrics Collection

Metrics collection is automatically triggered after successful backup completion when InfluxDB is configured:

# Backup workflow with automatic metrics collection
from grafana_backup.save import main as save_backup

# Configure InfluxDB in settings
settings['INFLUXDB_HOST'] = 'influxdb.example.com'
settings['INFLUXDB_DATABASE'] = 'grafana_metrics'

# Backup process automatically sends metrics if InfluxDB is configured
save_args = {
    'save': True,
    '--components': None,
    '--no-archive': False,
    '--config': None
}

save_backup(save_args, settings)
# 1. Performs backup operations
# 2. Creates archive (if enabled)
# 3. Uploads to cloud storage (if configured)
# 4. Sends metrics to InfluxDB (if configured)

Collected Metrics

The monitoring system collects comprehensive metrics about backup operations:

Operation Metrics

Operation type: backup, restore, delete
Operation status: success, failure, partial
Start time: ISO timestamp of operation start
End time: ISO timestamp of operation completion
Duration: Total operation duration in seconds

Component Metrics

Components processed: List of Grafana components included
Component counts: Number of items backed up per component type
Component timing: Time spent processing each component type
Component status: Success/failure status per component

System Metrics

Grafana instance: Source Grafana server information
Archive size: Size of created backup archive
File counts: Number of files created per component
Error counts: Number of errors encountered during operation

Performance Metrics

API response times: Grafana API call performance
Data transfer rates: Backup and upload throughput
Resource utilization: Memory and disk usage during operations
Concurrent operations: Number of parallel operations performed

Usage Examples

Basic Monitoring Setup

from grafana_backup.save import main as save_backup
from grafana_backup.grafanaSettings import main as load_config

# Load configuration with InfluxDB settings
settings = load_config('/path/to/grafanaSettings.json')

# Ensure InfluxDB is configured
settings.update({
    'INFLUXDB_HOST': 'influxdb.example.com',
    'INFLUXDB_PORT': 8086,
    'INFLUXDB_USERNAME': 'monitoring',
    'INFLUXDB_PASSWORD': 'secure_password',
    'INFLUXDB_DATABASE': 'grafana_metrics',
    'INFLUXDB_MEASUREMENT': 'grafana_backup'
})

# Perform backup with automatic metrics collection
save_args = {
    'save': True,
    '--components': None,
    '--no-archive': False,
    '--config': None
}

save_backup(save_args, settings)

Manual Metrics Sending

from grafana_backup.influx import main as send_metrics

# Send metrics manually after operations
metrics_args = {
    'save': True,
    '--components': 'dashboards,datasources',
    '--config': None
}

send_metrics(metrics_args, settings)

Metrics Collection for All Operations

# Backup operations automatically collect metrics
save_backup(save_args, settings)

# Restore operations can also collect metrics (if implemented)
from grafana_backup.restore import main as restore_backup
restore_args = {
    'restore': True,
    '<archive_file>': 'backup_202501011200.tar.gz',
    '--components': None,
    '--config': None
}
restore_backup(restore_args, settings)

# Delete operations can collect metrics (if implemented)
from grafana_backup.delete import main as delete_components
delete_args = {
    'delete': True,
    '--components': 'snapshots',
    '--config': None
}
delete_components(delete_args, settings)

InfluxDB Data Schema

Measurement Structure

Metrics are stored in InfluxDB using a structured measurement format:

measurement: grafana_backup (configurable via INFLUXDB_MEASUREMENT)

tags:
  - operation_type: "backup" | "restore" | "delete"
  - operation_status: "success" | "failure" | "partial"
  - grafana_host: Grafana server hostname
  - components: Comma-separated list of components processed
  - archive_created: "true" | "false"
  - cloud_upload: "s3" | "azure" | "gcs" | "none"

fields:
  - duration: Operation duration in seconds (float)
  - start_time: Operation start timestamp (string)
  - end_time: Operation end timestamp (string)
  - dashboard_count: Number of dashboards processed (integer)
  - datasource_count: Number of datasources processed (integer)
  - folder_count: Number of folders processed (integer)
  - user_count: Number of users processed (integer)
  - team_count: Number of teams processed (integer)
  - alert_count: Number of alerts processed (integer)
  - snapshot_count: Number of snapshots processed (integer)
  - annotation_count: Number of annotations processed (integer)
  - library_element_count: Number of library elements processed (integer)
  - archive_size_bytes: Size of created archive in bytes (integer)
  - total_files: Total number of files created (integer)
  - error_count: Number of errors encountered (integer)
  - api_calls: Number of Grafana API calls made (integer)
  - avg_api_response_time: Average API response time in milliseconds (float)

Example InfluxDB Query

Query backup operation metrics:

-- Get recent backup operations
SELECT * FROM grafana_backup 
WHERE time > now() - 24h 
AND operation_type = 'backup'

-- Calculate average backup duration by component set
SELECT MEAN(duration) as avg_duration
FROM grafana_backup 
WHERE time > now() - 7d 
AND operation_type = 'backup'
GROUP BY components

-- Monitor backup success rate
SELECT COUNT(*) as total_operations,
       SUM(CASE WHEN operation_status = 'success' THEN 1 ELSE 0 END) as successful_operations
FROM grafana_backup 
WHERE time > now() - 30d

Monitoring Dashboards

Grafana Dashboard Integration

Create Grafana dashboards to visualize backup metrics:

Backup Operation Overview

Success rate: Percentage of successful backup operations
Operation frequency: Number of backups per day/week
Duration trends: Backup duration over time
Component breakdown: Items backed up by component type

Performance Monitoring

API performance: Grafana API response times
Throughput metrics: Data processing rates
Resource utilization: System resource usage during backups
Error tracking: Error counts and types over time

Operational Health

Last successful backup: Time since last successful backup
Backup size trends: Archive size growth over time
Component changes: Changes in component counts over time
Cloud upload status: Success rate of cloud storage uploads

Alerting Integration

Set up alerts based on backup metrics:

-- Alert on backup failures
SELECT COUNT(*) FROM grafana_backup 
WHERE time > now() - 6h 
AND operation_status != 'success'

-- Alert on backup duration anomalies
SELECT duration FROM grafana_backup 
WHERE time > now() - 1h 
AND duration > (SELECT MEAN(duration) * 2 FROM grafana_backup WHERE time > now() - 7d)

-- Alert on missing backups
SELECT COUNT(*) FROM grafana_backup 
WHERE time > now() - 25h 
AND operation_type = 'backup'
HAVING COUNT(*) = 0

Integration Benefits

Operational Visibility

InfluxDB integration provides comprehensive operational visibility:

Backup health monitoring: Track backup success rates and identify issues
Performance optimization: Identify bottlenecks and optimize backup processes
Capacity planning: Monitor backup size growth and resource requirements
Compliance reporting: Generate reports on backup frequency and success

Automation and Alerting

Enable automated monitoring and alerting:

Proactive issue detection: Alert on backup failures before they become critical
Performance regression detection: Identify performance degradation trends
Capacity alerts: Warning when backup sizes or durations exceed thresholds
Compliance monitoring: Ensure backup schedules meet organizational requirements

Best Practices

Monitoring Configuration

Dedicated database: Use a dedicated InfluxDB database for backup metrics
Retention policies: Configure appropriate data retention for metrics
Security: Use dedicated monitoring credentials with minimal required permissions
Network security: Secure InfluxDB communication with TLS when possible

Dashboard Design

Key metrics focus: Prioritize the most important operational metrics
Time range selection: Provide multiple time range options for analysis
Drill-down capability: Enable detailed investigation of issues
Alert integration: Link dashboards to alerting systems

Alerting Strategy

Threshold tuning: Set appropriate alert thresholds based on historical data
Alert fatigue prevention: Avoid overly sensitive alerts that create noise
Escalation procedures: Define clear escalation paths for different alert types
Documentation: Maintain runbooks for common alert scenarios

The monitoring integration provides essential operational visibility for production backup operations, enabling proactive management and continuous improvement of backup processes.

Install with Tessl CLI