0
# Data Source Management
1

2
Panoramix supports two types of data sources: SQL databases and Druid clusters. This module provides the core functionality for connecting to, managing, and synchronizing metadata from these data sources.
3

4
## Capabilities
5

6
### SQL Database Management
7

8
Register and manage connections to SQL databases using SQLAlchemy. Supports multiple database backends including PostgreSQL, MySQL, SQLite, and others supported by SQLAlchemy.
9

10
```python { .api }
11
class Database(Model, AuditMixin):
12
    """
13
    SQL database connection model.
14
    
15
    Attributes:
16
        id (int): Primary key
17
        database_name (str): Unique database identifier
18
        sqlalchemy_uri (str): SQLAlchemy connection string
19
    """
20
    
21
    def get_sqla_engine(self):
22
        """
23
        Get SQLAlchemy engine for this database.
24
        
25
        Returns:
26
            sqlalchemy.engine.Engine: Configured database engine
27
        """
28
    
29
    def get_table(self, table_name):
30
        """
31
        Get table metadata from the database.
32
        
33
        Args:
34
            table_name (str): Name of the table to retrieve
35
            
36
        Returns:
37
            sqlalchemy.Table: Table metadata object
38
        """
39
    
40
    def __repr__(self):
41
        """String representation of the database."""
42
        return self.database_name
43
```
44

45
Usage example:
46

47
```python
48
from panoramix.models import Database
49

50
# Create a database connection
51
db = Database(
52
    database_name='sales_db',
53
    sqlalchemy_uri='postgresql://user:pass@localhost/sales'
54
)
55

56
# Get database engine
57
engine = db.get_sqla_engine()
58

59
# Get table metadata
60
table_meta = db.get_table('customer_orders')
61
```
62

63
### Druid Cluster Management
64

65
Manage connections to Druid clusters for real-time analytics. Handles both coordinator and broker endpoints, with automatic datasource discovery and metadata synchronization.
66

67
```python { .api }
68
class Cluster(Model, AuditMixin):
69
    """
70
    Druid cluster configuration and management.
71
    
72
    Attributes:
73
        id (int): Primary key
74
        cluster_name (str): Unique cluster identifier
75
        coordinator_host (str): Druid coordinator hostname
76
        coordinator_port (int): Druid coordinator port
77
        coordinator_endpoint (str): Coordinator endpoint path
78
        broker_host (str): Druid broker hostname  
79
        broker_port (int): Druid broker port
80
        broker_endpoint (str): Broker endpoint path
81
        metadata_last_refreshed (datetime): Last metadata sync timestamp
82
    """
83
    
84
    def get_pydruid_client(self):
85
        """
86
        Get PyDruid client for querying this cluster.
87
        
88
        Returns:
89
            pydruid.client.Client: Configured PyDruid client
90
        """
91
    
92
    def refresh_datasources(self):
93
        """
94
        Sync datasources from the Druid cluster.
95
        
96
        Connects to the cluster coordinator to discover available
97
        datasources and updates the local metadata cache.
98
        """
99
    
100
    def __repr__(self):
101
        """String representation of the cluster."""
102
        return self.cluster_name
103
```
104

105
Usage example:
106

107
```python
108
from panoramix.models import Cluster
109

110
# Configure Druid cluster
111
cluster = Cluster(
112
    cluster_name='prod_cluster',
113
    coordinator_host='druid-coordinator.company.com',
114
    coordinator_port=8082,
115
    broker_host='druid-broker.company.com', 
116
    broker_port=8083
117
)
118

119
# Get PyDruid client
120
client = cluster.get_pydruid_client()
121

122
# Refresh datasource metadata
123
cluster.refresh_datasources()
124
```
125

126
### Connection Validation
127

128
Both database and cluster models include connection validation to ensure connectivity and proper configuration.
129

130
```python { .api }
131
# Health check endpoints
132
@app.route('/health')
133
def health():
134
    """System health check endpoint"""
135

136
@app.route('/ping') 
137
def ping():
138
    """Simple ping endpoint"""
139
```
140

141
## Integration with Flask-AppBuilder
142

143
Data source management integrates with Flask-AppBuilder to provide admin interfaces for managing databases and clusters through the web UI.
144

145
```python { .api }
146
class DatabaseView(ModelView):
147
    """Admin view for managing SQL databases"""
148

149
class ClusterModelView(ModelView): 
150
    """Admin view for managing Druid clusters"""
151
```
152

153
These views provide CRUD operations, connection testing, and metadata management through the web interface.