0
# Storage Backends
1
2
Pluggable storage backend implementations for different persistence requirements, from in-memory storage for temporary operations to Redis-based storage for distributed scenarios.
3
4
## Capabilities
5
6
### BaseStore Interface
7
8
Abstract base class defining the interface that all storage backends must implement.
9
10
```python { .api }
11
class BaseStore:
12
"""Reference store to be implemented in different backends."""
13
14
def __init__(self, *args: Any, adapter: Optional["Adapter"] = None,
15
name: str = "", **kwargs: Any) -> None:
16
"""
17
Init method for BaseStore.
18
19
Args:
20
adapter: Associated adapter instance
21
name: Store name (defaults to class name)
22
"""
23
24
adapter: Optional["Adapter"]
25
name: str
26
```
27
28
### Core Storage Operations
29
30
Abstract methods that all storage backends must implement.
31
32
```python { .api }
33
def get_all_model_names(self) -> Set[str]:
34
"""
35
Get all the model names stored.
36
37
Returns:
38
Set of all the model names
39
"""
40
```
41
42
```python { .api }
43
def get(self, *, model: Union[str, "DiffSyncModel", Type["DiffSyncModel"]],
44
identifier: Union[str, Dict]) -> "DiffSyncModel":
45
"""
46
Get one object from the data store based on its unique id.
47
48
Args:
49
model: DiffSyncModel class or instance, or modelname string, that defines the type of the object to retrieve
50
identifier: Unique ID of the object to retrieve, or dict of unique identifier keys/values
51
52
Raises:
53
ValueError: if obj is a str and identifier is a dict (can't convert dict into a uid str without a model class)
54
ObjectNotFound: if the requested object is not present
55
"""
56
```
57
58
```python { .api }
59
def get_all(self, *, model: Union[str, "DiffSyncModel", Type["DiffSyncModel"]]) -> List["DiffSyncModel"]:
60
"""
61
Get all objects of a given type.
62
63
Args:
64
model: DiffSyncModel class or instance, or modelname string, that defines the type of the objects to retrieve
65
66
Returns:
67
List of objects
68
"""
69
```
70
71
```python { .api }
72
def get_by_uids(self, *, uids: List[str],
73
model: Union[str, "DiffSyncModel", Type["DiffSyncModel"]]) -> List["DiffSyncModel"]:
74
"""
75
Get multiple objects from the store by their unique IDs/Keys and type.
76
77
Args:
78
uids: List of unique id / key identifying object in the database
79
model: DiffSyncModel class or instance, or modelname string, that defines the type of the objects to retrieve
80
81
Raises:
82
ObjectNotFound: if any of the requested UIDs are not found in the store
83
"""
84
```
85
86
```python { .api }
87
def add(self, *, obj: "DiffSyncModel") -> None:
88
"""
89
Add a DiffSyncModel object to the store.
90
91
Args:
92
obj: Object to store
93
94
Raises:
95
ObjectAlreadyExists: if a different object with the same uid is already present
96
"""
97
```
98
99
```python { .api }
100
def update(self, *, obj: "DiffSyncModel") -> None:
101
"""
102
Update a DiffSyncModel object to the store.
103
104
Args:
105
obj: Object to update
106
"""
107
```
108
109
```python { .api }
110
def remove(self, *, obj: "DiffSyncModel", remove_children: bool = False) -> None:
111
"""
112
Remove a DiffSyncModel object from the store.
113
114
Args:
115
obj: object to remove
116
remove_children: If True, also recursively remove any children of this object
117
118
Raises:
119
ObjectNotFound: if the object is not present
120
"""
121
```
122
123
```python { .api }
124
def count(self, *, model: Union[str, "DiffSyncModel", Type["DiffSyncModel"], None] = None) -> int:
125
"""Returns the number of elements of a specific model, or all elements in the store if not specified."""
126
```
127
128
### Convenience Methods
129
130
Common operations implemented in the base class using the abstract methods.
131
132
```python { .api }
133
def get_or_instantiate(self, *, model: Type["DiffSyncModel"], ids: Dict,
134
attrs: Optional[Dict] = None) -> Tuple["DiffSyncModel", bool]:
135
"""
136
Attempt to get the object with provided identifiers or instantiate it with provided identifiers and attrs.
137
138
Args:
139
model: The DiffSyncModel to get or create
140
ids: Identifiers for the DiffSyncModel to get or create with
141
attrs: Attributes when creating an object if it doesn't exist. Defaults to None
142
143
Returns:
144
Tuple of (existing or new object, whether it was created)
145
"""
146
```
147
148
```python { .api }
149
def get_or_add_model_instance(self, obj: "DiffSyncModel") -> Tuple["DiffSyncModel", bool]:
150
"""
151
Attempt to get the object with provided obj identifiers or instantiate obj.
152
153
Args:
154
obj: An obj of the DiffSyncModel to get or add
155
156
Returns:
157
Tuple of (existing or new object, whether it was added)
158
"""
159
```
160
161
```python { .api }
162
def update_or_instantiate(self, *, model: Type["DiffSyncModel"], ids: Dict,
163
attrs: Dict) -> Tuple["DiffSyncModel", bool]:
164
"""
165
Attempt to update an existing object with provided ids/attrs or instantiate it with provided identifiers and attrs.
166
167
Args:
168
model: The DiffSyncModel to update or create
169
ids: Identifiers for the DiffSyncModel to update or create with
170
attrs: Attributes when creating/updating an object if it doesn't exist. Pass in empty dict, if no specific attrs
171
172
Returns:
173
Tuple of (existing or new object, whether it was created)
174
"""
175
```
176
177
```python { .api }
178
def update_or_add_model_instance(self, obj: "DiffSyncModel") -> Tuple["DiffSyncModel", bool]:
179
"""
180
Attempt to update an existing object with provided ids/attrs or instantiate obj.
181
182
Args:
183
obj: An instance of the DiffSyncModel to update or create
184
185
Returns:
186
Tuple of (existing or new object, whether it was added)
187
"""
188
```
189
190
## LocalStore Implementation
191
192
In-memory storage backend using Python dictionaries. Default storage backend for Adapter instances.
193
194
```python { .api }
195
class LocalStore(BaseStore):
196
"""LocalStore class."""
197
198
def __init__(self, *args: Any, **kwargs: Any) -> None:
199
"""Init method for LocalStore."""
200
201
_data: Dict
202
```
203
204
LocalStore uses a nested dictionary structure (`defaultdict(dict)`) to organize data by model name and unique ID:
205
206
```
207
_data = {
208
"device": {
209
"router1": <Device instance>,
210
"switch1": <Device instance>
211
},
212
"interface": {
213
"router1__eth0": <Interface instance>,
214
"router1__eth1": <Interface instance>
215
}
216
}
217
```
218
219
### LocalStore Usage
220
221
```python
222
from diffsync import Adapter, LocalStore
223
224
# LocalStore is the default, these are equivalent:
225
adapter1 = MyAdapter()
226
adapter2 = MyAdapter(internal_storage_engine=LocalStore)
227
228
# Both adapters will use in-memory storage
229
adapter1.load()
230
adapter2.load()
231
232
# Data is stored in memory and will be lost when the process ends
233
print(f"Adapter1 has {len(adapter1)} objects")
234
print(f"Adapter2 has {len(adapter2)} objects")
235
```
236
237
## RedisStore Implementation
238
239
Redis-based storage backend for persistent and distributed storage scenarios.
240
241
```python { .api }
242
class RedisStore(BaseStore):
243
"""RedisStore class."""
244
245
def __init__(self, *args: Any, store_id: Optional[str] = None,
246
host: Optional[str] = None, port: int = 6379,
247
url: Optional[str] = None, db: int = 0, **kwargs: Any):
248
"""
249
Init method for RedisStore.
250
251
Args:
252
store_id: Optional unique identifier for this store instance
253
host: Redis server hostname
254
port: Redis server port (default: 6379)
255
url: Redis connection URL (alternative to host/port)
256
db: Redis database number (default: 0)
257
258
Raises:
259
ValueError: if both url and host are specified
260
ObjectStoreException: if Redis is unavailable
261
"""
262
263
_store: Redis
264
_store_id: str
265
_store_label: str
266
```
267
268
RedisStore requires the `redis` extra to be installed:
269
270
```bash
271
pip install diffsync[redis]
272
```
273
274
### RedisStore Key Structure
275
276
RedisStore organizes data using hierarchical Redis keys:
277
278
```
279
diffsync:<store_id>:<model_name>:<unique_id>
280
```
281
282
Examples:
283
- `diffsync:12345:device:router1`
284
- `diffsync:12345:interface:router1__eth0`
285
286
### RedisStore Usage
287
288
```python
289
from diffsync import Adapter
290
from diffsync.store.redis import RedisStore
291
292
# Connect to Redis on localhost
293
adapter = MyAdapter(
294
internal_storage_engine=RedisStore(host="localhost", port=6379, db=0)
295
)
296
297
# Connect using Redis URL
298
adapter = MyAdapter(
299
internal_storage_engine=RedisStore(url="redis://localhost:6379/0")
300
)
301
302
# Use specific store ID for multiple isolated datasets
303
adapter1 = MyAdapter(
304
internal_storage_engine=RedisStore(host="localhost", store_id="dataset1")
305
)
306
adapter2 = MyAdapter(
307
internal_storage_engine=RedisStore(host="localhost", store_id="dataset2")
308
)
309
310
# Load data - will be persisted to Redis
311
adapter1.load()
312
adapter2.load()
313
314
# Data persists across process restarts
315
print(f"Dataset1 has {len(adapter1)} objects")
316
print(f"Dataset2 has {len(adapter2)} objects")
317
```
318
319
### RedisStore Advanced Usage
320
321
```python
322
# Distributed scenario - multiple processes sharing data
323
class SharedAdapter(Adapter):
324
device = Device
325
top_level = ["device"]
326
327
def __init__(self, process_name):
328
# All processes use the same store_id to share data
329
super().__init__(
330
name=process_name,
331
internal_storage_engine=RedisStore(
332
host="redis-server.example.com",
333
store_id="shared_network_data"
334
)
335
)
336
337
# Process 1: Load initial data
338
process1 = SharedAdapter("loader")
339
process1.load() # Loads data into Redis
340
341
# Process 2: Access the same data
342
process2 = SharedAdapter("consumer")
343
devices = process2.get_all("device") # Reads from Redis
344
print(f"Found {len(devices)} devices loaded by process1")
345
346
# Process 3: Sync with external source
347
process3 = SharedAdapter("syncer")
348
external_source = ExternalAdapter()
349
external_source.load()
350
351
# Sync will update the shared Redis data
352
process3.sync_from(external_source)
353
```
354
355
## Custom Storage Backends
356
357
You can implement custom storage backends by subclassing BaseStore.
358
359
### Custom Backend Example
360
361
```python
362
import sqlite3
363
from diffsync.store import BaseStore
364
from diffsync.exceptions import ObjectNotFound, ObjectAlreadyExists
365
366
class SQLiteStore(BaseStore):
367
def __init__(self, database_path, **kwargs):
368
super().__init__(**kwargs)
369
self.db_path = database_path
370
self._init_database()
371
372
def _init_database(self):
373
with sqlite3.connect(self.db_path) as conn:
374
conn.execute('''
375
CREATE TABLE IF NOT EXISTS objects (
376
model_name TEXT,
377
unique_id TEXT,
378
data BLOB,
379
PRIMARY KEY (model_name, unique_id)
380
)
381
''')
382
383
def get_all_model_names(self):
384
with sqlite3.connect(self.db_path) as conn:
385
cursor = conn.execute("SELECT DISTINCT model_name FROM objects")
386
return {row[0] for row in cursor.fetchall()}
387
388
def get(self, *, model, identifier):
389
object_class, modelname = self._get_object_class_and_model(model)
390
uid = self._get_uid(model, object_class, identifier)
391
392
with sqlite3.connect(self.db_path) as conn:
393
cursor = conn.execute(
394
"SELECT data FROM objects WHERE model_name = ? AND unique_id = ?",
395
(modelname, uid)
396
)
397
row = cursor.fetchone()
398
if not row:
399
raise ObjectNotFound(f"{modelname} {uid} not found")
400
401
# Deserialize object from database
402
obj_data = pickle.loads(row[0])
403
obj_data.adapter = self.adapter
404
return obj_data
405
406
def add(self, *, obj):
407
modelname = obj.get_type()
408
uid = obj.get_unique_id()
409
410
# Check if object already exists
411
try:
412
existing = self.get(model=modelname, identifier=uid)
413
if existing is not obj:
414
raise ObjectAlreadyExists(f"Object {uid} already exists", obj)
415
return
416
except ObjectNotFound:
417
pass
418
419
# Serialize and store object
420
obj_copy = copy.copy(obj)
421
obj_copy.adapter = None
422
serialized = pickle.dumps(obj_copy)
423
424
with sqlite3.connect(self.db_path) as conn:
425
conn.execute(
426
"INSERT INTO objects (model_name, unique_id, data) VALUES (?, ?, ?)",
427
(modelname, uid, serialized)
428
)
429
430
# Implement other required methods...
431
432
# Usage
433
adapter = MyAdapter(
434
internal_storage_engine=SQLiteStore(database_path="network_data.db")
435
)
436
```
437
438
## Storage Backend Selection Guide
439
440
### LocalStore - Choose When:
441
- Working with temporary data that doesn't need persistence
442
- Single-process applications
443
- Development and testing scenarios
444
- Fast, in-memory operations are priority
445
- No sharing of data between processes needed
446
447
### RedisStore - Choose When:
448
- Data needs to persist across process restarts
449
- Multiple processes need to share the same dataset
450
- Distributed applications with multiple nodes
451
- Need atomic operations and Redis features
452
- Building caching layers or session storage
453
454
### Custom Backend - Choose When:
455
- Need integration with existing database systems
456
- Specific performance requirements (e.g., very large datasets)
457
- Special serialization or encryption requirements
458
- Integration with external storage services (S3, cloud databases)
459
- Complex querying requirements beyond simple key-value access
460
461
## Logging Configuration
462
463
Console logging setup for DiffSync operations, particularly useful when debugging storage backend operations.
464
465
```python { .api }
466
def enable_console_logging(verbosity: int = 0) -> None:
467
"""
468
Enable formatted logging to console with the specified verbosity.
469
470
Args:
471
verbosity: 0 for WARNING logs, 1 for INFO logs, 2 for DEBUG logs
472
"""
473
```
474
475
### Logging Usage
476
477
```python
478
from diffsync.logging import enable_console_logging
479
from diffsync import Adapter
480
from diffsync.store.redis import RedisStore
481
482
# Enable debug logging to see detailed storage operations
483
enable_console_logging(verbosity=2)
484
485
# Now all storage operations will be logged
486
adapter = MyAdapter(
487
internal_storage_engine=RedisStore(host="localhost")
488
)
489
adapter.load() # Will show detailed Redis operations
490
491
# Sync operations will show storage backend interactions
492
target = MyAdapter()
493
adapter.sync_to(target) # Will log all storage operations
494
```
495
496
## Types
497
498
```python { .api }
499
from typing import Any, Dict, List, Optional, Set, Tuple, Type, Union
500
from redis import Redis
501
502
# Storage backend configuration types
503
StorageConfig = Dict[str, Any]
504
ModelIdentifier = Union[str, Dict]
505
ModelSpec = Union[str, "DiffSyncModel", Type["DiffSyncModel"]]
506
```