0
# Custom Storage Framework
1

2
Interface and base classes for implementing custom storage backends, enabling integration with databases, cloud storage, network protocols, or specialized data structures. The framework provides both low-level buffer access and high-level Python string marshaling.
3

4
## Capabilities
5

6
### Storage Interface
7

8
Base interface defining the contract for custom storage implementations.
9

10
```python { .api }
11
class ICustomStorage:
12
    """Interface for custom storage implementations."""
13
    
14
    # Error constants
15
    NoError = 0
16
    InvalidPageError = 1
17
    IllegalStateError = 2
18
    
19
    # Page constants  
20
    EmptyPage = -0x1
21
    NewPage = -0x1
22
    
23
    def registerCallbacks(self, properties):
24
        """
25
        Register storage callbacks with index properties.
26
        
27
        Parameters:
28
        - properties (Property): Property object to configure
29
        
30
        Abstract method that must be implemented by subclasses.
31
        """
32
    
33
    def clear(self):
34
        """
35
        Clear all stored data.
36
        
37
        Abstract method that must be implemented by subclasses.
38
        """
39
    
40
    @property
41
    def hasData(self) -> bool:
42
        """
43
        Indicate whether storage contains data.
44
        
45
        Returns:
46
        bool: True if storage has existing data
47
        """
48
    
49
    def allocateBuffer(self, length: int):
50
        """
51
        Allocate buffer for data operations.
52
        
53
        Parameters:
54
        - length (int): Buffer size in bytes
55
        
56
        Returns:
57
        Buffer object for data operations
58
        """
59
```
60

61
### Low-Level Storage Base
62

63
Base class for implementing custom storage with raw C buffer access, providing maximum performance and control.
64

65
```python { .api }
66
class CustomStorageBase(ICustomStorage):
67
    """Base class for raw C buffer access custom storage."""
68
    
69
    def create(self, context, returnError):
70
        """
71
        Create storage backend.
72
        
73
        Parameters:
74
        - context: Storage context object
75
        - returnError: Error reporting callback
76
        
77
        Abstract method for storage initialization.
78
        """
79
    
80
    def destroy(self, context, returnError):
81
        """
82
        Destroy storage backend and cleanup resources.
83
        
84
        Parameters:
85
        - context: Storage context object  
86
        - returnError: Error reporting callback
87
        
88
        Abstract method for storage cleanup.
89
        """
90
    
91
    def loadByteArray(self, context, page, resultLen, resultData, returnError):
92
        """
93
        Load data from storage backend.
94
        
95
        Parameters:
96
        - context: Storage context object
97
        - page (int): Page identifier to load
98
        - resultLen: Output parameter for data length
99
        - resultData: Output parameter for data buffer
100
        - returnError: Error reporting callback
101
        
102
        Abstract method for data loading.
103
        """
104
    
105
    def storeByteArray(self, context, page, len, data, returnError):
106
        """
107
        Store data to storage backend.
108
        
109
        Parameters:
110
        - context: Storage context object
111
        - page (int): Page identifier for storage
112
        - len (int): Data length in bytes
113
        - data: Data buffer to store
114
        - returnError: Error reporting callback
115
        
116
        Abstract method for data storage.
117
        """
118
    
119
    def deleteByteArray(self, context, page, returnError):
120
        """
121
        Delete data from storage backend.
122
        
123
        Parameters:
124
        - context: Storage context object
125
        - page (int): Page identifier to delete
126
        - returnError: Error reporting callback
127
        
128
        Abstract method for data deletion.
129
        """
130
    
131
    def flush(self, context, returnError):
132
        """
133
        Flush pending operations to storage backend.
134
        
135
        Parameters:
136
        - context: Storage context object
137
        - returnError: Error reporting callback
138
        
139
        Abstract method for storage flushing.
140
        """
141
```
142

143
### High-Level Storage Base
144

145
Base class for implementing custom storage with Python string marshaling, providing easier implementation with automatic data conversion.
146

147
```python { .api }
148
class CustomStorage(CustomStorageBase):
149
    """Default custom storage with Python string marshaling."""
150
    
151
    def create(self, returnError):
152
        """
153
        Create storage backend.
154
        
155
        Parameters:
156
        - returnError: Error reporting callback
157
        
158
        Abstract method for storage initialization.
159
        Simplified interface without context parameter.
160
        """
161
    
162
    def destroy(self, returnError):
163
        """
164
        Destroy storage backend and cleanup resources.
165
        
166
        Parameters:
167
        - returnError: Error reporting callback
168
        
169
        Abstract method for storage cleanup.
170
        Simplified interface without context parameter.
171
        """
172
    
173
    def loadByteArray(self, page, returnError):
174
        """
175
        Load data from storage backend as Python string.
176
        
177
        Parameters:
178
        - page (int): Page identifier to load
179
        - returnError: Error reporting callback
180
        
181
        Returns:
182
        str: Loaded data as Python string
183
        
184
        Abstract method for data loading with automatic marshaling.
185
        """
186
    
187
    def storeByteArray(self, page, data, returnError):
188
        """
189
        Store Python string data to storage backend.
190
        
191
        Parameters:
192
        - page (int): Page identifier for storage
193
        - data (str): Python string data to store
194
        - returnError: Error reporting callback
195
        
196
        Abstract method for data storage with automatic marshaling.
197
        """
198
    
199
    def deleteByteArray(self, page, returnError):
200
        """
201
        Delete data from storage backend.
202
        
203
        Parameters:
204
        - page (int): Page identifier to delete
205
        - returnError: Error reporting callback
206
        
207
        Abstract method for data deletion.
208
        """
209
    
210
    def flush(self, returnError):
211
        """
212
        Flush pending operations to storage backend.
213
        
214
        Parameters:
215
        - returnError: Error reporting callback
216
        
217
        Abstract method for storage flushing.
218
        """
219
```
220

221
## Usage Examples
222

223
### Dictionary-Based Custom Storage
224

225
```python
226
from rtree.index import Index, Property, CustomStorage
227

228
class DictStorage(CustomStorage):
229
    """Simple dictionary-based storage for demonstration."""
230
    
231
    def __init__(self):
232
        super().__init__()
233
        self.data = {}
234
        self.has_data = False
235
    
236
    def create(self, returnError):
237
        """Initialize storage."""
238
        self.data.clear()
239
        self.has_data = True
240
        returnError.value = self.NoError
241
    
242
    def destroy(self, returnError):
243
        """Cleanup storage."""
244
        self.data.clear()
245
        self.has_data = False
246
        returnError.value = self.NoError
247
    
248
    def loadByteArray(self, page, returnError):
249
        """Load data for page."""
250
        if page in self.data:
251
            returnError.value = self.NoError
252
            return self.data[page]
253
        else:
254
            returnError.value = self.InvalidPageError
255
            return None
256
    
257
    def storeByteArray(self, page, data, returnError):
258
        """Store data for page."""
259
        self.data[page] = data
260
        returnError.value = self.NoError
261
    
262
    def deleteByteArray(self, page, returnError):
263
        """Delete data for page."""
264
        if page in self.data:
265
            del self.data[page]
266
            returnError.value = self.NoError
267
        else:
268
            returnError.value = self.InvalidPageError
269
    
270
    def flush(self, returnError):
271
        """Flush operations (no-op for dict)."""
272
        returnError.value = self.NoError
273
    
274
    @property
275
    def hasData(self):
276
        return self.has_data
277

278
# Use custom storage
279
storage = DictStorage()
280
prop = Property(storage=RT_Custom)
281
storage.registerCallbacks(prop)
282

283
idx = Index(properties=prop, storage=storage)
284
idx.insert(0, (0, 0, 10, 10))
285
```
286

287
### Database-Backed Storage
288

289
```python
290
import sqlite3
291
from rtree.index import Index, Property, CustomStorage
292

293
class SQLiteStorage(CustomStorage):
294
    """SQLite database storage backend."""
295
    
296
    def __init__(self, db_path=":memory:"):
297
        super().__init__()
298
        self.db_path = db_path
299
        self.conn = None
300
    
301
    def create(self, returnError):
302
        """Initialize database storage."""
303
        try:
304
            self.conn = sqlite3.connect(self.db_path)
305
            self.conn.execute("""
306
                CREATE TABLE IF NOT EXISTS rtree_pages (
307
                    page_id INTEGER PRIMARY KEY,
308
                    data BLOB
309
                )
310
            """)
311
            self.conn.commit()
312
            returnError.value = self.NoError
313
        except Exception:
314
            returnError.value = self.IllegalStateError
315
    
316
    def destroy(self, returnError):
317
        """Cleanup database connection."""
318
        try:
319
            if self.conn:
320
                self.conn.close()
321
                self.conn = None
322
            returnError.value = self.NoError
323
        except Exception:
324
            returnError.value = self.IllegalStateError
325
    
326
    def loadByteArray(self, page, returnError):
327
        """Load page data from database."""
328
        try:
329
            cursor = self.conn.execute(
330
                "SELECT data FROM rtree_pages WHERE page_id = ?", (page,)
331
            )
332
            row = cursor.fetchone()
333
            if row:
334
                returnError.value = self.NoError
335
                return row[0].decode('utf-8')
336
            else:
337
                returnError.value = self.InvalidPageError
338
                return None
339
        except Exception:
340
            returnError.value = self.IllegalStateError
341
            return None
342
    
343
    def storeByteArray(self, page, data, returnError):
344
        """Store page data to database."""
345
        try:
346
            self.conn.execute(
347
                "INSERT OR REPLACE INTO rtree_pages (page_id, data) VALUES (?, ?)",
348
                (page, data.encode('utf-8'))
349
            )
350
            returnError.value = self.NoError
351
        except Exception:
352
            returnError.value = self.IllegalStateError
353
    
354
    def deleteByteArray(self, page, returnError):
355
        """Delete page data from database."""
356
        try:
357
            cursor = self.conn.execute(
358
                "DELETE FROM rtree_pages WHERE page_id = ?", (page,)
359
            )
360
            if cursor.rowcount > 0:
361
                returnError.value = self.NoError
362
            else:
363
                returnError.value = self.InvalidPageError
364
        except Exception:
365
            returnError.value = self.IllegalStateError
366
    
367
    def flush(self, returnError):
368
        """Commit pending transactions."""
369
        try:
370
            self.conn.commit()
371
            returnError.value = self.NoError
372
        except Exception:
373
            returnError.value = self.IllegalStateError
374
    
375
    @property
376
    def hasData(self):
377
        if not self.conn:
378
            return False
379
        cursor = self.conn.execute("SELECT COUNT(*) FROM rtree_pages")
380
        return cursor.fetchone()[0] > 0
381

382
# Use SQLite storage
383
storage = SQLiteStorage("spatial_index.db")
384
prop = Property(storage=RT_Custom)
385
storage.registerCallbacks(prop)
386

387
idx = Index(properties=prop, storage=storage)
388
```
389

390
### Network Storage Backend
391

392
```python
393
import requests
394
from rtree.index import Index, Property, CustomStorage
395

396
class HTTPStorage(CustomStorage):
397
    """HTTP-based storage backend for distributed indexing."""
398
    
399
    def __init__(self, base_url, auth_token=None):
400
        super().__init__()
401
        self.base_url = base_url.rstrip('/')
402
        self.auth_token = auth_token
403
        self.headers = {}
404
        if auth_token:
405
            self.headers['Authorization'] = f'Bearer {auth_token}'
406
    
407
    def create(self, returnError):
408
        """Initialize remote storage."""
409
        try:
410
            response = requests.post(
411
                f"{self.base_url}/storage/create",
412
                headers=self.headers
413
            )
414
            if response.status_code == 200:
415
                returnError.value = self.NoError
416
            else:
417
                returnError.value = self.IllegalStateError
418
        except Exception:
419
            returnError.value = self.IllegalStateError
420
    
421
    def destroy(self, returnError):
422
        """Cleanup remote storage."""
423
        try:
424
            response = requests.delete(
425
                f"{self.base_url}/storage",
426
                headers=self.headers
427
            )
428
            returnError.value = self.NoError
429
        except Exception:
430
            returnError.value = self.IllegalStateError
431
    
432
    def loadByteArray(self, page, returnError):
433
        """Load page from remote storage."""
434
        try:
435
            response = requests.get(
436
                f"{self.base_url}/storage/pages/{page}",
437
                headers=self.headers
438
            )
439
            if response.status_code == 200:
440
                returnError.value = self.NoError
441
                return response.text
442
            elif response.status_code == 404:
443
                returnError.value = self.InvalidPageError
444
                return None
445
            else:
446
                returnError.value = self.IllegalStateError
447
                return None
448
        except Exception:
449
            returnError.value = self.IllegalStateError
450
            return None
451
    
452
    def storeByteArray(self, page, data, returnError):
453
        """Store page to remote storage."""
454
        try:
455
            response = requests.put(
456
                f"{self.base_url}/storage/pages/{page}",
457
                data=data,
458
                headers=self.headers
459
            )
460
            if response.status_code in (200, 201):
461
                returnError.value = self.NoError
462
            else:
463
                returnError.value = self.IllegalStateError
464
        except Exception:
465
            returnError.value = self.IllegalStateError
466
    
467
    def deleteByteArray(self, page, returnError):
468
        """Delete page from remote storage."""
469
        try:
470
            response = requests.delete(
471
                f"{self.base_url}/storage/pages/{page}",
472
                headers=self.headers
473
            )
474
            if response.status_code in (200, 204):
475
                returnError.value = self.NoError
476
            elif response.status_code == 404:
477
                returnError.value = self.InvalidPageError
478
            else:
479
                returnError.value = self.IllegalStateError
480
        except Exception:
481
            returnError.value = self.IllegalStateError
482
    
483
    def flush(self, returnError):
484
        """Flush pending operations."""
485
        try:
486
            response = requests.post(
487
                f"{self.base_url}/storage/flush",
488
                headers=self.headers
489
            )
490
            returnError.value = self.NoError
491
        except Exception:
492
            returnError.value = self.IllegalStateError
493
    
494
    @property
495
    def hasData(self):
496
        try:
497
            response = requests.head(
498
                f"{self.base_url}/storage",
499
                headers=self.headers
500
            )
501
            return response.status_code == 200
502
        except Exception:
503
            return False
504

505
# Use HTTP storage
506
storage = HTTPStorage("https://api.spatial-service.com", "your-auth-token")
507
prop = Property(storage=RT_Custom)
508
storage.registerCallbacks(prop)
509

510
idx = Index(properties=prop, storage=storage)
511
```
512

513
### Cached Storage Implementation
514

515
```python
516
import tempfile
517
import os
518
from rtree.index import Index, Property, CustomStorage
519

520
class CachedFileStorage(CustomStorage):
521
    """File storage with in-memory caching."""
522
    
523
    def __init__(self, base_dir=None, cache_size=1000):
524
        super().__init__()
525
        self.base_dir = base_dir or tempfile.mkdtemp()
526
        self.cache = {}
527
        self.cache_size = cache_size
528
        self.access_order = []
529
    
530
    def _cache_key(self, page):
531
        return f"page_{page}"
532
    
533
    def _evict_cache(self):
534
        """Evict oldest entries if cache is full."""
535
        while len(self.cache) >= self.cache_size:
536
            oldest = self.access_order.pop(0)
537
            if oldest in self.cache:
538
                del self.cache[oldest]
539
    
540
    def _update_access(self, key):
541
        """Update access order for LRU eviction."""
542
        if key in self.access_order:
543
            self.access_order.remove(key)
544
        self.access_order.append(key)
545
    
546
    def create(self, returnError):
547
        """Initialize file storage."""
548
        try:
549
            os.makedirs(self.base_dir, exist_ok=True)
550
            returnError.value = self.NoError
551
        except Exception:
552
            returnError.value = self.IllegalStateError
553
    
554
    def destroy(self, returnError):
555
        """Cleanup file storage."""
556
        try:
557
            self.cache.clear()
558
            self.access_order.clear()
559
            returnError.value = self.NoError
560
        except Exception:
561
            returnError.value = self.IllegalStateError
562
    
563
    def loadByteArray(self, page, returnError):
564
        """Load page with caching."""
565
        cache_key = self._cache_key(page)
566
        
567
        # Check cache first
568
        if cache_key in self.cache:
569
            self._update_access(cache_key)
570
            returnError.value = self.NoError
571
            return self.cache[cache_key]
572
        
573
        # Load from file
574
        try:
575
            file_path = os.path.join(self.base_dir, f"page_{page}.dat")
576
            if os.path.exists(file_path):
577
                with open(file_path, 'r') as f:
578
                    data = f.read()
579
                
580
                # Cache the data
581
                self._evict_cache()
582
                self.cache[cache_key] = data
583
                self._update_access(cache_key)
584
                
585
                returnError.value = self.NoError
586
                return data
587
            else:
588
                returnError.value = self.InvalidPageError
589
                return None
590
        except Exception:
591
            returnError.value = self.IllegalStateError
592
            return None
593
    
594
    def storeByteArray(self, page, data, returnError):
595
        """Store page with caching."""
596
        try:
597
            # Write to file
598
            file_path = os.path.join(self.base_dir, f"page_{page}.dat")
599
            with open(file_path, 'w') as f:
600
                f.write(data)
601
            
602
            # Update cache
603
            cache_key = self._cache_key(page)
604
            self._evict_cache()
605
            self.cache[cache_key] = data
606
            self._update_access(cache_key)
607
            
608
            returnError.value = self.NoError
609
        except Exception:
610
            returnError.value = self.IllegalStateError
611
    
612
    def deleteByteArray(self, page, returnError):
613
        """Delete page and remove from cache."""
614
        try:
615
            file_path = os.path.join(self.base_dir, f"page_{page}.dat")
616
            if os.path.exists(file_path):
617
                os.remove(file_path)
618
                
619
                # Remove from cache
620
                cache_key = self._cache_key(page)
621
                if cache_key in self.cache:
622
                    del self.cache[cache_key]
623
                    self.access_order.remove(cache_key)
624
                
625
                returnError.value = self.NoError
626
            else:
627
                returnError.value = self.InvalidPageError
628
        except Exception:
629
            returnError.value = self.IllegalStateError
630
    
631
    def flush(self, returnError):
632
        """Flush operation (files are written immediately)."""
633
        returnError.value = self.NoError
634
    
635
    @property
636
    def hasData(self):
637
        try:
638
            return len(os.listdir(self.base_dir)) > 0
639
        except Exception:
640
            return False
641

642
# Use cached file storage
643
storage = CachedFileStorage("/tmp/rtree_cache", cache_size=500)
644
prop = Property(storage=RT_Custom)
645
storage.registerCallbacks(prop)
646

647
idx = Index(properties=prop, storage=storage)
648
```
649

650
## Type Definitions
651

652
```python { .api }
653
class CustomStorageCallbacks:
654
    """C callback functions for custom storage (ctypes.Structure)."""
655
    # Internal structure for C API integration
656
    # Used by registerCallbacks() to configure storage backend
657
```
658

659
## Constants
660

661
```python { .api }
662
# Error codes for custom storage
663
NoError = 0
664
InvalidPageError = 1
665
IllegalStateError = 2
666

667
# Special page identifiers
668
EmptyPage = -0x1
669
NewPage = -0x1
670
```