0
# SqliteDict
1

2
A lightweight wrapper around Python's sqlite3 database with a simple, Pythonic dict-like interface and support for multi-thread access. SqliteDict enables persistent storage of arbitrary pickle-able objects as values with string keys, backed by SQLite database files.
3

4
## Package Information
5

6
- **Package Name**: sqlitedict
7
- **Package Type**: pypi
8
- **Language**: Python
9
- **Installation**: `pip install sqlitedict`
10

11
## Core Imports
12

13
```python
14
from sqlitedict import SqliteDict
15
```
16

17
Alternative import patterns:
18

19
```python
20
import sqlitedict
21
# Use sqlitedict.SqliteDict() or sqlitedict.open()
22
```
23

24
For version access and module constants:
25

26
```python
27
import sqlitedict
28
print(sqlitedict.__version__)  # '1.7.0'
29
print(sqlitedict.PICKLE_PROTOCOL)  # Highest pickle protocol available
30
```
31

32
## Basic Usage
33

34
```python
35
from sqlitedict import SqliteDict
36

37
# Create a persistent dictionary (autocommit for immediate persistence)
38
mydict = SqliteDict('./my_db.sqlite', autocommit=True)
39
mydict['some_key'] = any_picklable_object
40
print(mydict['some_key'])
41
print(len(mydict))  # all dict functions work
42

43
# Context manager usage (recommended for transaction control)
44
with SqliteDict('./my_db.sqlite') as mydict:  # no autocommit
45
    mydict['some_key'] = "first value"
46
    mydict['another_key'] = range(10)
47
    mydict.commit()  # explicit commit
48
    mydict['some_key'] = "new value"
49
    # no explicit commit here - changes lost on exit
50

51
# Re-open same database
52
with SqliteDict('./my_db.sqlite') as mydict:
53
    print(mydict['some_key'])  # outputs 'first value', not 'new value'
54
```
55

56
## Architecture
57

58
SqliteDict provides thread-safe access to SQLite databases through an internal queuing system. The architecture includes:
59

60
- **SqliteDict**: Main dict-like interface providing standard Python dictionary operations
61
- **SqliteMultithread**: Internal threaded SQLite wrapper that serializes concurrent requests
62
- **Custom serialization**: Configurable encode/decode functions (defaults to pickle)
63
- **Multi-table support**: Multiple SqliteDict instances can share the same database file with different table names
64

65
## Capabilities
66

67
### Core Dictionary Interface
68

69
Standard Python dictionary operations with persistence to SQLite database.
70

71
```python { .api }
72
class SqliteDict:
73
    def __init__(self, filename=None, tablename='unnamed', flag='c', 
74
                 autocommit=False, journal_mode="DELETE", encode=encode, decode=decode):
75
        """
76
        Initialize a thread-safe sqlite-backed dictionary.
77
        
78
        Parameters:
79
        - filename (str, optional): Database file path. If None, uses temporary file
80
        - tablename (str): Table name within database (default: 'unnamed')
81
        - flag (str): Access mode - 'c' (create/read/write), 'r' (readonly), 
82
                     'w' (overwrite table), 'n' (new database)
83
        - autocommit (bool): Auto-commit after each operation (default: False)
84
        - journal_mode (str): SQLite journal mode (default: "DELETE")
85
        - encode (function): Custom encoding function (default: pickle-based)
86
        - decode (function): Custom decoding function (default: pickle-based)
87
        """
88

89
    def __getitem__(self, key):
90
        """Get value by key. Raises KeyError if key not found."""
91

92
    def __setitem__(self, key, value):
93
        """Set value by key. Replaces existing value."""
94

95
    def __delitem__(self, key):
96
        """Delete key-value pair. Raises KeyError if key not found."""
97

98
    def __contains__(self, key):
99
        """Check if key exists in dictionary."""
100

101
    def __len__(self):
102
        """Return number of items. Note: slow for large databases."""
103

104
    def __bool__(self):
105
        """Return True if dictionary has any items."""
106

107
    def __iter__(self):
108
        """Return iterator over keys."""
109

110
    def keys(self):
111
        """Return view/list of all keys."""
112

113
    def values(self):
114
        """Return view/list of all values (deserialized)."""
115

116
    def items(self):
117
        """Return view/list of (key, value) tuples."""
118

119
    def iterkeys(self):
120
        """Return iterator over keys (Python 2 compatibility)."""
121

122
    def itervalues(self):
123
        """Return iterator over values (Python 2 compatibility)."""
124

125
    def iteritems(self):
126
        """Return iterator over (key, value) tuples (Python 2 compatibility)."""
127

128
    def update(self, items=(), **kwds):
129
        """Update dictionary with items from mapping or iterable."""
130

131
    def clear(self):
132
        """Remove all items from table."""
133
```
134

135
### Database Management
136

137
Transaction control and database lifecycle management.
138

139
```python { .api }
140
class SqliteDict:
141
    def commit(self, blocking=True):
142
        """
143
        Persist all data to disk.
144
        
145
        Parameters:
146
        - blocking (bool): Wait for completion when True, queue when False
147
        """
148

149
    def close(self, do_log=True, force=False):
150
        """
151
        Close database connection and cleanup resources.
152
        
153
        Parameters:
154
        - do_log (bool): Whether to log the close operation
155
        - force (bool): Force close even if operations pending
156
        """
157

158
    def terminate(self):
159
        """Close connection and delete underlying database file permanently."""
160

161
    def sync(self):
162
        """Alias for commit() method."""
163
```
164

165
### Context Manager Support
166

167
Automatic resource management and transaction handling.
168

169
```python { .api }
170
class SqliteDict:
171
    def __enter__(self):
172
        """Context manager entry. Returns self."""
173

174
    def __exit__(self, *exc_info):
175
        """Context manager exit. Automatically calls close()."""
176
```
177

178
### Static Utilities
179

180
Database introspection and factory functions.
181

182
```python { .api }
183
def open(*args, **kwargs):
184
    """
185
    Factory function to create SqliteDict instance.
186
    
187
    Returns:
188
    SqliteDict instance with same parameters as constructor
189
    """
190

191
class SqliteDict:
192
    @staticmethod
193
    def get_tablenames(filename):
194
        """
195
        Get list of table names in SQLite database file.
196
        
197
        Parameters:
198
        - filename (str): Path to SQLite database file
199
        
200
        Returns:
201
        List of table name strings
202
        
203
        Raises:
204
        IOError: If file does not exist
205
        """
206
```
207

208
### Custom Serialization
209

210
Default and custom serialization functions for data persistence.
211

212
```python { .api }
213
def encode(obj):
214
    """
215
    Default encoding function using pickle.
216
    
217
    Parameters:
218
    - obj: Any pickle-able Python object
219
    
220
    Returns:
221
    sqlite3.Binary object suitable for SQLite storage
222
    """
223

224
def decode(obj):
225
    """
226
    Default decoding function using pickle.
227
    
228
    Parameters:
229
    - obj: Binary data from SQLite
230
    
231
    Returns:
232
    Deserialized Python object
233
    """
234

235
# Lower-level pickle functions (also exposed by module)
236
def dumps(obj, protocol):
237
    """
238
    Serialize object using pickle with specified protocol.
239
    
240
    Parameters:
241
    - obj: Object to serialize
242
    - protocol: Pickle protocol version (typically PICKLE_PROTOCOL constant)
243
    
244
    Returns:
245
    Bytes object containing pickled data
246
    """
247

248
def loads(data):
249
    """
250
    Deserialize object from pickle data.
251
    
252
    Parameters:
253
    - data: Bytes object containing pickled data
254
    
255
    Returns:
256
    Deserialized Python object
257
    """
258
```
259

260
### String Representation
261

262
Display and debugging support.
263

264
```python { .api }
265
class SqliteDict:
266
    def __str__(self):
267
        """Return string representation of SqliteDict instance."""
268

269
    def __repr__(self):
270
        """Return developer-friendly string representation."""
271
```
272

273
## Types
274

275
```python { .api }
276
class SqliteDict:
277
    """
278
    Main dictionary-like class with SQLite persistence.
279
    
280
    Inherits from UserDict (Python 3) or DictMixin (Python 2) to provide
281
    standard dictionary interface. All standard dict operations are supported.
282
    """
283
    VALID_FLAGS = ['c', 'r', 'w', 'n']  # Valid flag values for constructor
284

285
class SqliteMultithread(Thread):
286
    """Internal thread-safe SQLite wrapper (not typically used directly)."""
287

288
# Module-level constants
289
__version__ = str  # Version string, e.g. '1.7.0'
290
PICKLE_PROTOCOL = int  # Highest available pickle protocol for current Python version
291
```
292

293
## Advanced Usage Examples
294

295
### Custom Serialization
296

297
```python
298
import json
299
from sqlitedict import SqliteDict
300

301
# Use JSON instead of pickle
302
mydict = SqliteDict('./my_db.sqlite', encode=json.dumps, decode=json.loads)
303
mydict['data'] = {'key': 'value', 'number': 42}
304
```
305

306
### Compression with Custom Serialization
307

308
```python
309
import zlib
310
import pickle
311
import sqlite3
312
from sqlitedict import SqliteDict
313

314
def compress_encode(obj):
315
    return sqlite3.Binary(zlib.compress(pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)))
316

317
def compress_decode(obj):
318
    return pickle.loads(zlib.decompress(bytes(obj)))
319

320
mydict = SqliteDict('./compressed_db.sqlite', encode=compress_encode, decode=compress_decode)
321
```
322

323
### Multiple Tables in Same Database
324

325
```python
326
from sqlitedict import SqliteDict
327

328
# Different tables in same database file
329
users = SqliteDict('./app.db', tablename='users', autocommit=True)
330
sessions = SqliteDict('./app.db', tablename='sessions', autocommit=True)
331

332
users['user123'] = {'name': 'Alice', 'email': 'alice@example.com'}
333
sessions['session456'] = {'user_id': 'user123', 'expires': '2024-12-31'}
334

335
# List all tables in database
336
tables = SqliteDict.get_tablenames('./app.db')
337
print(tables)  # ['users', 'sessions']
338
```
339

340
### Read-Only Access
341

342
```python
343
from sqlitedict import SqliteDict
344

345
# Open existing database in read-only mode
346
readonly_dict = SqliteDict('./existing.db', flag='r')
347
value = readonly_dict['some_key']
348
# readonly_dict['key'] = 'value'  # Would raise RuntimeError
349
```
350

351
### Performance Considerations
352

353
```python
354
from sqlitedict import SqliteDict
355

356
# For batch operations, disable autocommit
357
with SqliteDict('./batch.db', autocommit=False) as batch_dict:
358
    for i in range(1000):
359
        batch_dict[f'key_{i}'] = f'value_{i}'
360
    batch_dict.commit()  # Single commit for all operations
361

362
# For frequent single operations, enable autocommit
363
frequent_dict = SqliteDict('./frequent.db', autocommit=True)
364
frequent_dict['immediate'] = 'persisted_immediately'
365
```
366

367
## Error Handling
368

369
Common exceptions and error patterns:
370

371
- **RuntimeError**: Invalid flags ('c', 'r', 'w', 'n'), read-only mode violations, missing directories
372
- **KeyError**: Standard dictionary behavior for missing keys in `__getitem__`, `__delitem__`
373
- **IOError**: File access issues in `get_tablenames()` when database file doesn't exist
374
- **ImportError**: Raised during module import if Python version < 2.5
375
- **SQLite exceptions**: Various sqlite3 exceptions may propagate from underlying database operations
376

377
## Thread Safety
378

379
SqliteDict is thread-safe and allows concurrent access from multiple threads to the same instance. However, concurrent requests are serialized internally, so multi-threading doesn't provide performance benefits - it's a workaround for SQLite's threading limitations in Python.
380

381
## Important Notes
382

383
**Mutable Object Limitations**: SqliteDict cannot detect when mutable objects are modified in-place. Always reassign modified objects:
384

385
```python
386
# Wrong - changes not persisted
387
val = mydict.get('key', [])
388
val.append(1)  # SQLite DB not updated
389

390
# Correct - explicitly reassign
391
val = mydict.get('key', [])
392
val.append(1)
393
mydict['key'] = val  # Now updated in database
394
```
395

396
**Performance**: `len()` operation is slow for large databases as it performs a full table scan.
397

398
## Python Version Compatibility
399

400
SqliteDict supports Python 2.5+ and Python 3.3+. Key compatibility considerations:
401

402
- **Python 2 vs 3**: Automatic handling of string types and pickle modules
403
- **Method behavior**: `keys()`, `values()`, `items()` return views in Python 3, lists in Python 2
404
- **Iterator methods**: `iterkeys()`, `itervalues()`, `iteritems()` provided for Python 2 compatibility
405
- **Minimum version**: Python 2.5 minimum (ImportError raised on older versions)