0
# Storage Backends
1

2
Multiple storage backend implementations with their specific configurations and capabilities for different deployment scenarios. Huey supports various storage backends to accommodate different infrastructure requirements and performance needs.
3

4
## Capabilities
5

6
### Redis Storage Backends
7

8
Redis-based storage backends offering high performance, persistence, and distributed capabilities.
9

10
```python { .api }
11
class RedisHuey(Huey):
12
    """
13
    Huey with Redis storage backend.
14
    
15
    Storage parameters:
16
    - host (str): Redis server host (default: 'localhost')
17
    - port (int): Redis server port (default: 6379)
18
    - db (int): Redis database number (default: 0)
19
    - password (str): Redis password (optional)
20
    - connection_pool: Custom Redis connection pool (optional)
21
    - url (str): Redis URL (optional, overrides other connection params)
22
    - client_name (str): Redis client name (optional)
23
    - health_check_interval (int): Connection health check interval (default: 0)
24
    """
25

26
class RedisExpireHuey(RedisHuey):
27
    """
28
    Redis storage with automatic result expiration.
29
    
30
    Additional parameters:
31
    - expire_time (int): Result expiration time in seconds (default: 86400)
32
    """
33

34
class PriorityRedisHuey(RedisHuey):
35
    """
36
    Redis storage with priority queue support.
37
    
38
    Features:
39
    - Priority-based task ordering
40
    - Higher priority tasks execute first
41
    - Maintains FIFO order within same priority level
42
    """
43

44
class PriorityRedisExpireHuey(PriorityRedisHuey):
45
    """
46
    Redis storage with both priority queues and result expiration.
47
    
48
    Combines features of PriorityRedisHuey and RedisExpireHuey.
49
    """
50
```
51

52
### SQLite Storage Backend
53

54
File-based SQLite storage for single-node deployments without external dependencies.
55

56
```python { .api }
57
class SqliteHuey(Huey):
58
    """
59
    Huey with SQLite storage backend.
60
    
61
    Storage parameters:
62
    - filename (str): SQLite database file path (default: 'huey.db')
63
    - cache_mb (int): SQLite cache size in MB (default: 8)
64
    - fsync (bool): Force filesystem sync (default: False)
65
    - timeout (int): Database lock timeout in seconds (default: 60)
66
    - strict_fifo (bool): Maintain strict FIFO ordering (default: False)
67
    """
68
```
69

70
### Memory Storage Backend
71

72
In-memory storage for development, testing, and immediate execution modes.
73

74
```python { .api }
75
class MemoryHuey(Huey):
76
    """
77
    Huey with in-memory storage backend.
78
    
79
    Features:
80
    - No persistence (data lost on restart)
81
    - Very fast operations
82
    - Suitable for testing and immediate mode
83
    - Thread-safe operations
84
    """
85
```
86

87
### File Storage Backend
88

89
File-system based storage using directories and files for task and result storage.
90

91
```python { .api }
92
class FileHuey(Huey):
93
    """
94
    Huey with file-system storage backend.
95
    
96
    Storage parameters:
97
    - path (str): Base directory for storage (default: '/tmp/huey')
98
    - levels (int): Directory nesting levels (default: 2)
99
    - use_thread_lock (bool): Use thread locks (default: True)
100
    """
101
```
102

103
### BlackHole Storage Backend
104

105
Storage backend that discards all data, useful for testing and benchmarking.
106

107
```python { .api }
108
class BlackHoleHuey(Huey):
109
    """
110
    Huey with BlackHole storage backend.
111
    
112
    Features:
113
    - Discards all tasks and results
114
    - No storage overhead
115
    - Useful for performance testing
116
    - Tasks appear to execute but do nothing
117
    """
118
```
119

120
### Custom Storage Implementation
121

122
Base class for implementing custom storage backends.
123

124
```python { .api }
125
class BaseStorage:
126
    """
127
    Base storage interface for custom implementations.
128
    
129
    Required methods to implement:
130
    - enqueue(data, priority=None)
131
    - dequeue()
132
    - queue_size()
133
    - enqueued_items(limit=None)
134
    - flush_queue()
135
    - add_to_schedule(data, timestamp)
136
    - read_schedule(timestamp)
137
    - scheduled_items(limit=None)
138
    - schedule_size()
139
    - flush_schedule()
140
    - put_data(key, value)
141
    - peek_data(key)
142
    - pop_data(key)
143
    - has_data_for_key(key)
144
    - put_if_empty(key, value)
145
    - delete_data(key)
146
    - flush_all()
147
    """
148
    
149
    blocking = False  # Whether dequeue() blocks or polls
150
    priority = True   # Whether storage supports priority queues
151
```
152

153
## Usage Examples
154

155
### Redis Storage Configuration
156

157
```python
158
from huey import RedisHuey
159

160
# Basic Redis connection
161
huey = RedisHuey('my-app', host='localhost', port=6379, db=0)
162

163
# Redis with authentication
164
huey = RedisHuey('my-app', 
165
                 host='redis.example.com', 
166
                 port=6379, 
167
                 password='secret123',
168
                 db=1)
169

170
# Redis with URL
171
huey = RedisHuey('my-app', url='redis://user:pass@localhost:6379/0')
172

173
# Redis with connection pool
174
import redis
175
pool = redis.ConnectionPool(host='localhost', port=6379, db=0, max_connections=20)
176
huey = RedisHuey('my-app', connection_pool=pool)
177

178
# Redis with priority support
179
huey = PriorityRedisHuey('my-app', host='localhost')
180

181
@huey.task()
182
def normal_task():
183
    return "Normal priority"
184

185
@huey.task(priority=10)
186
def high_priority_task():
187
    return "High priority"
188

189
# High priority tasks execute first
190
normal_result = normal_task()
191
high_result = high_priority_task()
192
```
193

194
### SQLite Storage Configuration
195

196
```python
197
from huey import SqliteHuey
198

199
# Basic SQLite storage
200
huey = SqliteHuey('my-app', filename='tasks.db')
201

202
# SQLite with performance tuning
203
huey = SqliteHuey('my-app',
204
                  filename='/var/data/huey.db',
205
                  cache_mb=64,        # 64MB cache
206
                  fsync=True,         # Force sync for durability
207
                  timeout=120,        # 2 minute lock timeout
208
                  strict_fifo=True)   # Maintain strict ordering
209
```
210

211
### File Storage Configuration
212

213
```python
214
from huey import FileHuey
215

216
# Basic file storage
217
huey = FileHuey('my-app', path='/var/huey_data')
218

219
# File storage with custom configuration
220
huey = FileHuey('my-app',
221
                path='/tmp/huey_tasks',
222
                levels=3,           # 3 levels of subdirectories
223
                use_thread_lock=True)
224
```
225

226
### Memory Storage for Testing
227

228
```python
229
from huey import MemoryHuey
230

231
# Memory storage for unit tests
232
def test_task_execution():
233
    huey = MemoryHuey('test-app', immediate=True)
234
    
235
    @huey.task()
236
    def add_numbers(a, b):
237
        return a + b
238
    
239
    result = add_numbers(2, 3)
240
    assert result() == 5
241

242
# Memory storage for development
243
huey = MemoryHuey('dev-app')
244

245
@huey.task()
246
def debug_task(data):
247
    print(f"Processing: {data}")
248
    return f"Processed: {data}"
249
```
250

251
### Storage Backend Comparison
252

253
```python
254
# Production: Redis for distributed systems
255
production_huey = RedisHuey('prod-app',
256
                           host='redis-cluster.example.com',
257
                           port=6379,
258
                           db=0)
259

260
# Development: SQLite for simplicity
261
dev_huey = SqliteHuey('dev-app', filename='dev.db')
262

263
# Testing: Memory for speed
264
test_huey = MemoryHuey('test-app', immediate=True)
265

266
# Benchmarking: BlackHole to test task overhead
267
benchmark_huey = BlackHoleHuey('benchmark-app')
268

269
# File storage for simple single-node deployments
270
simple_huey = FileHuey('simple-app', path='/opt/tasks')
271
```
272

273
### Custom Storage Backend
274

275
```python
276
from huey.storage import BaseStorage
277
import sqlite3
278
import json
279
import time
280

281
class CustomSqliteStorage(BaseStorage):
282
    def __init__(self, name, filename='custom.db'):
283
        super().__init__(name)
284
        self.filename = filename
285
        self._initialize_db()
286
    
287
    def _initialize_db(self):
288
        conn = sqlite3.connect(self.filename)
289
        conn.execute('''
290
            CREATE TABLE IF NOT EXISTS tasks (
291
                id INTEGER PRIMARY KEY AUTOINCREMENT,
292
                data TEXT NOT NULL,
293
                priority INTEGER DEFAULT 0,
294
                created_at REAL NOT NULL
295
            )
296
        ''')
297
        conn.commit()
298
        conn.close()
299
    
300
    def enqueue(self, data, priority=None):
301
        conn = sqlite3.connect(self.filename)
302
        conn.execute(
303
            'INSERT INTO tasks (data, priority, created_at) VALUES (?, ?, ?)',
304
            (data.decode('utf-8'), priority or 0, time.time())
305
        )
306
        conn.commit()
307
        conn.close()
308
    
309
    def dequeue(self):
310
        conn = sqlite3.connect(self.filename)
311
        cursor = conn.execute('''
312
            SELECT id, data FROM tasks 
313
            ORDER BY priority DESC, created_at ASC 
314
            LIMIT 1
315
        ''')
316
        row = cursor.fetchone()
317
        if row:
318
            task_id, data = row
319
            conn.execute('DELETE FROM tasks WHERE id = ?', (task_id,))
320
            conn.commit()
321
            conn.close()
322
            return data.encode('utf-8')
323
        conn.close()
324
        return None
325
    
326
    def queue_size(self):
327
        conn = sqlite3.connect(self.filename)
328
        cursor = conn.execute('SELECT COUNT(*) FROM tasks')
329
        count = cursor.fetchone()[0]
330
        conn.close()
331
        return count
332

333
# Use custom storage
334
class CustomHuey(Huey):
335
    def get_storage(self, **kwargs):
336
        return CustomSqliteStorage(self.name, **kwargs)
337

338
huey = CustomHuey('custom-app', filename='custom_tasks.db')
339
```
340

341
### Storage Selection Guidelines
342

343
```python
344
# Choose storage based on requirements:
345

346
# High throughput, distributed: Redis
347
if high_throughput and distributed:
348
    huey = PriorityRedisHuey('app', host='redis-cluster')
349

350
# Simple single-node: SQLite
351
elif single_node and persistent:
352
    huey = SqliteHuey('app', filename='tasks.db')
353

354
# Development/testing: Memory
355
elif development or testing:
356
    huey = MemoryHuey('app', immediate=True)
357

358
# No external dependencies: File
359
elif no_dependencies:
360
    huey = FileHuey('app', path='/var/huey')
361

362
# Performance testing: BlackHole
363
elif benchmarking:
364
    huey = BlackHoleHuey('app')
365
```