0
# DiskCache
1

2
DiskCache is a comprehensive disk-based caching library for Python that provides high-performance persistent storage using SQLite databases and memory-mapped files. It offers thread-safe and process-safe caching with multiple eviction policies (LRU, LFU), supports various data structures including Cache, FanoutCache, Deque, and Index, and provides Django integration. The library is designed for applications requiring gigabytes of cache storage with performance that matches or exceeds in-memory solutions like Memcached, while being completely written in pure Python with no external dependencies.
3

4
## Package Information
5

6
- **Package Name**: diskcache
7
- **Language**: Python
8
- **Installation**: `pip install diskcache`
9

10
## Core Imports
11

12
```python
13
import diskcache
14
```
15

16
Main classes:
17

18
```python
19
from diskcache import Cache, FanoutCache, Deque, Index
20
```
21

22
Constants and settings:
23

24
```python
25
from diskcache import DEFAULT_SETTINGS, EVICTION_POLICY, ENOVAL, UNKNOWN
26
```
27

28
Recipe classes for synchronization:
29

30
```python
31
from diskcache import Lock, RLock, BoundedSemaphore, Averager
32
```
33

34
Recipe functions:
35

36
```python
37
from diskcache import throttle, barrier, memoize_stampede
38
```
39

40
Django integration (if Django is available):
41

42
```python
43
from diskcache import DjangoCache
44
```
45

46
## Basic Usage
47

48
```python
49
import diskcache
50

51
# Create a simple cache
52
cache = diskcache.Cache('/tmp/mycache')
53

54
# Store and retrieve values
55
cache.set('key', 'value')
56
value = cache.get('key')
57

58
# Dict-like interface
59
cache['another_key'] = {'data': [1, 2, 3]}
60
data = cache['another_key']
61

62
# Context manager for transactions
63
with cache.transact():
64
    cache.set('a', 1)
65
    cache.set('b', 2)
66

67
# Use as a decorator for memoization
68
@cache.memoize()
69
def expensive_function(x, y):
70
    # ... expensive computation
71
    return x * y
72

73
# Clean up
74
cache.close()
75
```
76

77
## Architecture
78

79
DiskCache provides multiple data structure classes built on a common foundation:
80

81
- **Cache**: Thread-safe, process-safe disk cache with SQLite backend
82
- **FanoutCache**: Sharded cache that distributes keys across multiple Cache instances for higher throughput
83
- **Deque**: Persistent double-ended queue with collections.abc.Sequence interface
84
- **Index**: Persistent mapping/dictionary with collections.abc.MutableMapping interface
85
- **Disk/JSONDisk**: Serialization engines supporting various storage modes
86
- **Recipe Classes**: Synchronization primitives (Lock, RLock, BoundedSemaphore, Averager)
87
- **Django Integration**: Full Django cache backend compatibility
88

89
All classes support context management, atomic transactions, and provide comprehensive cache management with statistics, eviction policies, and tag-based operations.
90

91
## Capabilities
92

93
### Core Caching
94

95
Primary caching functionality with the Cache class, supporting all standard cache operations, atomic transactions, queue operations, and comprehensive cache management with statistics and eviction policies.
96

97
```python { .api }
98
class Cache:
99
    def __init__(self, directory=None, timeout=60, disk=Disk, **settings): ...
100
    def set(self, key, value, expire=None, read=False, tag=None, retry=False): ...
101
    def get(self, key, default=None, read=False, expire_time=False, tag=False, retry=False): ...
102
    def delete(self, key, retry=False): ...
103
    def clear(self, retry=False): ...
104
    def close(self): ...
105
```
106

107
[Core Caching](./core-caching.md)
108

109
### Sharded Caching
110

111
High-throughput caching using FanoutCache which automatically distributes keys across multiple Cache instances for improved performance and scalability.
112

113
```python { .api }
114
class FanoutCache:
115
    def __init__(self, directory=None, shards=8, timeout=0.010, disk=Disk, **settings): ...
116
    def set(self, key, value, expire=None, read=False, tag=None, retry=False): ...
117
    def get(self, key, default=None, read=False, expire_time=False, tag=False, retry=False): ...
118
    def cache(self, name, timeout=60, disk=None, **settings): ...
119
    def deque(self, name, maxlen=None): ...
120
    def index(self, name): ...
121
```
122

123
[Sharded Caching](./fanout-cache.md)
124

125
### Persistent Data Structures
126

127
Disk-backed data structures (Deque and Index) that provide familiar interfaces while persisting data to disk, supporting full sequence and mapping operations with atomic transactions.
128

129
```python { .api }
130
class Deque:
131
    def __init__(self, iterable=(), directory=None, maxlen=None): ...
132
    def append(self, value): ...
133
    def appendleft(self, value): ...
134
    def pop(self): ...
135
    def popleft(self): ...
136

137
class Index:
138
    def __init__(self, *args, **kwargs): ...
139
    def __getitem__(self, key): ...
140
    def __setitem__(self, key, value): ...
141
    def keys(self): ...
142
    def values(self): ...
143
    def items(self): ...
144
```
145

146
[Persistent Data Structures](./persistent-data-structures.md)
147

148
### Synchronization Primitives
149

150
Thread-safe and process-safe synchronization primitives including locks, semaphores, averager, and various decorators for throttling and cache stampede protection.
151

152
```python { .api }
153
class Lock:
154
    def __init__(self, cache, key, expire=None, tag=None): ...
155
    def acquire(self): ...
156
    def release(self): ...
157

158
class RLock:
159
    def __init__(self, cache, key, expire=None, tag=None): ...
160
    def acquire(self): ...
161
    def release(self): ...
162

163
class BoundedSemaphore:
164
    def __init__(self, cache, key, value=1, expire=None, tag=None): ...
165
    def acquire(self): ...
166
    def release(self): ...
167

168
class Averager:
169
    def __init__(self, cache, key, expire=None, tag=None): ...
170
    def add(self, value): ...
171
    def get(self): ...
172
    def pop(self): ...
173
```
174

175
[Synchronization Primitives](./synchronization-primitives.md)
176

177
### Recipe Functions
178

179
Decorator functions for advanced caching patterns including throttling, serialization barriers, and memoization with cache stampede protection.
180

181
```python { .api }
182
def throttle(cache, count, seconds, name=None, expire=None, tag=None, 
183
             time_func=time.time, sleep_func=time.sleep): ...
184
def barrier(cache, lock_factory, name=None, expire=None, tag=None): ...
185
def memoize_stampede(cache, expire, name=None, typed=False, tag=None, 
186
                     beta=1, ignore=()): ...
187
```
188

189
[Recipe Functions](./recipe-functions.md)
190

191
### Disk Serialization
192

193
Serialization engines that handle conversion between Python objects and disk storage, supporting multiple storage modes including pickle, JSON with compression, and raw binary data.
194

195
```python { .api }
196
class Disk:
197
    def __init__(self, directory, min_file_size=0, pickle_protocol=0): ...
198
    def put(self, key): ...
199
    def get(self, key, raw): ...
200
    def store(self, value, read, key=UNKNOWN): ...
201
    def fetch(self, mode, filename, value, read): ...
202

203
class JSONDisk:
204
    def __init__(self, directory, compress_level=1, **kwargs): ...
205
    # Inherits all Disk methods with JSON serialization
206
```
207

208
[Disk Serialization](./disk-serialization.md)
209

210
### Django Integration
211

212
Full Django cache backend compatibility providing seamless integration with Django applications while maintaining all DiskCache features and performance benefits.
213

214
```python { .api }
215
class DjangoCache:
216
    def __init__(self, directory, params): ...
217
    def add(self, key, value, timeout=DEFAULT_TIMEOUT, version=None): ...
218
    def get(self, key, default=None, version=None): ...
219
    def set(self, key, value, timeout=DEFAULT_TIMEOUT, version=None): ...
220
    def delete(self, key, version=None): ...
221
    def clear(self): ...
222
```
223

224
[Django Integration](./django-integration.md)
225

226
## Constants and Settings
227

228
```python { .api }
229
DEFAULT_SETTINGS = {
230
    'statistics': 0,
231
    'tag_index': 0,
232
    'eviction_policy': 'least-recently-stored',
233
    'size_limit': 2**30,  # 1GB
234
    'cull_limit': 10,
235
    'sqlite_auto_vacuum': 1,
236
    'sqlite_cache_size': 2**13,
237
    'sqlite_journal_mode': 'wal',
238
    'sqlite_mmap_size': 2**26,
239
    'sqlite_synchronous': 1,
240
    'disk_min_file_size': 2**15,
241
    'disk_pickle_protocol': pickle.HIGHEST_PROTOCOL,
242
}
243

244
EVICTION_POLICY = {
245
    'none': {...},
246
    'least-recently-stored': {...},
247
    'least-recently-used': {...},
248
    'least-frequently-used': {...},
249
}
250

251
ENOVAL = Constant('ENOVAL')  # Special constant for "no value"
252
UNKNOWN = Constant('UNKNOWN')  # Special constant for unknown values
253
```
254

255
## Exceptions
256

257
```python { .api }
258
class Timeout(Exception):
259
    """Database timeout exception."""
260

261
class UnknownFileWarning(UserWarning):
262
    """Warning for unknown files in cache directory."""
263

264
class EmptyDirWarning(UserWarning):
265
    """Warning for empty cache directories."""
266
```