0
# Low-level Operations and Utilities
1

2
Advanced methods for direct data access, manual expiration management, and internal storage operations.
3

4
## Capabilities
5

6
### Raw Data Access
7

8
Direct access to the internal datum objects and file paths.
9

10
```javascript { .api }
11
/**
12
 * Gets the complete datum object for a key (includes key, value, and TTL)
13
 * @param key - The key to get the datum for
14
 * @returns Promise resolving to datum object or empty object if not found
15
 */
16
function getDatum(key: string): Promise<any>;
17

18
/**
19
 * Gets the raw JSON string content of a storage file
20
 * @param key - The key to get raw data for
21
 * @returns Promise resolving to raw JSON string
22
 */
23
function getRawDatum(key: string): Promise<string>;
24

25
/**
26
 * Gets just the value from a datum object
27
 * @param key - The key to get the value for
28
 * @returns Promise resolving to the stored value or undefined
29
 */
30
async function getDatumValue(key: string): Promise<any>;
31

32
/**
33
 * Gets the filesystem path where a key's data is stored (synchronous)
34
 * @param key - The key to get the path for
35
 * @returns Absolute filesystem path to the storage file
36
 */
37
function getDatumPath(key: string): string;
38
```
39

40
**Usage Examples:**
41

42
```javascript
43
const storage = require('node-persist');
44
await storage.init();
45

46
await storage.setItem('user', { name: 'Alice' }, { ttl: 60000 });
47

48
// Get complete datum with metadata
49
const datum = await storage.getDatum('user');
50
console.log(datum);
51
// { key: 'user', value: { name: 'Alice' }, ttl: 1640995200000 }
52

53
// Get raw JSON string
54
const rawData = await storage.getRawDatum('user');
55
console.log(rawData);
56
// '{"key":"user","value":{"name":"Alice"},"ttl":1640995200000}'
57

58
// Get just the value
59
const value = await storage.getDatumValue('user');
60
console.log(value); // { name: 'Alice' }
61

62
// Get file path
63
const filePath = storage.getDatumPath('user');
64
console.log(filePath);
65
// '/path/to/storage/abcd1234...' (SHA256 hash of key)
66
```
67

68
### Manual Expiration Management
69

70
Control over the automatic TTL cleanup process.
71

72
```javascript { .api }
73
/**
74
 * Manually removes all expired items from storage
75
 * @returns Promise resolving when cleanup is complete
76
 */
77
async function removeExpiredItems(): Promise<void>;
78

79
/**
80
 * Starts the automatic expired item cleanup interval
81
 * Uses the expiredInterval option setting
82
 */
83
function startExpiredKeysInterval(): void;
84

85
/**
86
 * Stops the automatic expired item cleanup interval
87
 */
88
function stopExpiredKeysInterval(): void;
89
```
90

91
**Usage Examples:**
92

93
```javascript
94
// Manual cleanup
95
await storage.removeExpiredItems();
96
console.log('All expired items removed');
97

98
// Control automatic cleanup
99
storage.stopExpiredKeysInterval(); // Stop automatic cleanup
100
// ... do some work ...
101
storage.startExpiredKeysInterval(); // Resume automatic cleanup
102

103
// Custom cleanup schedule
104
storage.stopExpiredKeysInterval();
105
setInterval(async () => {
106
  console.log('Running custom cleanup...');
107
  await storage.removeExpiredItems();
108
}, 30000); // Every 30 seconds
109
```
110

111
### Write Queue Management
112

113
Control over the write queue system for performance optimization.
114

115
```javascript { .api }
116
/**
117
 * Starts the write queue processing interval
118
 * Processes pending writes at the configured interval
119
 */
120
function startWriteQueueInterval(): void;
121

122
/**
123
 * Stops the write queue processing interval
124
 * Note: This may leave pending writes unprocessed
125
 */
126
function stopWriteQueueInterval(): void;
127
```
128

129
**Usage Examples:**
130

131
```javascript
132
// Temporarily disable write queue processing for bulk operations
133
storage.stopWriteQueueInterval();
134

135
// Perform many write operations
136
for (let i = 0; i < 1000; i++) {
137
  await storage.setItem(`item${i}`, { data: i });
138
}
139

140
// Resume write queue processing
141
storage.startWriteQueueInterval();
142
```
143

144
### Configuration Updates
145

146
Dynamic configuration changes after initialization.
147

148
```javascript { .api }
149
/**
150
 * Updates storage configuration options
151
 * @param options - New configuration options to apply
152
 */
153
function setOptions(options: StorageOptions): void;
154
```
155

156
**Usage Examples:**
157

158
```javascript
159
// Change logging level
160
storage.setOptions({ logging: true });
161

162
// Update TTL default
163
storage.setOptions({ ttl: 3600000 }); // 1 hour
164

165
// Change write queue behavior
166
storage.setOptions({ 
167
  writeQueueWriteOnlyLast: false,
168
  writeQueueIntervalMs: 500 
169
});
170

171
// Update file descriptor limit
172
storage.setOptions({ maxFileDescriptors: 256 });
173
```
174

175
## Advanced Usage Patterns
176

177
### Storage File Inspection
178

179
```javascript
180
// Inspect storage files directly
181
const allData = await storage.data();
182

183
for (const datum of allData) {
184
  const filePath = storage.getDatumPath(datum.key);
185
  console.log(`Key: ${datum.key}`);
186
  console.log(`File: ${filePath}`);
187
  
188
  if (datum.ttl) {
189
    const expiresAt = new Date(datum.ttl);
190
    const isExpired = Date.now() > datum.ttl;
191
    console.log(`Expires: ${expiresAt.toISOString()} (${isExpired ? 'EXPIRED' : 'valid'})`);
192
  }
193
}
194
```
195

196
### Custom Cleanup Logic
197

198
```javascript
199
// Implement custom expiration logic
200
async function customCleanup() {
201
  const allData = await storage.data();
202
  
203
  for (const datum of allData) {
204
    // Custom logic: remove items older than 7 days regardless of TTL
205
    const sevenDaysAgo = Date.now() - (7 * 24 * 60 * 60 * 1000);
206
    const fileStats = fs.statSync(storage.getDatumPath(datum.key));
207
    
208
    if (fileStats.mtime.getTime() < sevenDaysAgo) {
209
      await storage.removeItem(datum.key);
210
      console.log(`Removed old item: ${datum.key}`);
211
    }
212
  }
213
}
214

215
// Run custom cleanup
216
await customCleanup();
217
```
218

219
### Performance Monitoring
220

221
```javascript
222
// Monitor write queue size
223
const originalSetItem = storage.setItem;
224
storage.setItem = async function(key, value, options) {
225
  const startTime = Date.now();
226
  const result = await originalSetItem.call(this, key, value, options);
227
  const duration = Date.now() - startTime;
228
  
229
  if (duration > 100) {
230
    console.warn(`Slow write detected: ${key} took ${duration}ms`);
231
  }
232
  
233
  return result;
234
};
235
```
236

237
## Important Notes
238

239
### Thread Safety
240

241
- node-persist is designed for single-process use
242
- Multiple processes writing to the same storage directory can cause corruption
243
- Use file locking or separate directories for multi-process scenarios
244

245
### File System Considerations
246

247
- Storage files use SHA256 hashes as filenames
248
- Each key creates one file in the storage directory
249
- Large numbers of keys can impact filesystem performance
250
- Consider directory cleanup for temporary data
251

252
### Memory Usage
253

254
- Methods like `data()`, `keys()`, and `values()` load all data into memory
255
- For large datasets, consider using streaming approaches or filtering
256
- Write queue batching helps reduce memory usage for frequent writes