0
# Blob Granules
1
2
FoundationDB Blob Granules provide a storage layer for large objects and historical data, designed for cost-effective storage of infrequently accessed data. Blob granules enable separation of compute and storage, allowing for efficient archival and analytical workloads.
3
4
## Core Concepts
5
6
- **Blob Granule**: A storage unit containing historical key-value data and metadata
7
- **Blobbification**: Process of converting regular FoundationDB data to blob storage
8
- **Granule Range**: Key range covered by a specific blob granule
9
- **Purge Operations**: Administrative functions to clean up old blob data
10
- **Read Context**: Configuration for reading blob granule files
11
12
## Capabilities
13
14
### Database-Level Blob Operations
15
16
Administrative functions for managing blob granules at the database level.
17
18
```c { .api }
19
// C API - Database blob granule operations
20
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_database_purge_blob_granules(FDBDatabase* db,
21
uint8_t const* begin_key_name,
22
int begin_key_name_length,
23
uint8_t const* end_key_name,
24
int end_key_name_length,
25
int64_t purge_version,
26
fdb_bool_t force);
27
28
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_database_wait_purge_granules_complete(FDBDatabase* db,
29
uint8_t const* purge_key_name,
30
int purge_key_name_length);
31
32
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_database_blobbify_range(FDBDatabase* db,
33
uint8_t const* begin_key_name,
34
int begin_key_name_length,
35
uint8_t const* end_key_name,
36
int end_key_name_length);
37
38
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_database_blobbify_range_blocking(FDBDatabase* db,
39
uint8_t const* begin_key_name,
40
int begin_key_name_length,
41
uint8_t const* end_key_name,
42
int end_key_name_length);
43
44
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_database_unblobbify_range(FDBDatabase* db,
45
uint8_t const* begin_key_name,
46
int begin_key_name_length,
47
uint8_t const* end_key_name,
48
int end_key_name_length);
49
50
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_database_list_blobbified_ranges(FDBDatabase* db,
51
uint8_t const* begin_key_name,
52
int begin_key_name_length,
53
uint8_t const* end_key_name,
54
int end_key_name_length,
55
int rangeLimit);
56
57
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_database_verify_blob_range(FDBDatabase* db,
58
uint8_t const* begin_key_name,
59
int begin_key_name_length,
60
uint8_t const* end_key_name,
61
int end_key_name_length,
62
int64_t version);
63
64
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_database_flush_blob_range(FDBDatabase* db,
65
uint8_t const* begin_key_name,
66
int begin_key_name_length,
67
uint8_t const* end_key_name,
68
int end_key_name_length,
69
fdb_bool_t compact,
70
int64_t version);
71
```
72
73
### Tenant-Level Blob Operations
74
75
Blob granule operations scoped to specific tenants.
76
77
```c { .api }
78
// C API - Tenant blob granule operations
79
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_tenant_purge_blob_granules(FDBTenant* tenant,
80
uint8_t const* begin_key_name,
81
int begin_key_name_length,
82
uint8_t const* end_key_name,
83
int end_key_name_length,
84
int64_t purge_version,
85
fdb_bool_t force);
86
87
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_tenant_wait_purge_granules_complete(FDBTenant* tenant,
88
uint8_t const* purge_key_name,
89
int purge_key_name_length);
90
91
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_tenant_blobbify_range(FDBTenant* tenant,
92
uint8_t const* begin_key_name,
93
int begin_key_name_length,
94
uint8_t const* end_key_name,
95
int end_key_name_length);
96
97
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_tenant_blobbify_range_blocking(FDBTenant* tenant,
98
uint8_t const* begin_key_name,
99
int begin_key_name_length,
100
uint8_t const* end_key_name,
101
int end_key_name_length);
102
103
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_tenant_unblobbify_range(FDBTenant* tenant,
104
uint8_t const* begin_key_name,
105
int begin_key_name_length,
106
uint8_t const* end_key_name,
107
int end_key_name_length);
108
109
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_tenant_list_blobbified_ranges(FDBTenant* tenant,
110
uint8_t const* begin_key_name,
111
int begin_key_name_length,
112
uint8_t const* end_key_name,
113
int end_key_name_length,
114
int rangeLimit);
115
116
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_tenant_verify_blob_range(FDBTenant* tenant,
117
uint8_t const* begin_key_name,
118
int begin_key_name_length,
119
uint8_t const* end_key_name,
120
int end_key_name_length,
121
int64_t version);
122
123
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_tenant_flush_blob_range(FDBTenant* tenant,
124
uint8_t const* begin_key_name,
125
int begin_key_name_length,
126
uint8_t const* end_key_name,
127
int end_key_name_length,
128
fdb_bool_t compact,
129
int64_t version);
130
```
131
132
### Transaction-Level Blob Operations
133
134
Reading and querying blob granules within transactions.
135
136
```c { .api }
137
// C API - Transaction blob granule operations
138
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_transaction_get_blob_granule_ranges(FDBTransaction* tr,
139
uint8_t const* begin_key_name,
140
int begin_key_name_length,
141
uint8_t const* end_key_name,
142
int end_key_name_length,
143
int rangeLimit);
144
145
DLLEXPORT WARN_UNUSED_RESULT FDBResult* fdb_transaction_read_blob_granules(FDBTransaction* tr,
146
uint8_t const* begin_key_name,
147
int begin_key_name_length,
148
uint8_t const* end_key_name,
149
int end_key_name_length,
150
int64_t beginVersion,
151
int64_t readVersion,
152
FDBReadBlobGranuleContext granuleContext);
153
154
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_transaction_summarize_blob_granules(FDBTransaction* tr,
155
uint8_t const* begin_key_name,
156
int begin_key_name_length,
157
uint8_t const* end_key_name,
158
int end_key_name_length,
159
int64_t summaryVersion,
160
int rangeLimit);
161
162
DLLEXPORT WARN_UNUSED_RESULT FDBFuture* fdb_transaction_read_blob_granules_description(FDBTransaction* tr,
163
uint8_t const* begin_key_name,
164
int begin_key_name_length,
165
uint8_t const* end_key_name,
166
int end_key_name_length,
167
int64_t begin_version,
168
int64_t read_version);
169
```
170
171
### Blob Granule Data Types
172
173
```c { .api }
174
// C API - Blob granule data structures
175
typedef struct keyrange {
176
const uint8_t* begin_key;
177
int begin_key_length;
178
const uint8_t* end_key;
179
int end_key_length;
180
} FDBKeyRange;
181
182
typedef struct granulesummary {
183
FDBKeyRange key_range;
184
int64_t snapshot_version;
185
int64_t snapshot_size;
186
int64_t delta_version;
187
int64_t delta_size;
188
} FDBGranuleSummary;
189
190
typedef struct bgtenantprefix {
191
fdb_bool_t present;
192
FDBKey prefix;
193
} FDBBGTenantPrefix;
194
195
typedef struct readgranulecontext {
196
void* userContext;
197
198
// File loading callbacks
199
int64_t (*start_load_f)(const char* filename, int filename_length,
200
int64_t offset, int64_t length, void* context);
201
uint8_t* (*get_load_f)(int64_t loadId, void* context);
202
void (*free_load_f)(int64_t loadId, void* context);
203
204
// Configuration options
205
fdb_bool_t debugNoMaterialize;
206
int granuleParallelism;
207
} FDBReadBlobGranuleContext;
208
209
typedef enum {
210
FDB_BG_MUTATION_TYPE_SET_VALUE = 0,
211
FDB_BG_MUTATION_TYPE_CLEAR_RANGE = 1
212
} FDBBGMutationType;
213
214
typedef struct bgmutation {
215
FDBBGMutationType type;
216
FDBKey param1;
217
FDBKey param2; // Only used for clear_range
218
} FDBBGMutation;
219
220
typedef struct bgfiledescription {
221
FDBBGFilePointer snapshot_file_pointer;
222
int delta_file_count;
223
FDBBGFilePointer* delta_files;
224
int memory_mutation_count;
225
FDBBGMutation* memory_mutations;
226
FDBBGTenantPrefix tenant_prefix;
227
} FDBBGFileDescription;
228
```
229
230
### Blob Granule File Parsing
231
232
Functions for parsing blob granule files and extracting data.
233
234
```c { .api }
235
// C API - Blob granule file parsing
236
DLLEXPORT WARN_UNUSED_RESULT FDBResult* fdb_readbg_parse_snapshot_file(const uint8_t* file_data,
237
int file_len,
238
FDBBGTenantPrefix const* tenant_prefix,
239
FDBBGEncryptionCtx const* encryption_ctx);
240
241
DLLEXPORT WARN_UNUSED_RESULT FDBResult* fdb_readbg_parse_delta_file(const uint8_t* file_data,
242
int file_len,
243
FDBBGTenantPrefix const* tenant_prefix,
244
FDBBGEncryptionCtx const* encryption_ctx);
245
246
// Future result extraction for blob granules
247
DLLEXPORT WARN_UNUSED_RESULT fdb_error_t fdb_future_get_granule_summary_array(FDBFuture* f,
248
FDBGranuleSummary const** out_summaries,
249
int* out_count);
250
251
DLLEXPORT WARN_UNUSED_RESULT fdb_error_t fdb_future_readbg_get_descriptions(FDBFuture* f,
252
FDBBGFileDescription** out,
253
int* desc_count);
254
255
// Result management
256
DLLEXPORT void fdb_result_destroy(FDBResult* r);
257
DLLEXPORT WARN_UNUSED_RESULT fdb_error_t fdb_result_get_keyvalue_array(FDBResult* r,
258
FDBKeyValue const** out_kv,
259
int* out_count);
260
```
261
262
## Usage Patterns
263
264
### Converting Range to Blob Storage
265
266
```c
267
// Example: Convert a key range to blob storage
268
FDBDatabase* db = /* ... */;
269
const uint8_t* begin_key = (const uint8_t*)"user_archive/";
270
const uint8_t* end_key = (const uint8_t*)"user_archive0";
271
272
// Initiate blobbification (asynchronous)
273
FDBFuture* blobbify_future = fdb_database_blobbify_range(
274
db, begin_key, strlen((char*)begin_key),
275
end_key, strlen((char*)end_key)
276
);
277
278
// Wait for completion
279
fdb_error_t err = fdb_future_block_until_ready(blobbify_future);
280
if (err == 0) {
281
printf("Range successfully converted to blob storage\n");
282
}
283
284
fdb_future_destroy(blobbify_future);
285
```
286
287
### Reading Historical Data from Blob Granules
288
289
```c
290
// Example: Read historical data from blob granules
291
FDBTransaction* tr = /* ... */;
292
FDBReadBlobGranuleContext context = {
293
.userContext = NULL,
294
.start_load_f = my_start_load_function,
295
.get_load_f = my_get_load_function,
296
.free_load_f = my_free_load_function,
297
.debugNoMaterialize = false,
298
.granuleParallelism = 4
299
};
300
301
const uint8_t* begin_key = (const uint8_t*)"historical/";
302
const uint8_t* end_key = (const uint8_t*)"historical0";
303
int64_t begin_version = 1000000;
304
int64_t read_version = 2000000;
305
306
FDBResult* result = fdb_transaction_read_blob_granules(
307
tr,
308
begin_key, strlen((char*)begin_key),
309
end_key, strlen((char*)end_key),
310
begin_version,
311
read_version,
312
context
313
);
314
315
// Extract key-value pairs from result
316
FDBKeyValue const* kvs;
317
int count;
318
fdb_error_t err = fdb_result_get_keyvalue_array(result, &kvs, &count);
319
320
if (err == 0) {
321
for (int i = 0; i < count; i++) {
322
printf("Key: %.*s, Value: %.*s\n",
323
kvs[i].key_length, kvs[i].key,
324
kvs[i].value_length, kvs[i].value);
325
}
326
}
327
328
fdb_result_destroy(result);
329
```
330
331
### Purging Old Blob Data
332
333
```c
334
// Example: Purge old blob granule data
335
FDBDatabase* db = /* ... */;
336
const uint8_t* begin_key = (const uint8_t*)"archive/";
337
const uint8_t* end_key = (const uint8_t*)"archive0";
338
int64_t purge_version = 1000000; // Purge data older than this version
339
340
FDBFuture* purge_future = fdb_database_purge_blob_granules(
341
db,
342
begin_key, strlen((char*)begin_key),
343
end_key, strlen((char*)end_key),
344
purge_version,
345
false // force = false
346
);
347
348
// Wait for purge to complete
349
fdb_error_t err = fdb_future_block_until_ready(purge_future);
350
if (err == 0) {
351
printf("Purge operation completed\n");
352
}
353
354
fdb_future_destroy(purge_future);
355
```
356
357
## Best Practices
358
359
### Blob Granule Design Patterns
360
361
1. **Archival Pattern**: Move old transactional data to blob storage
362
2. **Analytics Pattern**: Store analytical datasets in blob granules
363
3. **Backup Pattern**: Use blob granules for point-in-time backups
364
4. **Tiered Storage**: Separate hot and cold data using blob granules
365
366
### Performance Considerations
367
368
- Blob granules are optimized for large sequential reads
369
- Use appropriate granule parallelism for read operations
370
- Consider network latency when accessing blob files
371
- Purge old data regularly to manage storage costs
372
373
### Error Handling
374
375
Blob granule operations can fail due to various reasons:
376
- Network issues accessing blob storage
377
- File corruption or missing files
378
- Version conflicts during reads
379
- Administrative restrictions on purge operations
380
381
Always check return codes and implement appropriate retry logic for transient failures.