0
# Index Management
1
2
PyMilvus provides comprehensive index management for optimizing vector and scalar search performance. This covers vector index types, scalar indexes, index parameters, performance tuning, and maintenance operations.
3
4
## Vector Index Types
5
6
### FLAT Index (Exact Search)
7
8
```python { .api }
9
# FLAT index provides exact search results with 100% recall
10
flat_index_params = {
11
"index_type": "FLAT",
12
"metric_type": "L2", # or "IP", "COSINE"
13
"params": {} # No additional parameters needed
14
}
15
16
client.create_index("collection", "vector_field", flat_index_params)
17
```
18
19
**Characteristics:**
20
- **Accuracy**: 100% recall (exact search)
21
- **Performance**: Slower for large datasets, O(n) complexity
22
- **Memory**: Stores all vectors in memory
23
- **Best for**: Small datasets (<10K vectors), high accuracy requirements
24
25
### IVF_FLAT Index (Inverted File)
26
27
```python { .api }
28
# IVF_FLAT partitions vectors into clusters for faster search
29
ivf_flat_params = {
30
"index_type": "IVF_FLAT",
31
"metric_type": "L2",
32
"params": {
33
"nlist": 1024 # Number of clusters (16-16384, default: 1024)
34
}
35
}
36
37
client.create_index("large_collection", "embedding", ivf_flat_params)
38
```
39
40
**Parameters:**
41
- `nlist`: Number of clusters (more clusters = faster search but more memory)
42
43
**Search Parameters:**
44
```python { .api }
45
# Search parameters for IVF_FLAT
46
search_params = {
47
"nprobe": 16 # Number of clusters to search (1 to nlist)
48
}
49
50
results = client.search(
51
"large_collection",
52
data=[query_vector],
53
search_params=search_params,
54
limit=10
55
)
56
```
57
58
**Characteristics:**
59
- **Accuracy**: High recall with proper nprobe tuning
60
- **Performance**: Good balance of speed and accuracy
61
- **Memory**: More memory efficient than FLAT for large datasets
62
- **Best for**: Medium to large datasets (10K-10M vectors)
63
64
### IVF_PQ Index (Product Quantization)
65
66
```python { .api }
67
# IVF_PQ combines clustering with product quantization for memory efficiency
68
ivf_pq_params = {
69
"index_type": "IVF_PQ",
70
"metric_type": "L2",
71
"params": {
72
"nlist": 2048, # Number of clusters
73
"m": 16, # Number of PQ segments (dim must be divisible by m)
74
"nbits": 8 # Bits per PQ centroid (4-16, default: 8)
75
}
76
}
77
78
client.create_index("huge_collection", "vector", ivf_pq_params)
79
```
80
81
**Parameters:**
82
- `nlist`: Number of clusters (similar to IVF_FLAT)
83
- `m`: PQ segments count (vector dimension must be divisible by m)
84
- `nbits`: Bits per segment (4, 6, 8, 10, 12, 16)
85
86
**Search Parameters:**
87
```python { .api }
88
search_params = {"nprobe": 32} # Usually need higher nprobe for good recall
89
```
90
91
**Characteristics:**
92
- **Accuracy**: Good recall with proper tuning (slightly lower than IVF_FLAT)
93
- **Performance**: Fast search with compressed vectors
94
- **Memory**: Very memory efficient (4x-32x compression)
95
- **Best for**: Very large datasets (10M+ vectors) with memory constraints
96
97
### HNSW Index (Hierarchical NSW)
98
99
```python { .api }
100
# HNSW builds a multi-layer graph for fast approximate search
101
hnsw_params = {
102
"index_type": "HNSW",
103
"metric_type": "L2",
104
"params": {
105
"M": 16, # Max bidirectional links (4-64, default: 16)
106
"efConstruction": 200 # Construction time/quality tradeoff (8-512, default: 200)
107
}
108
}
109
110
client.create_index("fast_search_collection", "embedding", hnsw_params)
111
```
112
113
**Parameters:**
114
- `M`: Maximum bidirectional links per node (higher = better recall, more memory)
115
- `efConstruction`: Size of dynamic candidate list (higher = better quality, slower build)
116
117
**Search Parameters:**
118
```python { .api }
119
search_params = {
120
"ef": 64 # Search scope (ef >= limit, higher = better recall)
121
}
122
123
# ef should be at least equal to the search limit
124
results = client.search(
125
"fast_search_collection",
126
data=[query_vector],
127
search_params=search_params,
128
limit=10 # ef should be >= 10
129
)
130
```
131
132
**Characteristics:**
133
- **Accuracy**: Excellent recall with proper ef tuning
134
- **Performance**: Very fast search, especially for high-dimensional data
135
- **Memory**: Higher memory usage than IVF indexes
136
- **Best for**: Real-time applications requiring low latency
137
138
### Sparse Vector Indexes
139
140
```python { .api }
141
# SPARSE_INVERTED_INDEX for sparse vectors (BM25, TF-IDF)
142
sparse_index_params = {
143
"index_type": "SPARSE_INVERTED_INDEX",
144
"metric_type": "IP", # Inner Product for sparse vectors
145
"params": {
146
"drop_ratio_build": 0.2 # Drop tokens with low frequency during build
147
}
148
}
149
150
client.create_index("text_collection", "sparse_embedding", sparse_index_params)
151
```
152
153
**Search Parameters:**
154
```python { .api }
155
sparse_search_params = {
156
"drop_ratio_search": 0.1 # Drop low-weight tokens during search
157
}
158
```
159
160
## Scalar Indexes
161
162
### String Field Indexes
163
164
```python { .api }
165
# TRIE index for VARCHAR fields (prefix matching, equality)
166
trie_params = {"index_type": "TRIE"}
167
client.create_index("products", "category", trie_params)
168
169
# Examples of TRIE index benefits
170
results = client.query("products", "category == 'electronics'") # Fast equality
171
results = client.query("products", "category like 'elect%'") # Fast prefix matching
172
```
173
174
### Numeric Field Indexes
175
176
```python { .api }
177
# STL_SORT index for numeric fields (range queries, sorting)
178
sort_params = {"index_type": "STL_SORT"}
179
client.create_index("products", "price", sort_params)
180
181
# Examples of STL_SORT benefits
182
results = client.query("products", "price > 100 and price < 500") # Fast range queries
183
results = client.query("products", "price between 50 and 200") # Optimized range
184
```
185
186
### JSON Field Indexes
187
188
```python { .api }
189
# INVERTED index for JSON fields (key-value queries)
190
inverted_params = {"index_type": "INVERTED"}
191
client.create_index("documents", "metadata", inverted_params)
192
193
# JSON queries benefit from INVERTED index
194
results = client.query("documents", "metadata['category'] == 'tech'")
195
results = client.query("documents", "json_contains(metadata['tags'], 'AI')")
196
```
197
198
### Array Field Indexes
199
200
```python { .api }
201
# INVERTED index also works for ARRAY fields
202
client.create_index("articles", "tags", {"index_type": "INVERTED"})
203
204
# Array queries with index optimization
205
results = client.query("articles", "array_contains(tags, 'machine-learning')")
206
results = client.query("articles", "array_contains_all(tags, ['AI', 'Python'])")
207
```
208
209
## Index Operations
210
211
### Creating Indexes
212
213
```python { .api }
214
from pymilvus import MilvusClient, Collection
215
216
# Using MilvusClient
217
client = MilvusClient()
218
219
def create_index(
220
collection_name: str,
221
field_name: str,
222
index_params: Dict[str, Any],
223
timeout: Optional[float] = None,
224
**kwargs
225
) -> None
226
```
227
228
```python { .api }
229
# Using Collection (ORM)
230
collection = Collection("my_collection")
231
232
def create_index(
233
field_name: str,
234
index_params: Dict[str, Any],
235
timeout: Optional[float] = None,
236
**kwargs
237
) -> None
238
```
239
240
**Examples:**
241
```python
242
# Create multiple indexes on different fields
243
index_operations = [
244
("vector_field", {
245
"index_type": "HNSW",
246
"metric_type": "COSINE",
247
"params": {"M": 32, "efConstruction": 400}
248
}),
249
("category", {"index_type": "TRIE"}),
250
("price", {"index_type": "STL_SORT"}),
251
("metadata", {"index_type": "INVERTED"})
252
]
253
254
for field_name, params in index_operations:
255
client.create_index("products", field_name, params)
256
print(f"Created index on {field_name}")
257
```
258
259
### Index Information and Management
260
261
```python { .api }
262
# List all indexes for a collection
263
indexes = client.list_indexes("products")
264
print(f"Indexes: {indexes}")
265
266
# Describe specific index
267
index_info = client.describe_index("products", "vector_field")
268
print(f"Index type: {index_info['index_type']}")
269
print(f"Metric type: {index_info['metric_type']}")
270
print(f"Parameters: {index_info['params']}")
271
272
# Drop index
273
client.drop_index("products", "old_field")
274
275
# Check if index exists (Collection ORM)
276
collection = Collection("products")
277
has_vector_index = collection.has_index("vector_field")
278
```
279
280
### Index Building Progress
281
282
```python { .api }
283
from pymilvus import utility
284
285
# Monitor index building progress
286
progress = utility.index_building_progress("products", "vector_field")
287
print(f"Index progress: {progress['indexed_rows']}/{progress['total_rows']} ({progress['progress']}%)")
288
289
# Wait for index building to complete
290
utility.wait_for_index_building_complete("products", "vector_field", timeout=300)
291
print("Index building completed")
292
293
# Alternative: using Collection
294
collection = Collection("products")
295
collection.create_index("new_vector", hnsw_params)
296
297
# Monitor with polling
298
import time
299
while True:
300
progress = utility.index_building_progress("products", "new_vector")
301
if progress['progress'] == 100:
302
print("Index building completed")
303
break
304
print(f"Progress: {progress['progress']}%")
305
time.sleep(5)
306
```
307
308
## Performance Optimization
309
310
### Index Parameter Tuning
311
312
```python { .api }
313
def optimize_hnsw_parameters(collection_name: str, vector_field: str, dataset_size: int, dimension: int):
314
"""Optimize HNSW parameters based on dataset characteristics"""
315
316
# M parameter guidelines
317
if dataset_size < 100000:
318
M = 16 # Default for small datasets
319
elif dataset_size < 1000000:
320
M = 32 # Better connectivity for medium datasets
321
else:
322
M = 64 # Maximum connectivity for large datasets
323
324
# efConstruction guidelines
325
if dimension < 128:
326
efConstruction = 200 # Default for low-dimensional data
327
elif dimension < 512:
328
efConstruction = 400 # Higher for medium dimensions
329
else:
330
efConstruction = 800 # Maximum for high dimensions
331
332
params = {
333
"index_type": "HNSW",
334
"metric_type": "L2",
335
"params": {
336
"M": M,
337
"efConstruction": efConstruction
338
}
339
}
340
341
print(f"Optimized HNSW params for {dataset_size} vectors, {dimension}D: M={M}, efConstruction={efConstruction}")
342
343
client.create_index(collection_name, vector_field, params)
344
345
return params
346
347
# Usage
348
hnsw_params = optimize_hnsw_parameters("large_collection", "embedding", 5000000, 768)
349
```
350
351
```python { .api }
352
def optimize_ivf_parameters(dataset_size: int, memory_constraint: bool = False):
353
"""Optimize IVF parameters based on dataset size and memory"""
354
355
if dataset_size < 10000:
356
# Use FLAT for small datasets
357
return {
358
"index_type": "FLAT",
359
"metric_type": "L2",
360
"params": {}
361
}
362
363
# nlist guidelines: sqrt(N) to N/39
364
nlist = min(16384, max(16, int(dataset_size ** 0.5)))
365
366
if memory_constraint and dataset_size > 1000000:
367
# Use IVF_PQ for memory efficiency
368
return {
369
"index_type": "IVF_PQ",
370
"metric_type": "L2",
371
"params": {
372
"nlist": nlist,
373
"m": 16, # Adjust based on dimension
374
"nbits": 8 # 8 bits provides good compression
375
}
376
}
377
else:
378
# Use IVF_FLAT for better accuracy
379
return {
380
"index_type": "IVF_FLAT",
381
"metric_type": "L2",
382
"params": {"nlist": nlist}
383
}
384
385
# Create optimized index
386
params = optimize_ivf_parameters(dataset_size=2000000, memory_constraint=True)
387
client.create_index("vectors", "embedding", params)
388
```
389
390
### Search Parameter Optimization
391
392
```python { .api }
393
def find_optimal_search_params(collection_name: str, vector_field: str, test_vectors: List[List[float]], target_recall: float = 0.95):
394
"""Find optimal search parameters for target recall"""
395
396
# Get index information
397
index_info = client.describe_index(collection_name, vector_field)
398
index_type = index_info['index_type']
399
400
if index_type == "HNSW":
401
# Test different ef values
402
ef_values = [32, 64, 128, 256, 512]
403
404
for ef in ef_values:
405
start_time = time.time()
406
results = client.search(
407
collection_name,
408
data=test_vectors,
409
search_params={"ef": ef},
410
limit=10
411
)
412
search_time = (time.time() - start_time) / len(test_vectors)
413
414
# In practice, calculate recall against ground truth
415
# recall = calculate_recall(results, ground_truth)
416
417
print(f"ef={ef}: {search_time:.4f}s per query")
418
419
# Return first ef that meets target (you'd implement recall calculation)
420
if ef >= 64: # Placeholder for recall check
421
return {"ef": ef}
422
423
elif index_type in ["IVF_FLAT", "IVF_PQ"]:
424
# Test different nprobe values
425
nlist = index_info['params'].get('nlist', 1024)
426
nprobe_values = [max(1, nlist//64), max(1, nlist//32), max(1, nlist//16), max(1, nlist//8)]
427
428
for nprobe in nprobe_values:
429
start_time = time.time()
430
results = client.search(
431
collection_name,
432
data=test_vectors,
433
search_params={"nprobe": nprobe},
434
limit=10
435
)
436
search_time = (time.time() - start_time) / len(test_vectors)
437
438
print(f"nprobe={nprobe}: {search_time:.4f}s per query")
439
440
# Return reasonable nprobe value
441
if nprobe >= nlist // 32:
442
return {"nprobe": nprobe}
443
444
return {}
445
446
# Find optimal parameters
447
optimal_params = find_optimal_search_params("products", "embedding", test_query_vectors)
448
print(f"Optimal search params: {optimal_params}")
449
```
450
451
## Multi-Field Index Strategy
452
453
### Comprehensive Indexing Plan
454
455
```python { .api }
456
def create_comprehensive_indexes(collection_name: str):
457
"""Create optimized indexes for all searchable fields"""
458
459
# Get collection schema
460
schema_info = client.describe_collection(collection_name)
461
462
index_plan = []
463
464
for field in schema_info['schema']['fields']:
465
field_name = field['name']
466
field_type = field['type']
467
468
if field_type == 'FloatVector':
469
# Vector field - use HNSW for fast search
470
index_plan.append((field_name, {
471
"index_type": "HNSW",
472
"metric_type": "L2",
473
"params": {"M": 32, "efConstruction": 400}
474
}))
475
476
elif field_type == 'SparseFloatVector':
477
# Sparse vector - use sparse index
478
index_plan.append((field_name, {
479
"index_type": "SPARSE_INVERTED_INDEX",
480
"metric_type": "IP",
481
"params": {"drop_ratio_build": 0.2}
482
}))
483
484
elif field_type == 'VarChar':
485
# String field - use TRIE for prefix/equality searches
486
max_length = field.get('params', {}).get('max_length', 0)
487
if max_length > 0 and max_length <= 1000: # Don't index very long text
488
index_plan.append((field_name, {"index_type": "TRIE"}))
489
490
elif field_type in ['Int64', 'Int32', 'Float', 'Double']:
491
# Numeric fields - use STL_SORT for range queries
492
index_plan.append((field_name, {"index_type": "STL_SORT"}))
493
494
elif field_type in ['JSON', 'Array']:
495
# Complex fields - use INVERTED for key-value/array searches
496
index_plan.append((field_name, {"index_type": "INVERTED"}))
497
498
# Execute index creation plan
499
for field_name, index_params in index_plan:
500
try:
501
client.create_index(collection_name, field_name, index_params)
502
print(f"✓ Created {index_params['index_type']} index on {field_name}")
503
except Exception as e:
504
print(f"✗ Failed to create index on {field_name}: {e}")
505
506
return index_plan
507
508
# Create all recommended indexes
509
plan = create_comprehensive_indexes("comprehensive_collection")
510
```
511
512
### Index Maintenance
513
514
```python { .api }
515
def maintain_indexes(collection_name: str):
516
"""Perform index maintenance operations"""
517
518
# List current indexes
519
indexes = client.list_indexes(collection_name)
520
print(f"Current indexes: {indexes}")
521
522
# Check index status and rebuild if necessary
523
for field_name in indexes:
524
try:
525
# Get index information
526
index_info = client.describe_index(collection_name, field_name)
527
print(f"Index {field_name}: {index_info['index_type']}")
528
529
# Monitor index building progress
530
progress = utility.index_building_progress(collection_name, field_name)
531
532
if progress['state'] == 'Failed':
533
print(f"Index {field_name} build failed, rebuilding...")
534
535
# Drop and recreate failed index
536
client.drop_index(collection_name, field_name)
537
538
# Recreate with same parameters
539
client.create_index(collection_name, field_name, {
540
"index_type": index_info['index_type'],
541
"metric_type": index_info.get('metric_type', 'L2'),
542
"params": index_info['params']
543
})
544
545
except Exception as e:
546
print(f"Error checking index {field_name}: {e}")
547
548
# Perform maintenance
549
maintain_indexes("production_collection")
550
```
551
552
### Index Performance Monitoring
553
554
```python { .api }
555
def benchmark_index_performance(collection_name: str, test_queries: List[List[float]]):
556
"""Benchmark search performance with different index configurations"""
557
558
# Get current index info
559
index_info = client.describe_index(collection_name, "vector")
560
current_type = index_info['index_type']
561
562
print(f"Benchmarking {current_type} index...")
563
564
# Test different search parameter values
565
if current_type == "HNSW":
566
ef_values = [32, 64, 128, 256]
567
568
for ef in ef_values:
569
start_time = time.time()
570
571
# Run multiple test queries
572
total_results = 0
573
for query_vector in test_queries:
574
results = client.search(
575
collection_name,
576
data=[query_vector],
577
search_params={"ef": ef},
578
limit=10
579
)
580
total_results += len(results[0])
581
582
avg_time = (time.time() - start_time) / len(test_queries)
583
584
print(f"ef={ef}: {avg_time:.4f}s per query, {total_results} total results")
585
586
elif current_type in ["IVF_FLAT", "IVF_PQ"]:
587
nlist = index_info['params'].get('nlist', 1024)
588
nprobe_values = [max(1, nlist//64), max(1, nlist//32), max(1, nlist//16)]
589
590
for nprobe in nprobe_values:
591
start_time = time.time()
592
593
for query_vector in test_queries:
594
results = client.search(
595
collection_name,
596
data=[query_vector],
597
search_params={"nprobe": nprobe},
598
limit=10
599
)
600
601
avg_time = (time.time() - start_time) / len(test_queries)
602
print(f"nprobe={nprobe}: {avg_time:.4f}s per query")
603
604
# Generate test queries for benchmarking
605
import numpy as np
606
test_vectors = [np.random.rand(768).tolist() for _ in range(100)]
607
benchmark_index_performance("benchmark_collection", test_vectors)
608
```
609
610
## Advanced Index Patterns
611
612
### Hybrid Collection Indexing
613
614
```python { .api }
615
def setup_hybrid_collection_indexes(collection_name: str):
616
"""Setup optimized indexes for hybrid search collection"""
617
618
# Dense vector index for semantic similarity
619
client.create_index(
620
collection_name,
621
"dense_vector",
622
{
623
"index_type": "HNSW",
624
"metric_type": "COSINE", # Good for normalized embeddings
625
"params": {"M": 48, "efConstruction": 500}
626
}
627
)
628
629
# Sparse vector index for lexical matching
630
client.create_index(
631
collection_name,
632
"sparse_vector",
633
{
634
"index_type": "SPARSE_INVERTED_INDEX",
635
"metric_type": "IP",
636
"params": {"drop_ratio_build": 0.3}
637
}
638
)
639
640
# Scalar indexes for filtering
641
client.create_index(collection_name, "category", {"index_type": "TRIE"})
642
client.create_index(collection_name, "timestamp", {"index_type": "STL_SORT"})
643
client.create_index(collection_name, "metadata", {"index_type": "INVERTED"})
644
645
print("Hybrid collection indexes created successfully")
646
647
# Setup indexes for multi-vector search
648
setup_hybrid_collection_indexes("documents")
649
```
650
651
Index management in PyMilvus provides powerful capabilities for optimizing search performance across different data types and access patterns. Proper index selection and tuning are crucial for achieving optimal performance in production vector database applications.