0
# Index Optimization
1
2
The Optimizer class provides tools for improving index performance through graph structure optimization and search parameter tuning. It reconstructs indexes with optimized graph connectivity and automatically adjusts search coefficients for better accuracy-speed trade-offs.
3
4
## Capabilities
5
6
### Optimizer Configuration
7
8
Create and configure optimization parameters for index reconstruction and performance tuning.
9
10
```python { .api }
11
class Optimizer:
12
def __init__(self, num_of_outgoings=-1, num_of_incomings=-1, num_of_queries=-1,
13
num_of_objects=-1, low_accuracy_from=-1.0, low_accuracy_to=-1.0,
14
high_accuracy_from=-1.0, high_accuracy_to=-1.0, gt_epsilon=-1.0, margin=-1.0, log_disabled=False):
15
"""
16
Create optimizer with specified parameters.
17
18
Args:
19
num_of_outgoings (int): Number of outgoing edges per node (-1 for auto)
20
num_of_incomings (int): Number of incoming edges per node (-1 for auto)
21
num_of_queries (int): Number of queries for optimization (-1 for auto)
22
num_of_objects (int): Number of objects for optimization (-1 for auto)
23
low_accuracy_from (float): Starting low accuracy range (-1.0 for auto)
24
low_accuracy_to (float): Ending low accuracy range (-1.0 for auto)
25
high_accuracy_from (float): Starting high accuracy range (-1.0 for auto)
26
high_accuracy_to (float): Ending high accuracy range (-1.0 for auto)
27
gt_epsilon (float): Ground truth epsilon parameter (-1.0 for auto)
28
margin (float): Optimization margin parameter (-1.0 for auto)
29
log_disabled (bool): Disable progress logging (default: False)
30
"""
31
```
32
33
### Index Reconstruction
34
35
Reconstruct indexes with optimized graph structure for improved search performance.
36
37
```python { .api }
38
class Optimizer:
39
def execute(self, in_path, out_path):
40
"""
41
Reconstruct index with optimization and adjust search coefficients.
42
43
Args:
44
in_path (str): Input index path
45
out_path (str): Output optimized index path
46
47
Returns:
48
None
49
"""
50
```
51
52
### Search Parameter Optimization
53
54
Optimize search coefficients and parameters for existing indexes.
55
56
```python { .api }
57
class Optimizer:
58
def adjust_search_coefficients(self, path):
59
"""
60
Optimize search coefficients for existing index.
61
62
Args:
63
path (str): Index path to optimize
64
65
Returns:
66
None
67
"""
68
69
def optimize_search_parameters(self, path):
70
"""
71
Optimize search parameters for better performance.
72
73
Args:
74
path (str): Index path to optimize
75
76
Returns:
77
None
78
"""
79
80
def optimize_number_of_edges_for_anng(self, path, num_of_queries=-1, num_of_results=-1, num_of_threads=-1, target_accuracy=-1, target_num_of_objects=-1, num_of_sample_objects=-1, max_num_of_edges=-1):
81
"""
82
Optimize edge numbers for ANNG (Approximate Nearest Neighbor Graph).
83
84
Args:
85
path (str): Index path to optimize
86
num_of_queries (int): Number of queries for optimization (-1 for auto)
87
num_of_results (int): Number of results per query (-1 for auto)
88
num_of_threads (int): Number of threads for optimization (-1 for auto)
89
target_accuracy (float): Target accuracy level (-1 for auto)
90
target_num_of_objects (int): Target number of objects (-1 for auto)
91
num_of_sample_objects (int): Number of sample objects (-1 for auto)
92
max_num_of_edges (int): Maximum number of edges (-1 for auto)
93
94
Returns:
95
int: Optimized number of edges
96
"""
97
```
98
99
### Parameter Configuration
100
101
Configure optimization parameters and processing modes for fine-tuned control.
102
103
```python { .api }
104
class Optimizer:
105
def set(self, num_of_outgoings=-1, num_of_incomings=-1, num_of_queries=-1,
106
num_of_objects=-1, low_accuracy_from=-1.0, low_accuracy_to=-1.0,
107
high_accuracy_from=-1.0, high_accuracy_to=-1.0, gt_epsilon=-1.0, margin=-1.0):
108
"""
109
Set optimization parameters.
110
111
Args:
112
num_of_outgoings (int): Number of outgoing edges per node (-1 for auto)
113
num_of_incomings (int): Number of incoming edges per node (-1 for auto)
114
num_of_queries (int): Number of queries for optimization (-1 for auto)
115
num_of_objects (int): Number of objects for optimization (-1 for auto)
116
low_accuracy_from (float): Starting low accuracy range (-1.0 for auto)
117
low_accuracy_to (float): Ending low accuracy range (-1.0 for auto)
118
high_accuracy_from (float): Starting high accuracy range (-1.0 for auto)
119
high_accuracy_to (float): Ending high accuracy range (-1.0 for auto)
120
gt_epsilon (float): Ground truth epsilon parameter (-1.0 for auto)
121
margin (float): Optimization margin parameter (-1.0 for auto)
122
123
Returns:
124
None
125
"""
126
127
def set_processing_modes(self, shortcut_reduction=True, search_parameter_optimization=True,
128
prefetch_parameter_optimization=True, accuracy_table_generation=True):
129
"""
130
Configure optimization processing modes.
131
132
Args:
133
shortcut_reduction (bool): Enable shortcut reduction (default: True)
134
search_parameter_optimization (bool): Enable search parameter optimization (default: True)
135
prefetch_parameter_optimization (bool): Enable prefetch optimization (default: True)
136
accuracy_table_generation (bool): Enable accuracy table generation (default: True)
137
138
Returns:
139
None
140
"""
141
```
142
143
## Usage Examples
144
145
### Basic Index Optimization
146
147
```python
148
import ngtpy
149
150
# Create optimizer with default parameters
151
optimizer = ngtpy.Optimizer(log_disabled=False)
152
153
# Reconstruct index with optimization
154
optimizer.execute("original_index", "optimized_index")
155
156
print("Index optimization completed")
157
```
158
159
### Custom Optimization Parameters
160
161
```python
162
import ngtpy
163
164
# Create optimizer with custom parameters for high accuracy
165
optimizer = ngtpy.Optimizer(
166
num_of_outgoings=20, # More outgoing edges
167
num_of_incomings=150, # More incoming edges
168
num_of_queries=1000, # More queries for optimization
169
base_accuracy_from=0.8, # Start from 80% accuracy
170
base_accuracy_to=0.95, # Target 95% accuracy
171
log_disabled=False
172
)
173
174
# Set processing modes
175
optimizer.set_processing_modes(
176
shortcut_reduction=True,
177
search_parameter_optimization=True,
178
prefetch_parameter_optimization=True,
179
accuracy_table_generation=True
180
)
181
182
# Execute optimization
183
optimizer.execute("large_index", "high_accuracy_index")
184
185
print("High-accuracy optimization completed")
186
```
187
188
### Search Parameter Tuning
189
190
```python
191
import ngtpy
192
193
# Create optimizer for search parameter tuning only
194
optimizer = ngtpy.Optimizer(
195
num_of_queries=500,
196
num_of_results=10,
197
log_disabled=False
198
)
199
200
# Optimize search coefficients for existing index
201
optimizer.adjust_search_coefficients("existing_index")
202
203
# Optimize search parameters
204
optimizer.optimize_search_parameters("existing_index")
205
206
print("Search parameters optimized")
207
```
208
209
### ANNG Edge Optimization
210
211
```python
212
import ngtpy
213
214
# Create optimizer for ANNG edge optimization
215
optimizer = ngtpy.Optimizer(log_disabled=False)
216
217
# Optimize number of edges with accuracy table
218
optimizer.optimize_number_of_edges_for_anng(
219
"anng_index",
220
num_of_queries=2000,
221
accuracy_table_path="accuracy_results.tsv"
222
)
223
224
print("ANNG edge optimization completed")
225
```
226
227
### Progressive Optimization
228
229
```python
230
import ngtpy
231
232
# Stage 1: Basic optimization
233
basic_optimizer = ngtpy.Optimizer(
234
num_of_outgoings=15,
235
num_of_incomings=100,
236
log_disabled=False
237
)
238
basic_optimizer.execute("raw_index", "basic_optimized")
239
240
# Stage 2: Search parameter optimization
241
search_optimizer = ngtpy.Optimizer(
242
num_of_queries=1500,
243
num_of_results=20,
244
base_accuracy_from=0.85,
245
base_accuracy_to=0.95
246
)
247
search_optimizer.adjust_search_coefficients("basic_optimized")
248
249
# Stage 3: ANNG fine-tuning
250
search_optimizer.optimize_number_of_edges_for_anng(
251
"basic_optimized",
252
num_of_queries=1000
253
)
254
255
print("Progressive optimization completed")
256
```
257
258
### Memory-Optimized Configuration
259
260
```python
261
import ngtpy
262
263
# Configure optimizer for memory efficiency
264
memory_optimizer = ngtpy.Optimizer(
265
num_of_outgoings=8, # Fewer outgoing edges
266
num_of_incomings=60, # Fewer incoming edges
267
ratio=0.02, # Lower ratio for compression
268
margin=0.2, # Tighter margin
269
log_disabled=False
270
)
271
272
# Disable some processing modes to save memory
273
memory_optimizer.set_processing_modes(
274
shortcut_reduction=True,
275
search_parameter_optimization=False, # Skip to save memory
276
prefetch_parameter_optimization=False, # Skip to save memory
277
accuracy_table_generation=True
278
)
279
280
memory_optimizer.execute("large_index", "memory_optimized_index")
281
282
print("Memory-optimized index created")
283
```