0
# Hyperparameter Tuning
1

2
Advanced hyperparameter optimization framework with multiple search algorithms, flexible search space definitions, and seamless integration with popular ML frameworks. FLAML's tuning module provides efficient optimization strategies that balance exploration and exploitation.
3

4
## Capabilities
5

6
### Core Tuning Function
7

8
Main function for running hyperparameter optimization experiments.
9

10
```python { .api }
11
def run(trainable, search_space=None, searcher=None, scheduler=None,
12
        time_budget_s=None, num_samples=None, config=None, **kwargs):
13
    """
14
    Run hyperparameter tuning experiment.
15
    
16
    Args:
17
        trainable (callable): Function to optimize that takes config dict and returns metrics
18
        search_space (dict): Search space specification
19
        searcher: Search algorithm instance (BlendSearch, CFO, FLOW2, etc.)
20
        scheduler: Scheduler for early stopping and resource allocation
21
        time_budget_s (float): Time budget in seconds
22
        num_samples (int): Maximum number of trials to run
23
        config (dict): Fixed configuration parameters
24
        **kwargs: Additional arguments
25
        
26
    Returns:
27
        Analysis object with optimization results
28
    """
29

30
def report(**metrics):
31
    """
32
    Report metrics from within trainable function.
33
    
34
    Args:
35
        **metrics: Metric name-value pairs to report
36
    """
37
```
38

39
### Search Algorithms
40

41
#### BlendSearch
42

43
Advanced search algorithm that blends local and global search strategies for efficient hyperparameter optimization.
44

45
```python { .api }
46
class BlendSearch:
47
    def __init__(self, metric, mode="min", space=None, low_cost_partial_config=None,
48
                 cat_hp_cost=None, points_to_evaluate=None, time_budget_s=None,
49
                 resource_attr=None, global_search_alg=None, config_constraints=None,
50
                 metric_constraints=None, seed=None):
51
        """
52
        Initialize BlendSearch algorithm.
53
        
54
        Args:
55
            metric (str): Metric name to optimize
56
            mode (str): Optimization mode - 'min' or 'max'
57
            space (dict): Search space configuration
58
            low_cost_partial_config (dict): Initial low-cost configuration
59
            cat_hp_cost (dict): Cost specification for categorical hyperparameters
60
            points_to_evaluate (list): Initial points to evaluate
61
            time_budget_s (float): Time budget in seconds
62
            resource_attr (str): Resource dimension name for multi-fidelity optimization
63
            global_search_alg: Global search algorithm class to use
64
            config_constraints (list): Configuration constraint functions
65
            metric_constraints (list): Metric constraint specifications
66
            seed (int): Random seed for reproducibility
67
        """
68
        
69
    def suggest(self, trial_id):
70
        """
71
        Suggest next configuration to evaluate.
72
        
73
        Args:
74
            trial_id (str): Unique trial identifier
75
            
76
        Returns:
77
            dict: Configuration to evaluate
78
        """
79
        
80
    def on_trial_result(self, trial_id, result):
81
        """
82
        Process trial result.
83
        
84
        Args:
85
            trial_id (str): Trial identifier
86
            result (dict): Trial results including metrics
87
        """
88
        
89
    def on_trial_complete(self, trial_id, result=None, error=False):
90
        """
91
        Handle trial completion.
92
        
93
        Args:
94
            trial_id (str): Trial identifier
95
            result (dict): Final trial results
96
            error (bool): Whether trial ended in error
97
        """
98
        
99
    def save(self, checkpoint_path):
100
        """Save searcher state to checkpoint."""
101
        
102
    def restore(self, checkpoint_path):
103
        """Restore searcher state from checkpoint."""
104
```
105

106
#### Other Search Algorithms
107

108
```python { .api }
109
class CFO:
110
    """Cost-Frugal Optimization searcher (alias for BlendSearch with specific defaults)."""
111

112
class FLOW2:
113
    """Fast Local Search algorithm with adaptive step sizes."""
114
    
115
    def __init__(self, metric, mode="min", **kwargs):
116
        """Initialize FLOW2 searcher."""
117

118
class RandomSearch:
119
    """Random sampling baseline for hyperparameter optimization."""
120
    
121
    def __init__(self, metric, mode="min", **kwargs):
122
        """Initialize random search."""
123

124
class ChampionFrontierSearcher:
125
    """Online searcher for champion frontier optimization."""
126
```
127

128
### Search Space Definition
129

130
Functions and classes for defining hyperparameter search spaces.
131

132
```python { .api }
133
def uniform(low, high):
134
    """
135
    Uniform distribution sampling.
136
    
137
    Args:
138
        low (float): Lower bound
139
        high (float): Upper bound
140
        
141
    Returns:
142
        dict: Uniform distribution specification
143
    """
144

145
def loguniform(low, high):
146
    """
147
    Log-uniform distribution sampling.
148
    
149
    Args:
150
        low (float): Lower bound (log scale)
151
        high (float): Upper bound (log scale)
152
        
153
    Returns:
154
        dict: Log-uniform distribution specification
155
    """
156

157
def randint(low, high):
158
    """
159
    Random integer sampling.
160
    
161
    Args:
162
        low (int): Lower bound (inclusive)
163
        high (int): Upper bound (exclusive)
164
        
165
    Returns:
166
        dict: Random integer specification
167
    """
168

169
def lograndint(low, high):
170
    """
171
    Log-scale random integer sampling.
172
    
173
    Args:
174
        low (int): Lower bound
175
        high (int): Upper bound
176
        
177
    Returns:
178
        dict: Log-scale integer specification
179
    """
180

181
def quniform(low, high, q):
182
    """
183
    Quantized uniform distribution.
184
    
185
    Args:
186
        low (float): Lower bound
187
        high (float): Upper bound
188
        q (float): Quantization step
189
        
190
    Returns:
191
        dict: Quantized uniform specification
192
    """
193

194
def qloguniform(low, high, q):
195
    """Quantized log-uniform distribution."""
196

197
def qrandint(low, high, q):
198
    """Quantized random integer."""
199

200
def qlograndint(low, high, q):
201
    """Quantized log-scale random integer."""
202

203
def randn(mean, sd):
204
    """
205
    Normal distribution sampling.
206
    
207
    Args:
208
        mean (float): Mean value
209
        sd (float): Standard deviation
210
        
211
    Returns:
212
        dict: Normal distribution specification
213
    """
214

215
def qrandn(mean, sd, q):
216
    """Quantized normal distribution."""
217

218
def choice(categories):
219
    """
220
    Categorical choice sampling.
221
    
222
    Args:
223
        categories (list): List of categorical values
224
        
225
    Returns:
226
        dict: Categorical choice specification
227
    """
228
```
229

230
### Search Space Classes
231

232
```python { .api }
233
class Categorical:
234
    """Categorical parameter space for discrete choices."""
235
    
236
    def __init__(self, categories, ordered=None):
237
        """
238
        Initialize categorical parameter.
239
        
240
        Args:
241
            categories (list): Available categorical values
242
            ordered (bool): Whether categories have natural ordering
243
        """
244

245
class Float:
246
    """Continuous float parameter space."""
247
    
248
    def __init__(self, lower, upper, log=False):
249
        """
250
        Initialize float parameter.
251
        
252
        Args:
253
            lower (float): Lower bound
254
            upper (float): Upper bound
255
            log (bool): Whether to use log scale
256
        """
257

258
class PolynomialExpansionSet:
259
    """Polynomial expansion parameter set for feature interactions."""
260

261
def polynomial_expansion_set(feature_names, interaction_terms=2):
262
    """
263
    Create polynomial expansion set.
264
    
265
    Args:
266
        feature_names (list): Base feature names
267
        interaction_terms (int): Maximum interaction order
268
        
269
    Returns:
270
        PolynomialExpansionSet: Polynomial expansion specification
271
    """
272
```
273

274
### Trial Management
275

276
```python { .api }
277
class Trial:
278
    """Individual trial/experiment representation."""
279
    
280
    @property
281
    def trial_id(self):
282
        """Unique trial identifier."""
283
        
284
    @property
285
    def config(self):
286
        """Trial configuration parameters."""
287
        
288
    @property
289
    def last_result(self):
290
        """Last reported result from trial."""
291
```
292

293
### Usage Examples
294

295
#### Basic Hyperparameter Tuning
296
```python
297
from flaml.tune import run
298
from flaml.tune.searcher import BlendSearch
299
import numpy as np
300
from sklearn.ensemble import RandomForestClassifier
301
from sklearn.model_selection import cross_val_score
302

303
def train_rf(config):
304
    """Training function for Random Forest."""
305
    model = RandomForestClassifier(
306
        n_estimators=config["n_estimators"],
307
        max_depth=config["max_depth"],
308
        min_samples_split=config["min_samples_split"],
309
        random_state=42
310
    )
311
    
312
    # Cross-validation score
313
    scores = cross_val_score(model, X_train, y_train, cv=3)
314
    return {"accuracy": scores.mean()}
315

316
# Define search space
317
search_space = {
318
    "n_estimators": {"_type": "randint", "_value": [10, 100]},
319
    "max_depth": {"_type": "randint", "_value": [3, 20]},
320
    "min_samples_split": {"_type": "randint", "_value": [2, 20]}
321
}
322

323
# Run optimization
324
analysis = run(
325
    train_rf,
326
    search_space,
327
    searcher=BlendSearch(metric="accuracy", mode="max"),
328
    time_budget_s=300,
329
    num_samples=50
330
)
331

332
print(f"Best config: {analysis.best_config}")
333
print(f"Best accuracy: {analysis.best_result['accuracy']}")
334
```
335

336
#### Advanced Search with Constraints
337
```python
338
from flaml.tune import run, uniform, loguniform, choice
339
from flaml.tune.searcher import BlendSearch
340

341
def objective(config):
342
    """Complex objective function with multiple metrics."""
343
    # Your model training code here
344
    accuracy = train_and_evaluate(config)
345
    training_time = get_training_time()
346
    model_size = get_model_size(config)
347
    
348
    return {
349
        "accuracy": accuracy,
350
        "training_time": training_time,
351
        "model_size": model_size
352
    }
353

354
# Search space using convenience functions
355
search_space = {
356
    "learning_rate": loguniform(0.001, 0.1),
357
    "batch_size": choice([16, 32, 64, 128]),
358
    "hidden_dim": uniform(64, 512),
359
    "dropout": uniform(0.0, 0.5)
360
}
361

362
# Constraints: accuracy > 0.8 and training_time < 300
363
def config_constraint(config):
364
    # Custom configuration validation
365
    return config["batch_size"] <= 64 if config["hidden_dim"] > 256 else True
366

367
metric_constraints = [
368
    ("accuracy", ">=", 0.8),
369
    ("training_time", "<=", 300)
370
]
371

372
# Run with constraints
373
analysis = run(
374
    objective,
375
    search_space,
376
    searcher=BlendSearch(
377
        metric="accuracy",
378
        mode="max",
379
        config_constraints=[config_constraint],
380
        metric_constraints=metric_constraints
381
    ),
382
    time_budget_s=1800
383
)
384
```
385

386
#### Multi-Fidelity Optimization
387
```python
388
from flaml.tune import run
389
from flaml.tune.searcher import BlendSearch
390

391
def train_with_budget(config):
392
    """Training function with resource budget."""
393
    epochs = config.get("epochs", 100)  # Resource dimension
394
    
395
    model = create_model(config)
396
    for epoch in range(epochs):
397
        model.train_one_epoch()
398
        
399
        # Report intermediate results
400
        if epoch % 10 == 0:
401
            val_loss = model.evaluate()
402
            report(loss=val_loss, epoch=epoch)
403
    
404
    final_loss = model.evaluate()
405
    return {"loss": final_loss}
406

407
# Multi-fidelity search space
408
search_space = {
409
    "lr": loguniform(1e-4, 1e-1),
410
    "batch_size": choice([16, 32, 64]),
411
    "epochs": randint(10, 200)  # Resource attribute
412
}
413

414
analysis = run(
415
    train_with_budget,
416
    search_space,
417
    searcher=BlendSearch(
418
        metric="loss", 
419
        mode="min",
420
        resource_attr="epochs",
421
        time_budget_s=3600
422
    )
423
)
424
```
425

426
#### Integration with Ray Tune
427
```python
428
from flaml.tune.searcher import BlendSearchTuner
429
import ray
430
from ray import tune
431

432
# Initialize Ray
433
ray.init()
434

435
# Use FLAML searcher in Ray Tune
436
tune.run(
437
    trainable_fn,
438
    config=search_space,
439
    search_alg=BlendSearchTuner(
440
        metric="accuracy",
441
        mode="max"
442
    ),
443
    num_samples=100,
444
    time_budget_s=1800
445
)
446
```
447

448
### Advanced Search Space Classes
449

450
Advanced search space components for complex parameter definitions.
451

452
```python { .api }
453
class PolynomialExpansionSet:
454
    """Polynomial expansion set for hierarchical search spaces."""
455
    def __init__(self, init_monomials=None, **kwargs): ...
456
    def add_monomial(self, monomial): ...
457
    def expand(self, degree): ...
458

459
def polynomial_expansion_set(init_monomials=None, **kwargs):
460
    """
461
    Create polynomial expansion set for search space.
462
    
463
    Args:
464
        init_monomials (list): Initial monomials for expansion
465
        **kwargs: Additional parameters
466
        
467
    Returns:
468
        PolynomialExpansionSet: Polynomial expansion set instance
469
    """
470
```
471

472
### Trial Management
473

474
Classes for managing individual trials in hyperparameter optimization.
475

476
```python { .api }
477
class Trial:
478
    """Individual trial in hyperparameter optimization experiment."""
479
    
480
    def __init__(self, config, trial_id=None):
481
        """
482
        Initialize trial.
483
        
484
        Args:
485
            config (dict): Trial configuration parameters
486
            trial_id (str): Unique trial identifier
487
        """
488
    
489
    @property
490
    def config(self):
491
        """dict: Trial configuration parameters"""
492
    
493
    @property
494
    def trial_id(self):
495
        """str: Unique trial identifier"""
496
    
497
    @property
498
    def status(self):
499
        """str: Current trial status"""
500
    
501
    def set_status(self, status):
502
        """Set trial status."""
503
```
504

505
## Constants and Utilities
506

507
```python { .api }
508
INCUMBENT_RESULT = "INCUMBENT_RESULT"  # Special result indicator for best configuration
509

510
def choice(*categories):
511
    """
512
    Create categorical choice parameter.
513
    
514
    Args:
515
        *categories: Available categories to choose from
516
        
517
    Returns:
518
        Choice parameter specification
519
    """
520
```
521

522
## Integration Features
523

524
- **Ray Tune Compatibility**: Use FLAML searchers within Ray Tune framework
525
- **Early Stopping**: Integration with schedulers for efficient resource allocation  
526
- **Checkpointing**: Save and restore optimization state
527
- **Multi-Metric Optimization**: Support for multiple objectives and constraints
528
- **Distributed Execution**: Scale optimization across multiple workers
529
- **Custom Search Spaces**: Flexible parameter space definitions