Cross-Validated Model Selection Workflow

Build a utility that tunes a classification estimator using the package's model selection tools and reports evaluation diagnostics.

Capabilities

Stratified evaluation

• Splits input features and labels into user-specified folds using a label-aware splitter from the package; supports optional group labels to keep grouped samples in the same fold. @test
• Computes the chosen scoring metric on each fold and reports both the list of scores and their mean. @test

Hyperparameter search

• Runs a cross-validated search over the provided hyperparameter grid for the estimator, honoring the same fold configuration and random seed. @test
• Refits the estimator on the full dataset with the best parameters and exposes the corresponding validation score. @test

Learning curve diagnostics

• Computes training sizes plus paired training and validation scores via the package's learning-curve helper, reusing the evaluation splitter. @test
• When no training sizes are provided, uses a small-to-full range of at least five evenly spaced fractions of the dataset. @test

Result bundle

• Returns a serializable summary containing best_params, best_score, fold_scores (list of per-fold metrics), cv_mean_score, learning_curve (train_sizes, train_scores, val_scores), and the refit estimator. @test

Implementation

@generates

API

from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple

def run_model_selection(
    X: Sequence[Sequence[float]],
    y: Sequence[Any],
    *,
    estimator: Any,
    param_grid: Dict[str, Iterable[Any]],
    scoring: str,
    cv_folds: int = 5,
    group_labels: Optional[Sequence[Any]] = None,
    learning_curve_sizes: Optional[Sequence[float]] = None,
    random_state: Optional[int] = None,
) -> Dict[str, Any]:
    """Fits and evaluates the estimator using package-native model selection utilities.
    Returns keys: best_params, best_score, fold_scores, cv_mean_score, learning_curve, estimator."""

Dependencies { .dependencies }

scikit-learn { .dependency }

Machine learning algorithms, model selection tools, and metrics.