tessl/pypi-feature-engine
Python library with 44+ transformers for feature engineering and selection following scikit-learn API
—
Pending
Quality
Pending
Does it follow best practices?
Impact
Pending
No eval scenarios have been run
imputation.mddocs/
Missing Data Imputation
Transformers for handling missing values in numerical and categorical variables using statistical methods, arbitrary values, random sampling, and missing data indicators.
Capabilities
Mean and Median Imputation
Replaces missing data by the mean or median value of numerical variables.
class MeanMedianImputer:
def __init__(self, imputation_method='median', variables=None):
"""
Initialize MeanMedianImputer.
Parameters:
- imputation_method (str): 'mean' or 'median'
- variables (list): List of numerical variables to impute. If None, selects all numerical variables
"""
def fit(self, X, y=None):
"""
Learn mean/median values for each variable.
Parameters:
- X (pandas.DataFrame): Training dataset
- y (pandas.Series, optional): Target variable (not used)
Returns:
- self
"""
def transform(self, X):
"""
Impute missing data using learned parameters.
Parameters:
- X (pandas.DataFrame): Dataset to transform
Returns:
- pandas.DataFrame: Transformed dataset with imputed values
"""
def fit_transform(self, X, y=None):
"""Fit to data, then transform it."""Usage Example:
from feature_engine.imputation import MeanMedianImputer
import pandas as pd
# Sample data with missing values
data = {'var1': [1.0, 2.0, None, 4.0], 'var2': [10, None, 30, 40]}
df = pd.DataFrame(data)
# Mean imputation
imputer = MeanMedianImputer(imputation_method='mean')
df_imputed = imputer.fit_transform(df)
# Median imputation (default)
imputer = MeanMedianImputer()
df_imputed = imputer.fit_transform(df)
# Access learned parameters
print(imputer.imputer_dict_) # {'var1': 2.33, 'var2': 26.67}Arbitrary Number Imputation
Replaces missing data by an arbitrary value determined by the user for numerical variables.
class ArbitraryNumberImputer:
def __init__(self, arbitrary_number=999, variables=None, imputer_dict=None):
"""
Initialize ArbitraryNumberImputer.
Parameters:
- arbitrary_number (int/float): Number to replace missing data (ignored if imputer_dict provided)
- variables (list): List of variables to impute. If None, selects all numerical variables
- imputer_dict (dict): Dictionary mapping variables to imputation values
"""
def fit(self, X, y=None):
"""
Validate input data (no parameters learned).
Parameters:
- X (pandas.DataFrame): Training dataset
- y (pandas.Series, optional): Target variable (not used)
Returns:
- self
"""
def transform(self, X):
"""
Impute missing data with arbitrary values.
Parameters:
- X (pandas.DataFrame): Dataset to transform
Returns:
- pandas.DataFrame: Transformed dataset with imputed values
"""
def fit_transform(self, X, y=None):
"""Fit to data, then transform it."""Usage Example:
from feature_engine.imputation import ArbitraryNumberImputer
# Single value for all variables
imputer = ArbitraryNumberImputer(arbitrary_number=-999)
df_imputed = imputer.fit_transform(df)
# Different values per variable
imputer = ArbitraryNumberImputer(
imputer_dict={'var1': 0, 'var2': -1, 'var3': 99}
)
df_imputed = imputer.fit_transform(df)Categorical Variable Imputation
Replaces missing data in categorical variables by an arbitrary value or the most frequent category.
class CategoricalImputer:
def __init__(self, imputation_method='missing', fill_value='Missing',
variables=None, return_object=False, ignore_format=False):
"""
Initialize CategoricalImputer.
Parameters:
- imputation_method (str): 'missing' (use fill_value) or 'frequent' (use mode)
- fill_value (str/int/float): Value to replace missing data when method='missing'
- variables (list): List of categorical variables to impute. If None, selects all object variables
- return_object (bool): Whether to return variables as object dtype
- ignore_format (bool): Whether to ignore variable format and accept numerical variables
"""
def fit(self, X, y=None):
"""
Learn most frequent category or assign arbitrary value per variable.
Parameters:
- X (pandas.DataFrame): Training dataset
- y (pandas.Series, optional): Target variable (not used)
Returns:
- self
"""
def transform(self, X):
"""
Impute missing data in categorical variables.
Parameters:
- X (pandas.DataFrame): Dataset to transform
Returns:
- pandas.DataFrame: Transformed dataset with imputed values
"""
def fit_transform(self, X, y=None):
"""Fit to data, then transform it."""Usage Example:
from feature_engine.imputation import CategoricalImputer
# Impute with most frequent category
imputer = CategoricalImputer(imputation_method='frequent')
df_imputed = imputer.fit_transform(df)
# Impute with custom value
imputer = CategoricalImputer(
imputation_method='missing',
fill_value='Unknown'
)
df_imputed = imputer.fit_transform(df)End Tail Imputation
Replaces missing data by values at either tail of the distribution for numerical variables.
class EndTailImputer:
def __init__(self, imputation_method='gaussian', tail='right', fold=3, variables=None):
"""
Initialize EndTailImputer.
Parameters:
- imputation_method (str): 'gaussian' (mean ± fold*std), 'iqr' (Q1/Q3 ± fold*IQR), 'max' (fold*max/min)
- tail (str): 'right' (upper tail) or 'left' (lower tail)
- fold (int/float): Factor to multiply std, IQR or max values
- variables (list): List of numerical variables to impute
"""
def fit(self, X, y=None):
"""
Learn values at end of distribution for each variable.
Parameters:
- X (pandas.DataFrame): Training dataset
- y (pandas.Series, optional): Target variable (not used)
Returns:
- self
"""
def transform(self, X):
"""
Impute missing data with end tail values.
Parameters:
- X (pandas.DataFrame): Dataset to transform
Returns:
- pandas.DataFrame: Transformed dataset with imputed values
"""
def fit_transform(self, X, y=None):
"""Fit to data, then transform it."""Usage Example:
from feature_engine.imputation import EndTailImputer
# Right tail using IQR method
imputer = EndTailImputer(
imputation_method='iqr',
tail='right',
fold=3
)
df_imputed = imputer.fit_transform(df)
# Left tail using gaussian method
imputer = EndTailImputer(
imputation_method='gaussian',
tail='left',
fold=2
)
df_imputed = imputer.fit_transform(df)Missing Data Indicators
Adds binary variables that indicate if data was missing for each variable.
class AddMissingIndicator:
def __init__(self, missing_only=True, variables=None):
"""
Initialize AddMissingIndicator.
Parameters:
- missing_only (bool): Whether to add indicators only for variables with missing data in train set
- variables (list): List of variables to create indicators for. If None, evaluates all variables
"""
def fit(self, X, y=None):
"""
Find variables for which missing indicators will be created.
Parameters:
- X (pandas.DataFrame): Training dataset
- y (pandas.Series, optional): Target variable (not used)
Returns:
- self
"""
def transform(self, X):
"""
Add binary missing indicators to dataset.
Parameters:
- X (pandas.DataFrame): Dataset to transform
Returns:
- pandas.DataFrame: Dataset with additional binary indicator columns
"""
def fit_transform(self, X, y=None):
"""Fit to data, then transform it."""Usage Example:
from feature_engine.imputation import AddMissingIndicator
# Add indicators only for variables with missing data
indicator = AddMissingIndicator(missing_only=True)
df_with_indicators = indicator.fit_transform(df)
# Creates new columns like 'var1_na', 'var2_na' where missing data existed
print(df_with_indicators.columns) # Original columns + indicator columnsRandom Sample Imputation
Replaces missing data with random sample extracted from the variables in the training set.
class RandomSampleImputer:
def __init__(self, variables=None, random_state=None, seed='general', seeding_method='add'):
"""
Initialize RandomSampleImputer.
Parameters:
- variables (list): List of variables to be imputed. If None, selects all variables with missing data
- random_state (int/str/list): Random state for sampling reproducibility
- seed (str): 'general' (single seed) or 'observation' (seed per observation)
- seeding_method (str): 'add' or 'multiply' when combining seeds
"""
def fit(self, X, y=None):
"""
Store copy of training dataset for sampling.
Parameters:
- X (pandas.DataFrame): Training dataset
- y (pandas.Series, optional): Target variable (not used)
Returns:
- self
"""
def transform(self, X):
"""
Impute missing data with random samples from training set.
Parameters:
- X (pandas.DataFrame): Dataset to transform
Returns:
- pandas.DataFrame: Transformed dataset with imputed values
"""
def fit_transform(self, X, y=None):
"""Fit to data, then transform it."""Usage Example:
from feature_engine.imputation import RandomSampleImputer
# Random sampling with fixed seed
imputer = RandomSampleImputer(random_state=42, seed='general')
df_imputed = imputer.fit_transform(df)
# Different seed per observation
imputer = RandomSampleImputer(
random_state=42,
seed='observation',
seeding_method='add'
)
df_imputed = imputer.fit_transform(df)Drop Missing Data
Deletes rows containing missing values, similar to pandas.dropna().
class DropMissingData:
def __init__(self, missing_only=True, threshold=None, variables=None):
"""
Initialize DropMissingData.
Parameters:
- missing_only (bool): If True, consider only variables with missing data in train set
- threshold (int/float): Percentage (0-1) or count of non-NA values required to keep row
- variables (list): List of variables to evaluate for missing data. If None, uses all variables
"""
def fit(self, X, y=None):
"""
Find variables for missing data evaluation.
Parameters:
- X (pandas.DataFrame): Training dataset
- y (pandas.Series, optional): Target variable (not used)
Returns:
- self
"""
def transform(self, X):
"""
Remove rows with missing data based on specified criteria.
Parameters:
- X (pandas.DataFrame): Dataset to transform
Returns:
- pandas.DataFrame: Dataset with rows containing missing data removed
"""
def fit_transform(self, X, y=None):
"""Fit to data, then transform it."""
def return_na_data(self, X):
"""
Return subset of dataframe with rows that would be removed.
Parameters:
- X (pandas.DataFrame): Dataset to evaluate
Returns:
- pandas.DataFrame: Rows that contain missing data
"""Usage Example:
from feature_engine.imputation import DropMissingData
# Drop rows with any missing data
dropper = DropMissingData()
df_clean = dropper.fit_transform(df)
# Keep rows with at least 80% non-missing data
dropper = DropMissingData(threshold=0.8)
df_clean = dropper.fit_transform(df)
# See which rows would be dropped
rows_to_drop = dropper.return_na_data(df)Common Attributes
All imputation transformers share these fitted attributes:
variables_(list): Variables that will be transformedn_features_in_(int): Number of features in training setimputer_dict_(dict): Dictionary with imputation values per variable (where applicable)
Install with Tessl CLI
npx tessl i tessl/pypi-feature-engine