Preprocessing API

The preprocessing API transforms tabular data before model selection. Most users configure preprocessing through mamut.wrapper.Mamut, which forwards preprocessing keyword arguments to mamut.preprocessing.preprocessing.Preprocessor.

Preprocessor

class mamut.preprocessing.preprocessing.Preprocessor(numeric_features=None, categorical_features=None, num_imputation='knn', cat_imputation='most_frequent', scaling='standard', feature_selection=False, pca=False, profile='generic_ohe', outlier_removal=False, imbalanced_resampling=True, resampling_strategy='SMOTE', skew_threshold=1, pca_threshold=0.95, selection_threshold=0.05, imbalance_threshold=0.1, random_state=42)[source]

Bases: object

A class used to preprocess data for machine learning models.

numeric_features

List of numeric feature names.

Type:

Optional[List[str]]

categorical_features

List of categorical feature names.

Type:

Optional[List[str]]

num_imputation

Method for numeric imputation.

Type:

Literal[“iterative”, “knn”, “mean”, “median”, “constant”]

cat_imputation

Method for categorical imputation.

Type:

Literal[“most_frequent”, “constant”]

scaling

Method for scaling numeric features.

Type:

Literal[“standard”, “robust”]

feature_selection

Whether to perform feature selection.

Type:

bool

pca

Whether to perform PCA for feature extraction.

Type:

bool

outlier_removal

Whether to remove rows flagged by IsolationForest during preprocessing.

Type:

bool

imbalanced_resampling

Whether to perform resampling to handle imbalanced data.

Type:

bool

resampling_strategy

Strategy for resampling imbalanced data.

Type:

Literal[“SMOTE”, “undersample”, “combine”]

skew_threshold

Threshold for skewness correction.

Type:

float

pca_threshold

Threshold for PCA feature extraction.

Type:

float

selection_threshold

Threshold for feature selection.

Type:

float

imbalance_threshold

Threshold for detecting imbalanced data.

Type:

float

random_state

Random state for reproducibility.

Type:

Optional[int]

fit_transform(X: pd.DataFrame, y: pd.Series)[source]

Fits the preprocessor and transforms the data.

transform(X: pd.DataFrame) → np.ndarray[source]

Transforms the data using the fitted preprocessor.

report() → dict[source]

Returns a report of the preprocessing steps.

_check_fitted()[source]

Checks if the preprocessor has been fitted.

Common Methods

Preprocessor.fit_transform(X, y)[source]

Fits the preprocessor and transforms the data.

Parameters:

  • X (DataFrame) – The input features.

  • y (Series) – The target variable.

Return type:

(ndarray, ndarray, Pipeline)

Returns:

  • np.ndarray – Transformed features.

  • np.ndarray – Transformed target variable.

  • Pipeline – The fitted pipeline.

Preprocessor.transform(X)[source]

Transforms the data using the fitted preprocessor.

Parameters:

X (DataFrame) – The input features.

Returns:

Transformed features.

Return type:

ndarray

Preprocessor.report()[source]

Returns a report of the preprocessing steps.

Returns:

A dictionary containing the report of the preprocessing steps.

Return type:

dict

Handlers

The handler functions below are lower-level building blocks used by Preprocessor.

mamut.preprocessing.handlers.handle_outliers(X, y, feature_names, contamination=0.01, random_state=42)[source]

Handles outliers in the dataset using IsolationForest.

Parameters:

  • X (DataFrame) – pd.DataFrame Feature matrix.

  • y (Series) – pd.Series Target array.

  • feature_names (List[str]) – List[str] Names of the features in the dataset.

  • contamination (float) – float The proportion of outliers in the data.

  • random_state (int) – int Seed for reproducibility.

Returns:

pd.DataFrame

Feature matrix with outliers removed.

y_filtered: pd.Series

Target array with outliers removed.

transformer: IsolationForest

Fitted IsolationForest model.

Return type:

(DataFrame, Series, IsolationForest)

mamut.preprocessing.handlers.handle_imbalanced(X, y, strategy, random_state=42)[source]

Balances an imbalanced dataset using techniques from imbalanced-learn.

Parameters:

  • X (DataFrame) – pd.DataFrame Feature matrix.

  • y (Series) – pd.Series Target array.

  • strategy (Literal['SMOTE', 'undersample', 'combine']) – Literal[“SMOTE”, “undersample”, “combine”] Resampling method to use. Options: - ‘SMOTE’: Synthetic Minority Oversampling Technique. - ‘undersample’: Random undersampling of majority class. - ‘combine’: SMOTE with Tomek links.

  • random_state – int Seed for reproducibility.

Returns:

pd.DataFrame

Feature matrix after resampling.

y_resampled: pd.Series

Target array after resampling.

transformer: object

Fitted resampling method instance.

Return type:

(DataFrame, Series, object)

mamut.preprocessing.handlers.handle_skewed(X, feature_names, threshold=1)[source]

Handles skewed features in the dataset using PowerTransformer.

Parameters:

  • X (DataFrame) – pd.DataFrame Feature matrix.

  • feature_names (List[str]) – List[str] Names of the features in the dataset.

  • threshold (float) – float Threshold for skewness.

Returns:

pd.DataFrame

Feature matrix with skewed features transformed.

transformer: PowerTransformer

Fitted PowerTransformer model.

skewed_feature_names: List[str]

Names of the skewed features that were transformed.

lambdas: List[float]

Lambda values for the transformed features.

Return type:

(DataFrame, PowerTransformer, List[str])

mamut.preprocessing.handlers.handle_missing_numeric(X, feature_names, strategy)[source]

Handles missing numeric values in the dataset using specified imputation strategy.

Parameters:

  • X (DataFrame) – pd.DataFrame Feature matrix.

  • feature_names (List[str]) – List[str] Names of the numeric features in the dataset.

  • strategy (Literal['iterative', 'knn', 'mean', 'median', 'constant']) – Literal[“iterative”, “knn”, “mean”, “median”, “constant”] Imputation strategy to use.

Returns:

pd.DataFrame

Feature matrix with missing numeric values imputed.

imputer: object

Fitted imputer model.

Return type:

(DataFrame, object)

mamut.preprocessing.handlers.handle_missing_categorical(X, feature_names, strategy)[source]

Handles missing categorical values in the dataset using specified imputation strategy.

Parameters:

  • X (DataFrame) – pd.DataFrame Feature matrix.

  • feature_names (List[str]) – List[str] Names of the categorical features in the dataset.

  • strategy (Literal['most_frequent', 'constant']) – Literal[“most_frequent”, “constant”] Imputation strategy to use.

Returns:

pd.DataFrame

Feature matrix with missing categorical values imputed.

imputer: SimpleImputer

Fitted SimpleImputer model.

Return type:

(DataFrame, SimpleImputer)

mamut.preprocessing.handlers.handle_categorical(X, feature_names)[source]

Handles categorical features in the dataset using OneHotEncoder.

Parameters:

  • X (DataFrame) – pd.DataFrame Feature matrix.

  • feature_names (List[str]) – List[str] Names of the categorical features in the dataset.

Returns:

pd.DataFrame

Feature matrix with categorical features encoded.

encoder: OneHotEncoder

Fitted OneHotEncoder model.

ohe_feature_names: List[str]

Names of the one-hot encoded features.

Return type:

(DataFrame, OneHotEncoder)

mamut.preprocessing.handlers.handle_scaling(X, feature_names, strategy)[source]

Handles scaling of features in the dataset using specified scaling strategy.

Parameters:

  • X (DataFrame) – pd.DataFrame Feature matrix.

  • feature_names (List[str]) – List[str] Names of the features to be scaled.

  • strategy (Literal['standard', 'robust']) – Literal[“standard”, “robust”] Scaling strategy to use.

Returns:

pd.DataFrame

Feature matrix with scaled features.

scaler: object

Fitted scaler instance.

Return type:

(DataFrame, object)

mamut.preprocessing.handlers.handle_selection(X, y, threshold=0.05, random_state=42)[source]

Handles feature selection using ExtraTreesClassifier.

Parameters:

  • X (DataFrame) – pd.DataFrame Feature matrix.

  • y (Series) – pd.Series Target array.

  • threshold (float) – float Threshold for feature selection.

  • random_state (int) – int Seed for reproducibility.

Returns:

pd.DataFrame

Feature matrix with selected features.

selector: SelectFromModel

Fitted SelectFromModel instance.

selected_features: List[str]

Names of the selected features.

feature_importances: np.ndarray

Feature importances from the ExtraTreesClassifier.

Return type:

(DataFrame, SelectFromModel, List[str])

mamut.preprocessing.handlers.handle_extraction(X, threshold=0.95, random_state=42)[source]

Handles feature extraction using PCA.

Parameters:

  • X (DataFrame) – pd.DataFrame Feature matrix.

  • threshold (float) – float Threshold for PCA.

  • random_state (int) – int Seed for reproducibility.

Returns:

np.ndarray

Feature matrix after PCA transformation.

extractor: PCA

Fitted PCA instance.

loadings: np.ndarray

Loadings of the PCA components.

Return type:

(ndarray, PCA)

Settings