FEA allow any resampler in the BalancedBaggingClassifier (#808)

doc/bibtex/refs.bib

@@ -244,3 +244,32 @@ @article{wilson1997improved
   pages={1--34},
   year={1997}
 }
+
+@inproceedings{wang2009diversity,
+  title={Diversity analysis on imbalanced data sets by using ensemble models},
+  author={Wang, Shuo and Yao, Xin},
+  booktitle={2009 IEEE symposium on computational intelligence and data mining},
+  pages={324--331},
+  year={2009},
+  organization={IEEE}
+}
+
+@article{hido2009roughly,
+  title={Roughly balanced bagging for imbalanced data},
+  author={Hido, Shohei and Kashima, Hisashi and Takahashi, Yutaka},
+  journal={Statistical Analysis and Data Mining: The ASA Data Science Journal},
+  volume={2},
+  number={5-6},
+  pages={412--426},
+  year={2009},
+  publisher={Wiley Online Library}
+}
+
+@article{maclin1997empirical,
+  title={An empirical evaluation of bagging and boosting},
+  author={Maclin, Richard and Opitz, David},
+  journal={AAAI/IAAI},
+  volume={1997},
+  pages={546--551},
+  year={1997}
+}

doc/ensemble.rst

@@ -18,9 +18,9 @@ Bagging classifier
 
 In ensemble classifiers, bagging methods build several estimators on different
 randomly selected subset of data. In scikit-learn, this classifier is named
-``BaggingClassifier``. However, this classifier does not allow to balance each
-subset of data. Therefore, when training on imbalanced data set, this
-classifier will favor the majority classes::
+:class:`~sklearn.ensemble.BaggingClassifier`. However, this classifier does not
+allow to balance each subset of data. Therefore, when training on imbalanced
+data set, this classifier will favor the majority classes::
 
   >>> from sklearn.datasets import make_classification
   >>> X, y = make_classification(n_samples=10000, n_features=2, n_informative=2,
@@ -41,14 +41,13 @@ classifier will favor the majority classes::
   >>> balanced_accuracy_score(y_test, y_pred)  # doctest: +ELLIPSIS
   0.77...
 
-:class:`BalancedBaggingClassifier` allows to resample each subset of data
-before to train each estimator of the ensemble. In short, it combines the
-output of an :class:`EasyEnsemble` sampler with an ensemble of classifiers
-(i.e. ``BaggingClassifier``). Therefore, :class:`BalancedBaggingClassifier`
-takes the same parameters than the scikit-learn
-``BaggingClassifier``. Additionally, there is two additional parameters,
-``sampling_strategy`` and ``replacement`` to control the behaviour of the
-random under-sampler::
+In :class:`BalancedBaggingClassifier`, each bootstrap sample will be further
+resampled to achieve the `sampling_strategy` desired. Therefore,
+:class:`BalancedBaggingClassifier` takes the same parameters than the
+scikit-learn :class:`~sklearn.ensemble.BaggingClassifier`. In addition, the
+sampling is controlled by the parameter `sampler` or the two parameters
+`sampling_strategy` and `replacement`, if one wants to use the
+:class:`~imblearn.under_sampling.RandomUnderSampler`::
 
   >>> from imblearn.ensemble import BalancedBaggingClassifier
   >>> bbc = BalancedBaggingClassifier(base_estimator=DecisionTreeClassifier(),
@@ -61,6 +60,12 @@ random under-sampler::
   >>> balanced_accuracy_score(y_test, y_pred)  # doctest: +ELLIPSIS
   0.8...
 
+Changing the `sampler` will give rise to different known implementation
+:cite:`maclin1997empirical`, :cite:`hido2009roughly`,
+:cite:`wang2009diversity`. You can refer to the following example shows in
+practice these different methods:
+:ref:`sphx_glr_auto_examples_ensemble_plot_bagging_classifier.py`
+
 .. _forest:
 
 Forest of randomized trees
@@ -69,8 +74,7 @@ Forest of randomized trees
 :class:`BalancedRandomForestClassifier` is another ensemble method in which
 each tree of the forest will be provided a balanced bootstrap sample
 :cite:`chen2004using`. This class provides all functionality of the
-:class:`~sklearn.ensemble.RandomForestClassifier` and notably the
-`feature_importances_` attributes::
+:class:`~sklearn.ensemble.RandomForestClassifier`::
 
   >>> from imblearn.ensemble import BalancedRandomForestClassifier
   >>> brf = BalancedRandomForestClassifier(n_estimators=100, random_state=0)
@@ -99,11 +103,11 @@ a boosting iteration :cite:`seiffert2009rusboost`::
   >>> balanced_accuracy_score(y_test, y_pred)  # doctest: +ELLIPSIS
   0...
 
-A specific method which uses ``AdaBoost`` as learners in the bagging classifier
-is called EasyEnsemble. The :class:`EasyEnsembleClassifier` allows to bag
-AdaBoost learners which are trained on balanced bootstrap samples
-:cite:`liu2008exploratory`. Similarly to the :class:`BalancedBaggingClassifier`
-API, one can construct the ensemble as::
+A specific method which uses :class:`~sklearn.ensemble.AdaBoostClassifier` as
+learners in the bagging classifier is called "EasyEnsemble". The
+:class:`EasyEnsembleClassifier` allows to bag AdaBoost learners which are
+trained on balanced bootstrap samples :cite:`liu2008exploratory`. Similarly to
+the :class:`BalancedBaggingClassifier` API, one can construct the ensemble as::
 
   >>> from imblearn.ensemble import EasyEnsembleClassifier
   >>> eec = EasyEnsembleClassifier(random_state=0)

doc/whats_new/v0.8.rst

@@ -24,6 +24,11 @@ New features
   only containing categorical features.
   :pr:`802` by :user:`Guillaume Lemaitre <glemaitre>`.
 
+- Add the possibility to pass any type of samplers in
+  :class:`imblearn.ensemble.BalancedBaggingClassifier` unlocking the
+  implementation of methods based on resampled bagging.
+  :pr:`808` by :user:`Guillaume Lemaitre <glemaitre>`.
+
 Enhancements
 ............
 

examples/ensemble/plot_bagging_classifier.py

@@ -0,0 +1,175 @@
+"""
+=================================
+Bagging classifiers using sampler
+=================================
+
+In this example, we show how
+:class:`~imblearn.ensemble.BalancedBaggingClassifier` can be used to create a
+large variety of classifiers by giving different samplers.
+
+We will give several examples that have been published in the passed year.
+"""
+
+# Authors: Guillaume Lemaitre <[email protected]>
+# License: MIT
+
+# %%
+print(__doc__)
+
+# %% [markdown]
+# Generate an imbalanced dataset
+# ------------------------------
+#
+# For this example, we will create a synthetic dataset using the function
+# :func:`~sklearn.datasets.make_classification`. The problem will be a toy
+# classification problem with a ratio of 1:9 between the two classes.
+
+# %%
+from sklearn.datasets import make_classification
+
+X, y = make_classification(
+    n_samples=10_000,
+    n_features=10,
+    weights=[0.1, 0.9],
+    class_sep=0.5,
+    random_state=0,
+)
+
+# %%
+import pandas as pd
+
+pd.Series(y).value_counts(normalize=True)
+
+# %% [markdown]
+# In the following sections, we will show a couple of algorithms that have
+# been proposed over the years. We intend to illustrate how one can reuse the
+# :class:`~imblearn.ensemble.BalancedBaggingClassifier` by passing different
+# sampler.
+
+# %%
+from sklearn.model_selection import cross_validate
+from sklearn.ensemble import BaggingClassifier
+
+ebb = BaggingClassifier()
+cv_results = cross_validate(ebb, X, y, scoring="balanced_accuracy")
+
+print(f"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}")
+
+# %% [markdown]
+# Exactly Balanced Bagging and Over-Bagging
+# -----------------------------------------
+#
+# The :class:`~imblearn.ensemble.BalancedBaggingClassifier` can use in
+# conjunction with a :class:`~imblearn.under_sampling.RandomUnderSampler` or
+# :class:`~imblearn.over_sampling.RandomOverSampler`. These methods are
+# referred as Exactly Balanced Bagging and Over-Bagging, respectively and have
+# been proposed first in [1]_.
+
+# %%
+from imblearn.ensemble import BalancedBaggingClassifier
+from imblearn.under_sampling import RandomUnderSampler
+
+# Exactly Balanced Bagging
+ebb = BalancedBaggingClassifier(sampler=RandomUnderSampler())
+cv_results = cross_validate(ebb, X, y, scoring="balanced_accuracy")
+
+print(f"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}")
+
+# %%
+from imblearn.over_sampling import RandomOverSampler
+
+# Over-bagging
+over_bagging = BalancedBaggingClassifier(sampler=RandomOverSampler())
+cv_results = cross_validate(over_bagging, X, y, scoring="balanced_accuracy")
+
+print(f"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}")
+
+# %% [markdown]
+# SMOTE-Bagging
+# -------------
+#
+# Instead of using a :class:`~imblearn.over_sampling.RandomOverSampler` that
+# make a bootstrap, an alternative is to use
+# :class:`~imblearn.over_sampling.SMOTE` as an over-sampler. This is known as
+# SMOTE-Bagging [2]_.
+
+# %%
+from imblearn.over_sampling import SMOTE
+
+# SMOTE-Bagging
+smote_bagging = BalancedBaggingClassifier(sampler=SMOTE())
+cv_results = cross_validate(smote_bagging, X, y, scoring="balanced_accuracy")
+
+print(f"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}")
+
+# %% [markdown]
+# Roughly Balanced Bagging
+# ------------------------
+# While using a :class:`~imblearn.under_sampling.RandomUnderSampler` or
+# :class:`~imblearn.over_sampling.RandomOverSampler` will create exactly the
+# desired number of samples, it does not follow the statistical spirit wanted
+# in the bagging framework. The authors in [3]_ proposes to use a negative
+# binomial distribution to compute the number of samples of the majority
+# class to be selected and then perform a random under-sampling.
+#
+# Here, we illustrate this method by implementing a function in charge of
+# resampling and use the :class:`~imblearn.FunctionSampler` to integrate it
+# within a :class:`~imblearn.pipeline.Pipeline` and
+# :class:`~sklearn.model_selection.cross_validate`.
+
+# %%
+from collections import Counter
+import numpy as np
+from imblearn import FunctionSampler
+
+
+def roughly_balanced_bagging(X, y, replace=False):
+    """Implementation of Roughly Balanced Bagging for binary problem."""
+    # find the minority and majority classes
+    class_counts = Counter(y)
+    majority_class = max(class_counts, key=class_counts.get)
+    minority_class = min(class_counts, key=class_counts.get)
+
+    # compute the number of sample to draw from the majority class using
+    # a negative binomial distribution
+    n_minority_class = class_counts[minority_class]
+    n_majority_resampled = np.random.negative_binomial(n=n_minority_class, p=0.5)
+
+    # draw randomly with or without replacement
+    majority_indices = np.random.choice(
+        np.flatnonzero(y == majority_class),
+        size=n_majority_resampled,
+        replace=replace,
+    )
+    minority_indices = np.random.choice(
+        np.flatnonzero(y == minority_class),
+        size=n_minority_class,
+        replace=replace,
+    )
+    indices = np.hstack([majority_indices, minority_indices])
+
+    return X[indices], y[indices]
+
+
+# Roughly Balanced Bagging
+rbb = BalancedBaggingClassifier(
+    sampler=FunctionSampler(func=roughly_balanced_bagging, kw_args={"replace": True})
+)
+cv_results = cross_validate(rbb, X, y, scoring="balanced_accuracy")
+
+print(f"{cv_results['test_score'].mean():.3f} +/- {cv_results['test_score'].std():.3f}")
+
+
+# %% [markdown]
+# .. topic:: References:
+#
+#    .. [1] R. Maclin, and D. Opitz. "An empirical evaluation of bagging and
+#           boosting." AAAI/IAAI 1997 (1997): 546-551.
+#
+#    .. [2] S. Wang, and X. Yao. "Diversity analysis on imbalanced data sets by
+#           using ensemble models." 2009 IEEE symposium on computational
+#           intelligence and data mining. IEEE, 2009.
+#
+#    .. [3] S. Hido, H. Kashima, and Y. Takahashi. "Roughly balanced bagging
+#          for imbalanced data." Statistical Analysis and Data Mining: The ASA
+#          Data Science Journal 2.5‐6 (2009): 412-426.

examples/ensemble/plot_comparison_ensemble_classifier.py

@@ -3,15 +3,14 @@
 Compare ensemble classifiers using resampling
 =============================================
 
-Ensembling classifiers have shown to improve classification performance compare
+Ensemble classifiers have shown to improve classification performance compare
 to single learner. However, they will be affected by class imbalance. This
 example shows the benefit of balancing the training set before to learn
 learners. We are making the comparison with non-balanced ensemble methods.
 
 We make a comparison using the balanced accuracy and geometric mean which are
 metrics widely used in the literature to evaluate models learned on imbalanced
 set.
-
 """
 
 # Authors: Guillaume Lemaitre <[email protected]>