Skip to content

Commit

Permalink
Rename 'true' to 'observed'
Browse files Browse the repository at this point in the history
  • Loading branch information
ogrisel committed Aug 27, 2024
1 parent e2608e2 commit 71dffb8
Showing 1 changed file with 40 additions and 38 deletions.
78 changes: 40 additions & 38 deletions content/python_files/build_calibration_curve.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,80 +11,82 @@
# discuss the implications of the data modeling process on the calibration curve.
#
# So let's first generate some predictions. The generative process is located in the
# file `_generate_predictions.py`. This process stores the true labels and the
# file `_generate_predictions.py`. This process stores the observed labels and the
# predicted probability estimates of several models into the `predictions` folder.

# %%
# Make sure to have scikit-learn >= 1.5
import sklearn

sklearn.__version__

# %%
# Equivalent to the magic command "%run _generate_predictions.py" but it allows this
# file to be executed as a Python script.
# %% Equivalent to the magic command "%run _generate_predictions.py" but it
# allows this file to be executed as a Python script.
from IPython import get_ipython

ipython = get_ipython()
ipython.run_line_magic("run", "../python_files/_generate_predictions.py")

# %% [markdown]
#
# Let's load a the true labels and the predicted probabilities of one of the models.
# Let's load a the observed labels and the predicted probabilities of one of
# the models.

# %%
import numpy as np

y_true = np.load("../predictions/y_true.npy")
y_observed = np.load("../predictions/y_true.npy")
y_prob = np.load("../predictions/y_prob_2.npy")

# %%
y_true
y_observed

# %% [markdown]
#
# `y_true` contains the true labels. In this case, we face a binary classification
# problem.
# `y_observed` contains the observed labels. In this case, we face a binary
# classification problem.

# %%
y_prob

# %% [markdown]
#
# `y_prob` contains the predicted probabilities of the positive class estimated by a
# given model.
# `y_prob` contains the predicted probabilities of the positive class estimated
# by a given model.
#
# As discussed earlier, we could evaluate the discriminative power of the model using
# a metric such as the ROC-AUC score.
# As discussed earlier, we could evaluate the discriminative power of the model
# using a metric such as the ROC-AUC score.

# %%
from sklearn.metrics import roc_auc_score

print(f"ROC AUC: {roc_auc_score(y_true, y_prob):.2f}")
print(f"ROC AUC: {roc_auc_score(y_observed, y_prob):.2f}")

# %% [markdown]
#
# The score is much above 0.5 and it means that our model is able to discriminate
# between the two classes. However, this score does not tell us if the predicted
# probabilities are well calibrated.
# The score is much above 0.5 and it means that our model is able to
# discriminate between the two classes. However, this score does not tell us if
# the predicted probabilities are well calibrated.
#
# We can provide an example of a well-calibrated probability: for a given sample, the
# predicted probabilities is considered well-calibrated if the proportion of samples
# where this probability is predicted is equal to the probability itself.
# We can provide an example of a well-calibrated probability: for a given
# sample, the predicted probabilities is considered well-calibrated if the
# proportion of samples where this probability is predicted is equal to the
# probability itself.
#
# It is therefore possible to group predictions into bins based on their predicted
# probabilities and to calculate the proportion of positive samples in each bin and
# compare it to the average predicted probability in the bin.
# It is therefore possible to group predictions into bins based on their
# predicted probabilities and to calculate the proportion of positive samples
# in each bin and compare it to the average predicted probability in the bin.
#
# Scikit-learn provide a utility to plot this graphical representation. It is available
# through the class `sklearn.calibration.CalibrationDisplay`.
# Scikit-learn provide a utility to plot this graphical representation. It is
# available through the class `sklearn.calibration.CalibrationDisplay`.

# %%
import matplotlib.pyplot as plt
from sklearn.calibration import CalibrationDisplay

n_bins = 10
disp = CalibrationDisplay.from_predictions(
y_true, y_prob, n_bins=n_bins, strategy="uniform"
y_observed, y_prob, n_bins=n_bins, strategy="uniform"
)
_ = disp.ax_.set(xlim=(0, 1), ylim=(0, 1), aspect="equal")

Expand All @@ -107,16 +109,16 @@

# %% [markdown]
#
# 2. Create a DataFrame with the true labels, predicted probabilities, and bin
# 2. Create a DataFrame with the observed labels, predicted probabilities, and bin
# identifier.

# %%
# TODO

# %% [markdown]
#
# 3. Group the DataFrame by the bin identifier and calculate the mean of the true
# labels and the predicted probabilities.
# 3. Group the DataFrame by the bin identifier and calculate the mean of the
# observed labels and the predicted probabilities.

# %%
# TODO
Expand Down Expand Up @@ -148,13 +150,13 @@

# %% [markdown]
#
# 2. Create a DataFrame with the true labels, predicted probabilities, and bin
# identifier.
# 2. Create a DataFrame with the observed labels, predicted probabilities, and
# bin identifier.

# %%
predictions = pd.DataFrame(
{
"y_true": y_true,
"y_observed": y_observed,
"y_prob": y_prob,
"bin_identifier": bin_identifier,
}
Expand All @@ -163,8 +165,8 @@

# %% [markdown]
#
# 3. Group the DataFrame by the bin identifier and calculate the mean of the true
# labels and the predicted probabilities.
# 3. Group the DataFrame by the bin identifier and calculate the mean of the
# observed labels and the predicted probabilities.

# %%
mean_per_bin = predictions.groupby("bin_identifier", observed=True).mean()
Expand All @@ -176,7 +178,7 @@

# %%
_, ax = plt.subplots()
mean_per_bin.plot("y_prob", "y_true", ax=ax)
mean_per_bin.plot("y_prob", "y_observed", ax=ax)
ax.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
ax.legend()
_ = ax.set(
Expand All @@ -200,7 +202,7 @@

# %%
disp = CalibrationDisplay.from_predictions(
y_true, y_prob, n_bins=n_bins, strategy="quantile"
y_observed, y_prob, n_bins=n_bins, strategy="quantile"
)
_ = disp.ax_.set(xlim=(0, 1), ylim=(0, 1), aspect="equal")

Expand All @@ -225,14 +227,14 @@
bin_identifier = pd.cut(y_prob, bins=bins)
predictions = pd.DataFrame(
{
"y_true": y_true,
"y_observed": y_observed,
"y_prob": y_prob,
"bin_identifier": bin_identifier,
}
)
mean_per_bin = predictions.groupby("bin_identifier", observed=True).mean()
_, ax = plt.subplots()
mean_per_bin.plot("y_prob", "y_true", ax=ax)
mean_per_bin.plot("y_prob", "y_observed", ax=ax)
ax.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
ax.legend()
_ = ax.set(
Expand Down

0 comments on commit 71dffb8

Please sign in to comment.