4.7.1 Model Evaluation Metrics

What does a metric do?

A model can be evaluated in several reasonable ways. A classifier may have high accuracy but miss most examples of a rare class. A regressor may have a small average error but still make a few very large mistakes. The metrics disagree because they care about different behaviour.

A metric turns a set of predictions and true answers into a number. In an IOAI task, that number determines the score, so the task's metric is part of the problem—not an afterthought.

Approaching a Typical Problem explains where evaluation fits into the workflow. This module focuses on how common metrics behave and how to compute them. Clustering metrics are introduced separately in K-Means Clustering.

Start with the model output

Different metrics expect different forms of output.

The same classifier may provide all three classification outputs. Logistic regression, for example, produces probabilities; those probabilities can rank samples or be converted to labels with a threshold.

Four shared questions distinguish the metrics:

Every metric card below uses the same fields: input, behaviour, direction, suitable uses, risks and scikit-learn name.

Metrics in IOAI tasks

IOAI tasks show why the exact definition matters.

Some tasks use one sklearn.metrics function. Others add weights, average groups or define a new formula. Save the Factory even passed labels—not probabilities—to roc_auc_score. Small details can change the meaning of the score.

The confusion matrix

Most label-based classification metrics come from one shared table. In binary classification, one class is called positive, usually the class we want to detect, and the other is negative.

The first word says whether the prediction was correct; the second says which class was predicted. For $K$ classes, entry $C_{jk}$ of the $K\times K$ confusion matrix counts samples of true class $j$ predicted as class $k$. Correct predictions lie on the diagonal.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

cancer = load_breast_cancer(as_frame=True)
X = cancer.data[["mean radius", "mean texture"]]
y = 1 - cancer.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(confusion_matrix(y_test, y_pred))
# [[87  3]
#  [ 8 45]]
tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()

y3_true = [0, 0, 0, 0, 0, 1, 1, 1, 2, 2]
y3_pred = [0, 0, 0, 0, 0, 1, 1, 0, 2, 0]
print(confusion_matrix(y3_true, y3_pred))
# [[5 0 0]
#  [1 2 0]
#  [1 0 1]]

Accuracy and balanced accuracy

Accuracy

In the three-class example, accuracy is $(5+2+1)/10=0.8$. An always-largest-class classifier can therefore look strong without learning the other classes.

Balanced accuracy

The class recalls in the example are $1$, $2/3$ and $1/2$, so balanced accuracy is $0.722$. An always-one-class classifier has balanced accuracy $1/K$.

from sklearn.metrics import accuracy_score, balanced_accuracy_score

print(round(accuracy_score(y_test, y_pred), 3))            # 0.923
print(round(balanced_accuracy_score(y_test, y_pred), 3))   # 0.908
print(accuracy_score(y3_true, y3_pred))                    # 0.800
print(round(balanced_accuracy_score(y3_true, y3_pred), 3)) # 0.722

Averaging accuracy over other groups

Balanced accuracy averages group accuracy over true classes. The same idea can use other groups.

• Robot Chasing averages accuracy over six robots. With 600 samples per robot, this equals ordinary accuracy.
• Operation Night Watch gives equal weight to the old-class and new-class groups, even though they contain different numbers of clips and classes.

import numpy as np

def mean_group_accuracy(y_true, y_pred, groups):
    """Accuracy inside each group, averaged with equal weights."""
    y_true, y_pred, groups = map(np.asarray, (y_true, y_pred, groups))
    scores = [np.mean(y_pred[groups == g] == y_true[groups == g])
              for g in np.unique(groups)]
    return float(np.mean(scores))

print(round(mean_group_accuracy(y3_true, y3_pred, y3_true), 3))  # 0.722
is_old = np.asarray(y3_true) < 2
print(mean_group_accuracy(y3_true, y3_pred, is_old))             # 0.6875

The old group has $7/8$ correct and the new group $1/2$, giving $(0.875+0.5)/2=0.6875$.

Precision, recall and F1

These metrics focus on the positive class and ignore TN.

Precision

Recall

The negative class's recall, $\mathrm{TN}/(\mathrm{TN}+\mathrm{FP})$, is specificity or the true negative rate.

F1

For the tumour model, precision is $45/48=0.938$, recall is $45/53=0.849$, and F1 is $90/101=0.891$.

from sklearn.metrics import f1_score, precision_score, recall_score

print(round(precision_score(y_test, y_pred), 3))  # 0.938
print(round(recall_score(y_test, y_pred), 3))     # 0.849
print(round(f1_score(y_test, y_pred), 3))         # 0.891

proba = model.predict_proba(X_test)[:, 1]
top1 = (proba == proba.max()).astype(int)
print(precision_score(y_test, top1))         # 1.000
print(round(recall_score(y_test, top1), 3))  # 0.019
print(round(f1_score(y_test, top1), 3))      # 0.037

If no sample is predicted positive, precision becomes $0/0$. Scikit-learn returns 0 with an UndefinedMetricWarning; zero_division can set 0, 1 or np.nan. Recall is similarly undefined when the true positive class is absent. pos_label chooses the positive class; its default is 1.

Averaging over classes

For multiclass data, each class can be treated as positive in turn before the results are combined.

For this example, macro F1 is 0.767, weighted F1 is 0.790 and micro F1 is 0.800. In single-label classification, micro F1 equals accuracy. Macro recall equals balanced accuracy.

from sklearn.metrics import classification_report

print(classification_report(y3_true, y3_pred, digits=3))
for average in ["macro", "weighted", "micro"]:
    print(average, round(f1_score(y3_true, y3_pred,
                                  average=average), 3))
# macro 0.767
# weighted 0.790
# micro 0.800

Help BOBAI and Synthetic Speech Detector use macro F1, so small classes count as much as large ones. Synthetic Speech Detector averages the F1 of bonafide and spoof samples; this is not default binary F1. For the tumour example, class F1 values are [0.941, 0.891], giving macro F1 0.916 instead of positive-class F1 0.891.

Thresholds, ROC curves and AUC

Most classifiers first produce a score or probability. A decision threshold turns it into a label: predict positive when the score is at least $t$.

Lower thresholds usually find more positives but also create more false alarms.

proba = model.predict_proba(X_test)[:, 1]
for t in [0.1, 0.3, 0.5, 0.7, 0.9]:
    pred_t = (proba >= t).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_test, pred_t).ravel()
    print(f"t={t:.1f} TP={tp:2} FP={fp:2} "
          f"TPR={tp/(tp+fn):.3f} FPR={fp/(fp+tn):.3f}")
# t=0.1 TP=52 FP=34 TPR=0.981 FPR=0.378
# t=0.3 TP=48 FP= 7 TPR=0.906 FPR=0.078
# t=0.5 TP=45 FP= 3 TPR=0.849 FPR=0.033
# t=0.7 TP=39 FP= 1 TPR=0.736 FPR=0.011
# t=0.9 TP=32 FP= 0 TPR=0.604 FPR=0.000

ROC AUC

ROC AUC depends only on score order, so decision_function and probabilities give the same value when they preserve that order.

from sklearn.metrics import roc_auc_score

print(round(roc_auc_score(y_test, proba), 3))  # 0.970

pos = proba[y_test.to_numpy() == 1]
neg = proba[y_test.to_numpy() == 0]
diff = pos[:, None] - neg[None, :]
print(round(np.mean(diff > 0) + 0.5 * np.mean(diff == 0), 3))  # 0.970

ROC AUC of hard labels

If roc_auc_score receives 0/1 labels instead of scores, its area is

$$\frac{\mathrm{TPR}+\mathrm{TNR}}{2},$$

which is exactly balanced accuracy.

print(round(roc_auc_score(y_test, y_pred), 4))            # 0.9079
print(round(balanced_accuracy_score(y_test, y_pred), 4))  # 0.9079

Save the Factory evaluated model.predict(...) with roc_auc_score, so its reported ROC AUC was balanced accuracy, as noted in Decision Trees.

Precision–recall curves and average precision

ROC divides false positives by all negatives, so many false positives can look small when negatives are extremely common. The precision–recall curve makes that failure more visible.

OptionalWorked example: ROC AUC and AP with rare positives2 min readShowHide

from sklearn.datasets import make_classification
from sklearn.metrics import average_precision_score

X_r, y_r = make_classification(
    n_samples=6000, weights=[0.95], random_state=0
)
X_tr, X_rest, y_tr, y_rest = train_test_split(
    X_r, y_r, test_size=0.5, stratify=y_r, random_state=0
)
X_val, X_te, y_val, y_te = train_test_split(
    X_rest, y_rest, test_size=0.5, stratify=y_rest, random_state=0
)
clf = LogisticRegression().fit(X_tr, y_tr)
s_val = clf.predict_proba(X_val)[:, 1]
s_te = clf.predict_proba(X_te)[:, 1]

print(round(y_te.mean(), 3))                          # 0.054
print(round(roc_auc_score(y_te, s_te), 3))            # 0.914
print(round(average_precision_score(y_te, s_te), 3))  # 0.476

Average precision can be written as

$$\mathrm{AP}=\sum_m(R_m-R_{m-1})P_m,$$

where thresholds run from high to low. Here AUC is 0.914 but AP is 0.476; at threshold 0.5, precision is 0.583 and recall only 0.259.

Hide this optional section

Choosing a threshold

A label metric such as F1 depends on the threshold. Choose it using validation data or cross-validation, never the test set.

from sklearn.metrics import precision_recall_curve

prec, rec, thr = precision_recall_curve(y_val, s_val)
f1 = 2 * prec * rec / np.maximum(prec + rec, 1e-12)
best = thr[np.argmax(f1[:-1])]
print(round(best, 3))  # 0.173
for t in [0.5, best]:
    print(round(f1_score(y_te, (s_te >= t).astype(int)), 3))
# 0.359
# 0.484

Threshold 0.173 raises test F1 from 0.359 to 0.484: recall rises from 0.259 to 0.667 while precision falls from 0.583 to 0.380. TunedThresholdClassifierCV performs this search with cross-validation.

Log loss

Log loss works with any number of classes. It is also the training loss of logistic regression.

from sklearn.metrics import log_loss

y_ll = np.array([1] * 50 + [0] * 50)
p_ll = np.where(y_ll == 1, 0.9, 0.1)
print(round(log_loss(y_ll, p_ll), 3))  # 0.105
p_ll[0] = 0.001
print(round(log_loss(y_ll, p_ll), 3))  # 0.173
p_ll[0] = 0.0
print(round(log_loss(y_ll, p_ll), 3))  # 0.465

A true-class probability of exactly 0 would have infinite loss. Scikit-learn clips it near machine precision, about $2.2\times10^{-16}$, producing a sample loss near 36. Never submit exact 0 or 1 probabilities for a log-loss task. Loss Functions compares log loss with other training losses.

Regression metrics

For residual $r_i=y_i-\hat y_i$, regression metrics differ mainly in how quickly the penalty grows and whether the scale of the label matters.

MAE, MSE and RMSE

All are lower-is-better. Use MAE when a mistake twice as large should cost about twice as much. Use RMSE when occasional large mistakes deserve extra emphasis.

from sklearn.metrics import (mean_absolute_error, mean_squared_error,
                             root_mean_squared_error)

y_reg = np.array([50, 60, 70, 80, 90])
pred_a = np.array([52, 58, 72, 78, 92])
pred_b = np.array([52, 58, 72, 78, 110])
for pred in [pred_a, pred_b]:
    print(mean_absolute_error(y_reg, pred),
          mean_squared_error(y_reg, pred),
          round(root_mean_squared_error(y_reg, pred), 2))
# 2.0 4.0 2.0
# 5.6 83.2 9.12

Changing one error from 2 to 20 multiplies MAE by 2.8 but RMSE by 4.6.

The coefficient of determination

On one fixed set, $R^2$ and MSE rank models identically. $R^2$ can be negative on new data because the model does not know the test labels' mean.

from sklearn.datasets import load_diabetes
from sklearn.dummy import DummyRegressor

Xd, yd = load_diabetes(return_X_y=True)
Xd_tr, Xd_te, yd_tr, yd_te = train_test_split(
    Xd, yd, test_size=0.2, random_state=0
)
mean_model = DummyRegressor(strategy="mean").fit(Xd_tr, yd_tr)
print(round(mean_model.score(Xd_tr, yd_tr), 3))  # 0.0000
print(round(mean_model.score(Xd_te, yd_te), 4))  # -0.0013

from sklearn.datasets import load_iris
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

iris = load_iris(as_frame=True).frame
X_i = iris[["petal length (cm)"]]
y_i = iris["petal width (cm)"]
X_itr, X_ite, y_itr, y_ite = train_test_split(
    X_i, y_i, test_size=0.3, random_state=0
)
reg = LinearRegression().fit(X_itr, y_itr)
w_pred = reg.predict(X_ite)
print(round(r2_score(y_ite, w_pred), 3),
      round(root_mean_squared_error(y_ite, w_pred), 3))  # 0.896 0.231

setosa = (iris.loc[X_ite.index, "target"] == 0).to_numpy()
print(round(r2_score(y_ite[setosa], w_pred[setosa]), 3),
      round(root_mean_squared_error(y_ite[setosa],
                                    w_pred[setosa]), 3))  # -0.215 0.132

Relative errors

Chicken Counting uses $\exp(-\mathrm{MAPE})$, so its score is higher when relative error is lower.

from sklearn.metrics import mean_absolute_percentage_error

def chicken_score(y_true, y_pred):
    y_true, y_pred = np.asarray(y_true), np.asarray(y_pred)
    relative_error = np.abs(y_true - y_pred) / y_true
    return float(np.exp(-np.mean(relative_error)))

counts = np.array([40, 25, 10])
pred_counts = np.array([36, 27, 13])
mape = mean_absolute_percentage_error(counts, pred_counts)
print(round(mape, 3), round(chicken_score(counts, pred_counts), 3))
# 0.160 0.852

The error of 3 on 10 counts three times as much as the error of 4 on 40. Predicting 0 can undercount by at most 100%, but overcounting is unbounded: predicting 30 instead of 10 gives relative error 2.

Choosing a metric

Use the task statement first. When designing an evaluation, this flow gives a starting point.

Classification
├─ Final labels matter
│  ├─ Classes have similar size and importance → accuracy
│  ├─ Every class should count equally → balanced accuracy or macro F1
│  ├─ False positives matter most → precision
│  ├─ False negatives matter most → recall
│  └─ Need one balance of precision and recall → F1 or F-beta
├─ Ranking across thresholds matters → ROC AUC
├─ Positives are rare and their quality matters → average precision
└─ Probability quality matters → log loss

Regression
├─ Typical absolute error matters → MAE
├─ Large errors deserve extra punishment → RMSE
├─ Need performance relative to label variation → R²
└─ Error relative to target size matters → MAPE or a task formula

No metric is universally best. The right one is the metric whose rewards and penalties match the task.

Metrics in scikit-learn

Metric functions take true values first: f1_score(y_true, y_pred). Swapping the arguments of precision_score, for example, computes recall instead.

Scoring strings

Cross-validation and tuning tools accept metric names through scoring. A scorer such as "roc_auc" calls predict_proba or decision_function itself.

from sklearn.model_selection import cross_val_score

for scoring in ["accuracy", "f1_macro", "roc_auc", "neg_log_loss"]:
    scores = cross_val_score(model, X_train, y_train, cv=5,
                             scoring=scoring)
    print(f"{scoring:13}{scores.mean():.3f}")
# accuracy      0.878
# f1_macro      0.866
# roc_auc       0.947
# neg_log_loss -0.279

Scorers always treat larger values as better. Scikit-learn therefore negates errors: "neg_log_loss" is minus log loss and "neg_root_mean_squared_error" is minus RMSE. The best value is closest to zero. sklearn.metrics.get_scorer_names() lists every string.

Custom metrics

make_scorer adapts a function of (y_true, y_pred) for cross-validation. Set greater_is_better=False for an error, or response_method="predict_proba" when the function needs probabilities.

from sklearn.metrics import make_scorer

chicken_scorer = make_scorer(chicken_score)
scores = cross_val_score(LinearRegression(), X_itr, y_itr, cv=5,
                         scoring=chicken_scorer)
print(scores.round(3))  # [0.890 0.809 0.841 0.782 0.785]

The scorer can be passed to Hyperparameter Tuning and Cross-Validation. When a metric needs extra information such as robot IDs, it can be clearer to predict on validation data and call mean_group_accuracy directly.

The whole idea in one view

task goal
   ↓
choose what the metric should reward
   ↓
match the metric to the model output
labels → accuracy, balanced accuracy, precision, recall, F1
scores → ROC AUC, average precision
probabilities → log loss
numbers → MAE, RMSE, R², relative error
   ↓
reproduce the exact metric on validation data
   ↓
compare models and tune thresholds without touching the test set

The formula matters, but its meaning matters more: a metric decides which mistakes count, how strongly they count and which model looks best.