4.3.1 Loss Functions

What is a loss function?

A model turns features $x$ into a prediction $\hat{y}$. During training, the true answer $y$ is known, so the model needs a way to answer one question:

How bad was this prediction?

A loss function gives that answer:

$$\ell(y,\hat{y}).$$

A smaller loss means a better prediction according to the rule we chose. Training repeatedly makes predictions, calculates their losses and changes the model parameters to reduce them.

features x → prediction ŷ
                    ↓
true answer y → loss ℓ(y, ŷ)
                    ↓
          adjust the model

The central idea of this chapter is:

Every model encountered so far has used a loss, even when it was not emphasised. Linear regression minimises squared error, while logistic regression minimises log loss. Losses guide the fitting of model parameters; they are not hyperparameters themselves.

Loss, cost, objective and metric

These four terms describe different stages of training and evaluation.

Here, $R(\theta)$ is a penalty on the model parameters and $\lambda$ controls its strength. The $\alpha$ used by Ridge and Lasso and the C setting of logistic regression affect this part of the objective. In informal discussion, people often use “loss” for the pointwise loss, dataset cost or full objective when the intended meaning is clear.

The metric belongs to a separate step:

prediction errors
      ↓
pointwise losses
      ↓ average over training data
cost
      ↓ add any parameter penalty
training objective

fitted model → untouched data → evaluation metric

Loss and metric sometimes match directly, while others are closely aligned—for example, minimising MSE also minimises RMSE on the same data. They do not have to align so closely. A classifier may train with log loss and be evaluated with accuracy, F1 or ROC AUC.

Choosing a loss changes what the model learns

A loss does more than measure a finished prediction. It decides which prediction training prefers.

Consider a model that must use one constant $c$ for the labels

$$2,\ 3,\ 4,\ 5,\ 36.$$

The last value is an outlier. Different losses choose different best constants:

Replacing 36 with 6 moves the mean from 10 to 4 but leaves the median at 4. Squared error allows the extreme label to pull the prediction strongly; absolute error does not.

import numpy as np

y = np.array([2.0, 3.0, 4.0, 5.0, 36.0])
c = np.linspace(0, 40, 40001)
R = y - c[:, None]
A = np.abs(R)

losses = {
    "squared": R ** 2,
    "absolute": A,
    "Huber, delta=3": np.where(A <= 3, 0.5 * R ** 2,
                                3 * (A - 1.5)),
    "pinball, tau=0.75": np.maximum(0.75 * R, -0.25 * R),
}

for name, values in losses.items():
    best = c[np.argmin(values.mean(axis=1))]
    print(f"{name}: {best:.3f}")

A common framework for understanding losses

The guide can grow to include many more losses, but each one can be understood through the same questions.

Every loss card below follows this schema. New losses can be added by filling in the same fields rather than creating a new explanation structure.

How losses differ

Before looking at individual losses, four shared ideas explain most of their behaviour.

Error shape

For regression, define the residual

$$r=y-\hat{y}.$$

Squared error grows as $r^2$, while absolute error grows as $|r|$. An error of 1 therefore contributes 1 to either loss, but an error of 10 contributes 100 to squared error and only 10 to absolute error.

Three samples make the effect concrete:

The squared-error cost is $10/3\approx3.33$, while the absolute-error cost is $4/3\approx1.33$. The third sample accounts for 90% of the first cost and 75% of the second. The shape of the loss decides which mistakes dominate training.

Robustness to outliers

An outlier often creates an unusually large residual. A rapidly growing loss lets that sample dominate the objective. A linearly growing loss still counts the error, but limits how quickly its influence grows.

squared error → large residuals grow quadratically → sensitive
absolute error → large residuals grow linearly    → more robust
Huber loss    → quadratic nearby, linear far away → compromise

Robustness is not automatically better. If a large error is genuinely much more costly than a small one, squared error may express the task correctly.

Symmetric and asymmetric costs

A symmetric loss charges the same amount for overpredicting by 5 and underpredicting by 5. Squared and absolute error are symmetric.

An asymmetric loss charges the two directions differently. Pinball loss can make underprediction more expensive than overprediction, which is useful for demand forecasts, safety margins and prediction intervals.

Smoothness and surrogate losses

Gradient-based training needs a loss that changes as the model parameters change. Accuracy and 0–1 error are flat for most small parameter changes, then jump when a sample crosses the decision boundary. Their gradients therefore provide almost no guidance.

A surrogate loss is an easier-to-optimise replacement that encourages the behaviour the metric needs:

classification error or accuracy → log loss or hinge loss
F1                               → probability loss, then tune a threshold
ROC AUC                          → a score-producing loss, then evaluate ranking

The training loss should guide optimisation; the metric should express the final goal.

Regression losses

The regression catalogue uses the shared residual $r=y-\hat{y}$. The cards make their differences visible without repeating the general theory.

Squared error

The dataset average of squared error is MSE; RMSE is its square root and returns to the units of the label.

Absolute error

With $\tau=0.5$, pinball loss is one half of absolute error, so both target the median.

Huber loss

Huber loss joins a quadratic centre to linear tails:

$$\ell_\delta(r)= \begin{cases} \frac{1}{2}r^2, & |r|\le\delta,\\ \delta\left(|r|-\frac{1}{2}\delta\right), & |r|>\delta. \end{cases}$$

HuberRegressor estimates a scale and applies its epsilon threshold, 1.35 by default, to scaled residuals. The threshold therefore does not directly depend on the units of $y$.

Pinball or quantile loss

For a quantile level $0<\tau<1$,

$$\ell_\tau(r)=\max\bigl(\tau r,(\tau-1)r\bigr).$$

Models at $\tau=0.05$ and $\tau=0.95$ estimate the endpoints of a 90% conditional prediction interval.

import numpy as np
from sklearn.linear_model import (
    HuberRegressor, LinearRegression, QuantileRegressor,
)

rng = np.random.default_rng(0)
x = rng.uniform(0, 10, size=100)
y = 2 * x + 1 + rng.normal(0, 1, size=100)
bad = rng.choice(100, size=10, replace=False)
y[bad] -= 30
X = x.reshape(-1, 1)

models = [
    LinearRegression(),
    HuberRegressor(),
    QuantileRegressor(quantile=0.5, alpha=0),
]
for model in models:
    model.fit(X, y)
    print(type(model).__name__, round(model.coef_[0], 3),
          round(model.intercept_, 3))
# LinearRegression 2.241 -3.4
# HuberRegressor 1.99 0.856
# QuantileRegressor 1.973 1.118

Classification losses

Classification losses can consume different kinds of model output:

• a class, such as 0 or 1;
• a probability $\hat{p}$ between 0 and 1; or
• a real-valued score $f(x)$ whose sign selects the class.

For binary score-based classification, code labels as $y\in\{-1,+1\}$ and define the margin

$$m=yf(x).$$

A positive margin is correct, a negative margin is wrong, and a large positive margin is confidently or safely correct. In scikit-learn, decision_function returns such scores for compatible estimators.

Zero–one loss

Zero–one loss explains what accuracy counts, but its poor optimisation behaviour motivates the surrogate losses below.

Binary cross-entropy or log loss

For a probability $\hat{p}=P(y=1\mid x)$ and label $y\in\{0,1\}$,

$$\ell(y,\hat{p})=-\left[y\log\hat{p}+(1-y)\log(1-\hat{p})\right].$$

With signed labels and margin $m$, the same loss is $\log(1+e^{-m})$.

Multiclass cross-entropy

If the model predicts probabilities $\hat{p}_1,\ldots,\hat{p}_K$ for $K$ classes and the true class is $y$,

$$\ell(y,\hat{p})=-\log\hat{p}_y.$$

This is the loss used by multiclass logistic regression.

Hinge loss

A correctly classified sample with $0<m<1$ still has positive hinge loss because it lies inside the desired safety margin.

Exponential loss

For each extra unit of negative margin, exponential loss is multiplied by $e\approx2.718$.

Comparing the margin losses

The table and figure show why “classification loss” is not one behaviour. Hinge loss stops caring after margin 1, log loss continues to reward greater confidence gently, and exponential loss lets badly misclassified samples dominate.

f = np.linspace(-5, 5, 100001)
for p in [0.7, 0.9]:
    log = p * np.log(1 + np.exp(-f)) + (1 - p) * np.log(1 + np.exp(f))
    hinge = p * np.maximum(0, 1 - f) + (1 - p) * np.maximum(0, 1 + f)
    f_log, f_hinge = f[np.argmin(log)], f[np.argmin(hinge)]
    p_back = 1 / (1 + np.exp(-f_log))
    print(p, round(f_log, 3), round(p_back, 3), round(f_hinge, 3))
# 0.7 0.847 0.7 1.0
# 0.9 2.197 0.9 1.0

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0
)
for loss in ["hinge", "log_loss"]:
    clf = make_pipeline(
        StandardScaler(), SGDClassifier(loss=loss, random_state=0)
    )
    clf.fit(X_train, y_train)
    print(loss, round(clf.score(X_test, y_test), 3),
          hasattr(clf, "predict_proba"))
# hinge 0.965 False
# log_loss 0.965 True

Weighted losses

Weighting is not a separate loss family. It modifies any existing loss by deciding which samples matter more:

$$L_w=\frac{\sum_{i=1}^{n}w_i\ell(y_i,\hat{y}_i)}{\sum_{i=1}^{n}w_i}.$$

A sample with a larger non-negative weight $w_i$ has more influence on the fitted parameters. The underlying loss could be squared error, cross-entropy, hinge loss or something else.

Sample weights and class weights

• A sample weight assigns importance to one particular training example.
• A class weight gives the same weight to every example of one class.

Scikit-learn commonly accepts weights through fit(X, y, sample_weight=w). For LogisticRegression, weight 2 on one sample produces the same fitted model as including that sample twice.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X = StandardScaler().fit_transform(X)
X_rep = np.vstack([X, X[:1]])
y_rep = np.append(y, y[0])
w = np.ones(len(y))
w[0] = 2

opts = dict(tol=1e-10, max_iter=10_000)
a = LogisticRegression(**opts).fit(X_rep, y_rep)
b = LogisticRegression(**opts).fit(X, y, sample_weight=w)
print(np.allclose(a.coef_, b.coef_),
      np.allclose(a.intercept_, b.intercept_))  # True True

With class_weight="balanced", scikit-learn gives class $k$ the weight

$$w_k=\frac{n}{K n_k},$$

where $K$ is the number of classes and $n_k$ is the number of samples in class $k$. Every class then has the same total weight, $n/K$.

from sklearn.datasets import make_classification
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.utils.class_weight import compute_class_weight

X, y = make_classification(
    n_samples=2000, n_features=10, n_informative=4, n_redundant=0,
    weights=[0.9], class_sep=1.5, random_state=0,
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0
)

print(np.bincount(y_train))
print(compute_class_weight(
    "balanced", classes=np.array([0, 1]), y=y_train
).round(3))

for cw in [None, "balanced"]:
    clf = make_pipeline(
        StandardScaler(), LogisticRegression(class_weight=cw)
    )
    clf.fit(X_train, y_train)
    pred = clf.predict(X_test)
    print(cw, round(recall_score(y_test, pred), 3),
          round(precision_score(y_test, pred), 3))
# [1337  163]
# [0.561 4.601]
# None 0.204 0.846
# balanced 0.63 0.209

Loss and evaluation metric

Training needs a quantity that can guide optimisation. Evaluation needs a quantity that reflects the task's goal. Sometimes one function can do both jobs; sometimes a smooth surrogate and a later decision rule are needed.

Accuracy, F1 and ROC AUC change in steps as predictions change, so gradient methods do not directly minimise them. Train with a surrogate, then tune any threshold on validation data as described in Model Evaluation Metrics.

The same loss ideas appear across model families:

• Decision trees use split criteria related to the prediction error in their leaves. Squared and absolute regression criteria produce means and medians; classification impurity is covered in Decision Trees.
• Gradient boosting fits new trees using derivatives of the chosen loss, as shown in XGBoost.
• Neural networks use the same principles through functions such as nn.MSELoss, nn.L1Loss and nn.CrossEntropyLoss. The deep-learning loss module will build on this framework.

Choosing a loss

The cards provide detail; this decision flow gives a practical starting point.

Regression
├─ Need the ordinary conditional average? → squared error
├─ Need the median or stronger outlier resistance? → absolute error
├─ Want smooth small-error fitting with robust tails? → Huber
└─ Need a quantile, interval or asymmetric penalty? → pinball

Classification
├─ Need probabilities? → binary or multiclass cross-entropy
├─ Need only a separating boundary? → hinge is possible
└─ Using AdaBoost-style training? → exponential loss

Unequal sample or class importance? → weight the chosen loss
Step-based metric such as F1?        → train a surrogate, tune threshold later

The default loss is usually a sensible baseline. Change it when the target quantity, outlier behaviour, direction of error, probability requirement or metric gives a specific reason.

The whole idea in one view

loss for one prediction
        ↓ average across samples
training cost
        ↓ add any parameter penalty
objective minimised by training

choosing the loss decides
├─ what quantity the model predicts
├─ how quickly penalties grow
├─ whether outliers dominate
├─ whether error direction matters
└─ what optimisation signal the model receives

weights modify an existing loss
metric evaluates the final behaviour

Once these questions are understood, a new loss is no longer an isolated formula. It is another card in the same framework.