5.2.4 Tensors, Matrices, and Related Mathematical Functions

Why learn the maths of tensors?

In Backpropagation, we followed gradients backwards through a network, and in Multi-Layer Perceptrons we wrote a whole layer as one matrix multiplication. Both relied on a small vocabulary: shapes, matrix products, sums over one dimension, and a few functions such as exp and log. This module collects that vocabulary in one place.

It matters because most bugs in deep learning code are not about ideas but about shapes. A tensor that is (32, 10) where you expected (10, 32), a softmax over the wrong dimension, or a sum that silently broadcasts into a square matrix: none of these always raises an error, but all of them break a model. When you know what each operation does to a shape, you can read an unfamiliar training notebook and predict what every line produces.

You already know the basics of arrays from NumPy Basics, and PyTorch tensors behave almost identically, as we saw in PyTorch Basics. Here we go one step further: what the operations mean mathematically, and how they are used inside a network.

What you need to know

By the end of this module, you should be comfortable with these ideas:

You do not need eigenvalues, determinants or matrix inverses for anything in this part of the guide. They are important in other areas of maths, but training a neural network only uses the operations above.

Learn it properly

This page shows each idea with a short PyTorch example. For the geometric picture behind them, which makes the rules much easier to remember, use one of these.

Pick one main course

You do not need a full linear algebra course for this module. One short series, watched properly, is enough.

Our pick: 3Blue1Brown's Essence of linear algebra

3Blue1Brown's Essence of linear algebra turns vectors and matrices into pictures. For this module, watch:

• Chapter 1: Vectors
• Chapter 2: Linear combinations, span, and basis vectors
• Chapter 3: Linear transformations and matrices
• Chapter 4: Matrix multiplication as composition
• Chapter 9: Dot products and duality

Chapter 4 shows that applying one matrix after another is the same as applying their product, which is the picture behind the fact, proved with algebra in the MLP module, that two linear layers collapse into one. The other chapters are worth watching later, but you do not need them now.

If you prefer something else

• A textbook with code: the Linear Algebra and Data Manipulation sections of Dive into Deep Learning cover the same ground as this page, in PyTorch.
• The official tutorial: PyTorch's Tensors page, if you want more practice creating and indexing tensors.
• The exact rules: PyTorch's broadcasting semantics page, for when a shape surprises you.

A quick tour

The rest of this page goes through the operations in the order a network uses them: first the shapes of the data, then the products inside a layer, then the reductions and functions at the end of a network. Each example shows the shape of the result, because that is what you should learn to predict.

Scalars, vectors, matrices and tensors

These words describe how many dimensions a collection of numbers has:

• A scalar is a single number: shape (), 0 dimensions.
• A vector is a list of numbers: shape (n,), 1 dimension.
• A matrix is a table: shape (rows, columns), 2 dimensions.
• A tensor is the general word for any number of dimensions. Scalars, vectors and matrices are all tensors.

import torch

temperature = torch.tensor(21.5)                # a scalar
scores = torch.tensor([72.0, 85.0, 90.0])       # a vector
table = torch.tensor([[1.0, 2.0, 3.0],
                      [4.0, 5.0, 6.0]])         # a matrix

temperature.shape, scores.shape, table.shape

(torch.Size([]), torch.Size([3]), torch.Size([2, 3]))

.ndim gives the number of dimensions: 0, 1 and 2 here. In maths we write a vector as $x \in \mathbb{R}^n$ and a matrix as $A \in \mathbb{R}^{m \times n}$, which is just the shape written as a superscript.

The batch dimension

Networks almost never look at one sample at a time. They process a batch, and by convention the batch is the first dimension. A batch of 32 colour images of 28 × 28 pixels is a 4-dimensional tensor:

images = torch.zeros(32, 3, 28, 28)  # (batch, channels, h, w)
images.shape

torch.Size([32, 3, 28, 28])

The same pattern holds everywhere: tabular data is (batch, features), text is often (batch, tokens, embedding size). When you read someone else's code, the first question to ask about any tensor is "which dimension is which?". We will meet image and text tensors properly in the computer vision and NLP parts of the guide; for now, just remember that the batch comes first.

The dot product

The dot product of two vectors of the same length multiplies matching entries and adds the results:

$$u \cdot v = u_1 v_1 + u_2 v_2 + \dots + u_n v_n = \sum_{i=1}^{n} u_i v_i.$$

u = torch.tensor([1.0, 2.0, 3.0])
v = torch.tensor([4.0, 5.0, 6.0])
torch.dot(u, v)                        # 1*4 + 2*5 + 3*6

tensor(32.)

The same number comes out of (u * v).sum(): * multiplies matching entries, and sum adds them. A neuron's weighted sum, $w^\top x$, is exactly a dot product. The dot product is also large when two vectors point the same way and near zero when they are unrelated, which is why it later measures how similar two embeddings are.

Matrix multiplication

To multiply a matrix $A$ of shape $(m, n)$ by a matrix $B$ of shape $(n, p)$, we take the dot product of every row of $A$ with every column of $B$. Entry $(i, j)$ of the result is

$$(AB)_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj},$$

and the result has shape $(m, p)$. The rule for shapes is worth memorising: the inner sizes must match, and the outer sizes are what you get.

$$(m, \underline{n}) \times (\underline{n}, p) \;\rightarrow\; (m, p)$$

In PyTorch, @ is matrix multiplication:

A = torch.ones(2, 3)
B = torch.ones(3, 4)
(A @ B).shape                          # (2, 3) @ (3, 4) -> (2, 4)

torch.Size([2, 4])

If the inner sizes do not match, PyTorch stops with an error that tells you both shapes:

A @ torch.ones(2, 4)                   # inner sizes 3 and 2 differ

RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x3 and 2x4)

Do not confuse @ with *. * multiplies matching entries (the elementwise or Hadamard product) and needs compatible shapes, while @ combines rows with columns. For the same square matrix t, t * t squares each entry but t @ t is a genuinely different matrix.

A linear layer as matrix multiplication

Now we can read a linear layer precisely. For a batch $X$ of shape $(n, d)$, a layer with $m$ neurons computes

$$Y = X W^\top + b,$$

where PyTorch's nn.Linear stores $W$ with shape $(m, d)$, one row per neuron, and $b$ with shape $(m,)$. The transpose turns $W$ into $(d, m)$ so the inner sizes match:

$$(n, d) \times (d, m) \;\rightarrow\; (n, m).$$

X = torch.zeros(8, 5)                  # 8 samples, 5 features
W = torch.zeros(3, 5)                  # 3 neurons, 5 weights each
(X @ W.T).shape                        # (8, 5) @ (5, 3) -> (8, 3)

torch.Size([8, 3])

Some books write the same layer as $W x + b$ for a single sample $x$ written as a column. That is the same computation, arranged differently. The MLP module checks with code that nn.Linear gives exactly X @ W.T + b.

Transpose

The transpose swaps rows and columns: entry $(i, j)$ moves to $(j, i)$, and a shape $(m, n)$ becomes $(n, m)$. For a 2-D tensor, .T does it:

table.T

tensor([[1., 4.],
        [2., 5.],
        [3., 6.]])

The first row of table, [1, 2, 3], is now the first column. For tensors with more dimensions, transpose(dim0, dim1) swaps any two chosen dimensions; you will need it for images and text later, but not yet.

Reshape, view and flatten

Reshaping keeps the same numbers in the same order but arranges them into a different shape. The total number of entries must stay the same:

table.reshape(3, 2)                    # 6 numbers: (2, 3) -> (3, 2)

tensor([[1., 2.],
        [3., 4.],
        [5., 6.]])

Compare this with table.T above: the transpose moved numbers to new positions, while reshape just refilled the new shape row by row. They are different operations, even though both produce shape (3, 2).

A -1 in the shape means "work this size out for me", so table.reshape(-1) gives the flat vector tensor([1., 2., 3., 4., 5., 6.]). The most common reshape in practice is flattening each image into one long vector before an MLP:

batch = torch.zeros(32, 1, 28, 28)     # 32 greyscale 28x28 images
flat = batch.flatten(start_dim=1)      # keep dim 0, merge the rest
flat.shape                             # 1 * 28 * 28 = 784

torch.Size([32, 784])

start_dim=1 keeps the batch dimension and merges everything after it. You will also see .view(...) in older code. It does the same job as reshape, but only works when the numbers are stored in memory in the right order; after a transpose they are not:

table.T.view(6)

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

The error message gives the fix. Use reshape and you never need to think about memory layout.

Broadcasting with tensors

Adding a bias to every row of a layer's output means adding a tensor of shape (3,) to one of shape (n, 3). Broadcasting makes this work without copying the bias n times. You met it in NumPy Basics; the rules are the same:

1. Line up the shapes from the right.
2. Two sizes are compatible if they are equal, or if one of them is 1 (or missing).
3. A size-1 dimension is stretched to match the other.

X = torch.tensor([[1.0, 10.0],
                  [2.0, 20.0],
                  [3.0, 30.0],
                  [4.0, 40.0]])        # (4, 2)
bias = torch.tensor([0.5, -1.0])       # (2,) -> stretched to (4, 2)
X + bias

tensor([[ 1.5000,  9.0000],
        [ 2.5000, 19.0000],
        [ 3.5000, 29.0000],
        [ 4.5000, 39.0000]])

The same rule lets us centre each feature: subtract the column means from every row.

col_means = X.mean(dim=0)              # (2,): one mean per column
X - col_means                          # (4, 2) - (2,) -> (4, 2)

tensor([[ -1.5000, -15.0000],
        [ -0.5000,  -5.0000],
        [  0.5000,   5.0000],
        [  1.5000,  15.0000]])

Broadcasting is convenient, but it can also hide mistakes. Adding a tensor of shape (4,) to one of shape (4, 1) does not raise an error: both stretch, and the result is a (4, 4) matrix. If a loss suddenly becomes a matrix, or is far too large, look for exactly this.

Reductions over a dimension

A reduction turns many numbers into fewer: a sum, a mean, a maximum. With a 2-D tensor we almost always want to reduce along one dimension, chosen with dim. A good way to remember it: dim is the dimension that disappears.

Let us use a batch of 2 samples with scores for 3 classes, the typical output of a classifier:

logits = torch.tensor([[2.0, 1.0, 0.1],
                       [0.5, 2.5, 0.3]])   # (2 samples, 3 classes)
logits.sum(dim=1)                          # add across the classes

tensor([3.1000, 3.3000])

dim=1 (the classes) disappeared, leaving one total per sample. logits.mean(dim=0) would instead average over the samples, giving one value per class: tensor([1.2500, 1.7500, 0.2000]).

Often we want the result to keep its dimension, with size 1, so that it broadcasts correctly against the original tensor. That is what keepdim=True does:

logits.sum(dim=1, keepdim=True)        # (2, 3) -> (2, 1), not (2,)

tensor([[3.1000],
        [3.3000]])

Now logits / logits.sum(dim=1, keepdim=True) divides each row by its own total, which is exactly what softmax will need below. Without keepdim, the shapes (2, 3) and (2,) would not line up.

For classification, the reduction you will use most is argmax: the position of the largest value, which is the predicted class.

logits.argmax(dim=1)                   # the best class for each sample

tensor([0, 1])

logits.max(dim=1) returns both the largest values and their positions, as .values and .indices.

Elementwise functions

Functions such as $e^x$, $\log x$ and $\sqrt{x}$ apply to every entry separately, so the shape does not change:

torch.exp(torch.tensor([0.0, 1.0, 2.0]))     # e^0, e^1, e^2

tensor([1.0000, 2.7183, 7.3891])

Two facts about them cause trouble in practice. torch.log(torch.tensor(0.0)) is -inf, and torch.exp of a large number overflows to inf, because a number such as $e^{1000}$ is far larger than any number a computer can store. Both matter for the next two sections. (torch.sqrt of a negative number gives nan, "not a number", for the same kind of reason.)

Activation functions are elementwise too: ReLU replaces every negative entry with 0. Activation Functions covers them.

Softmax

A classifier produces one score per class, the logits. Softmax turns them into probabilities that are positive and add up to 1:

$$\mathrm{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j} e^{z_j}}.$$

The exponential makes every score positive, and dividing by the total makes them add up to 1. Let us build it from the pieces we already have, for the batch of logits from above:

exps = torch.exp(logits)                       # (2, 3), all positive
row_totals = exps.sum(dim=1, keepdim=True)     # (2, 1)
probs = exps / row_totals                      # (2, 3) / (2, 1)
probs

tensor([[0.6590, 0.2424, 0.0986],
        [0.1086, 0.8025, 0.0889]])

Each row now adds up to 1, and the largest logit got the largest probability. PyTorch's torch.softmax(logits, dim=1) gives exactly the same result. The dim matters: it must be the classes dimension. With dim=0, PyTorch would normalise each class across the samples instead, and the rows would no longer add up to 1.

Numerical stability and log-sum-exp

Our home-made softmax has a hidden bug. If the logits are large, exp overflows:

big = torch.tensor([1000.0, 999.0, 998.0])
torch.exp(big) / torch.exp(big).sum()          # inf / inf

tensor([nan, nan, nan])

The fix uses a neat property: subtracting the same number from every logit does not change the softmax, because the factor $e^{-c}$ cancels between the top and the bottom:

$$\frac{e^{z_i - c}}{\sum_j e^{z_j - c}} = \frac{e^{-c}\, e^{z_i}}{e^{-c} \sum_j e^{z_j}} = \frac{e^{z_i}}{\sum_j e^{z_j}}.$$

So we subtract the largest logit first. The biggest exponent is then $e^0 = 1$, and nothing can overflow:

shifted = big - big.max()                      # [0, -1, -2]
torch.exp(shifted) / torch.exp(shifted).sum()

tensor([0.6652, 0.2447, 0.0900])

torch.softmax does this for you, which is one reason to use the built-in function instead of writing your own.

Losses usually need the logarithm of the probabilities, and taking log(softmax(z)) in two steps can hit log(0) = -inf when a probability underflows to zero. The stable, combined version is log-softmax:

$$\log \mathrm{softmax}(z)_i = z_i - \log \sum_j e^{z_j},$$

and the term $\log \sum_j e^{z_j}$, computed with the same shifting trick, is called log-sum-exp:

torch.logsumexp(big, dim=0)

tensor(1000.4076)

Where the naive version would overflow, torch.logsumexp returns the correct value, about $1000 + \log(1 + e^{-1} + e^{-2})$. This is why PyTorch's classification losses take raw logits rather than probabilities: they apply log-softmax internally, in the stable way. Loss Functions builds on exactly this. For your own code, the practical rule is simple: use torch.softmax, torch.log_softmax and the built-in losses, and never write torch.log(torch.softmax(...)).

Norms

A norm measures the size of a vector. The most common is the L2 norm, the ordinary length:

$$\lVert x \rVert_2 = \sqrt{x_1^2 + x_2^2 + \dots + x_n^2}.$$

p = torch.tensor([3.0, 4.0])
torch.linalg.vector_norm(p)                    # sqrt(9 + 16)

tensor(5.)

The norm of a difference is the distance between two points, so torch.linalg.vector_norm(p - q) is the straight-line distance used by K-Nearest Neighbors. The L1 norm, $\lVert x \rVert_1 = \sum_i |x_i|$, adds absolute values instead; pass ord=1 to get it (7 for p above).

Norms appear in deep learning in two more places that you will meet later: weight decay, which penalises large weights to reduce overfitting, and gradient clipping, which shrinks a gradient whose norm is too large. You do not need either yet; just recognise the word when you see it.

Common mistakes

Most of these mistakes come from shapes. The messages are exactly what PyTorch prints.

The single most useful debugging habit: print .shape at every step until the shapes match what you expected.

How to practise

• Before running each cell of this page, write down the shape you expect, then check. Predicting shapes is the skill.
• HackerRank's NumPy problems and CodeChef's NumPy practice practise the same operations (dot, matrix products, reshape, reductions); every one translates directly to PyTorch. They are in the practice table below.
• Do the exercises at the end of the d2l Linear Algebra section.

Check yourself

What comes next

• Activation Functions: the elementwise functions between layers, and why ReLU became the default.
• Loss Functions: how log-softmax and log-sum-exp turn into the cross-entropy loss.