6.2.2 Pooling

Why learn pooling?

In Convolutional Layers, we saw that a layer of 16 kernels with padding turned the macaw photo into 16 feature maps of 430 × 500, over three million numbers from a single layer. A network stacks many such layers, and if every feature map stayed the size of the image, the network would be slow and would need a great deal of memory.

Pooling shrinks feature maps on purpose. It slides a small window over each map, as a convolution does, but instead of multiplying by learned weights it simply summarises the window: by its largest value (max pooling) or by its mean (average pooling). A 2 × 2 window that jumps 2 pixels at a time halves the height and the width. Pooling has no parameters at all, and it appears in almost every classic image network, including the ones in Convolutional Neural Networks.

What you need to know

By the end of this module, you should be comfortable with these ideas:

Learn it properly

Pooling is the simplest layer in an image network, and one course section covers it. Read it, then use this page to see the numbers and the shapes.

Pick one main course

One short resource is enough here.

Our pick: Dive into Deep Learning, section 7.5

Read Pooling, section 7.5 of Dive into Deep Learning, which follows straight on from the sections recommended in Convolutional Layers. It explains why pooling makes a network less sensitive to exactly where a pattern is, computes max and average pooling by hand and with a short function, and then shows padding, stride and several channels with PyTorch's pooling layer. Choose the PyTorch tab for the code.

If you prefer something else

• Course notes: the Pooling Layer section of Stanford CS231n's Convolutional Neural Networks notes is short, with a picture of max pooling, and mentions the move towards networks that use few or no pooling layers.
• A video: deeplizard's Max Pooling in Convolutional Neural Networks explained works through examples of max pooling and explains why it is used. Watch the first seven minutes; the code demo at the end uses Keras, which the contest machines do not have.

A quick tour

The rest of this page pools the feature map we computed by hand in Convolutional Layers, then looks at the pooling layers in PyTorch and at why networks use them. Go through your chosen course first, then run each cell in a notebook as you read.

Max pooling by hand

Here is the 4 × 4 feature map that the vertical edge detector produced from our made-up 6 × 6 image in Convolutional Layers:

The usual pooling window is 2 × 2 with a stride of 2, so the windows do not overlap: they cut the map into four blocks of four values. Max pooling keeps the largest value of each block. The top-left block holds 11, 16, 16 and 13, so its output is 16; the top-right block holds 11, 3, 4 and 0, so its output is 11. All four:

Average pooling

Average pooling keeps the mean of each block instead. The top-left block gives $(11 + 16 + 16 + 13) / 4 = 14$:

Both results are 2 × 2, a quarter of the size. Max pooling answers "how strongly does the pattern appear anywhere in this block?", while average pooling answers "how much of it is there on average?". For a feature map from an edge detector, the first is usually the more useful question, which is why max pooling is the more common choice between convolutional layers. Step through both:

Pooling in PyTorch

PyTorch has both as layers, nn.MaxPool2d and nn.AvgPool2d. The first argument is the window size, and the stride is the same as the window size unless you say otherwise. Like convolutional layers, they take batches of shape (N, C, H, W):

import torch
from torch import nn

feature_map = torch.tensor([[11, 16, 11, 3],
                            [16, 13, 4, 0],
                            [15, 7, 0, 0],
                            [10, 2, -1, 0]], dtype=torch.float32)
feature_map = feature_map.unsqueeze(0).unsqueeze(0)   # (1, 1, 4, 4)

max_pool = nn.MaxPool2d(kernel_size=2)      # stride 2 by default
average_pool = nn.AvgPool2d(kernel_size=2)
print(max_pool(feature_map))
print(average_pool(feature_map))

tensor([[[[16., 11.],
          [15.,  0.]]]])
tensor([[[[14.0000,  4.5000],
          [ 8.5000, -0.2500]]]])

The same tables as by hand. Pooling layers have no weights, so there is nothing to train:

len(list(max_pool.parameters()))

0

The effect on shape

Pooling works on each channel separately: values from one channel are never mixed with values from another. So the number of channels stays the same and only the height and width shrink. Here is a batch shaped like the 16 feature maps of the macaw photo from the last module:

torch.manual_seed(0)
feature_maps = torch.rand(1, 16, 430, 500)
pooled = max_pool(feature_maps)
print(pooled.shape)
print(feature_maps.numel(), pooled.numel())

torch.Size([1, 16, 215, 250])
3440000 860000

Four times fewer numbers for the next layer to process. The output size follows the same formula as a convolution, $\lfloor (H + 2P - K) / S \rfloor + 1$, with the window size as $K$: $\lfloor (430 - 2) / 2 \rfloor + 1 = 215$. When a size is odd, the last row or column does not fill a window and is simply dropped:

odd = torch.rand(1, 1, 215, 250)
max_pool(odd).shape

torch.Size([1, 1, 107, 125])

215 rows become 107, not 107.5: the last row was never looked at. That rarely matters, but it is why image networks often use sizes such as 32, 64 or 224, which can be halved many times without a remainder.

Why networks pool

Pooling throws information away, so it is worth being clear about what it buys. There are three things.

Smaller feature maps. Every 2 × 2 pooling layer divides the number of positions by four, and with it the computation and memory of every later layer. This is the main practical reason.

Tolerance to small shifts. Max pooling only reports the largest value in each window, not where in the window it was. If a pattern moves by a pixel but stays inside the same window, the pooled output does not change. We can see this with a feature map that has a single strong response:

def one_spot(row, col):
    spot = torch.zeros(1, 1, 4, 4)
    spot[0, 0, row, col] = 9.0
    return spot

for row, col in [(0, 0), (1, 1), (0, 2)]:
    pooled_spot = max_pool(one_spot(row, col))
    print((row, col), pooled_spot[0, 0].tolist())

(0, 0) [[9.0, 0.0], [0.0, 0.0]]
(1, 1) [[9.0, 0.0], [0.0, 0.0]]
(0, 2) [[0.0, 9.0], [0.0, 0.0]]

Moving the spot from (0, 0) to (1, 1) leaves the output unchanged, because both positions are in the top-left window. Moving it to (0, 2) crosses into the next window, and the output changes. So the tolerance is real but limited to shifts within a window; a stack of several pooling layers tolerates larger shifts. Photos of the same object are rarely aligned to the exact pixel, so this helps a classifier.

More context per value. After pooling, each value stands for a 2 × 2 block of the map before it, so a 3 × 3 kernel in the next layer covers what used to be 6 × 6. The region of the input image that can affect one value is called its receptive field. Two 3 × 3 convolutions in a row give each output a receptive field of 5 × 5 pixels. With a 2 × 2 pooling layer between them, it grows to 8 × 8, and with another pooling layer and a third convolution, to 18 × 18. Deep in a network, each value can therefore respond to a large part of the image, such as a whole eye or a whole beak, although every kernel is only 3 × 3.

Global average pooling

At the end of an image network, the feature maps have to become one row of numbers per image for the final nn.Linear layer. One way is to flatten them, but then the size of the nn.Linear depends on the size of the image. Global average pooling averages each channel over its whole map, so each channel becomes one number, whatever the height and width. In PyTorch it is nn.AdaptiveAvgPool2d(1), where 1 is the output size you ask for: 1 × 1.

global_pool = nn.AdaptiveAvgPool2d(1)
for height, width in [(430, 500), (100, 80)]:
    maps = torch.rand(2, 16, height, width)
    pooled_maps = global_pool(maps)              # (2, 16, 1, 1)
    print(pooled_maps.shape, pooled_maps.flatten(1).shape)

torch.Size([2, 16, 1, 1]) torch.Size([2, 16])
torch.Size([2, 16, 1, 1]) torch.Size([2, 16])

Both image sizes give 16 numbers per image, so the same classifier layer, nn.Linear(16, n_classes), works for both. flatten(1) keeps the batch dimension and joins the rest. The pooled value is exactly the mean of each channel:

by_hand = maps.mean(dim=(2, 3))                  # mean over H and W
torch.allclose(pooled_maps.flatten(1), by_hand)

True

"Adaptive" means that you give the output size and PyTorch works out the window size for you; nn.AdaptiveAvgPool2d(2) would give 2 × 2 per channel. Global average pooling has a second benefit: it has no parameters, while a flattened 16 × 107 × 125 map would need a classifier with over 200,000 weights per class.

Strided convolutions instead of pooling

A convolution with stride 2 also halves the height and width, as Convolutional Layers showed. The difference is that it learns how to summarise each region, instead of always taking the maximum or the mean. Many modern networks therefore shrink their feature maps mostly with strided convolutions.

from torchvision.models import resnet18

resnet = resnet18()          # random weights, no download
print(resnet.maxpool)
print(resnet.layer2[0].conv1)
print(resnet.avgpool)

MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
AdaptiveAvgPool2d(output_size=(1, 1))

Both styles work. For the small networks you will build first, a convolution followed by max pooling is simple and reliable, and it is what Convolutional Neural Networks uses.

Common mistakes

The messages are exactly what PyTorch prints.

How to practise

There is no section on pooling on the practice sites we use (CodeChef, CS50 and HackerRank), so practise with small experiments:

• Write max_pool_by_hand(feature_map) for a 2-dimensional tensor with two loops, like the convolve function in Convolutional Layers, and check it against nn.MaxPool2d(2).
• Apply a 3 × 3 vertical edge kernel to the grayscale macaw photo, then max pooling, three times in a row. Show the result after each round with plt.imshow. How much of the photo can you still recognise?
• Compare nn.MaxPool2d(2) with nn.MaxPool2d(kernel_size=3, stride=2, padding=1) on a map of 430 × 500. Predict both output shapes first with the formula.

Check yourself

What comes next

• Reading and Manipulating Images with Pillow: before building a network, the practical side of getting images off the disk and into the right shape.
• Convolutional Neural Networks: convolution, ReLU and pooling stacked into a network and trained on handwritten digits.