Convolutional networks: output shapes and parameter counts

Compute how convolution and pooling change the size of an image, and why sharing weights saves parameters.

def conv_out(n, k, stride=1, pad=0):
    return (n + 2 * pad - k) // stride + 1


size, channels = 28, 1
size = conv_out(size, 5)
conv1_params = (5 * 5 * channels + 1) * 6
channels = 6
print(size, channels, conv1_params)
size = conv_out(size, 2, stride=2)
print(size, channels)
size = conv_out(size, 5)
conv2_params = (5 * 5 * channels + 1) * 16
channels = 16
print(size, channels, conv2_params)
size = conv_out(size, 2, stride=2)
print(size, channels, size * size * channels)
print((28 * 28) * (24 * 24 * 6) + 24 * 24 * 6)
print(256 * 120 + 120)

24 6 156
12 6
8 16 2416
4 16 256
2712960
30840

Explanation

A convolutional neural network (CNN) is built for data with a grid structure, such as images. A convolutional layer slides small filters over the input. Each filter has a fixed number of weights and is applied at every position, so the same weights are reused: this is weight sharing. A layer with c_out filters of size k × k on an input with c_in channels has (k·k·c_in + 1)·c_out parameters, including one bias per filter, however large the image is.

The size of the output depends on the input size n, the filter size k, the stride s (how far the filter moves), and the padding p added around the border: out = floor((n + 2p − k) / s) + 1. Pooling layers shrink the maps further, for example 2 × 2 max pooling with stride 2 halves the width and height. Each output value depends only on a small neighbourhood of its input, so the network is built from local features that later layers combine into larger patterns. Deep learning libraries actually compute cross-correlation, which does not flip the filter, and call it convolution.

The program follows a small LeNet-style network on a 28 × 28 grayscale image. The first layer has 6 filters of size 5 × 5 and needs only 156 parameters. A fully connected layer producing the same 24 × 24 × 6 output from all 784 inputs would need 2,712,960. The second convolution, with 16 filters, adds 2,416 parameters. After pooling, 4 × 4 × 16 = 256 values feed into a dense layer with 120 units.

—

—

24 6 156

24 6 156
12 6
8 16 2416
4 16 256

24 6 156
12 6
8 16 2416
4 16 256
2712960

24 6 156
12 6
8 16 2416
4 16 256
2712960
30840

Cost and common mistakes

A convolution layer costs O(out_h · out_w · c_out · k² · c_in) multiply-adds, and stores only O(k² · c_in · c_out) parameters.

• Shape mistakes are the most common bug in CNNs. Compute the output size of every layer before training.
• Filters are learned, not designed. The 5 × 5 size is a hyperparameter you choose.
• Pooling and strides discard spatial detail. Too much downsampling can lose information needed for fine tasks.

Try it yourself

What output size does a 3 × 3 filter with padding 1 and stride 1 give for a 32 × 32 input?

Further reading

• Stanford CS231n: Convolutional Neural Networks for Visual Recognition (external)
• Deep Learning (Goodfellow, Bengio, Courville) (external)
• Dive into Deep Learning (external)