Standard fully connected networks fail when processing high-dimensional data like images because they ignore spatial relationships. Convolutional Neural Networks (CNNs) solve this by looking at local pixel blocks.
The core of a CNN is the Convolutional Layer. Instead of connecting every pixel to every neuron, a sliding matrix called a Kernel (or filter) traverses the image. As it slides, it performs dot products on local region matrices, highlighting spatial features like edges, textures, or shapes.
Pooling layers reduce the spatial size of representation matrices to cut down parameters and computational load. The most popular method is Max Pooling, which slides a filter across the output and selects the maximum value from each block, keeping only the most dominant features.
A 2×2 pooling window with stride 2 slides across the 4×4 input, selecting the maximum value in each region.
A fully connected layer connects every input to every neuron. For a modest 224 x 224 colour image there are 224 x 224 x 3 = 150,528 input values; a single dense layer with 1,000 neurons would need over 150 million weights, and it would learn nothing about the fact that neighbouring pixels belong together. Worse, a cat in the top-left corner and the same cat in the bottom-right would look like entirely different inputs. CNNs fix both problems with two ideas: local connectivity (each output looks at a small neighbourhood) and weight sharing (the same small kernel is reused at every position). A 3 x 3 kernel has just 9 weights regardless of image size, and because it is applied everywhere, a feature it detects is detected wherever it appears.
Take a 3 x 3 patch of pixel brightness and a vertical-edge kernel (a Sobel-style filter). The output value is the sum of element-wise products:
patch = [[10, 10, 0], kernel = [[-1, 0, 1],
[10, 10, 0], [-2, 0, 2],
[10, 10, 0]] [-1, 0, 1]]
output = (10*-1 + 10*0 + 0*1)
+ (10*-2 + 10*0 + 0*2)
+ (10*-1 + 10*0 + 0*1)
= -10 - 20 - 10 = -40The large-magnitude result signals a strong vertical edge: bright on the left, dark on the right. On a flat region where all pixels are equal, the positive and negative columns cancel and the output is 0. Sliding the kernel over the whole image produces a feature map showing where that pattern occurs. In a CNN you do not design these kernels; the network learns their values by gradient descent. (Strictly speaking, deep learning libraries compute cross-correlation, which does not flip the kernel; since the kernel is learned, the distinction does not matter in practice.)
The formula from the exercise, O = floor((I - K + 2P) / S) + 1, deserves a few more cases:
Padding does two jobs: it preserves spatial size so you can stack many layers, and it lets border pixels take part in as many windows as interior pixels.
Real images have channels (3 for RGB), and a convolutional layer has many filters. Each filter spans all the input channels and produces one output channel, so a layer with 16 filters turns a 3-channel image into a 16-channel stack of feature maps. The number of learnable parameters in a convolutional layer is (K x K x C_in + 1) x C_out, where the +1 is each filter's bias. For 3 x 3 kernels mapping 3 channels to 16 that is (27 + 1) x 16 = 448 parameters, tiny compared with the dense layer above.
Stacking layers builds up abstraction. Early filters tend to respond to edges and colour blobs; middle layers combine them into textures and parts such as corners or eyes; deep layers respond to whole objects. This works because each successive layer sees a larger region of the original image, called its receptive field. Two stacked 3 x 3 convolutions see a 5 x 5 region, and pooling grows the receptive field faster still.
A classic small CNN for 28 x 28 greyscale digits alternates convolution, a non-linearity and pooling, then flattens into a dense classifier. Tracking shapes is the key skill:
import torch
import torch.nn as nn
model = nn.Sequential(
nn.Conv2d(1, 8, kernel_size=3, padding=1), # (N,1,28,28) -> (N,8,28,28)
nn.ReLU(),
nn.MaxPool2d(2), # -> (N,8,14,14)
nn.Conv2d(8, 16, kernel_size=3, padding=1), # -> (N,16,14,14)
nn.ReLU(),
nn.MaxPool2d(2), # -> (N,16,7,7)
nn.Flatten(), # -> (N,784)
nn.Linear(16 * 7 * 7, 10), # -> (N,10) class scores
)
x = torch.randn(4, 1, 28, 28) # a batch of 4 fake images
print(model(x).shape) # torch.Size([4, 10])
print(sum(p.numel() for p in model.parameters())) # 9098The parameter count is 80 (first conv) + 1,168 (second conv) + 7,850 (the linear layer) = 9,098. Notice that the final dense layer holds most of the weights, and that PyTorch expects image batches in the order (batch, channels, height, width).
Max pooling keeps the strongest response in each window. It has no learnable parameters, halves the height and width when the window and stride are both 2, and gives a small degree of translation tolerance: if a feature moves by one pixel inside a window, the pooled output often stays the same. The price is that precise position information is discarded, which is acceptable for classification but is why tasks like segmentation use architectures that recover resolution. Average pooling takes the mean instead and is often used as a global pooling step at the end of a network.
Linear layer must equal channels x height x width after the last pooling. Print the shape or run a dummy batch to find it.Calculate the output spatial size (O) for a convolution operation with the following settings:
Next, we will apply convolutional concepts in a project where we train a model on real images!