Convolutional Neural Networks

• CNNs transform images into hierarchies of features.
• The goal is to understand tensor shapes, parameters, and gradient flow.
• By the end, you will be able to design and diagnose a small CNN.

• Five parts; each one builds only on the previous ones.
• Every new operation is first shown on a tiny numeric example.
• Check questions close each block.

• A camera provides an array of intensities, not objects.
• The model must separate changes in the object from changes in image acquisition.
• The output must match the task and its decision tolerance.

• Classification: one label (or a vector of independent attributes).
• Detection: a variable number of boxes, each with a class.
• Segmentation: a class for every pixel. Regression: continuous quantities.

• Position, scale, rotation, and deformation of the object.
• Illumination, exposure, noise, and background.
• Occlusion and within-class variability.

• A grayscale image is a matrix of numbers, one per pixel.
• A colour image adds a third axis: channels. In PyTorch the order is C × H × W.
• Everything a CNN does is arithmetic on such arrays.

• Weighted sum of inputs plus a bias: z = w^Tx + b.
• An activation function φ turns z into the output y = φ(z).
• Learning means finding w and b that reduce the loss.

• Two linear layers in a row are again one linear map: W₂W₁ = W.
• An activation between them breaks this collapse; then depth adds expressive power.
• ReLU = max(0, z) is the default in CNNs: cheap, with well-behaved gradients.

• The image is flattened into one long vector.
• Every input pixel is connected to every unit of the next layer.
• 224 × 224 × 3 inputs → 1000 units: 150 million weights in one layer.
• Flattening also throws away which pixels are neighbours.

• A fully connected layer has a separate weight for every pixel index.
• A pattern seen at position A teaches nothing about position B.
• We want one detector that is reused everywhere in the image.

• Local connectivity: an output depends on a small neighbourhood only.
• Weight sharing: the same small kernel is applied at every position.
• Hierarchy: stacked layers compose edges into parts and parts into objects.

• Take a small window of the input, e.g. 3 × 3 pixels.
• Multiply it cell by cell with a kernel of the same size and add up the products.
• The result is one number of the output map. The kernel values are learned weights.

Start with a 1D signal: the kernel w = [−1, 0, 1] is placed over three neighbouring inputs, gives one output, and then moves one cell to the right.

• In 2D the window is a 3 × 3 patch; the kernel is a 3 × 3 matrix.
• Multiply the 9 pairs of matching cells and add them: one number.
• In general: y_i,j = Σ_u Σ_v x_{i+u, j+v} · k_u,v.

The window moves left to right, then down by one row. Each position fills one cell of the output; the kernel never changes.

• The output is large where the input patch looks like the kernel.
• [−1, 0, 1] in every row compares right with left: a vertical-edge detector.
• A CNN does not get such kernels by hand; it learns them from data.

• One 3 × 3 kernel slides over the whole image, row by row; its output is a feature map.
• Flat regions give values near zero (grey); edges give strong responses. Try other kernels, or drop your own image.
• Libraries actually compute cross-correlation (no kernel flip); for learned kernels the difference does not matter.

• The output has as many values as there are valid window positions.
• Without padding the window must stay inside the input.
• Each layer therefore shrinks the signal by K − 1.

• Padding adds P extra values (usually zeros) at each border.
• With P = (K − 1)/2 and stride 1 the output keeps the input size (“same”).
• Border outputs partly see artificial values; keep this in mind when measuring near edges.

• Stride S is the step of the window; S = 2 skips every second position.
• It filters and downsamples in one operation, so later layers are cheaper.
• Large strides can skip small objects and cause aliasing.

• Dilation D inserts gaps between the kernel taps.
• The kernel covers a span of D(K − 1) + 1 inputs with only K weights.
• It enlarges the view without downsampling (common in segmentation).

• All four effects combine into one formula, applied separately to height and width.
• Move the sliders and predict the output size before you look.

• An RGB input has three channels, so the filter also has three 3 × 3 slices.
• Filter depth always equals the number of input channels C_in.
• All 3 · 3 · 3 = 27 products are summed: one filter gives one output channel.

• A layer has C_out independent filters, each looking for a different pattern.
• Their feature maps are stacked into the output tensor C_out × H_o × W_o.
• The next layer sees C_out as its number of input channels.

• A batch of images is a 4D tensor N × C × H × W.
• The weight tensor of a convolution is C_out × C_in × K × K.
• Spatial output size follows the formula; the channel count is chosen by the designer.

conv = nn.Conv2d(in_channels=3, out_channels=32,
                 kernel_size=3, stride=2, padding=1)

x.shape            # (N, 3, 128, 128)
conv(x).shape      # (N, 32, 64, 64)    ⌊(128+2−3)/2⌋+1 = 64
conv.weight.shape  # (32, 3, 3, 3)
conv.bias.shape    # (32,)

• Weights: C_out filters of size K × K × C_in, plus one bias per filter.
• The count does not depend on the image resolution H × W.
• Thanks to locality and sharing, it is orders of magnitude smaller than a dense layer.

• Few parameters do not mean little computation: each weight is used at every output position.
• Activations grow with resolution and must be stored for the backward pass.
• High-resolution layers dominate cost and memory; low-resolution layers dominate parameters.

• Each output channel is the response map of one learned detector.
• Some channels have a simple meaning (edges, blobs); many do not.
• Their meaning emerges together with the layers that read them.

• Convolution is linear: two convolutions in a row equal one larger convolution.
• A ReLU after each convolution prevents this collapse, exactly as for neurons.
• The standard block: convolution → normalization → ReLU.

• The receptive field is the input region that can influence one output element.
• One 3 × 3 layer sees 3 pixels per axis; two layers see 5; three see 7.
• Deep layers can therefore react to larger structures than any single kernel.

• Stride and pooling make later layers take bigger steps in the input.
• The receptive field then grows multiplicatively, not just by K − 1 per layer.
• The theoretical field is larger than the region that effectively matters.

• Early layers: small receptive field → edges and colours.
• Middle layers combine them into corners, arcs, and textures, then into parts.
• Late layers carry object- and task-specific information.

• A 1 × 1 kernel looks at a single pixel, but across all its channels.
• It is a small dense layer applied identically at every pixel.
• Used to reduce or increase the number of channels cheaply (bottlenecks).

• Pooling slides a window like convolution but uses a fixed rule instead of weights.
• Max pooling keeps the strongest response; average pooling keeps the mean.
• It halves the resolution and adds tolerance to small shifts.

• Both halve the resolution.
• Pooling has no learnable weights; a strided convolution learns what to keep.
• The choice affects cost, aliasing, and localization accuracy.

• Each channel is averaged over all positions into a single number.
• The result is a vector of length C, regardless of the input resolution.
• It replaces a large flatten layer before classification, but discards location.

• Subtract the per-channel mean and divide by the standard deviation.
• Use the statistics of the pretraining data (e.g. ImageNet) or of the training set.
• Store them with the model; inference must use exactly the same values.

• The same idea applied inside the network, to the activations of each channel.
• During training, statistics come from the current minibatch; γ, β are learned.
• At inference, stored running averages are used instead.

• Three conv blocks halve the resolution and increase the number of channels.
• Global average pooling turns the last feature map into a vector.
• A linear layer maps it to one score per class.

• The last linear layer produces one real number per class: the logits.
• Larger logit = more evidence for that class.
• Logits are not probabilities yet.

• Softmax turns logits into a distribution over mutually exclusive classes.
• Sigmoid squashes each logit separately; use it for independent labels.
• Move the logits: softmax always sums to one, sigmoid outputs do not.

• A loss compares the prediction with the target and returns one number.
• Its gradient tells the optimizer how to change the weights.
• It must match the output type; a reporting metric need not be a good loss.

• Classification loss: minus the log-probability of the correct class.
• Confident correct predictions cost little; confident mistakes cost a lot.
• Most libraries take logits directly; do not apply softmax twice.

• MSE penalizes large errors quadratically, so outliers dominate.
• MAE grows linearly: robust, but its gradient magnitude never shrinks.
• Huber is quadratic near zero and linear for large errors.

• Pixel-wise cross-entropy is classification at every pixel.
• When the object covers few pixels, the background dominates the loss.
• Dice loss measures region overlap and is insensitive to this imbalance; either loss works on its own.

• Imagine the loss as a function of a single weight w.
• The derivative is the slope; stepping against it lowers the loss.
• The learning rate η sets the size of the step. Press → to take a step.

• Too small: training is slow and may stall.
• Too large: steps overshoot the minimum, the loss oscillates or becomes NaN.
• In practice: try values on a log scale (10⁻⁴, 10⁻³, 10⁻²) and watch the curve.

• The forward pass computes the prediction and stores intermediate values.
• The backward pass applies the chain rule from the loss back to every weight.
• One backward pass gives the gradient for all millions of weights at once.

• The same kernel produced every output value.
• Its gradient is therefore a sum of contributions from all positions.
• A pattern that matters anywhere in the image updates the same weights.

• The gradient is estimated from a minibatch of B images, not from the whole set.
• One iteration = one minibatch = one weight update; one epoch = one pass over the data.
• The number of updates matters more than the epoch count alone.

• Minibatch gradients are noisy; momentum averages them over recent steps.
• AdamW additionally adapts the step size to each parameter.
• Both paths below are computed live on the same elongated loss surface.

• Warm-up protects early training from steps that are too large.
• Cosine or step decay refines the final stage with smaller steps.
• Recheck the learning rate whenever the batch size changes.

• Train is used to learn parameters.
• Validation guides model and hyperparameter selection.
• Test is used only once, for the final independent estimate.

• Images of the same product must not appear in more than one split.
• Fit preprocessing statistics only on the training set.
• Temporal and spatial correlation can inflate the measured performance.

• Training loss keeps falling while validation loss starts to rise.
• The model memorizes random or source-specific details of the training set.
• Keep the checkpoint with the best validation result (early stopping).

• Both training and validation results are poor.
• Causes: too small a model, too short training, a bad learning rate, or wrong inputs.
• Stronger regularization does not solve underfitting.

• Random transformations create new training samples; the label must stay correct.
• Apply geometry consistently to the image, mask, and boxes.
• Photometric changes simulate the camera and illumination.

• Accuracy can hide complete failure on a rare class.
• Use per-class recall, precision, and a confusion matrix.
• Sampling, a weighted loss, or focal loss may help.

• A pretrained backbone already provides general visual features.
• Train the new head first, then carefully unfreeze the backbone.
• The domain shift determines how many layers should be adapted.

• Weight decay limits the magnitude of weights.
• Dropout randomly disables activations during training.
• For CNNs, sound augmentation often matters more than a high dropout rate.

• Track training and validation loss as well as task metrics.
• A plateau may expose a limit in data, representation, or optimization.
• A sudden NaN indicates a numerical error or an excessive step size.

• Rows are true classes, columns are predictions; off-diagonal cells are errors.
• A row-normalized matrix helps when classes are imbalanced.
• Inspect the actual images behind the largest confusions.

• Group errors by illumination, camera, class, and geometry.
• Each group leads to a different pipeline correction.
• One aggregate number cannot explain operational risk.

• Store the seed, split, dataset version, and configuration.
• A checkpoint must contain both weights and normalization.
• Report variation across runs when the dataset is small.

• Parameters determine model size, not latency.
• FLOPs ignore memory transfers and hardware behaviour.
• Measure latency on the target device with a realistic batch size.

• Start with a simple pretrained backbone and a short training run.
• Sanity checks: one batch must be fit almost perfectly; random labels must not generalize.
• Look at augmented inputs before training; then change one thing at a time.

• Define the error tolerance and the unit used for data splitting.
• Build a baseline and a catalogue of failures.
• Iterate on data and measurement first, then on architecture.

• Convolution slides one small learned detector over the whole image.
• Output shape, parameters, and receptive field can be computed before training.
• A good experiment is measurable, reproducible, and analyzes its errors.

• You can calculate the shape and parameters of a convolutional layer.
• You can distinguish a logit, activation, loss, and metric.
• Next, we will combine a backbone, neck, and head for specific tasks.