CNN Architectures

Backbones, Necks, and Heads for Machine Vision

Matouš Cejnek

Department of Instrumentation and Control Engineering
Faculty of Mechanical Engineering, CTU in Prague

→ Space next step · ← back · F fullscreen · O overview · H help

Why This Lecture

Lecture 1 built one small CNN for one task. Real systems reuse a few proven designs.
The goal is to read an architecture as tensors flowing through named blocks.
By the end, you can choose a backbone, neck, and head for a given vision problem.

Roadmap

Seven parts: the system view first, then each building block, then design decisions.
Every block is introduced through the problem it solves.
Check questions close each part.

Recap: What a CNN Already Does

Stacked conv blocks lower the resolution and raise the number of channels.
Strides 4, 8, 16, 32 give feature maps at several scales — but we used only the last one.
A small head (GAP + linear) turned the last map into class scores.

Four Tasks, Four Output Shapes

The same image can be asked four different questions.
Each answer has a different tensor shape — detection even a variable length.
The architecture must produce exactly that shape.

Do Not Train Everything From Scratch

A deep feature extractor needs a lot of data and compute to train well.
Good features (edges, textures, parts) are useful for almost every visual task.
So we train it once on a large dataset and reuse it; only the task-specific part is new.

Backbone, Neck, Head

Backbone: generic feature extractor, usually pretrained.
Neck: optionally fuses maps of different scales.
Head: turns features into the required output structure.

Feature Levels and Strides

A backbone exposes several maps, named by stage: C2 … C5.
Stride = how many input pixels one cell of the map corresponds to.
Fine levels keep small details; coarse levels see large context.

Check question

You switch from classifying parts to segmenting their defects. Which blocks change?

The head certainly (scores → per-pixel mask), usually also the neck (fine levels are needed). The pretrained backbone can stay.

Seven Years of Backbones

Accuracy rose by 20 points; compute first exploded, then dropped below AlexNet.
Each step solved a concrete problem: depth, optimization, cost.
We follow that story: VGG → Inception → ResNet → DenseNet → MobileNet → EfficientNet.

VGG: One Block, Repeated

Five stages of identical 3 × 3 convolutions, each followed by 2 × 2 max pooling.
Very regular and easy to read — still a good mental model.
Its weakness is the end: three huge fully connected layers.

Why Only 3 × 3 Kernels?

Two stacked 3 × 3 layers see the same 5 × 5 region as one 5 × 5 layer (lecture 1).
They need 28 % fewer weights and add a second nonlinearity.
Three 3 × 3 layers replace a 7 × 7 in the same way.

Where VGG's Parameters Are

The convolutions hold only 15 M of 138 M parameters.
The first dense layer alone (7 · 7 · 512 → 4096) holds 103 M.
Later networks replace it with global average pooling — the head from lecture 1.

Inception: Several Kernel Sizes at Once

Objects appear at different sizes; which kernel size is right is not known in advance.
Inception runs 1 × 1, 3 × 3, 5 × 5, and pooling branches in parallel on the same input.
The results are concatenated along the channel axis.

The 1 × 1 Reduction Trick

A 5 × 5 convolution on 192 channels is expensive.
A 1 × 1 convolution first reduces the channels (lecture 1: it mixes channels cheaply).
The same output costs about ten times less.

Check question

When does processing several kernel sizes in parallel pay off?

When relevant structures have very different sizes within one image and a single kernel size would cover them poorly.

Deeper Is Not Automatically Better

Stacking more plain layers made even the training error worse.
This is not overfitting: the deeper model could copy the shallow one, but the optimizer does not find it.
We need a way to make very deep networks trainable.

Why Plain Deep Networks Are Hard to Train

Backpropagation multiplies local derivatives layer by layer (lecture 1).
If each factor is a bit below one, the gradient fades before it reaches the early layers.
A shortcut that passes the gradient unchanged keeps all layers learning.

The Residual Block

The block adds its input to the output of two convolutions.
The layers only learn what to change; doing nothing is easy.
This single idea enabled networks with 50, 100, and more layers.

When Shapes Do Not Match

Adding requires identical shapes (lecture 1: shapes are part of the design).
At a stage boundary, the main path halves the resolution and doubles the channels.
A 1 × 1 convolution with stride 2 on the shortcut adapts the input.

The Bottleneck Block

Deep ResNets have 256 or more channels; a 3 × 3 there is costly.
1 × 1 squeezes to 64 channels, the 3 × 3 works there, 1 × 1 expands back.
17 × fewer weights per block — this is what ResNet-50 and deeper use.

The ResNet Family

ResNet-18/34 use basic blocks; ResNet-50/101/152 use bottlenecks.
Accuracy grows with depth, but each step costs more for less gain.
ResNet-50 is the most common default backbone to start with.

Two Ways to Merge Tensors

Adding keeps the number of channels and requires identical shapes.
Concatenating stacks the channels and only requires the same height and width.
Every architecture diagram is built from these two merges — look for them.

DenseNet: Concatenate Everything

Inside a block, each layer receives the concatenated outputs of all earlier layers.
It adds only a few new channels (growth rate k), so it stays small.
Few parameters, but the growing concatenations cost memory bandwidth.

Depthwise Separable Convolution

A standard convolution filters space and mixes channels in one expensive step.
Split it: a depthwise 3 × 3 per channel, then a pointwise 1 × 1 to mix channels.
Almost the same expressive power for about 1/8 of the cost.

MobileNetV2: The Inverted Residual

Expand a narrow tensor, filter it with a cheap depthwise conv, project it back.
The skip connection joins the narrow ends — the reverse of a bottleneck.
Built for phones and embedded devices.

Two Knobs: Width and Resolution

Width α scales all channel counts; cost and parameters scale with α².
Input resolution scales cost quadratically but not the parameters.
Try it: which knob would you turn for small defects, which for a tight latency budget?

EfficientNet: Scale Everything Together

Making a network only wider, only deeper, or only higher-resolution saturates quickly.
EfficientNet grows all three together with fixed ratios (compound scaling).
B0 … B7 form one family from small to large.

Reusing a Pretrained Backbone

How much to fine-tune depends on dataset size and on how close the domain is to ImageNet.
Industrial images (grayscale, X-ray, microscopy) are often far from it.
Always compare with a frozen backbone: it is the cheapest baseline.

Check question

An inspection camera runs on an embedded board without a GPU. Which backbone family do you try first, and what do you measure?

MobileNet or EfficientNet-B0 (depthwise separable). Measure end-to-end latency on the board itself, not FLOPs on a PC.

The Classification Head

Global average pooling turns the last map into one vector per image.
A linear layer maps it to class logits; softmax gives probabilities.
Dropout before the linear layer helps on small datasets.

GAP versus Flatten

Flattening keeps position but makes the head 50 × larger.
GAP discards position and works for any input resolution.
For classification GAP is the standard; for localization neither is enough.

One Feature Vector, Several Head Types

The output activation and the loss must match what the label means (lecture 1).
Several heads can share one backbone (multi-task learning).
The head is small: changing it is cheap.

Embedding Heads: Learning Similarity

Instead of class scores, the head outputs a vector (embedding).
Two images pass through the same backbone; their distance measures similarity.
New classes need no retraining: just compare with stored examples.

What an Embedding Space Is Good For

Training arranges the space so that similar images are close.
Then search, clustering, and anomaly detection are just distance computations.
Useful in inspection: only “good” parts are needed to flag unusual ones.

Check question

A line produces 300 part types and new types appear every month. Classification head or embedding head?

Embedding head: a new type only needs reference images in the database, not a new output neuron and retraining.

Detection Changes the Output

The number of objects is unknown and different in every image.
Each object needs a class, a confidence score, and four box coordinates.
A fixed-size output layer cannot produce this directly.

Describing a Box

Two corners, or a centre plus width and height: the same box.
Datasets and libraries differ (pixels vs. 0–1, corners vs. centre).
A wrong format is a common, silent bug.

Intersection over Union

IoU measures how well a predicted box overlaps the true one: 0 = disjoint, 1 = identical.
It decides whether a detection counts as correct, and drives duplicate removal.
Move the prediction and watch how fast IoU drops.

Two Families of Detectors

Two-stage: first propose likely regions, then classify each one.
One-stage: predict boxes densely at every location in a single pass.
Both need a final filtering step.

From R-CNN to Faster R-CNN

R-CNN ran a whole CNN on 2000 crops per image.
Fast R-CNN ran the CNN once and cropped the feature map instead.
Faster R-CNN also learned the proposals: fully end to end.

Region Proposal Network

A small conv network slides over the feature map.
At each location it scores a set of reference boxes (anchors) and refines them.
The best few hundred become proposals for the second stage.

RoI Pooling and RoI Align

Proposals have different sizes, but the second-stage head needs a fixed-size input.
RoI pooling snaps the box to the grid; RoI Align samples it exactly.
The difference matters for masks and small objects.

One-Stage Detection: Predict on a Grid

The final feature map is a grid; each cell is responsible for objects centred in it.
Each cell predicts boxes, confidences, and class scores directly.
One forward pass, no proposals: YOLO, SSD.

Anchors: Predict Corrections, Not Boxes

Each cell has a few reference boxes of typical sizes and aspect ratios.
The network predicts small offsets from the best-matching anchor.
Small, normalized targets are much easier to learn than raw coordinates.

Objectness and the Background Class

Most candidate boxes contain no object.
YOLO adds an objectness score; SSD and RetinaNet add a background class.
Either way, a score threshold removes most candidates.

From Ten Thousand Candidates to a Few Boxes

Dense detectors score every anchor at every location of several maps.
A score threshold keeps the confident ones.
Several boxes still cover the same object — they must be merged.

Non-Maximum Suppression

Take the highest-scoring box and keep it.
Remove all remaining boxes that overlap it with IoU above a threshold.
Repeat until no boxes are left. Press → to run it.

When NMS Removes Real Objects

Neighbouring objects of the same class can overlap strongly.
NMS cannot tell a duplicate from a neighbour — only the IoU threshold decides.
Move the threshold and watch the second cat.

Focal Loss: Too Many Easy Negatives

A dense detector sees thousands of easy background boxes per image.
Their small losses add up and dominate cross-entropy (lecture 1: class imbalance).
Focal loss down-weights examples the model already gets right.

Measuring Detection: Precision, Recall, AP

A detection is a true positive if it matches an unmatched object with IoU ≥ 0.5.
Sweeping the score threshold traces a precision–recall curve.
Average precision (AP) summarizes it; mAP averages over classes.

Check question

Your detector misses 10-pixel defects. Deeper backbone, or a higher-resolution feature map?

Higher resolution: at stride 32 the defect is a fraction of one cell and is averaged away. Depth adds semantics, not detail.

Why Small Objects Get Lost

On a stride-32 map, one cell summarizes 32 × 32 input pixels.
An object smaller than a cell is mixed with its background.
Small objects need fine maps; large objects need wide context.

Detail versus Meaning

Shallow maps are sharp but semantically weak.
Deep maps are semantically strong but coarse.
A neck brings the meaning of deep maps down to the fine levels.

Feature Pyramid Network, Step by Step

Start from the deepest map; upsample it and add the next finer backbone map.
1 × 1 lateral convolutions make the channel counts equal (so they can be added).
The result: a pyramid where every level is both fine and semantic.

Which Level Detects Which Object

Small objects are assigned to P3, large ones to P5.
Each level then only handles a limited range of sizes.
The head is shared: it learns one rule applied at every scale.

PANet and BiFPN

FPN sends information only from deep to fine levels.
PANet adds a bottom-up path so fine details reach the deep levels too.
BiFPN repeats weighted two-way fusion; more fusion is not free — measure it.

Check question

All objects in your data are between 8 and 20 pixels. What would you change in a standard FPN detector?

Add or emphasize a finer level (P2, stride 4), drop P5–P7, and check anchor sizes; also consider a higher input resolution.

Kinds of Segmentation

Semantic: every pixel gets a class; objects of one class merge.
Instance: every object gets its own mask.
Panoptic: both at once — countable objects plus background regions.

Real Segmentation Tasks

Thin cracks on steel: tiny, elongated, extremely imbalanced.
Apples among leaves: occlusion, varying size, similar colours.
The output must be as sharp as the annotation.

The Segmentation Output

The network outputs C logit maps, one per class.
Softmax across channels at each pixel gives class probabilities.
Argmax per pixel gives the label map.

Why the Backbone Alone Is Not Enough

The last backbone map has stride 32: a 331-pixel image becomes 10 × 10 cells.
Upsampled back, the mask is blocky and small objects vanish.
We need to recover resolution — and the detail lost on the way down.

Upsampling

Nearest neighbour and bilinear interpolation have no parameters.
Transposed convolution learns the upsampling but can create checkerboard patterns.
Bilinear resize followed by a convolution is a safe default.

Encoder–Decoder with Skip Connections

The encoder is a backbone; the decoder upsamples step by step.
Skip connections concatenate encoder maps of the same size into the decoder.
This is U-Net (2015) — originally for medical images, now a general pattern.

U-Net Shapes

level	encoder output	decoder input (after concat)	decoder output
1	64 × 256 × 256	64 + 64 = 128 × 256 × 256	64 × 256 × 256
2	128 × 128 × 128	128 + 128 = 256 × 128 × 128	128 × 128 × 128
3	256 × 64 × 64	256 + 256 = 512 × 64 × 64	256 × 64 × 64
4	512 × 32 × 32	512 + 512 = 1024 × 32 × 32	512 × 32 × 32
bottleneck	1024 × 16 × 16	–	upsampled to 512 × 32 × 32
output			1 × 1 conv → C × 256 × 256

Read from the bottom up: each decoder level upsamples, concatenates the encoder map of the same size, then convolves. Padded convolutions keep the sizes aligned.

Instance Segmentation: Mask R-CNN

Detect each object first, then segment inside its box.
A small mask branch is added next to the box head of Faster R-CNN.
RoI Align (Part 4) is what makes these masks accurate.

Measuring Segmentation

Pixel accuracy is misleading when the object is small (lecture 1: imbalance).
IoU and Dice measure the overlap of the predicted and true regions.
Report them per class; for thin structures also check boundaries.

Check question

For the steel billet cracks: semantic or instance segmentation, which decoder, and which metric?

Semantic (one crack class), U-Net-style decoder with skips to keep 1-pixel detail, Dice or IoU of the crack class (maybe with a boundary tolerance) — never pixel accuracy.

Channel Attention: Squeeze and Excitation

Not every channel is useful for every image.
SE summarizes each channel (GAP), computes a weight per channel, and rescales.
Cheap to add to any block; typically +1 point on ImageNet.

Spatial Attention

The same idea for positions: which parts of the map matter?
Pool over channels, convolve, and squash into a 0–1 weight per pixel.
The feature map is multiplied by this mask.

CBAM: What, Then Where

Channel attention first, spatial attention second.
A drop-in module after convolutional blocks.
Attention reweights features — it cannot recover detail that was never resolved.

Self-Attention versus Convolution

A convolution only sees a fixed local neighbourhood.
Self-attention lets every position look at every other, weighted by content.
Global context at a quadratic cost; modern models mix both.

Choosing an Architecture

Decide in this order; each answer narrows the next choice.
Start from the simplest standard model that can produce the output.
Change one block at a time, driven by the errors you observe.

Three Common Situations

Small dataset

pretrained backbone, simple head
freeze first, fine-tune later
split by part / batch / day
better labels beat a fancier neck

Small objects

higher input resolution or tiling
use a fine level (P3 or P2)
avoid early aggressive striding
report recall by object size

Edge deployment

MobileNet / EfficientNet backbone
static shapes, supported operators
measure end-to-end latency on the device
check quantized accuracy per class

Ablations: Which Change Helped?

variant	mAP (all)	recall, objects < 16 px	latency
baseline: ResNet-50, P3–P5	0.61	0.42	18 ms
+ input 640 → 960	0.64	0.58	39 ms
+ P2 level instead	0.63	0.55	24 ms
+ CBAM in the neck	0.62	0.44	20 ms
ResNet-101 backbone	0.62	0.43	29 ms

Illustrative numbers. Change one thing per row, report quality and cost, and look at the metric that matches your problem — here, small-object recall.

Reading a Network Diagram

Annotate every node with C × H × W before trusting a diagram.
For every merge, decide whether it adds or concatenates.
Work along the diagram; press → to check each shape.

Common Design Mistakes

Downsampling too early: the defect is gone before any layer can see it.
A head that does not match the annotation (boxes for amorphous regions, masks for counts).
Copying the biggest model from a paper instead of the simplest that fits.
Benchmarking the network alone, ignoring preprocessing, NMS, and data transfer.
Judging a change by one aggregate number instead of the errors that matter.

Check question

A camera must find hairline cracks on a moving strip at 30 FPS. Sketch the backbone, neck, head, input size, and metric.

Tiles at full resolution, efficient backbone, U-Net-style decoder or P2-level head, Dice / recall on cracks, latency measured end to end on the target hardware.

A Short History

Most building blocks used today appeared between 2012 and 2018.
Later work refines efficiency, scaling, and training — and adds transformers.
Knowing the blocks lets you read new architectures quickly.

What to Remember

Backbone, neck, and head solve different parts of the problem.
Spatial scale decides most localization problems.
Choose by output, object size, budget, and observed errors — not by popularity.

Before the Next Lecture

So far, every output lived in a single image plane.
Next, the course turns to geometry: camera models, multi-view geometry, and conventional visual SLAM.
Learned 3D reconstruction comes later and builds on both: CNN features and SLAM poses.