LambdaLabTM
Artificial Intelligence · Class 10 · Unit 5 · Computer Vision
class 10 aipractical unit⏱️ 9 min read

The Convolution Operator

Every filter you have applied on Instagram or Snapchat is this one operation. It is also how a CNN pulls features out of a picture, so it is worth doing slowly.

Where this is examined
In the syllabus, Image Features & Convolution Operator and Convolution Neural Network are listed under “to be assessed through Practicals” — the four theory marks for this unit come from the lessons before this one. Learn this for the practical, and because a CNN makes no sense without it.

1Image features

Feature
In computer vision and image processing, a feature is a piece of information that is relevant for solving the computational task. Features may be specific structures in the image such as points, edges or objects.

Imagine a security camera image, and six small patches cut out of it. Your job is to find where each patch came from. Some are far easier than others:

  • Flat patches — a stretch of plain wall or sky. Almost impossible: wherever you slide the patch it looks the same, so it could have come from anywhere in that area.
  • Edges — the side of a building. Better: you can find roughly where it belongs, but not exactly, because the pattern is identical all the way along the edge.
  • Corners — where two edges meet. Easiest: move the patch in any direction at all and it looks different, so its position is unique.
The conclusion
Corners are the best features to extract from an image, followed by edges. Flat areas are the worst, because they look the same everywhere.

Finding a jigsaw piece's place

A piece of plain blue sky could go almost anywhere and you leave it till last. A piece with a straight roofline narrows it to one row. A piece with the corner of a window on it goes in exactly one place, first time.

2Convolution

Computers store images as numbers. If you change those numbers, the image changes — that is the basis of all image editing. Convolution is the operation that changes them in an organised way.

Convolution
A simple mathematical operation fundamental to many image processing operators. An image convolution is an element-wise multiplication of the image array and another array called the kernel, followed by a sum. It takes two arrays of numbers and produces a third.
Kernel
A kernel is a matrix which is slid across the image and multiplied with the input, so that the output is enhanced in a certain desirable manner. Each kernel has different values for the different kinds of effect you want to apply.

3Watch it happen

The kernel sits over a small patch, the overlapping numbers are multiplied pair by pair, the products are added up, and that single result is written into the output. Then the kernel moves one step and does it again.

A kernel, sliding

The small grid is the kernel. Watch the feature map build up underneath as it travels across the image.

The kernel sits over a 3 × 3 patch, multiplies the nine overlapping numbers pair by pair, adds them up, and writes the single result into the feature map below.

Edge detect. The flat areas come out near black and the outlines light up. This is what 'extracting features' means — the sky is thrown away and the edges are kept. Notice the feature map is 14 × 14 where the input was 16 × 16: a 3 × 3 kernel cannot hang off the edge, so the output comes out smaller.

Why the output is smaller
The centre of the kernel is lined up with each pixel in turn, so at the very edge of the image part of the kernel would hang off the side. It cannot, so the outermost row and column never get a turn and the output image comes out smaller than the input.
To keep them the same size you extend the edge values out by one before convolving — and while extending, the added pixel values are considered zero.

4What it is for

  1. Convolution is a common tool used for image editing.
  2. It is an element-wise multiplication of an image and a kernel to get the desired output.
  3. In computer vision applications it is used inside a Convolutional Neural Network to extract image features.

5Practice

1
What is a kernel, and what does convolution do with it? (Practice)
2 marks
▶Show the answer

A kernel is a matrix that is slid across the image and multiplied with the input so that the output is enhanced in some desirable way; different kernel values give different effects.
Convolution is the element-wise multiplication of the image array with that kernel, followed by a sum — producing a third array. In computer vision it is used to extract features from images.

2
Which makes a better feature in an image, an edge or a corner, and why? (Practice)
2 marks
▶Show the answer

A corner. Wherever you move a patch containing a corner, it looks different, so its location in the image is unique and easy to pin down. A patch containing only an edge looks the same all the way along that edge, so you can find its approximate position but not its exact one. Flat areas are worst of all, because they look identical everywhere.

6Check yourself

Quick Check

Which makes the best feature to extract from an image?

Quick Check

What is a convolution?

Quick Check

Why is the output of a convolution smaller than the input?

← Previous
🎨 Grayscale and RGB Images
Next →
Layers of a CNN 🧠