Convolution Fundamental I

Convolution Fundamental I

Foundations of CNNs

Learning to implement the foundational layers of CNN's (pooling,convolutions) and to stack them properly in a deep network to solve multi-class image classification problems.

Computer vision

Computer vision is from the applications that are rapidly active thanks to deep learning

One of the applications of computer vision that are using deep learning includes:

Self driving cars

Face recognition

Deep learning also is making new arts to be created to in computer vision as we will see.

Rabid changes to computer vison are making new applications that weren't possible a few years ago.

Computer vison deep leraning techniques are always evolving making a new architectures which can help us in other areas other than computer vision

For example, Andrew Ng took some ideas of computer vision and applied it in speech recognition

Examples of a computer vision problems includes:

Image classification

Object detection

Detect object and localize them

Neural style transfer

Changes the style of an image using another image.

On of the challenges of computer vision problem that images can be so large and we want a fast and accurate algorithm to work with that.

For example,a 1000*1000 image will represent 3 million feature/input to the full connected neural network. If the following hidden layer contains 1000, Then we we will want to learn weight of the shape [1000,3 million] which is 3 billion parameter only in the first layer and that's so computationally expensive!

On of the solutions is to build this using convolution layers instead of the fully connected layers.

Edge detection example

The convolution operation is one of the fundamentals blocks of a CNN. One of the examples about convolution is the image edge detection operation.

Early layers of CNN might detect edges then the middle layers will detect parts of objects and the layers will put the these parts together to produce an output

In an image we can detect vertical edges, horizontal edges,or full edge detector

Vertical edge detection

An example of convolution operation to detect vertical edges:

In the last exmple a 6*6 matrix convolved with 3*3 filter/kernel gives us a 4*4 matrix

If you make the convolution operation in TensorFlow you will fin the function tf.nn.conv2d. In keras you will fin conv2d function.

The vertical edge detection filter will find a 3*3 place in an image where there are a bright region followed by a dark region

If we applied this filter to a white region followed by a dark region,it should find the edges in between the two colors as a positive value. But if we applied the same filter to a dark region followed by a white region it will give us negative values. To solve this we can use the abs function to make it positive.

Horizontal edge detection

Filter would be like this

There are a lot of ways we can put number inside the horizontal of vertical edge detections. For example here are the vertical Sobel filter(The idea is taking care of the middle row)

Also something called Scharr filter(The idea is taking great care of the middle row)

What we learned in the deep learning is that we don't need to hand craft these numbers, we can treat them as weights and then learn them. It can learn horizontal, vertical ,angled, or any edge type automatically ranther than getting them by hand.

Padding

In order to use deep neural networks we really need to use paddings

In the last section we saw that a 6*6 matrix convolved with 3*3 filter/kernel gives us a 4*4 matrix.

To give it a general rule, if a matrix n*n is convolved with f*f filter/kernel give us n-f+1,n-f+1 matrix.

The convolution operation shrinks the matrix if f>1

We want to apply convolution operation multiple times, but if the image shrinks we will lose a lot of data on this process. Also the edges pixels are uses less than other pixels in an image.

So the problems with convolutions are:

shrinks output

throwing away a lot of information that are in the edges.

To solve these problems we can pad the input image before convolution by adding some rows and columns to it. We will call the padding amount p the number of row/columns that we will insert in top, bottom , left and right of the image.

In almost all the cases the padding values are zeros

The general rule now, if a matrix n*n is convolved with f*f filter/kernel and padding p give us n+2p-f+1,n+2p-f+1 matrix

If n=6,f=3, and p=1 The n the output image will have n+2p-f+1=6+2-3+1=6. We maintain the size of the image.

Same convolutions is a convolution with a pad so that output size is the same as the input size. Its given by the equation:

In computer vision f is usually odd. Some of the reasons is that its have a center value.

Strided convolution

Strided convolution is another piece that are used in CNNs

We will call stride s

When we are making the convolution operation we used s to tell us the number of pixels we will jump when we are convolving filter/kernel. The last examples we described s was 1

Now the general rule are:

if a matrix n*n is convolved with f*f filter/kernel and padding p and stride s it give us (n+2p-f)/s+1, (n+2p-f)/s+1 matrix

In case (n+2p-f)/s+1 is fraction we can take floor of this value.

In math textbooks the conv operation is filpping the filter before using it. What we were doing is called cross-correlation operation but the state of art of deep learning is using this as conv operation.

Same convolutions is a convolution with a pad so that output size is the same as the input size. Its given by the equation:

Convolution over volumes

We see how convolution works with 2D images, now lets see if we want to convolve 3D iamges(RGB image)

We will convolve an image of height,width,# of channels with a filter of a height, width,same # of cahnnels. Hint hat the image number channels and the filter number of channels are the same.

We can call this as stacked filters for each channel!

Example

input image: 6*6*3

Filter:3*3*3

Result image: 4*4*1

In the last result p=0,s=1

Hint the output here is only 2D

We can use multiple filters to detect multiple features or edges. Example

Input image: 6*6*3

10 Filters: 3*3*3

Result image: 4*4*10

In the last result p=0,s=1

One Layer of a Convolutional Network

First we convolve some filters to a given input and then add a bias to each filter output and then get RELU of the result. Example:

Input iamge: 6*6*3 # a0

10 Filters: 3*3*3 #w1

Result image: 4*4*10 #w1a0

Add b(bias) with 10*1 will get us: 4*4*10 image #w1a0+b

Apply RELU will get us: 4*4*10 image #A1=RELU(w1a0+b)

In the last result p=0,s=1

Hint number of parameters here are; (3*3*3*10)+10=280

The last example forms a layer in the CNN

Hint that no matter how the size of the input, the number of the parameters for the same filter will still the same. That makes it less prune to overfitting

Here are some notation we will use. If layer l is a conv layer:

A simple convolution network example

Lets build a big example.

Input Image are: a0=39*39*3

n0=39 and nc0=3

First layer(Conv layer):

f1=3,s1=1,and p1=0

number of filters=10

Then output are a1=37*37*10

n1=37 and nc1-10

second layer(Conv layer):

f2=5,s2=2,p2=0

number of filters=20

The output are a2=17*17*20

n2=17,nc2=20

Hint shrinking goes much faster because the stride is 2

Third layer(Conv layer):

f3=5,s3=2,p2=0

number of filters=40

The output are a3=7*7*40

n3=7,nc3=40

Forth layer(Fully connected softmax)

a3=7*7*40=1960 as a vector.

In the last example you seen that the image are getting smaller after each layer and that's the tread now.

Typesof layer in a convolutional network:

Convolution. #Conv

Pooling #Pool

Fully connected #FC

Pooling layers

Other than the conv layers,CNNs often uses pooling layers to reduce the size of the inputs, speed up computation, and to make some of the features it detects more robust.

Max pooling example:

This example has f=2,s=2 and p=0 hyperparameters

The max pooling is saying, if the feature is detected anywhere in this filter then keep a high number. But the main reason why people are using pooling because its works well in practice and reduce computations.

Max pooling has no parameters to leran

Example of Max pooling on 3D input:

Input: 4*4*10

Max pooling size=2 and stride=2

output 2*2*10

Average pooling is taking the averages of the values instead of taking the max values

Max pooling is used more often than average pooling in practice.

If stride of pooling equals the size, it will then apply the effect of shrinking.

Hyperparameters summary

f: filter size

s: stride

Padding are rarely uses here

Max or average pooling

Convolutional neural network example

Now we will deal with a full CNN example. This example is something like the LeNet-5 that was invented by Yann Lecun

Input image are: a2=32*32*3

n0=32 and nc0=3

First layer(Conv layer): #Conv1

f1=5,s1=1,and p1=0

number of filters=6

Then output are a1=28*28*6

n1=28,and nc1=6

Then apply(Max pooling): #Pool1

f1p=2 and s1p=2

The output are a1=14*14*16

Second layer(Conv layer):#Conv2

f2=5,s2=1,p2=0

number of filters=16

The output are a2=10*10*16

n2=10,nc2=16

Then apply(Max pooling):#pool2

f1p=2,and s1p=2

The output are a2=5*5*16

Third layer(Fully connected) #FC3

Number of neurous are 120

The output a3=120*1, 400 came from 5*5*16

Forth layer(Full connected) #FC4

Number of neurons are 84

The output a4=84*1

Fifth layer(Softmax)

Number of neurons is 10 if we need to identify for example the 10 digits

Hint a Conv1 and Pool1 is treated as one layer

Some statistics about the last example:

Hyperparameters are a lot. For choosing the value of each you should follow the guideline that we will discuss later or check the literature and takes some ideas and numbers from it.

Usually the input size decrease over layers while the number of filters incerease

A CNN usually consists of one or more convolution(Not just one as the shown examples) folowed by a pooling.

Fully connected layers has the most parameters in the network

To consider using these bolocks together you should look at other working examples firsts to get some intuitions

Why convollutions?

Two main advantages of Convs are:

Parameter sharing.

A feature detector(such as a vertical edge detector) that's useful in one part of the

image is probably useful in another part of the image

sparsity of connection.

In each layer, each output value depends only on a small number of inputs which

makes it translation

Putting it all together

Deep convolutional models: case studies

Learn about the practical tricks and methods used in deep CNNs straight from the research paper.

Why look at case studies?

We learned about Conv layer, pooling layer, and fully connected layers. It turns out that computer vision researchers spent the past few years on how to put these layers together.

To get some intuitions you have to see the examples that has been made.

Some neural networks architecture that works well in some tasks can also work well in other tasks.

Here are some classical CNN networks:

    LeNet-5

    AlexNet

    VGG

The best CNN architecture that won the last ImageNet competition is called ResNet and it has 152 layers!

There are also an architecture called Inception that was made by Google that are very useful and apply to your tasks.

Reading and trying the mentioned models can boost you and give you a lot of ideas to solve your task.

Classic networks

In this section we will talk about classic networks which are LeNet-5,AlexNet, and VGG

LeNet-5

The goal for this model was to identify handwritten digits in a 32*32*1 gray image. Here are the drawing of it:

This model was published in 1998. The last layer wasn't using softmax back then

It has 60K parameters.

The dimensions of the image decreases as the number of channel s increases.

ConvèPoolèConvèPoolèFCèFCèsoftmax this type of arrangement is quite common.

The activation function used in the paper was Sigmoid and Tanh. Modern implementation uses RELU in most of the cases.

[LeCun et al., 1998. Gradient-based learning applied to document recognition]

AlexNet

Named after Alex Krizhevsky who was the first author of this paper. The other authors includes Jeoffery Hinton.

The goal for the model was the ImageNet challenge which classifies images into 1000 classes. Here are the drawing of the model:

Summary:

ConvèMax-poolèConvèMax-poolèConvèConvèConvèMax-poolèFlattenèFCèFCèSoftmax

Similar to LeNet-5 but bigger.

Has 60 Million parameter compared to 60K parameter of LeNet-5

It used the RELU activation function.

The original paper contains Multiple GPUs and Local Response normalization(RN)

Multiple GPUs was used because the GPUs was so fast back then.

Researchers proved that Local Response normalization doesn't help much so far now

don't bother yoursef for understanding or implementing it.

This paper convinced the computer vision researchers that deep learning is so important.

VGG-16

A modification for AlexNet.

Instead of having a lot of hyperparameters lets have some simpler network.

Focus on having only these blocks:

CONV=3*3 filter, s=1, same

MAX-POOL=2*2,s=2

Here are the architecture:

This network is large even by modern standards. It has around 138 million parameters.

Most of the paramters are in the fully connected layers.

It has a total memory of 96MB per image for only forward propagation!

Most memory are in the earlier layers.

Number of filters increases from 64 to 128 to 256 to 512, 512 was made twich.

Pooling was the only one who is responsible for shrinking the dimensions.

There are another version called VGG-19 which is bigger version. But most people used the VGG-16 instead of the VGG-19 because it does the same.

VGG paper is attractive it tries to make some rules regarding using CNNs

Special Netwroks

Residual Networks(ResNets)

Very, very deep NNs are difficult to train because of vanishing and exploding gradients problems.

In this section we will learn about skip connection which makes you take the activation from one layer and suddenly feed it to another layer even much deeper in NN which allows you to train large NNs even with layers greater than 100.

Residual block

ResNets are built out of some Residual blocks.

They add a shorcut/skip connection before the second activation.

The authors of this block find that you can train a deeper NNs using stacking this block.

Residual Network

Are a NN that consists of some Residual blocks.

These networks can go deeper without hurting the performance. In the normal NN –Plain networks- the theory tell us that if we go deeperwe will get a better solution to our problem. but because of the vanishing and exploding gradients problems the performance of the network suffers as it goes deeper. Thanks to Residual Network we can go deeper as we want now.

On the left is the normal NN and on the right are the ResNet. As you can see the performance of RestNet increases as the nwtwork goes deeper.

In some cases going deeper won't effect the performance and that depends on the problem on your hand.

Some people are trying to train 1000 layer now which isn't used in practice

Why ResNets work

Lets see some example that illustrates why resNet work.

We have a big NN as the following:

XàBig NNàa[l]

Lets add two layers to this network as a residual block:

XàBig NNàa[l]àLayer1àLayer2àa[l+2]

And a[l] has a direct connection to a[a+2]

Suppose we are using RELU activations:

Then:

a[l+2]=g(z[l+2]+a[l])=g(w[l+2]a[l+1]+b[l+2]+a[l])

Then if we are using L2 regularization for example, w[l+2] will be zero. Lets say that b[l+2] will be zero too.

Then a[l+2]=g(a[l])=a[l] with no negative values.

This show that identity function is easy for a residual block to learn. And that why if can train deeper NNs.

Also that the two layers we added doesn't hurt the performance of big NN we made.

Hint: dimensions of z[l+2] and a[l] have to be the same in resNets. In case they have different dimension what we put a matrix parameters(Which can be learned or fixed)

a[l+2]=g(z[l+2]+ws*a[l]) #The added Ws should make the dimentions equal

ws also can be a zero padding

Using a skip-connection helps the gradient to backpropagate and thus heps you train deeper networks

Lets take a look at ResNet on images.

Here are the architecture of ResNet-34:

All the 3*3 Conv are same Convs

Keep it simple in design of the network

spatial size/2è #filters*2

No FC layers,No dropout is used

Two main types of blocks are used in a ResNet, depending mainly on whether the input/output dimensions are same of different. You are going to implement both of them.

The dotted lines is the case when the dimensions are different. To solve then they down sample the input by 2 and then pad zeros to match the two dimensions. There's another trick which is called bottleneck which we will explore later.

Useful concept(Spectrum of Depth)

Residual blocks types:

Identity block:

Hint the conv is followed by a batch norm BN before RELU. Dimensions here are the same.

This skip is over 2 layers. The skip connection can jump n connection where n>2

The convolutional block:

The conv can be bottleneck 1*1 conv

Network in Network and 1*1 convolutions

A 1*1 convolution- we also call it Network in Network- is so useful in many CNN models.

What does a 1*1 convolution do? Isn't it just multiplying by a number?

Let's first consider an example:

Input: 6*6*1

Conv:1*1*1 one filter. # The 1*1 Conv

Output: 6*6*1

Another example:

Input:6*6*32

Conv:1*1*32 5 filters. # The 1*1 Conv

Output: 6*6*5

It has been used in a lot of modern CNN implementations likes ResNet and Inception models.

A 1*1 convolution is sueful when:

we want to shrink the number of channels. We also call this feature transformation.

In the second discussed example above we have shrieked the input from 32 to 5

We will later see that by shrinking it can save a lot of computations

If we have specified the number of 1*1 Conv filters to be the same as the input number

of channels then the output will contain the same number of channels. Then 1*1 Conv     will act like a non linearity and will learn non linearity operator.

Replace fully connected layers with 1*1 convolutions as Yann LeCun believes they are the same.

In Convolutional Nets, there is no such thing as "fully-connected layers", There are only

convolution layers with 1*1 convolution kernel and a full connection table.

Inception network motivation

When you design a CNN you have to decide all the layers yourself. Will you pick a 3*3 Conv or 5*5 Conv or maybe a max pooling layer. You have so many choices.

What inception tells us is, Why not use all of them at once?

Inception module, naïve version:

Hint that max-pool are same here.

Input to the inception module are 28*28*192 and the output are 28*28*256

We have done all the Convs and pools we might want and will let the NN learn and decide which it want to use most.

The problem of computational cost in Inception model:

If we have just focused on a 5*5 Conv that we have done in the last example.

There are 32 same filters of 5*5, and the input are 28*28*192

output should be 28*28*32

The total number of multiples needed here are:

Number of output*Filter size*Filter size*Input dimensions

Which equals: 28*28*32*5*5*192=120Mil

120Mil multiply operation still a problem in the modern day computers.

Using a 1*1 convolution we can reduce 120mil to just 12 mil. Lets see how.

Using 1*1 convolution to reduce computational cost:

The new architecture are:

X0 shape is (28,28,192)

We then apply 16(1*1 Convolution)

That produces X1 of shape (28,28,16)

Hint, we have reduced the dimensions here.

Then apply 32(5*5 Convolution)

That produces X2 of shape(28,28,32)

Now lets calculate the number of multiplications:

For the first Conv: 28*28*16*1*1*192=2.5Mil

For the second Conv: 28*28*32*5*5*16=10Mil

So the total number are 12.5Mil approx. Which is so good compared to 120Mil

A 1*1 Conv here is called Bottlenect BN

It turns out that the 1*1 Conv won't hurt the performance.

Inception module,dimensions reduction version:

Example of inception model in Keras:

Inception network(GoodNet)

The inception network consist of concatenated blocks of the Inception module.

The name inception was taken from a name image which was taken from Inception movie

There are the full model:

Some times a Max-pool block is used before the inception module to reduce the dimensions of the inputs.

There are a 3 Sofmax branches at different positions to push the network toward its goal. and helps to ensure that the intermediate features are good enough to the network to learn and it turns out that softmax0 and softmax1 gives regularization effect.

Since the development of the inception module, the authors and the others have built another versions of this network. Like inception v2,v3 and v4. Also there are a network that has used the inception module and the ResNet together.