Hello, everyone, and welcome to this new and exciting session in which we are going to discuss the
Resnick model.
This model was first introduced in this paper entitled Deep Residual Learning for Image Recognition
by coming here and up till date.
That's about seven years later.
This model is still greatly used and the high performance is gotten when working with the rest net model
come due to the fact that the rest of model relies on this residual block right here, which permits
us get even better error rates as we could see here, as compared to the VG and Google Net models.
In this section, we are going to focus on understanding how this residual block works and how the resonant
model is constructed based off this model.
This curves right here depict the limit of models like the VG, which are just based on stacking up
conv layers to best understand this plot.
Recall that with the Alex net we had fewer number of layers.
So we we started out with Alex net fewer number of layers, and then we moved on to VG where we had
the VG 16 version, then the VG 19 version.
And then we expect that if we keep increasing this number of layers, then the error rates should be
dropping.
But what happens actually is the opposite.
So what goes on here is you see for this 20 layer network, we have a lower error rate as compared to
this 56 layer network.
So we expect that this instead should be lower than this because this will stack and or stack more conv
layers, but that's instead the opposite.
Now this same phenomenon is witness with the test set.
So here there's a test and there's a train.
So for the test tool, we have the 20 layer performing better than the 56 layer.
And so it's clear that just blindly stacking up conv layers wouldn't help in making this in dropping
the training and test errors even though they are more expensive.
And this is why the Resnick model introduces this residual learning which is based off the residual
block which we've seen already.
Here is the residual block.
Now note that this wet layer your others with layers, you are simply convolutional layers.
And so now unlike before where we'll just stack this year let's let's let's call this w LX for weight
layer we stack this weight layer and then we'll stack this other one and then we'll keep stacking up
that way.
We'll just keep stacking up like this.
Now what we do is we create a connection between the input and the output.
So we create this connection and then you're, there's some addition.
So we get this output and we add it with this other output right here to produce now this new output.
So here, if we suppose that this input is X, and then what goes on in here is F of X, So let's change
this to read.
What goes on in here is f of x, let's this.
F of X here we have this f of x, then our output can be given us h of x, which is simply equal f of
x plus x.
That's the input plus its output.
Now to produce this new output h of x.
But again, to better understand why we need this residual block right here, we need to first understand
why models which are based on just stacking up the conveyors like the VG actually under feet, even
when you add up or when you increase the number of layers.
Now the reason for that is exploding and or vanishing gradients.
So let's explain what it means for gradients to be vanishing.
Now recall that in the gradient descent process we have a weight.
And this weight, the weight is weight is updated is such that we take its previous value.
So we have the previous value of the weight minus the learning rates here, let's call it our LR generally
denoted alpha minus this learning rate times the partial derivative of the loss with respect to that
given weight.
Now, during the training process, in order to compute this partial derivative right here very efficiently,
the method used is back propagation, or one of the most common methods used is that of back propagation.
Now, the way back propagation works is that you have, say, this model, let's call this model M,
and then you have some input right here with the output.
Obviously you have what the model outputs and let's, let's call the model output Y cup and then Y.
That's not the model we have Y right here, which is what the model is expected to output.
Now it's this difference that produces the loss, and we're finding that partial derivative of this
difference with respect to each and every weight which makes up this model.
Then if we split this up into different layers, let's just say we have different layers like this.
Note that each layer is composed of several weights, but let's, let's say we just put this up like
this.
Now, the layers, this layer has its own weights.
But one point to note is that during the back propagation process, to obtain the partial derivative
of the loss with respect to this weight, here we make use of the partial derivative of the loss with
respect to weights, which come after the layer, which come after this layer right here.
So we to get this year, we will need this different partial derivative to get let's, let's get back
let's get back to get this for example, to get the partial derivative with respect to this weight here
we will need this others right here.
Now, the problem is if we have this year and and also before before going to explain the problem with
this is that we need to understand that to get for example, let's take this one to get, for example,
the partial derivative of the loss with respect to this different weights.
There are many weights in this layer.
Let's say weight for this layer is actually equal.
Some values, let's say alpha one times whatever value times of this partial derivative of the loss
with respect to this weights here.
So let's consider that this is the seat layer.
One, two, three, four, five, six, then L six.
Here we have WL seven, layer seven.
So because we're multiplying here, it means that if a while getting this partial derivative, we obtain
a value very close to zero.
If we get a value very close to zero say 0.000001, for example, it means that it's going to affect
this other partial derivative in the sense that this tool will be a very small value.
And if this partial derivatives are too small, then we will not get a change in this weight because
you have the new way you're trying to get being equal to previous weight minus a very small value here.
And so there will be no there will be little or no changes in the weights.
And that is why even though you're you keep increasing this number of layers, let's take this off.
Even though we keep increasing this number of layers, we cannot achieve better performances due to
this vanishing ingredient problem, as the model is now finding it difficult to update its weights such
that the training error can be decreased since the gradients right here are vanishing.
That is getting towards zero.
So now we've just seen that making our network deeper or increasing the number of layers makes it difficult
to propagate information from one far end to the other end.
And so what the authors suppose is that if the added layers can be constructed as identity mappings,
a deeper model should have training error no greater than its shallower counterpart.
So this means that you have this model, let's suppose this model and then this is the shallower model
and then this is the deeper model right here.
Are we saying that if we construct this deeper model such that it's identical to the shallower model?
So basically it's the same as this here changes in blue.
So we have this same shallower model and then the remaining layers here, this other layers here are
constructed such that this is the identity function of a group of identity functions which are stacked
together.
Then the training error of this one, although deeper, shouldn't be greater than this training error
right here or the training error of this shallower model.
And so this means that if we want to pass information from this point to this point right here, and
that those weights are the values in your damping, this information, then there is this path right
here which permits us.
Simply copy this input information to the output.
And obviously this is simply the identity function.
And so here, if after passing through, say, 20 layers and we get to this point, so let's consider
each and every one of these is a single layer.
So we've gone through 20 layers and then we've gotten to this point where the values we get here are
almost zero, such that when this information passes is going to be also or practically zero.
Then there is this path which at least restores this exact same input we have here.
And so this means that just as the authors of the papers supposed in this example we took here, if
we make our model or neural network deeper by adding these residual blocks, then there will be no increase
in the error rate.
And in practice, this instead leads to a decrease in the error rate, which is exactly what we want.
And one other argument which accounts for the fact that this residual blocks help in improving the performance
of the model is the fact that since we have several parts, this residual model now looks like a combination
of several shallow models.
So it looks like you're combining different.
Let's let's draw it better.
Let's get back.
It looks like we're combining this shallow model here with this other shallow model with this other
shallow model and then producing what we call an ensemble of models or an ensemble of shallow models,
to be more precise, which help in making the overall model much more performance as compared to when
we just have a single path.
So another way you can look at this is that for this, let's take this first shallow model.
You could have information which passes this way and then gets here and then goes this way.
So that's the first model.
And then another time you could have information which goes this way or goes this way, goes this way,
and you have this other model.
You could have another situation where the model goes this way, it goes this way, and then goes just
straight and giving us this other model later in this paper entitled Visualizing the Lost Landscape
of Neural Nets, Li Al produced this visualization here, which shows.
Resonate without SKIP connections and originate with Skip connections.
And it shows how easy it is or how easy it is for the waits to find their way or to get the optimal
weights which minimize the loss as compared to when there are no skip connections here.
It should also be noted that this addition we have here is an element wise addition, and so we have
to ensure that the dimensions of this input should match the dimensions of the output we have here for
this operation to be valid.
Now, in the cases where on the case where these two aren't equal, then we need to do some modifications
in this skip connection right here.
Now, when we look at this three models compared, we have this VG 19, we have this 34 layer plane
or convolution network.
Then we have this 34 layer residual network.
We find that we have this skip connections.
That's our residual block.
So this is our residual block right here.
We stack this residual blocks Now instead of just stacking the conf layers as we did with the Vgs,
now we stack the residual blocks and then sometimes you could see here, sometimes this line is solid,
sometimes it is dotted.
Now when it's data like this, it simply because there's going to be a change in the dimension.
So if you notice here, you find that every time we have this dotted lines, you find that there is
a change here, the number of channels.
So here you have 64 channels and then here you have a 128.
And so since we get in this 64 channel input and we want to match this with 128 output, we get in here,
then we need to do some adjustments here.
Now, as with senior or as we can see in the paper here, they actually two ways of making this adjustments.
The first way is this a the next one, this is B four.
The A the shortcut still perform identity mapping with extra zero entries padded for increasing the
dimensions.
So to get from 64 to 128, we add this extra zero entries.
Then either we do that or we take the B.
That's the projection shortcut which was presented in equation two is used to match the dimensions,
but this is actually done using one by one convolutions.
Now to better understand how the one by one convolutions work, let's take this example where we have
this input size ten by ten cannot size now since it's one by one, then the kernel size is one.
So this is we have just this one wait right here.
And then you see you just go through each and every pixel value.
So here we have this and then notice that the input is the same shape as the output.
And so if, for example, you have this input made of two channels, let's add a second channel.
Let's suppose this input has two channels.
So we have this one channel and this other channel right here then to obtain an output of say four channels.
So we have this input to channels.
So it's ten by ten by two, two channels.
And if we want to have this output to get to four channels, all we need to do now is just make use
of the one by one convolution and then we add the all.
We have four of these different weights or this four for all the different kernels.
Since it's obviously one by one.
If we three by three, then we would have something like this would have three of the four of this.
Now it's one by one.
We have this here for of this.
And then in that case, this one will give this output, then this other one will give this other channel
here.
This order.
One Let's change the color.
This other one will give another channel which will just add your.
Then this other one your stick that to be green will produce this other channel.
So that's how we can move from these two channels to four channels.
And so in the case of the rest nets where we want the inputs, again, some inputs, let's say 64 channel
input, and we want that at the end.
All we want, that this what we have here should match with this output, which is already 128 or 128
year.
Then we need we could add this one by one convolutions with a certain number of filters such that we
could get this desired number of channels here.
So here now we can have 128 one by one filters, and then this now will match up with this output right
here such that we could carry out the element wise addition.
And now the difference with the VG and other previous conf nets is that instead of making use of the
max pool, as you will have here, you see this pooling layer with pool size two.
What we do here is we use our three by three convolutional layer, but we use stripes.
So we specify the striped number of two and this permits us to down sample this feature maps right here.
Again, we could check this out here.
Let's suppose we have this three.
And if we were to do or if we want to downsample this, what we could do is increase number of stripes.
Let's take that to two.
And you see, this is going to be downsampled.
And if we take the pattern to one, you find that the output is half of what we have here.
Then the others also make use of augmentation and then batch normalization where they apply this batch
normalization layer right after each convolution and before the radio activation.
Now, batch normalization is this technique for accelerating deep neural network training by reducing
internal covariate shift.
To better understand marginalization.
We'll start by explaining the notion of covariate shift.
To better understand the notion of covariate shift.
Let's suppose that we're trying to build a model which classifies or which says whether an input image
like this one or say this one is that of a car.
So I call or it is not a car.
Now, if you build in this kind of system and then you start by or you create batches of these kinds
of toy cars and you pass through the model and model learns how to see this and know that it's a car
and see some other image and knows that that's not a car.
Then later on, when you take a car from this order distribution and you pass into our system, it becomes
difficult for the weights of this model to adapt to this change.
In distribution, though, the inputs are our cars.
And to visualize this, let's consider this plot right here.
What we're going to have is something like this.
So we'll have this year for car and then this for not car, and then we'll build this classifier or
this model which distinguishes a car and an image, which is not a car by, say, this function, for
example.
Now, when you bring some other distribution, like say, this distribution, you would have something
like this.
You see you have something like this here, this other distribution and not call this a not car is about
this.
And then you would need to have something like this to separate the cars from images which are not cars.
And this then makes it difficult for us to have a function which separates images, which are cars from
those which are not cars when these images come from this to different distribution.
Now, this shift here is known as the covariate shift.
And so that's why most times before passing the image into the model, what we do is we normalize this
input.
So let's suppose we have an input X, We generally carry out some normalization in order to account
for this covariate shift.
So now after normalization, what we're going to have is that all those images be from this distribution
or this order distribution right here will now have been normalized to reduce the effect of the shift.
And so now we could have our single or could have this our function which separates the cars from the
non cars and with much more ease.
Now, that said, what if this kind of covariate shift instead happens in the hidden layers?
That's these layers which make up the model right here.
So let's suppose that we have some confidence like this stacked with the activation functions and then
we have this weights, does this parameters which make up which are part of the layer now coming from
different distributions.
Then in this case we have an internal covariate shift.
And to remedy the situation, we now make use of the batch normalization and the algorithm for the batch
normalization is described in the paper.
So here we have a mini batch and we obtain its mean.
So here we try to obtain the average value of the different weights.
Then we also obtain the standard deviation, which is sigma and the variance which is sigma squared.
So basically we obtain the mean and we obtain the variance.
And it's this that we make use now to normalize our data.
So now you take every weight you subtract by the mean that's year, which is calculate a year, and
then you divide by this standard deviation.
And then we add the small epsilon to avoid having a very small number or zero at the denominator.
So that's it.
This is how the normalization or the batch normalization process goes on.
And it should be noted that there are other normalization techniques like the layer and group normalization,
which are kind of similar to this, but different in a sense that with a batch normalization, this
mean is calculated over a given mini batch.
So like, this is the mean of values calculated in the mean batch and the standard deviation from that
mini batch.
Now, after getting this new value of x x Chapo, what we now do is we multiply it by gamma and add
better.
Now this gamma.
And better actually trainable parameters.
So when working with batch normalization in, say, TensorFlow or PyTorch, you'll notice that the batch
layer will also have its parameters.
Now, you're the role of this gamma, and this better is to scale and shift.
And these parameters are learned along with the original model parameters and restore the representation
power of the network.
And so when we set Gamma to be the square root of the variance and beta to be this, to be the expectancy
or the mean of X, then we could recover the original activations if that were the optimal thing to
do.
So essentially what this saying here is if it's instead optimal for us not to use the batch normalization,
then we could adapt the value of gamma and better such that we get this original value of extra year.
And the way this can be done is quite simple.
All we need to do here is multiply this X by, let's say, gamma squared plus epsilon.
So we have gamma squared plus epsilon.
And then once we've multiplied this, you see when you take this year and multiply by this, you set
this to like this, do this, this cancels out with this and you're left with x I minus the mean.
Now when you live with XY minus the mean, if beta is equal to mean, then you will have x i minus the
mean.
Plus bitter, which in this case is the mean and you see it comes out and gives you x I which is this
original value of x.
So if we set our beta to be this our rather our gamma two with this and our beta to be the mean, then
in that case we drive our original value of x.
And so that's why you see these two parameters are trainable, such that we get the best values or the
most optimal values for gamma and beta.
Then there's also this initialization that's the model or the the network is trained from scratch.
Stochastic gradient descent is used with a mini batch size of 256 learning results from 0.1 and is divided
by ten when the error plateaus.
So basically when we get to the point where let's have this, when we get to a point where the arrow
starts to arrow starts to plateau, then at that point we could update the learning rate from 0.1 to
0.01.
And then if it plateaus, if it drops, you see any plateaus.
Let's go this way.
It drops and plateaus again.
Then we carry out this same computation.
Now, that said, is four over 60 times ten to the four iterations with decay and momentum I used and
there is no use of drop out.
So again, in testing here we have different skills which are used and averaged.
So we pass the image at these different scales and then the average value of the average scores recorded.
Now, before we move forward to check out some results, it's important to note here that after this
last Conv layer, we do not carry out flooding.
Instead, what is done here is average pooling in order to better understand how the global average
pooling works.
Let's consider this example from politico.com.
So right here, we're supposing that we have this as the output of the final layer.
Then what we do is instead of just flooding and that's just picking all those values and flattening
them out and then passing through a fully connected layer.
What we are going to do is depending on the number of neurons we want, the next fully connected layer,
we are going to create a certain number of channels.
So here, for example, we have this depth here.
The number of channels here is three.
We have the height and we have the width.
And then since we know what we're actually doing, global average pulling, then for each and every
one of these channels, for this channel, this channel and this channel, we're going to get the average
value.
So you have the average value.
Is this eight?
Yeah, the average value is three.
You're the average value is this five.
And so if you want a 1000 of this, then you should have a depth of 1000 right here And now with this
you see that it looks quite similar to the flood and layer as now you have all these different values.
There's different single values which cannot be passed into a fully connected layer.
Now, it should be noted that in certain tasks like in classification, which we are trying to do,
this global average pooling will be great as the position of the pixels don't really matter.
Now the simply means that you're since you get this average and you get this average, you get this
average, it means that this position of this pixel wouldn't be close to this other pixel, as in the
case of the flattening.
But since like in our case, we're interested in saying whether a person is angry, happy or sad, the
positions of this output values right here won't really matter as much as would would have mattered
in the case where we're dealing with an object detection or say, object counting problem, where the
particular position of the person or of whatever we are trying to detect actually counts.
So the better.
Explain this again, let's consider this two examples.
Example one example two right here for classification problems.
All we're interested in is in detecting that this person is happy.
So whether we have a face this way or this way, the position doesn't really matter.
All we're interested in is in knowing whether this person is happy, angry or sad.
Now, for object detection, the exact position of the person matters.
And so the exact.
Position of this neurons year will matter.
And so employing global average pulling for such tasks isn't a great idea.
So in summary, if you have a task where the position doesn't matter that much, then you could use
a global average pulling.
If not, then your advice to use the flatten layer from here you could see the different variants of
the rest nets.
You see here we have the 18 layer rest net as a rest net 18, the rest net 34 rest net 50 rest net 101
rest net 152.
And here we could see that with the plane networks, that's without the rest residual block you find
this 18 layer performing better than the 34 layer.
But once we have the rest net block, you find the 34 layer performing are better than the 18 layer,
meaning that we could now go deeper.
We also have this table right here, which shows the VG model, the Google Net, the glue unit, and
then the different plane networks and the residual networks shows clear that the rest net 150 performs
best regardless of the fact that it is deeper than the 101 and 50 counterparts.
And before we move on, also note that you're this rest net block, as you can see here, is composed
of these two conv layers for the 34 layer, while from this 50 to 150 layers, the resonant blocks are
composed of three layers or three conv layers, as you could see right here.