Hello, everyone, and welcome to this new and exciting session in which we are going to look at other
state of the art convolutional neural network based model.
And talking about state of the art models.
Ten years ago in the image Net, visual recognition challenged the Alex Net Convolutional neural network
beat all state of the art solutions previously or before this Alex net solution state of the art methods
achieved the top error rate of our top 5% error rate of 35 25.3%.
But with Alex net we dropped this error rate to 15.3%.
Now this is 25.2.
This breakthrough has led to a widespread adoption of convolutional neural networks in solving recognition
tasks like this one.
And although today we would hardly use this kind of model, that's to say the Alex Net model, we are
going to discuss this model because it was a precursor to most of the modern components we have today,
like the mobile nets, dense nets and efficient nets.
That said, we are going to see what makes our what made the Alex Net model so powerful.
The Alex Net model was first published in this paper entitled Image Net Classification with deep convolutional
neural networks.
Just from the title, you could get some idea that this was one of the first times.
Conf nets were used for this image net challenge.
This paper was by Alex Krinsky, Elias Sekhar and Jeffrey E Hinton.
In the abstract Do start by presenting the results.
As you could see here on test data, they achieved top one and top five error rates of 37.5% and 17%,
which is considerably better than the previous state of the art.
Now this model was 60 million parameter model composed of comb layers and max pulling layers with some
final three fully connected layers, as we shall see in the model under the model section shortly.
Now, you're 1000 ways off, Max, because we have 1000 classes.
They also try to reduce overfitting by implementing strategies like drop out and data augmentation.
That said, you are to discuss the data set which was used.
Obviously the image net, which is 15 million labeled high resolution image dataset, although the finally
used roughly 1.2 million training images, 50,000 validation and 150,000 for testing, then it's also
important to note that the images which we used for training were downsampled to 256 by 256 images.
As for the overall architecture, as we could see here, we have the Conv layers followed by max pulling
layers.
Sometimes you see your comp layer max pulling conf layer max pooling.
Then we have several conf layers, this max pooling layer and then we have three dense layers.
You see here we have this dance layer, this dense layer and this final dense layer with our 2000 way
output.
Then another point to note here is given that at a time many times the non linearity use was the trench
or the sigmoid.
Let's get back to this top here.
You see here to talk about the non linearity which was used just here.
The previously were the mostly used Tange or the sigmoid as the seer.
But it turns out that after working with the rail, Lou, the rail, if you can recall, we've seen
this in the previous section, the yellow is simply this function which takes in a value x, and then
if x is negative, the value is zero.
If x is positive, it remains the same value.
So basically we have x f of x f of x here.
Which is our review function, which is zero.
If X is less than zero and it is x if x is greater than or equals zero.
So this are really function right here.
And what the discovered was that the revenue permitted them to train their model much faster than this
previously used nonlinearity is like the tank of x.
So as you could see here, just after a few epochs, just after, let's say five epochs, the attain
this training error rate as compared to this other non linearity here.
So this is what we get when we use the rail, Lou.
And this is what we get when we use some other nonlinearity like the X and it's important to note that
till date most conf nets we build make use of this ratio nonlinearity.
Another thing that did to speed up the training was to use multiple GPUs.
Then the actual number of GPUs they use here is two and the device a method of communication between
these two GPUs to speed up calculation.
From here, the authors make use of this normalization strategy for regularization known as the local
response normalization, and this normalization strategy was used alongside the low nonlinearity.
So from here, we have some inputs.
Let's take this schematic from this post by Akhil and Wah, where it shows this even more clearly.
Here we have some input and then we normalize it based on its surroundings, hence the term local response
normalization here he explains that there is inter channel, local response normalization, and there
is intra channel local response normalization.
As you can see, this is all between pixels of a given channel or neurons of a given channel.
And here this is inter channel.
So this is carry out between pixels of different channels.
Now, that said, the exact mathematical formula used here is this one.
So here we have a given neuron and then we divide as value by this summation right here, which is the
square of some neighboring values.
And according to the author's terms here, you do see this sort of response.
Normalization implements a form of lateral inhibition.
So take note of this and it's inspired by the type found in real neurons creating competition for big
activities amongst neuron outputs computed using different channels.
So this means that if we consider this three neighboring channels from or rather the three neighboring
neurons from those three different channels, if we take this particular neuron right here and we try
to normalize it or perceptual normalization layer, given that it is surrounded by this pixel was value
is relatively high because of this squared term right here and this is the squared, right?
You're meaning that you take this value C one divided sum by sum summation and then you have this alpha,
obviously you have this K right here.
Let's admit that.
Let's just put it right here.
We have this alpha and then we have this value squared.
So obviously this would be a function of basically this will be this value here.
So when you square this value, it means that this overall value here will become very small, hence
the term lateral inhibition.
And so for a neuron to maintain a relatively high value after going through this local response normalization
layer right here, it has to ensure that it has one of the highest values among the surrounding neurons.
Nonetheless, this local response normalization as compared to other normalization techniques like the
batch normalization layer normalization and the group normalization hasn't proven to be very effective
when it comes to regularizing a neural network, and it's not used by modern conflicts.
We had seen in the previous sessions that the pulling layers permit us down sample information from
the inputs such that as we go deeper in the neural network, we have a reduced.
Number of features now in this paper that make use of this pulling layer, more specifically the max
pull layer.
And the way it works is quite straightforward.
So we're supposing we have this kind of input and then we have a three by three max per layer with initially
straight of one.
What we're going to have here is we have this positions which we are going to fix here.
Let's get back and then pick some values.
So let's say we have a value of one, two and then all this other values.
If we want to carry out the max pull operation with a straight of one, we'll start with this year.
See?
And because it is Max pull we have, we're going to pick the max of all this.
So we have the highest values, 11.
And so here we're going to have 11.
And then the next thing we'll do is we are going to shift this.
So we shift this here.
Since the start of one, we're going to go one step to the right.
And so we have this now here.
Notice how we still pick out three by three pixels.
Let's now take this one off.
So we've done the shift and now we have this position.
We take the max here.
The max here again is going to give us 11.
So we have 11 year and then we're going to do another shift.
So from here we're going to take this year, let's take this off.
We do this other shift and then we still have this.
The max here again is going to be 11.
So you see at the top we have 11, 11, 11.
And then from here, we'll move on to this next one.
So we will go downward, one step downward.
We'll have this here.
And the max here is going to be year 11 still.
So we're going to have 11 here and there will move this way.
This way.
We'll go downward and all of that.
So if we move this way, I think we should have 11 still the other way.
11 we go downward.
We still have 11 practically.
We will have 11 everywhere.
So we'll have 11 and your 11.
So this is going to be our output from this input year after the max pull operation.
Now, when we modify this strike number from 1 to 2, take this number from 1 to 2 as it was illustrated
in the paper, instead of having or instead of moving through one step, we move to two steps.
So if we take this off here, you'll see that we'll start with this.
So from the first one, we're going to get 11.
So this was try to equal two.
So the first one we get 11.
And then from here, instead of moving just one step, like previously, we had this, here we have
this year and then we move one step.
Now we're going to move two steps.
So we'll move this way.
And then again we have 11.
And then instead of going one step downward, we're going to go two steps downward.
And so we'll end up here.
And then we have a maximum of 11, and then we'll go two steps again this way.
So let's take this one off.
Take this one off.
We go two steps again, and then we get a maximum of 11 right here.
So as you could see here, when it talk of overlapping pulling, they actually use a stride of two,
just as we have described.
And then the found this to give them or to give an improvement in the results, though these improvements
aren't very much so in practice we generally use the classical max pulling with as equal one does stride
number equal one, and we also use two by two can our size.
So instead of using three by three kind of sizes, as you see here most times or in modern coordinates,
generally use two by two pull in size.
Now, getting back to the general architecture, we could see here that this very first coordinate has
a cannot size of 11 by 11.
And although these kinds of canals permit the network capture much larger spatial context, we'll see
that they are computationally much more expensive compared to these canals with smaller filter size.
And as we'll see in subsequent sections, the companies developed after this didn't use this kind of
large kennel sizes as they were able to make use of these kinds of smaller filters to still capture
this large spatial context.
The 11 by 11 filters capture then to overcome overfitting.
The others make use of data augmentation and the drop out technique.
So you could check out on our previous sessions where we talk about this to different techniques.
Now, that said, you would see your the training details.
And then one very interesting advantage of working with the 11 by 11 kennel size filters is the fact
that we could have visualizations like this.
So because those kennel sizes are large enough, we could visualize them in this manner.
And then clearly from here we see how our conv layer captures these kinds of low level features.
Like here we have a slanted line.
Here we have many slanted lines, We have this vertical line, we have this horizontal lines right here,
and then we have this this checkerboard pattern.
We have this colors, sometimes dual, sometimes single color.
And so we see how this first conv layers parameters capture low level features in this section that
discuss this record breaking results.
As you could see, we have this top one error rate, which for now or at that time was about 45.7%.
And then with this curve net model, this was dropped to 37.5%.
Then for the top five, we have dropped from 25.7 to 17%.
Now, they also developed this other variant, which comes with an even better top five error rate of
15.3%.
And so as you could see here, we're moving from this previous method that is a C plus F these, which
had 26.2 top 5% error rate to the CNN, which has 15.3% error rate.
Then in this section on qualitative results, we see the different inputs, the correct levels, and
then what the model predicts are the top five best predictions.
See, the model does well here.
Those were all your correct correct prediction.
Yours wrong.
You see, it predicts convertible wins actually agree all year.
It does this wrongly here it's also wrong.
But unlike your this level doesn't even occur among the top five best predictions.
And so that's it for this breakthrough model.
We are going to look at other Corvette models in the next sections.