Hello, everyone, and welcome to this session in which we treat this modern convolutional neural network
architecture known as the Efficient Nets.
In this efficient paper, the authors proposed a more controlled manner of designing convolutional neural
networks such that it suits our demands in accuracy and speed.
And as you can see in those plots, you see that we could choose suitable parameters such that we could
modify or increase our accuracy while taking note of how this affects the speed.
That said, in this section, we'll see how Minxin Tang and Kwok Lei build the system for automatically
scaling our convolutional neural networks much more efficiently.
Components are commonly developed at a fixed resource budget and then scaled up for better accuracy
if more resources are available.
So with the case of the rest nets we had rest net 34.
So your rest net 34.
Then after we had resonant 50 ref net say 152.
And depending on the kind of setting, we are going to pick this resonant model which will permit us
to run without any problems of latency while maintaining a reasonable accuracy.
So this means that if we are working in a high compute environment, then we could afford to work with
this.
Whereas if we are working a low compute environment, then we would have to work with this model with
fewer conv layers.
Now that said, in this paper, the authors propose a more systematic study of how this model's killing
can be done.
And unlike other methods where we just kill by increasing the depth here, the proposal scaling by increasing
the depth, increasing the width, the number of channels and the resolution that is the size of the
input image.
And so here the proposed new scaling method that uniformly scales all dimensions of this depth with
resolution using a simple yet highly effective compound coefficient.
You could see the results right here.
You see, for example, the rest at 50.
Let's extrapolate.
Let's let's let's let's pick this here.
Although this has more parameters on the rest that 50.
Let's take inside the B four because it has less parameters.
So you see it has fewer parameters than the rest at 50 but is an accuracy that's top one.
Accuracy on the image net is much greater than that of this resonant 50 year.
So we have the efficient net B version, which is about 83% to 1% accuracy.
While this is only at about 76% or one accuracy.
Now in this figure, we see how we have this baseline, some sort of baseline, like in the case of
the rest net, we could say this is rest net 18 and then we have this deeper model, the upscaling.
This could be rest net, say 50.
Now, in this case, they have this baseline.
First of all, this baseline is gotten by carrying out an automatic network architecture search.
So we we get this baseline and the different layers we have for this baseline.
And then note that this baseline has a depth.
As you could see, we ask, yes, depth.
And when we scale deeper, when we do carry out depth scaling, you see we have much more layers added
to this one.
And then when we do with scaling, we increase the number of channels.
So you see we have this smaller channels for the baseline and then the width scaling permits us to increase
this number of channels.
Then also we have the resolution scaling, which has to do with the inputs.
So here we have this input high times width.
And now after current resolution scaling, we see we increase this resolution.
This means that we may work with a base of 224 by 224.
And then after scaling, we may get to say 640 by 640.
Then from here we also have the compound scaling, which is what is used in this paper where we don't
only focus on the width or the depth or the resolution, but we scale all this systematically to achieve
the best possible results while maintaining reasonable speeds.
That said, we could see from this different plots that when you increase like here we have the width
that is the number of channels which is increased.
You notice that as we increase this number of channels, at some point it starts to plateau.
And then when we increase the the depth at some point to start to plateau, then when we also increase
this input size, that's a resolution.
At some point it starts to plateau.
And so this is why the authors propose a technique where we could combine all this such that we get
even better results.
And there we go.
We see the effect of compound scaling.
You see that we have this depth and then our resolution, you see when the depth is one and resolution
is one, we add this blue here, you see we have worse results here.
Whereas when we doubled this depth and then increase resolution by 1.3, you see we have this best results
right here.
That said, we will now dive a bit more deeper and look at this compound coefficient which the spoke
of at the very beginning.
So we go down here and we have this formula right here.
See this formula right here?
All right.
At this equation, which is equation three, where we have this depth, we have this different formulas,
the three formulas, the depth equal alpha times fee, and now these fees are user specified coefficient
that controls how many more resources are available for model scaling.
So this is some sort of scaling coefficient Right here is Phi Phi Phi.
And then here we have Alpha, Beta and Gamma.
Now this is designed such that alpha beta squared gamma squared is approximately equal to and alpha
greater.
On equal one beta is always greater than equal one and gamma always greater than equal one.
So now we are going to carry out a grid search.
So we're going to search for the best values for this alpha, beta and gamma and then fix them.
Obviously, they're constant.
So we're going to fix this and then now start varying for search that we carry out the scaling in a
more systematic manner.
And there we go to carry out the order to find the the values for alpha, beta and gamma, the fixed
fee to be equal one and then the obtain alpha 1.2, beta 1.1 and gamma 1.15.
All of these such that we have this constraint now they didn't fix alpha, beta and gamma as constants
and scale up the baseline network with the different files, as we already explained.
And it's based on this different values of PHI that we obtain the different versions of the efficient
net going from B one to B seven.
Now, before moving on, it's important to take note of this efficient net b zero, which is our baseline
network.
Remember, we have some baseline network which we have seen here.
Let's go this way.
We have this baseline network right here.
This one, this baseline network, which we are going to scale such that we have better results while
working with compute constraints.
Now, that said, let's crow, let's take this off and then scroll down back to this baseline model
which is given just here in this table.
You see we have this baseline model, efficient net B zero.
And then you'll notice first that the resolution is 224 by 224, meaning that we're going to start with
image sizes of 224 by 224 and not that different image sizes could be used for the different models,
although the best or the most adapted resolution for each model should be preferably used.
Now that said, here you see we have a usual conv layer and then we have this MDB conf right here.
Now before getting to the and b conf also note that after carrying out the neural architecture search,
the authors noticed that we could also make use of this five by five kernel or five by five kernel size
filters.
So unlike what we had discussed in previous sessions, this five by five kernel size filters are still
very useful.
Then get into the MDB conf we find here here the say its main building block.
It's is this mobile inverted bottleneck.
So we call the mobile inverted bottleneck which we found in sunlight.
All sunlight all is the mobile net version two paper we had seen already, to which they also add the
squeeze and excitation optimization.
Now in the mobile net version three, the Squeeze and excitation optimization was added.
So here we have basically the mobile net embedded residual block, which we have seen already.
And then if we check out this mobile net v three paper which you can feel free to look at, you would
have this squeeze and excitation.
Write your resume into this.
You see here we have the mobile net version two with bottleneck, we residual, there's residual.
Then our bottleneck as usual.
Here we have this low dimension input getting in and then it gets expanded and then we have this low
dimension output which is produced this final layer right here.
Now with this squeeze and excitation to better understand this squeezing excitation layer, we should
or we could get back to how the conv layers actually work.
You see that to get this output, let's take this off to get this output.
For example, we carry out multiplications and additions for each and every channel.
Your that's for each and every channel, the input and the filters which correspond to this channel.
And then to produce this negative one right here, all this are added up with equal weights.
So the output from this computation, let's call it alpha, will be put here, plus the output from
this computation.
Let's call it beta.
We will put your plus the output from this computation, let's call it gamma will be put here and we'll
get this value or this output of negative one at this position.
Now what the squeeze and excitation layer brings in is some weights on this addition operation right
here.
So instead of just having a weight of one, your one and your one, we're going to have some modified
parameter or some parameters added here such that certain channels influence the output more than some
others.
And so here we could have instead of one we can have a weight E, your alpha with B and your A C getting
back to this paper, the way this is done is as such we start by carrying out some pulling and the result
of this pulling will be one by one by C output.
Now C is the number of channels.
So if you're we have C channels here, we have this output your C, so this output here will be one
by one by C, you notice how this is small and then we have C the size.
The size C is exactly the same size right here.
So we have exactly this size is the same as this.
And therefore the height and width is one by one.
Now, once we get this, we pass this through two fully connected layers, you see with this railway
activation.
And then here we have this hard sigmoid activation after this fully connected layer.
And then here we get this output of this same number of channels C which will match with this one.
But now what we get here will be multiplied by each and every channel here.
So this now will serve as the weights will serve us, because remember, we designed this as a a alpha
plus beta plus gamma, and then we had a, B and then C, So this A, B, and C is actually this output
right here.
We're supposed to see that the channel size equal three.
So we have this three here and these are the values which we get after going through this fully connected
layer.
And then we take this now and multiply by each and every channel.
So if you break this up into three parts, let's let's remove this.
There is this and we could cut this, let's cut this into three parts.
So we have one, two, three.
So we suppose there's one by one by three.
So if we have this, this first part will multiply this chunk.
So we'll multiply this chunk and then this other part will multiply this next chunk.
And then this other part here we multiply this other chunk.
And so now we have this channels whose contribution to this output is now weighted.
That said, we also have the expansion factor.
There is six.
Then getting back to the result, we see how the efficient nets perform better than the corresponding
or other conflicts with similar number of parameters or even more number of parameters.
Like here we see how the efficient net B zero or outperforms the rest.
Net 50 though you see this great difference number of parameters as the efficient net is more efficient
as or has fewer parameters as compared to the rest.
Net 50.
We see rest net efficient at B one compared to rest net 152 C €60 million 7.8.
But this one is more accurate than the rest and 152 you could check out from this right up to efficient
net.
B seven You see we have this G pipe.
They are 9797, but this one has 557 million parameters.
Why?
This is only at 66 million parameters.
And we could also look at the floating point operations You see here we have fewer floating point operations
for the efficient net zero while still having higher accuracy.
We also see that if we scale the mobile nets and the rest nets will still they wouldn't still get better
results compared to the efficient net.
And this shows the power of the network architecture search, which was used in getting our baseline.
Now we'll go down and check out this year.
We have this results right here.
You see the class activation map, which is a visualization technique which permits practitioners understand
how the model or rather what portions of the inputs helped in producing the outputs shows clearly here
that when we use compound killing, we have the map which is more focused on relevant regions, as you
could see right here, as compared to the baseline model and this other models with the deeper with
depth scaling with scaling and resolution scaling.