In this lecture we are going to introduce a new type of layer which has been shown to be very helpful
in training convolution or known that works here's something to consider.
Recall that early on we looked at why it's important to normalize your data before passing it into a
model such as linear regression or logistic regression.
We do this by subtracting the mean and dividing by the standard deviation but because this is an operation
on the data we can only do it at the very first layer of the neural network.
What happens if the data becomes not normalized at some later layer then that layer has to deal with
UN normalized data which is not as good as normalized data.
So this begs the question how can we make the data at every layer normalized.
The answer is batch normalization.
If you recall inside the PI towards training loop we're actually doing Batch gradient descent on the
outer loop.
We loop through each epoch and on the inner loop we look at a chunk of data on each step and do gradient
descent with respect to just this chunk of data.
This works because we know that a batch of data will have approximately the same statistics as the full
dataset.
So what if we did something like this.
We inject a layer that does normalization for us for example in an n n instead of dense dense dense
we have batch of dense batch form dense that you know I'm dense at each of these layers.
All it does is take the batch of data subtract is mean and divide by its standard deviation.
In fact this is pretty much what black normalization does
but there's something extra that batch normalization does.
Here's another question to consider.
How do we know that normalization is actually good.
In fact we don't.
Perhaps there's a better amount of shifting and a better amount of scaling that we can use that leads
to a more optimal model.
And so this is the full description of the batch normalization layer it first does normalization using
the batch mean and the batch standard deviation.
Then it re scales the data and re shifts the data so that the final output has an optimal location and
an optimal scale.
Of course these new parameters gamma and beta are learned automatically using a gradient descent.
One alternative perspective on bash normalization is that it acts as a regular riser.
As you know we use regularization to prevent over fitting with methods such as drop out.
So how does batch normalization do regularization.
The idea is because each batch of data is slightly different.
It's going to have a different mean and standard deviation.
It's not the true mean and standard deviation of the entire dataset these differences are like noise
which slightly change what the known that work sees on each iteration.
In other words because they're known that work always has to learn noisy images it becomes impervious
to the noise
the final thing I want to mention in this lecture is that while we discussed the batch norm in the context
of a field for neuron that work maximum is not commonly applied in between dense layers as you recall.
This is the CNN section of the course.
And so the reason we're talking about batch normalization now is because batch norm is mostly used with
convolutions now adding national arm also adds some hyper parameter choices.
In particular you have to decide whether or not you should use nationalism and if you decide you should
then you have to decide where to put it.
Luckily we can just follow the conventions in this regard factional players are usually applied after
convolution layers well after or before depends on which way you look at it.
So maybe a better way of looking at it is that they go in between convolution layers.
One common pattern would be like this convolution to bash an arm to convolution to back storm back to
convolution back to back storm and then flatten and then your dense layers.
So that's just to give you an idea of what it might look like.
You're always encouraged to try things on your own but also to read about the most popular CNN such
as Fiji resident and inception to see how they're doing things.
This is the approach we're going to take in the next lecture when we improve our CFR 10 results.
By the way if you want to read more about batch form you can check out the paper included an extra reading
that T T.
The title is back to normalization accelerating deep network training by reducing internal covariance
shift.