Instructor: Hello and welcome to this tutorial.
All right,
so we are gonna start with the first file of our model
and the most important file, that's Model P. Y.
And this is in this file that we will implement
the brain of the whole model.
You know, the brain at the heart of the A3C model.
So that's in this file that we will make the neural network
which will of course contain some
convolution neural networks
because of course we're still doing some
deep reinforcement learning.
So our AI will still have eyes.
And inside this neural network,
we will integrate everything
that is related to the Active Create Mode.
And there is a bonus,
as I told you,
we are implementing one of the most powerful A3C models.
And what makes it that powerful
is that it will contain a Rec Renewal Network
and more precisely an LSTM, long short term memory,
so that we can learn the
temporal properties of what's going on in the game.
That is actually the
temporal properties of the input
so that the predictions can be even better.
So there we go.
We are implementing a very powerful model
that combines basically all the neural networks
that we saw in the deep learning course
that is a Artificial Renewal Network,
a Convolution Neural Network,
and a Record Renewal Network.
And at the heart of all these networks,
there is of course the A3C model
that will make the AI very powerful.
So let's do this.
Let's attack this model and implement it.
So we're gonna start by making two functions,
that are just some functions that will take care
of how we can initialize the weights
because you know we're gonna have some neural networks
and therefore we're gonna have weights.
And we just want to make these two functions first
so that we already have a tool
to integrate very easily inside the whole model,
the neural network.
So these two functions are gonna be
normalized columns initializer,
that is basically a function that can,
not only initialize some weights,
but set a specific variance of a tensor of weights.
So that's exactly what we're about to implement right now.
And then we will implement a second function
which will be the weights in it function
and that will basically initialize the weights
in an optimal way for the learning.
All right?
And then, once we're done with these two functions
we will start implementing the neural network.
So let's do it.
Let's quickly make these two functions.
So I'm starting with a deaf here.
Then I'm gonna give the name of these function
which is Normalized_Columns_Initializer.
There we go.
And this function is gonna take just two inputs.
First, it's going to be the weights we want to initialize
and the standard deviation,
because as I just said
we want to set a specific variance
for our tensor of weights.
And if you want to understand why we have to do this,
it's because you know,
when we make the neural network
there will be the actor and the critic
according to the A3C model,
and we will make two separate fully connected layers,
one for the actor and one for the critic.
And these two fully connected layers will have weights,
and we will set a standard deviation
for each of these two groups of weights.
And so what we'll do is,
we will set a small standard deviation for the actor.
It'll be around 0.01
and a big standard deviation for the critic,
which will be around 1, I think.
So that's why we're making this function
so that we can very easily set the standard deviation
for the weights we will initialize later
for the actor and the critic.
That's why we're doing this.
So now we are just going to set a default value,
but this will change afterwards
when we initialize the weights.
So let's choose so far, 1.0.
All right.
And now we're ready to define what's inside this function.
So what we'll first prepare is the output,
that we're gonna call Out.
So this Out variable
is what will be returned by this function.
And so Out, first what we're gonna do is initialize it.
So as you understood,
this output will be a tensive weight
that will have a specific standard deviation.
But before we take care of setting the standard deviation
we just want to initialize it,
and then we will set the standard deviation here.
Which is the argument, which is the input of this function.
So Out and to initialize a tensor of weights,
you might know how to do it, we already did it.
We're gonna use our Torch Library.
And from this Torch Library
we will take the RAND N function
which will initialize a Torch tensor
with random weights that follow a normal distribution.
So that's why it is called RAND N, N is for normal.
And so now what we simply need to input
is the number of elements that this tensor will contain.
And this number of elements is of course
the number of weights
because we're actually initializing
a tensor for these weights here.
And so to get this number of elements,
we can simply take our weights
and add dot to get size with parenthesis.
And this will give the number of elements in weights
so that it'll create a Torch tensor
of the same number of elements of our weights.
And it'll be initialized with random weights
following normal distribution.
All right?
And now it is time to
set the standard deviation we want to have.
That is this standard deviation here.
So what we're gonna do now is a simple normalization.
We have a Torch tensor of weight,
and now we want to normalize it.
And so to normalize it,
we will simply write the explicit computation.
And so what we simply need to do here
is take our output
then update it by multiplying it
by the standard deviation we want to have,
divided by this sum I've just mentioned.
And so to get the sum,
we're gonna use the square root function by Torch.
And so that's why I'm taking here Torch.SQRT
that's the square root function.
And inside we are going to input
the square root sum of the weights of our vector.
And so we take our output,
then we use the Power Function
to which we input 2
because we want to take the square of the sum,
and then we take therefore the sum.
And inside we are going to specify the index
of the column that contains the weight we want to sum
and then to get these weights separately,
because we want to sum them,
while we use the expand underscore
as function of our output, Out.
All right, so this will get the weights of Out
which so far was initialized as a Torch tensor of weights.
So that gets all these weights.
We take the sum of square
and then we take the square root to apply the normalization.
And the fact that we have this standard deviation
and numerator will make sure that can write it here.
Variants of Out will be equal
to square of the standard deviation.
This formula here will make sure
that this Tensor of weights that we initialized
will have a variance that will be equal to
the square of the standard deviation
that we input as argument.
And that is how we can set a specific standard deviation
for the future actor and critic that we will make soon.
And we will choose a small
standard deviation for the actor
and a large one for the critic.
And we will do this very easily thanks to this function.
All right.
And so now we have only one thing to do left.
It's of course to return the output
that is now normalized with this specific
standard deviation.
All right, so perfect.
That's the first function we had to make.
That's the first tool
we will be very happy to use to make the A3C Brain.
We have one more function to make now.
It's gonna be the weight and its function
and that's just a function that will, I remind,
initialize the weights to make the learning optimal.
So let's do this in the next tutorial.
And until then, enjoy AI.