-: Hello and welcome to this tutorial.
All right, so after we made
these four convolutions and the LSTM,
we now have an encoded state
that is going to be the input
of these two neural networks that we're gonna make
for the actor and the critic.
And speaking of them,
the only thing that we have to do now
is just create two linear full connections,
one for the actor and one for the critic.
But before we do that,
we need to get the number of possible actions.
And so I'm going to call a new variable here
that is not gonna be a variable of the object.
So I'm not gonna use self here
but I'm going to create the variable
none outputs (keyboard clicks)
which will represent the number of possible actions.
And to get it, well,
we can get it from the action space.
So we take our action space,
(keyboard clicks)
which will be the input of the INIT function
when we create the object
and then we add .n to get this number
of possible actions.
And now the actor and the critic
will take separately the same input
that is the output of this whole process here
with the convolutions and the LSTM.
So it will take the same input, which is an encoded state
but then they will have two different
linear full connections
so that we get eventually actually two neural networks.
One for the actor and one for the critic.
So let's make these two separate neural networks,
but since we already did the big job
with the encoding here,
well what with simply need to do
is just create two objects,
one linear full connection for the actor,
and one other linear full connection for the critic.
And so that's exactly what I'm gonna do.
I'm gonna create two objects now.
A first object for the linear full connection of the critic
which I'm gonna call,
"Critic_linear." (keyboard clicks)
And to create this linear full connection,
while you know how to do it,
we simply need to take the nn module
and then the linear class
to which we have to input well the input neurons,
which are the outputs of all this encoding here
with the convolutions and the LSTM.
That is
256
neurons.
So we input
256 here, (keyboard clicks)
and then we are gonna have one output,
because remember the output of the neural network
for the critic is the value of the V function
applied to the input state,
to the input and coded state that we made here.
So if we call the input state S
that is the output of all this.
Well, the output of the neural network
of the critic will be VS.
And therefore it has one dimension, it's just a value.
And so here we input one.
And remember VS is what is shared among the actors
so that they can get some common information
that they can use to play their action
in a more relevant way.
Okay. So that's for the neural network of the critic.
And now let's make the neural network of the actor
and therefore I'm adding here
self.actor, linear. (keyboard clicks)
-: And same, we already have the input encoded state.
So now we simply need to add a linear full connection.
And therefore same we take the nn module
then the linear class.
And now same.
This neural network of the actor
will take the encoded state that has a size
of 256.
So 256 here. (keyboard clicks)
But then the output is gonna be different
because, of course, you know it,
the output of the neural network
for the actor are the Q values.
The Q values of the input state
the one that will encode it here, and the action plate.
So again, if we call this encoded state that we made here S
and the action played A,
the output of this neural network, actor linear,
will be QSA.
And since you know,
we have one Q value for each action,
then therefore we have none outputs Q values.
And therefore the output here is gonna be
none outputs (keyboard clicks)
because none outputs is actually the number of Q values.
Okay, perfect.
So if you want, I can write for you
output here
for the critic is VS (keyboard clicks)
where S is the encoded state.
And for the actor,
the output is QSA. (keyboard clicks)
Alright? So that's very important
to understand this distinction here
and to understand that we therefore
have two separate neural networks,
one for the critic and one for the actor.
Okay, perfect.
So we are almost done with this INIT function.
Now the most important thing is done.
The only remaining thing that we have to do
is to initialize all the weights
of those two neural networks and all the buyers.
And of course to do that we're gonna use
the two functions that we made earlier.
That is the normalized_columns_initializer
and the weights INIT
So let's do that quickly.
It's gonna be pretty easy and pretty fast.
So first we're gonna initialize some random weights.
And to do this,
we're gonna apply the weights INIT function to our object.
So, here we have to take self to get our object
and to our object, we apply
the weights INIT function.
So therefore inside we just need to input
the weights INIT function. (keyboard clicks)
And there we go.
That will apply this function to our object.
And by doing this,
we are just initializing some random weights
to get a future optimal learning of these weights.
And now, what we have to do is make
a special normalization for the actor and the critic.
But remember, I think I told this in the previous tutorials,
we are not gonna set the same variance
for the actor and the critic.
The actor will get a small standard deviation,
a small variance, and the critic will get a big one.
And why do we do this?
What's the purpose of giving
a small standard deviation of the weights for the actor,
and a large standard deviation of the weight for the critic?
Well that allows to manage
to deal exploration vs exploitation.
That's exactly the purpose of doing this.
By giving a small variance the actor
and a large variance to the critic,
we will have a good management
of exploration vs exploitation.
So let's do this.
Let's first take care of the actor.
So we take self, our object
then we're gonna take the neural network of our actor,
which is actor linear. (keyboard clicks)
Then we are gonna access the weights
of this neural network of the actor.
And remember to access the data
of the weights we need to add .data.
Alright.
So with this we get the weights,
and now we're gonna use
our function normalized_columns_initializer.
So I'm copying this, pasting that here.
And we are going to enter as argument
the standard deviation we want these weights to have.
But first, remember, this function takes two arguments.
First, are the weights we want to initialize.
So we simply need to take that again
and paste that here.
And the second argument is this standard deviation,
we want these weights to have.
So as we said,
we want a small standard deviation for the actor,
and a small one is going to be 0.01.
Perfect.
So that's for the weights
of the neural network of the actor.
Now, let's take of the bias
of the neural network of the actor.
And therefore here we're gonna do almost the same thing.
We're going to copy this,
paste that below,
replace weight
by bias to access the bias.
And after data, we're simply going to add .fill.
And remember, inside we input zero,
because we want all the bias to be initialized with zero.
So, actually I don't think this line is necessary
because as you remember,
the bias are already initialized to zero
with this fill function in the weight INIT function.
So, you know, we're doing this just to make sure
that the bias are actually initialized to zero
but I think it's already done here.
But anyway, now we are 100% sure.
All right, and now we're gonna do the same for the critic.
So, let's be efficient and let's copy these two lines.
(keyboard clicks)
Let's paste them here.
And here we are just going to replace
actor by critic. (keyboard clicks)
Same here.
And now the only thing that we have to change
is just the standard deviation we want the weights
of the neural network for the critic to have.
And, as you remember,
we want this time a large standard deviation
and instead of open 01,
we will input one.
So there we go.
We have a small standard deviation
for the weights of the neural network of the actor
and a large standard deviation for the weights
of the neural network of the critic.
And, of course, let's not forget
to replace actor here by critic.
Alright. And now we're good.
Cool. So now we have two remaining thing to do.
First, is to initialize also the bias of the LSTM.
And to do this, we take our object self
because the LSTM belongs to our object.
Then we take our LSTM then dot.
And then we're gonna get the two types
of bias that are in the LSTM.
That's bias_ih. (keyboard clicks)
And the other one is bias_hh.
So that's the two types of bias in the LSTM.
And same, they're gonna be initialized to zero.
So first we access to the data,
and then we use
the fill_function (keyboard clicks)
to fill all these bias with zeros,
initialize them with zeros.
Alright.
And now for the second group of bias,
we adhere
the same
but we replace ih by hh. (keyboard clicks)
Alright.
So that initializes the bias of the LSTM with zeros.
And now, the last thing we need to do
is to use a method that is inherited from the nn module
that is the train method.
And basically that is just a method that puts the module
in train mode.
So what's the use of it?
While the use is that it allows to activate
if there is any, the dropouts
and the batch normalizations.
And so to use it, we just add
self.train (keyboard clicks)
and that puts the module in train mode.
Perfect.
So we are done with the INIT function.
We have our convolutions, we have the LSTM,
we have our two separate neural networks
for the critic and the actor.
And all the weights and bias are well initialized.
So that's all good.
We are ready to move on to the next step
which is to make the forward function
that will, of course, forward propagate the signal
from the very beginning with the original input images
throughout all the brain until we get the output.
So let's do that in the next tutorial.
And until then, enjoy AI.