Instructor: Hello and welcome to this tutorial.
Now we're gonna make the full loop
that will compute the policy loss
and the value loss.
And once we have these two losses,
we will be able to use our optimizer
to apply the (indistinct) in dissent
to reduce the losses.
All right, so there we go. We start here.
By the way, in the previous tutorial,
we implemented this section
and I forgot to remove the indent.
Sorry about that.
So starting from R here,
is not in the full loop
and now we're starting a new full loop.
So I'm starting here with four.
And now what we're gonna do,
is we're gonna start from
the last step that was done during the exploration
and we're gonna move backward in time.
So that's why here, I'm doing for I in a reversed range
lan rewards, because rewards is the list.
And since each step of the exploration
is associated to a reward,
because at each step we get reward,
when lan rewards is this number of steps.
And this reverse here, is used
so that we can move back in time.
So there we go.
And now, what we're gonna do,
is update the cumulative reward.
That is R.
And we're gonna update it this way.
That's actually the same
as what we did for doom.
It will be equal to Gamma,
which we get from our parameters.
So I'm taking params first, params dot Gamma times R
plus the reward of the step,
which we can get by taking the list reward,
and taking the index back.
So first this will be the reward of the last step.
Then it'll be the reward
of the previous step, and et cetera.
And each time we update R by multiplying it by Gamma,
and then adding this reward at this step.
And so by doing this,
remember, we will get in the end,
so I'm gonna write it as a comment,
we will get R, accumulative reward.
That will be equal
at the end of the loop to R zero,
the reward of step zero, plus Gamma
times R one, the reward of the first step,
plus Gamma squared
times R two, the reward of the second step,
plus, dot, dot, dot, plus Gamma
at the power of N minus one,
times the reward obtained at step
N minus one, where N is the number of steps.
But then, be careful at the end we will have Gamma.
At the power of number of steps,
times the value, the value of the V function
applied to the last state.
This, what we should get at the end.
And this, we will get that because remember here,
we got this value of the last step,
because this was done at the end of this fore loop here.
And so we got the value,
and we set R to be equal to that value.
So right now, R,
at the beginning of this second full loop here,
will be equal to this value of the last date,
but then by doing this,
this is what we'll get in the end.
R equal R zero, plus Gamma,
R one, plus Gamma squared R two plus Gamma
at the power of N minus one
times the rewarded step N minus one plus Gamma
the power of number
of steps times this value of the last date.
So that's the main thing to understand
in this computation of the cumulative reward.
And that's why it is important to start
from it by initializing R with the value here and
doing this reversed full loop to get this final equation.
Perfect. And now, now
that we have the right value for the cumulative reward
well, we will compute the advantage.
And the advantage here is just the advantage
of getting this reward compared to the value.
So I'm going to introduce a new variable, Advantage
and therefore it will be equal to this cumulative reward
minus the value of the V function obtained at the step I.
So therefore that is error minus values I.
Perfect. And now that we have the community reward
and the advantage. then we can get the value loss.
This is the first loss we can get now.
So we are gonna get our value loss variable
and this will be updated the following way.
Remember so far that the value loss was initialized to zero.
And so we're gonna take the value loss again
and add oh 0.5 times
the squared of the advantage so we can get it this way.
Advantage dot power two.
So that just mean the square of the Advantage
Advantage at the power of two.
And that is exactly the value loss
the loss generated by the predictions
of the value of the V function output by the critic.
And so it makes sense
that this is the value loss because remember
the advantage A of the action A
and the state S is the difference
between the Q value and the value of the V function.
And so when we play the optimal action, well
we get the stationary state with Q optimal
of the optimal action A star played in the state S
equals the optimal value V star of the state S.
So it's quite intuitive to understand that,
when the advantage is not equal to zero,
then there will be a difference between these two
and therefore that's how the loss is measured.
Okay, so value loss computed.
One loss down. We have now one more to go.
It has the policy loss
and that's exactly what we're going to compute right now.
And to compute it, we need to consider again
the generalized advantage estimation.
Because to compute the policy loss
we need the generalized advantage estimation.
And to get the generalized advantage estimation
we need first the temporal difference of the state values.
So we have multiple things to compute here
and we're gonna start with this temporal difference.
Once we get the temporal difference
we will get the generalized advantage estimation.
And once we get the generalized advantage estimation
we will get the policy loss.
All right, so let's start with the temporal difference TD.
So TD is equal to
the reward of the step I,
plus gamma which we get things to our params list.
So params dot gamma times the value of the step
I plus one and we add that data to access it
minus the value of the step I.
And same we add that data.
All right, so that's the formula of the temporal difference
of the state values.
And now we can update the generalized advantage estimation
and how is it updated? while we take our GAE
and we multiply it by Gamma,
params dot Gamma times tell
which we access with our parameters as well.
So we take params tell
and we add this temporal difference of the state values.
So be careful.
We are in a full loop and each time we multiply the GAE
by Gamma and by tell, and we add the temporal difference.
So it's important to understand
that at the end of this loop
well this generalized advantage estimation will be equal to
the sum on all the steps
of Gamma times tell at the power of I
times the temporal difference at the step I.
All right, so important to keep that in mind.
And now that we have the generalized advantage estimation
and the temporal difference
we can finally compute the policy loss.
So let's do this.
We are going to update our policy loss the following way
by taking the old policy loss.
And we subtract the log probabilities obtained
at the step I that we multiply
by this generalized advantage estimation that we have to put
in a variable because then we will compute the gradients.
So it has to be attached to gradients in the dynamic graph.
And then we add minus or 0.01 times the entropy
the entropy obtained at the step I in the full loop.
And again, now be careful.
This is the computation inside the full loop
which means that at the end of the full loop
what we will get is policy loss
equals minus sum over the steps
of the product log of the policy
at the step I times the generalized advantage estimation
plus this 0.01
times the entropy
at the step I. So there we go.
And now what is the policy at the step I well,
that's the soft max probabilities
of the actions and the entropy at the step I.
Well, you know what it is, it's what we computed earlier
and what we appended to the list.
So we already have that.
But this PII here is the soft max probability
of the actions.
And why do we put a minus here?
That's because the log
of the probability and the entropy are negated values.
And since we want to minimize their absolute value
we must see this loss as a [inaudible] likelihood
as opposed to a distance.
You know, we want to maximize the probability
of playing the action that will maximize the advantage.
That's the whole idea behind it.
We want to maximize the probability
of playing the action that will maximize the advantage.
And for those of you who might be wondering
what is the purpose of this
entropy coefficient that is this factor 0.01 here.
Well, the purpose of it is just to prevent
from falling too quickly
into a trap where we have a distribution
of probabilities with zeros for all the actions
except one which has a probability of one.
And if that happens, that would minimize the entropy.
So that's why we're adding this small coefficient
0.01 here that will make the entropy increase
in the gradient descent.
Okay? So now the good news is
that the most difficult part is done.
We have the two losses
and therefore what we only need to do now
and we already know how to do it
is to perform just a cast grid
in descent to reduce these two losses.
And so what we're gonna do now is
get out of this loop and we're gonna take our optimizer,
the one we made separately.
Then remember, the first thing we have to do is to
initialize all the gradient parameters to zero.
And to do this, we add dot
then the zero and the score grad method.
All right? So that's done then.
Now we're going to perform backward propagation
but we're gonna give twice as much
as I importance to the policy loss
than the value loss because the policy loss is smaller.
So to do this, we are gonna put in parenthesis
policy underscore loss plus
0.5 value loss.
So 0.5 times the value loss.
And we're gonna add here dot and we applies the
backward method to perform backward propagation.
And thanks to this trick here with the policy loss
plus health of the value loss, we give twice as
much importance to the policy loss than the value loss.
Okay? Then we're gonna use another trick
which is to prevent the gradient
from taking extremely large values
and therefore degenerate the algorithm.
And the trick to do that is to get first our torch library
then the NN module
from the torch library, then the details sub module.
And now we're gonna use the function clip
underscore grad underscore norm.
And we are going to input our model parameters
with a second input, which will be 40.
And that trick will basically make sure
that the gradients won't take extremely large values
and degenerate the algorithm.
And for those of you who might be wondering
what this 40 is exactly
well that just means that we're using this value
so that the norm of the gradient stays between zero and 40
and therefore that's how we prevent the gradient
from taking too large values.
Okay, so now we're almost done.
Remember we made this ensure shared grads function
at the beginning of the file, which is to ensure
that the agent and the shared model share the same gradient.
And to do this, to make sure of it
we can apply this function here.
And so we are gonna add ensure shared grad
to make sure that the model
and the shared model share the same gradient.
All right, so that's just a precaution.
I'm not sure that's totally necessary, but you know
at least we won't get any issue here.
Okay? And finally, last line of code.
We are of course going to perform the optimization step
to reduce the losses and you know how to do it.
Of course, we take our optimizer
and we add dot step with parenthesis,
and there we go. The training of our brains is over.
So congratulations.
I hope this wasn't too overwhelming.
Don't worry.
I will provide the code with all the comments.
So if you missed any detail
you can have a look at the comments.
And don't worry if you haven't understood anything
This is very advanced.
But, rest assured, this is also the most powerful.
Remember, who is it made from? The creator of PyTorch.
So, we are really working with the best here
the state of the art.
So it's totally normal
if you didn't get everything the first time.
But, by working on it many times
you will definitely get more and more comfortable.
So now we're done with the training.
So basically we made all the most important things.
You know, we made the brains by building the architectures
of the neural networks with the convolutions, the LSDM
and the fully connected layers.
We trained this brain by making this train code here.
So basically the heart of the algorithm is done.
You made the A three C, congratulations.
Now we have a few more things to do
but that is just to get the fun part, you know
we need to make this test dot py fl
which will test the agent
and provide the videos of the AI playing breakout.
So this will be very fun to watch.
We will not code all the lines of this test dot py fl
because as we said, we did the most important thing.
all related to A three C
but I will of course explain the code
and eventually we have this main dot py fl
which will execute the code.
And from the moment we execute this code
all the codes will be generated.
So the brains will be made, the training will happen
and the AI will play new games of breakout
and we will get all the videos.
So I can't wait to eventually watch them.
We are gonna see
if the AI is smart enough to catch the balls.
So now I'm gonna see you in the next tutorial for this test
dot py, so that we can test the AI on some new games.
And until then, enjoy AI.