-: Hello and welcome to this Bison tutorial.
All right, so in this new code section
we are going to implement experience replay.
So we're gonna make a new class
which we will call replay memory
and that will implement experience replay exactly
as you saw in the intuition lectures.
But first, let's give a quick reminder
about what is experience replay.
So you know, all this artificial intelligence is based
on mark of decision processes
and mark of decision processes consist
of looking at a series of events.
So the events are, you know, for example
going from one state ST to the next state, ST plus one.
But if the events were like that, well
since the next state is very correlated to the current state
well, the network would not be learning very well.
So for those coming from the Deep Learning course,
that's exactly the same
as where we learned our time series with only one time step.
It was not learning anything
because one time step was not sufficient enough
for a model to learn to understand long-term correlations.
So that's the same here
and that's why we have to implement experience replay.
So how does it work?
Well, that's very simple.
Instead of only considering the current states
that is only one state at time T,
we are going to consider more in the past.
So exactly like for our list TMs, and therefore our series
of events will not be ST and ST plus one.
This will be, for example, the 100 states in the past.
So ST minus 100, ST minus 99,
up to ST minus one, and then ST.
So in other words, we put the 100 last transitions
into what we call the memory.
And that way we have a long term memory
as opposed to a short term memory
or even should I say, an instant memory.
And that makes the whole deep key learning process
work much better.
And then, once we create this memory of the last 100 events
we will sample, that is we will take some random batches
of these transitions to make our next subjects.
That is our next move, by selecting the next action.
And therefore, in this replay memory class that
we're implementing for experience replay
we will make three functions.
First of all, the INIT function as usual.
That's the case for any class.
And so in this INIT function we will define the variables
that will be attached to the future instances of the class.
That is the future objects that will be created
from this class.
And so very simply, these variables will be the memory
of the 100 transitions, the 100 events, and the capacity
that is the 100 number.
You will be welcome to try a longer memory
by increasing the capacity.
So that's the first function, INIT function.
And then we'll make two other functions, one push function
to make sure that the memory doesn't ever contain more
than 100 transitions.
And for this, we'll use the capacity by just
doing one simple "if" condition
and then eventually we will make the sample function.
And that will be of course to sample some transitions
in this memory of the last 100 transitions.
All right, so let's start by introducing the class.
So as usual, we start with class
And then we give a name to the class.
So we call it replay memory.
And then in parentheses we input object, then column.
And then here we go.
We start with the first function, the INIT function.
So that's exactly the same as before.
We start with def, then two underscores, INIT,
two underscores again, and then the variables.
So there is of course self, which is the variable attached
to the future instances of the class, the future objects.
And then we're gonna have another variable
for you to be able to try some other experience replay,
some other memories.
And that's gonna be the capacity.
So this capacity will simply be the number 100
because we're gonna make experience replay
with the 100 last transitions.
All right, and then, colon, and here we go.
Let's go inside the function and let's define the variables
of our replay memory objects.
So, the first one will be self dot capacity.
(typing)
And as you probably understood
this will be the capacity that is the maximum number
of transitions we want to have in our memory of events.
And this will be equal
to the argument we will input when creating an object
of the replay memory class.
And therefore that is capacity.
That's the argument of the INIT function.
So capacity.
So again, not to be confused, self dot capacity is the name
of the variable that is attached to the object
and capacity here is the argument we will input
when creating an object of the replay memory class.
All right, and then we have a second variable.
That's of course the memory.
So self dot memory.
All right.
And so what will this memory variable be equal to?
Well, this memory is supposed to contain the last 100 events
and therefore this should be a simple list.
You know, a list which will contain the last 100 events,
the last 100 transitions.
And to initialize the list, there is nothing more simple.
We just add some brackets like that.
And here we go.
Our memory is initialized.
So of course, at the beginning of the experiments
or more precisely the beginning of the exploration
the memory will be an empty list.
And then we will put the transitions each time
we reach a future state.
And speaking of that, that's exactly what we'll do
with the next function.
That's what we're gonna call the push function.
We will make this push function to append the events
in this memory list.
And then we'll use the capacity to make sure
that this memory list always contains 100 events
and never more.
All right, so let's do this in the next tutorial.
And until then, enjoy AI.