-: Hello, and welcome back
to the course on artificial intelligence.
I hope you're enjoying the course so far.
And today we're talking about action selection policies.
All right, let's dive straight into it.
Previously, we talked about adding a neural network
to our simple Q learning
and so far we are getting quite into deep Q learning.
We've talked about the learning part quite a bit
including adding some elements to it.
And today we're talking about this part,
we're talking about the acting.
So let's have a look.
So here we've got what we discussed about the acting
that once you input the values, the parameters
or the vector describing the state
the agent is currently in that environment,
then that is after all the learning is done
or even before the learning is done,
basically we get all the Q values.
So we're not interested in the learning right now.
We're interested in acting.
So once we have these Q values,
how do we understand which one we need to use?
Well, if you think about it, Q values are simply,
these are the predictions for the Q values.
So as we did in the simple Q learning algorithm,
what did we do?
We just selected the one with the best,
with the highest Q value.
Once we have the one with the highest Q value,
we just take that action
because it just brings us the highest Q value
and that we know that Q values is calculated,
is immediate reward that we expect to receive
plus the DK factor terms the value of the next state.
And it's a recursive calculation.
So why not?
Why wouldn't you take the best Q value?
And that's kind of the end of it.
But as you can see here, it's not as simple.
Here, we're using a softmax function
and this is where we're going to talk
about action selection policies.
So here, in reality,
we don't have to have just a softmax function.
We can have different action selection policies.
For example, we've got epsilon greedy, epsilon soft
and we've got the softmax.
And those are kind of like the most commonly
used action selection policies.
Of course, they're others.
For instance, the most basic one is,
here's a very simple action selection policy.
Just select the best that one with the highest Q value.
But why doesn't that action policy fly
and why do we have different types of action policy,
action selection policies?
Well, it all boils down to exploration versus exploitation.
And that is the core of reinforcement learning
because we've already talked about this a little bit
that your agent, when it's operating in an environment,
it might predict certain Q values,
which might be good.
And it might not turn out great,
it might turn out that those Q values are bad
and it'll be forced to explore.
So if we, for instance in this case
predict that Q2 is the best one, and then it takes Q2,
takes action two,
so from here it takes action two
and then it gets a very negative reward.
Then the environment is forcing the agent to go and explore
because now it's gonna learn
that, Oh, actually I thought Q2 is gonna be very good
but it turned out very bad.
So the result turned out very bad.
So the network's gonna update itself.
So next time he's in the state,
he's gonna probably,
he might still choose Q2 if it was,
like if it was very, very favorable.
So you might think that's like,
he might need a couple of times, a couple of penalties,
punishments in order to learn that Q2 is a bad action.
But maybe he'll already soon learn
that, Okay I'm gonna take a different action,
I'm gonna take this action
because now it has the best Q value.
So sometimes the environment forces the agent
to take different, to explore different actions
but sometimes the agent might get,
find itself stuck in a local maximum.
It might find that it found,
like through its initial exploration,
it found that, oh this is a pretty cool action.
Like, I'm going to go right here.
And that's pretty cool action
but the problem is that it thinks it's the best action
simply because it hasn't explored,
it's explored going up,
it's gonna explored going left,
it's explored going right
but it hasn't explored going down
from that specific state that it's in.
And now that it's kind of like biased towards this action
and thinks it's a good action, it's gonna keep taking,
it's gonna keep getting, it's gonna keep taking this action,
it's gonna keep getting a good reward
but what if this action would've been even better
if this action would've been so much better
that if it knew about this action,
it would actually switch to this action.
But because it got stuck in a local maximum
and it's getting these good rewards,
it's just going to be reinforced.
This is going to keep reinforcing itself that
or the environment's going to reinforce it
that this is a good action to take, keep doing that.
But the reality is that there's this other action
that it hasn't found yet
or hasn't even explored that would've been much better.
And so what we want to do,
is we want to come up with an action selection policy
that allows our agent not to get stuck in a local maximum.
Yes, it's important to keep doing the good actions.
That's the exploitation part.
We wanna exploit what we've found
but at the same time, we still want to explore.
We never want to stop exploring.
Like in life, you never wanna stop learning,
you stop learning, you die.
There's a saying like that.
That when you're not growing, you're dying
or something like that.
So you want to keep learning
and your agent wants to keep learning.
And that's where these action selection policies come in.
So we've got three listed here.
So the first one is epsilon greedy.
It's a very simple one.
It sounds pretty complex in the sense
that like it's got a cool name
and usually things (indistinct) names are,
It's actually not.
So basically what it does, is it'll select the one
with the best Q value and like epsilon greedy,
you might hear it in other places.
It's just like a selection policy.
So in this case, we're using it to select our Q values,
our action.
So you'll select the one with the highest Q value,
all the time, except for epsilon percent of the time.
So for instance, if you set epsilon to 10%,
then you're going to, or 0.1,
then 10% of the time,
the action is going to be selected at random.
So 90% of the time,
you're still going to be selecting the best action
based on the highest Q value.
But 10% of the time is gonna be selecting a random action.
Uniform is is going to be absolutely
randomly taking an action
or if you said epsilon to 0.5 or 0.05,
that means that 95% of the time
the agent is gonna be taking the action
with the highest Q value,
but 5% of the time it's still going to be selecting
in a random action.
So it's going to be going out there and exploring.
So epsilon soft is very similar.
By the way, that's kind of like
why it's called epsilon greedy
because you're greedily selecting the action,
the good action except
for that little epsilon percent of the time.
So the lower the epsilon,
the more greedily you're selecting
that kind of the action that is the optimal action.
And the less you are leaving,
less chances you're leaving for exploration.
Epsilon soft is the opposite.
So basically you're selecting,
at random, you're selecting one
minus epsilon percent of the time.
So if your epsilon is like 0.1, so 10%,
then only 10% of the time you're taking this action
and 90% of the time you're selecting a random action.
So very simple, just inverted algorithms.
And softmax is kind of like the next step from,
or it's a more advanced version,
I would say, of the epsilon greedy algorithm.
Although they both have merit
and they both have place.
We are going to be using soft max in our coding,
in our practical set of things.
So that's why we're going to talk
in a bit more detail about softmax.
So let's have a look.
So let's move on to softmax.
Hopefully it's pretty clear about epsilon greedy.
So it's a pretty straightforward algorithm select.
This one most of the time
except for sometimes go and explore.
And now we also see why it's important
to do that exploration
so that we don't end up in local maximums
in our optimization process.
So now we're gonna talk a bit more about softmax,
there's a tutorial on softmax at the end of the course,
I think it's in annex number two
where we talk about the concept behind softmax.
I'm just going to refresh a little bit here.
So there we're talking about convolution neural networks.
And by the way, we are going to be covering convolution.
We're not covering convolution neural networks
in this section of the course.
In this section we're still using a vector
but in the next section of the course,
when we're creating an AI to play doom,
we are going to be using convolution neural networks.
So it could be beneficial for you
to look at convolution neural networks
and then take the softmax function
or you can learn a bit more about softmax
after you take the convolution neural networks annex
of the course later on.
But here's a quick refresher.
So here we've got a convolution neural network
which decides whether it's a dog or a cat.
So here we've got the voting process between these neurons
and this one says that it's got the features
the fluffy ears, pointed face type of thing
and the kind of like the features,
the types of eyes, the way the eyes look
all these features that belong to a dog.
So it's a 95% chance
that it's dog and the 5% chance that cat.
But the question is, how did we get,
and in that tutorial we're talking
about how did we get these values to add up to one.
Well, whatever the convolution
or our whole neural network,
so the convolution neural network
plus the fully connected layers, whatever it's sped out,
whatever the values it sped out,
we applied a softmax function over here.
This is where we're introduce the formula
for the softmax function.
This is what it looks like.
And then we got these values.
And so basically that's a quick refresher.
This is the formula for the softmax.
What it does, is it takes however many outputs you have,
doesn't matter.
It will take them and it will squash them all into values
between zero and one, regardless of how big they are.
Just by looking at this formula,
you can see that there's a total sum at the bottom.
So these values are gonna be zero,
between zero and one.
And also all these values are going to add up to one always.
And so that's very beneficial for us
because when we're using the softmax function,
what happens is we get these Q values,
we select this best Q value.
But in reality, what happens is, these Q values that we get,
they're actual numbers, right?
So there's some kind of numbers,
they don't have to add up to one,
they don't have to be between zero and one,
just some numbers.
But when we apply softmax,
we don't just select the best one,
We actually get numbers like that.
So we get numbers in the range between zero and one
and that also add up to one.
And so what other thing do we know that add up to one?
Well, probabilities,
we know that probabilities always have to add up to one.
So that is why we can say
here we've got Q values,
but here all of a sudden we've got probability.
So we can say that the likelihood
of this being the best action is 90%,
this best being best action is 5%, 2%, 3%.
Because we know the higher your Q value,
the better the action.
And so if we squashed them to zero to one,
then these become probabilities
and we can deal with them as such.
And therefore now is when the action is selected.
And that's how we come up with Q2.
But if you look at it closely,
this isn't a strict 100%
and these are not strict 0%.
So this is a 5%, 2%, 3%.
So the most natural way to apply the softmax
in order to preserve exploration in the algorithm,
is to use these exact probabilities
as how often we are going to be taking that action.
So these probabilities actually represent the distribution
of these actions that we're taking.
So basically softmax makes it very easy for us
to come up with a way
to combine exploitation and exploration.
So the best action
will always have the highest probability
because it has the highest Q value.
And therefore here we're going to be,
just we're gonna use these as our distribution
and we're gonna say,
"Okay we're gonna be taking Q2 90% of the time
but 5% of the time we're still gonna be taking Q1
and 2% of the time we're gonna take Q3
and 3% of the time we're gonna be taking Q4."
And the beauty here is also that as these values update
as the agent goes through the network
more and more and more,
it becomes more familiar with the environment
and therefore these updates.
So this value for instance might become,
like it might ascertain
that this value is actually less or this actually is higher.
And so these probability will also change
as an agent goes through.
So even though here we've got Q2,
nobody is to say
that sometimes 5% of the time to be more precise,
we'll be selecting Q1 as the action to take.
And sometimes or action one will be taking action one.
Sometimes we'll be taking action through two,
action three 2% of the time
and action four will be taking about 3% of the time.
So every action has a chance to play in this process
as long as we have enough iterations
and agent goes through lots and lots of times
through these states that they're in.
And that's how any kind of deep learning algorithm works
that you want to do this many, many, many times
so that you learn from experience.
And therefore, as you can see here,
it's a very natural transition to,
we're not just randomly like an epsilon greedy algorithm.
We're not just randomly selecting the actions,
we're selecting them based on their softmax values,
which makes it, like has some logic behind it,
not just at random 10% of the time,
we're selecting a random action
but there's some logic behind how we're doing it
and based on the Q values that we've explored.
And so, that's the action selection policy
that we are going to be using in this course.
You're welcome to definitely check out the epsilon greedy,
action selection policy if you like,
but we are going to be predominantly using
the softmax action selection policy.
And I've got an interesting reading for you.
So this is called Adaptive Epsilon Greedy Exploration
in Reinforcement Learning Based on Value Differences.
It's such 1,010 article.
And it's interesting because Mike Michelle,
I'm not sure how to pronounce it.
Michelle, Mickel topic introduces a different type
of algorithm.
So an adjusted epsilon greedy algorithm
and called the VDBE algorithm
or epsilon VDBE algorithm.
You can see it over here.
And he actually compares it
to the epsilon greedy and softmax.
And it's an epsilon greedy algorithm
which basically the main idea behind it,
is to adjust the value of epsilon
depending on the state the agent is in.
So if the agent is very certain about the state they're in,
then epsilon should be smaller.
So there should be less exploration.
If the agent is uncertain,
epsilon should be higher, should be more exploration.
So it is a 2010 article.
I'm not sure if this new proposed algorithm is widely used
or has been accepted in the community
or if artificial intelligence has moved
kind of away from this suggestion.
But nevertheless, it will definitely help you
reinforce your knowledge about action selection policies
which we discussed the epsilon greedy, the softmax,
will help you, it'll give you an opportunity
to compare the side by side,
and also see in which direction people actually think
when they want to improve artificial intelligence.
So if you're ever planning
on creating really interesting algorithms
that are pushing the edge
of ultra artificial intelligence
and pushing the envelope in this space,
then this could be a good way for you
to see in which direction people think sometimes
when they're trying to improve the norms
of artificial intelligence
or the norms that existed back then in 2010.
So there we go.
Hopefully you enjoyed today's tutorial
about the action selection policies,
and we learned about epsilon greedy, epsilon soft
and the softmax.
And now you are even more prepared
for the practical side of things.
And on that note, I look forward to seeing you next time.
And until then, enjoy AI.