-: Hello and welcome back
to the course on Artificial Intelligence.
Today we're continuing our journey
into the world of A3C
and we're talking about the asynchronous side of A3C.
So we have our abbreviation,
Asynchronous Advantage Actor Critic,
and today we are going to find out
what asynchronous here stands for, what it means.
And let's go back a step.
Let's look at what we started this whole course with.
We started with reinforcement learning
and the, what it's all about,
that the agent is in a certain state,
they observe the state,
they make certain decisions,
they take actions in that state,
and then the state is changed.
So they get into a new state, plus they get a reward.
So they get reward for taking that action,
or some sort of reward, which could be a penalty as well.
And they end up in a new state.
And based on that, now they take another action.
Again, they get reward and end up in a new state
and they take another action and so on.
And so that is the basis behind
all of reinforcement learning.
And that's what we've been using in Q-Learning,
in Deep Q-Learning, in Deep Evolution Q-Learning,
And that has allowed our agents to beat
gradually more complex and more complex environment.
But now we're going to introduce a even better concept
and even further, take this even to a further level
what A3C introduces through this asynchronous element is,
instead of having one agent attack the environment,
they have three agents or whatever number of agents,
so several agents attacking the same environment.
And the key here is that why it's called asynchronous,
is because they're initialized differently.
So their starting points are different.
So for instance, as you'll see from practical tutorials
you set a random seed and you set it differently
for each of the agents.
And that way, because their starting points are different,
they're going to first go through environments
in different ways
and then they're gonna explore in different ways.
And then in the next iterations,
they're also gonna explore in different ways.
And so for instance, we have three agents,
you are, all of a sudden you're getting triple
the amount of experience.
Instead of just one agent going through
and exploring an environment
and trying to understand how to operate
in that environment, you now have three,
or however many of them, going through that
and getting this experience.
And so they're so that each one of them is learning
through this bigger experience.
And apart from being,
just giving a broader range of experience,
it also reduces the chances of one agent
getting stuck in a local maximum.
So for instance, if one agent finds like a way to
beat the environment, which is not the most optimal,
because if it deviates to the left or to the right
from that solution that it found, it always gets like
gets more penalized.
It might get stuck in that local maximum.
It might just keep doing that
thinking that that's the optimal solution,
where, where it's actually not.
Well, the likelihood of several agents getting stuck
in that same local maximum is, decreases over,
decreases with the number of agents.
So the probability of one agent getting stuck
in a certain local maximum might be high,
but, or might have, might be a certain value.
But the probability when you have three of them
of all three of them getting stuck
in that local maximum is much lower.
And as long as they share experience between each other
they can help each other out.
So if one of them gets stuck, for instance
it's stuck in a local maximum,
just simply think that, that's the best one,
that's the best, that's the best solution
all the time and keeps doing that.
Well, as long as it interacts with the other agents,
so let's say this guy gets stuck in a local maximum
as long as it interacts with the other agents
through the way we build our whole algorithm,
A3C algorithm, they will help him out.
They will give him knowledge that actually no,
hey, you should explore this
or he will be likely, more likely to get out of that.
And also overall, the environment will know that,
hey, even though this is a great maximum,
these other agents have seen better options
and we should keep exploring
because there are, it looks like there are better options.
So in a, in a very short
kind of rough, intuitive understanding
that's, that, those are some of the advantages
of having, these asynchronous agents.
First of all, you like have more experience
to choose from and to learn from.
You could get to the solution faster.
And generally speaking, there's a lesser chance
of getting stuck in a certain local maximum.
So let's see how this all plays out
in this model that we've built so far.
As you remember, this is what we've gotten so far
through the actor-critic.
And this is, like, where it all ties in.
This is so far, as you remember from the previous tutorial,
we did introduce this, you know,
we had this already even in Deep Evolution Q-Learning.
So we just named actor now, but now we've introduced critic
but so far it doesn't really make sense.
What's the point of having this critic
and measuring the value of the state
or predicting the value of the state
using the same neural networks,
this same approach.
But now it's, this is, this is the part
where it's gonna start making more sense.
What we're going to do is we're going to replicate this
because now we have multiple agents.
So with multiple agents, this is
this is what it would look like.
So the, the first way of imagining it
is now we have these three things.
Well remember what we said
about them sharing the experience between each other.
So this is actually, right, right now
they're all independent.
You have one playing the game, another one playing the game
another one playing the game.
It's like, it's like launching your agent
on three different computers.
You put three different computers next to each other
and you launch them and you know, that's great.
Like indeed, you, you, like, you'll get,
you'll get more experience, you'll get like more variety,
especially if they're initialized differently.
So we're gonna assume from here that they're all initial
always initialized differently.
Even though we have the same picture here, we are going to
know that they're actually initialized differently.
So it's not gonna be like identical training,
identical learning from this game.
And so even if you, like, you put three computers
side by side and you launch them, yes,
you're gonna have more experience
because you're gonna have three agents playing
and also you're going to have a bigger variety
of possible solutions. So that's true.
But the problem is that they're not sharing
that experience among each other.
They're not learning from each other.
So they, they don't have that synergy
they don't have the advantage,
or the the extra power, that they would get
if they were cooperating.
You know, like how if you have,
if you have a team of people
they work better together than each one of them separately.
So like in a team here, you got 1+1+1, it, it's three.
But in a team 1+1+1 is not three, it's like 33
because they leverage each other's strengths
and mitigate each other's weaknesses.
And same thing here.
So if you put these three computers side by side,
yes, you'll, you'll have more experience,
more variety, and possibly with someone will
get to better solution than the other one.
That's great, but it'll be even better
if they start sharing that experience.
And how do they do that?
Well, it's through this V that we calculated.
So this V value, that's the output of our network
is actually like that.
So they have this same V.
So every time all these agents
they're contributing to the same critic.
They don't have separate critics, they have a common critic.
And that's the key of how
the actor-critic ties in with the synchronous.
So there's one critic that's watching as,
as they get experience.
So how do we calculate the V?
The we calculate the V through, as you remember,
we calculate the V through the values that we get.
So the rewards that we get through the environment.
And so as the agents explore their environment,
they are calculating, they're predicting the V
plus they have the V that they can calculate.
This is, this is all, all ties back
into what we've already discussed
in the previous sections of this course.
So they already have a V that they, that they can predict,
like expect, through the rewards
that they know that exist in this maze
and that they've already explored.
And as they explore them,
of course, that that value can change.
But also they have the V that
this V is the output of the neural network.
So as they're going through this,
they're going to be adjusting their neural networks
in order to better match that expected V.
So basically, this is shared.
The critic part is shared between the agents
and that is how they share the information
between each other.
That's how they are able to, kind of,
see what's going on in the environment
shared with each other and then use that,
as we'll see further in the next part, in the advantage,
see, use that in order to optimize
how they're behaving in that environment.
And the other thing to note here is, so this was A3C
This is like the core of A3C up to here.
This is a type version of A3C, but there's an actually
an even better implementation of this A3C
which you'll actually hear Atlan talk about
in the, one of the first tutorials
in the practical side of things.
And what he'll be talking about
is how the creator of PyTorch
actually made an adjustment to one of the codes
that was shared on GitHub where he took all of these,
as you can see right now, they have separate neural networks
and only, they share the V,
that adjustment that was made was
actually to take all of these neural networks
and put them into one, take them and put them together.
So ultimately, there's only one neural network
here shared among the agents.
So before they had, each one of them had one neural network
which was shared for the actor and for the critic.
One neural network shared for the actor, for the critic
one neural network shared for the actor, for.
Now they all have one neural network,
which is shared for the actor-critic,
actor-critic, actor-critic.
And then the critic is here, in common.
So let's see, let's let's move these pictures
to the left or here, so make some space.
And this, this is basically the architecture
or the structure,
that we are going to be using in the practical tutorials.
I know that like this, this may sound a bit overwhelming
at this stage, but we've got, we've got
one more A to talk about which is the advantage.
And there we, we'll see a bit better
in action how this is going.
So we'll, we'll talk about the intuition and action there.
But generally speaking, this is, this is what it is.
This is, there's one network which each of the agents use.
So they share, basically what that means
is that they share the weights.
The weights of the network are shared
between agents and when they update it,
they update the whole network, not just their own network.
And then they have outputs, they have, like,
these actions for each agent
and then they have the critic that is shared
which is gonna be monitored.
So I know all this is kind of like,
like there's a lot of stuff right now
but hopefully it's slowly coming together.
At least the main takeaway from here
is that the critic, because it's shared,
that's how the agents are able to make sure
that they're cooperating together
in order to get to the result much faster.
And then in the next tutorial we'll see even further
how all of this adds up, how this comes together.
And for now there's, we like, I would like to recommend
or we would like to recommend you a, an additional reading.
So this is a blog by Jaromir Janisch,
it's called "Let's Make an A3C: Implementation."
There's actually two parts, implementation and theory.
There's the link and it's very similar
to what Atlan will be implementing
in the practical side of the tutorial.
So it's not specifically for this tutorial,
just not specifically just for this tutorial,
but it's for this whole section, in encouragement there
some additional information, some additional insights there.
And so that's, that's why we're bringing it up here.
But nevertheless, in the next tutorial
we're going to start pulling all of this together,
everything we've discussed so far.
And I look forward to seeing you next time.
And until then, enjoy AI.