-: Hello and welcome back to the course
on artificial intelligence.
In today's tutorial, we're taking our first step
into the world of A3C.
And as a first step, we're going to find out
what this abbreviation stands for.
So A3C stands
for Asynchronous Advantage Actor-Critic algorithm.
Now, this is an algorithm which was developed
at Google DeepMind in 2016 by a group of researchers
and it's the cutting edge algorithm
for artificial intelligence to date.
Now it has multiple modifications
and we'll discuss that more in the course
especially in the practical tutorials.
But nevertheless, this algorithm blows everything else
including Deep Convolution Q-Learning networks
out of the water, completely out of the water.
And it is faster, it takes less time for training,
and gets better results.
So throughout this part of the course,
we'll be referencing, and we have already referenced,
but we'll be referencing even more,
a paper or the paper that was published
that first introduced A3C.
It's called "Asynchronous Methods
of Deep Reinforcement Learning" by Volodymyr Mnih
and others from Google DeepMind.
So I'm going to show you this paper now
so that you have a introduction to it.
So here is this paper.
I wanted to show it to you so that you can get a feel for it
and already get introduced to it a little bit.
And of course, it is highly recommended
to read through the paper and understand
what exactly they're talking about.
And you'll see that throughout the practical tutorials
Alan will be taking you through certain parts of the paper,
through certain paragraphs or sections
which will be relevant to what we'll be programming
at that point in time.
And what I wanted to point out here is like, as you can see,
a lot of research went into this,
but and there's a lot of references as well,
but like a part of that I really like is that at the end,
the very end, they compare the different algorithms,
compare the results.
And this is what I wanted to point out here.
So let's zoom in a little bit.
So here, as you can see, even in Google DeepMind,
they are training or they're evaluating the algorithms
on games just as we are doing in this course.
So exactly the same principle because games
are a simulated environment, or a small environment,
a confined environment with certain rules,
and they want to understand
how well this artificial intelligence is doing
in those games.
And here we've got exactly all those games
which you can find.
A lot of them, you can find on OpenAI Gym.
And the games that we've been working with, so for instance,
in this section we are working with Breakout,
so it's also here.
And so you can see that for Breakout,
they've got it in bold,
they've got the best algorithm highlighted.
So DQN, that's the algorithm we've been working with.
Then some other algorithms, and then here you've got A3C.
A3C with LSTM, long short-term memory.
So that's the one we'll be implementing
in this part of the course.
We'll have A3C with LSTM, which makes it even stronger.
So as you can see, Breakout, the best result is achieved
by A3C with LSTM.
So that's the score, 766.8, compared to the others.
And also you can see that for the most of them.
So if we now take like a bigger picture view,
you can see that most of the bold ones
are actually in this last column.
So yes, indeed, there are some games where other algorithms
are performing better.
But as you can see, DQN is actually not performing better
in any of the games.
But you can see that there are other algorithms.
Other algorithms perform better sometimes,
but A3C-LSTM performs the best in most cases.
So you can see that this is bold, this is bold, this one,
these ones, this one, and so on.
So you can see that A3C-LSTM is a really powerful algorithm.
It is indeed at the forefront of artificial intelligence
and that's exactly what we'll be implementing.
So very exciting section ahead.
Highly encourage you to go through this paper
and get a feel for what we're going to be talking about.
And then throughout this section
and throughout especially the practical side of the things,
practical side of STRAWs,
we're going to be going through this in detail.
We're actually going to be working
with their pseudocode here, which is available,
and we're going to be,
this outline will show you how to implement that
and how we're going to be working with that.
And on that note, I hope you're going to enjoy this paper
and I look forward to seeing you next time.
And until then, enjoy AI.