Speaker: Hello and welcome back to the course
on artificial intelligence.
In today's tutorial we're starting off this section
on deep convolutional key learning.
So let's have a look at what it's all about.
Previously, we talked about deep key learning.
So we had an environment with an agent
and we had a vector describing that environment
which was fed into a neural network.
And at the end we got the Q-values as our outputs.
And then of course we, we found
out how to tr- how the network is trained,
the learning part.
We found out how actions are decided
based on those Q-values,
that's a action part.
And we talked about the action selection policies and
and different things about, um,
how deep key learning works.
But here, the key concept for all of this is
how do we get from this, from the actual environment
and the states to the neural network?
Well, the transition is over here, the input vector.
So the input layer of our neural network,
and it is a vector.
So what we are looking at is, okay,
so we're actually, actually not the correct
it's not the correct term.
We're not looking at anything.
The agent basically has this information.
So the environment is passing it
this information saying, okay, you, the agent,
you're currently in this, this
your state is described by this vector.
In this simplified example, it's described
by this vector 'X' one of one, 'x' two of two.
So your coordinate's a one, two
and that is your whole state.
In a, in a more complex environment, the state
might involve other things that the agent can be observing.
But the point here is that it is passed as a vector.
And the thing is that that doesn't happen in real life,
in real life, except for GPS systems
and other things like that.
But in real life, what do we use most of the time?
We use our senses. We use our eyes.
Even in GPS, it's not built into our brain.
It's not telling us the coordinates through our brain.
And so it, we are still using our eyes to look at,
at the GPS and understand what's going on there.
And so this is kind of cheating for AI to be able to get,
like, information about the environment as a vector.
It's too simple.
It's not how it works in real life.
That's not how we as humans operate
and ultimately wanna create artificial intelligence
which can operate sim- in a similar fashion to humans
which is as, like can, can take on
the same challenges as humans.
And so in, in the human world, we don't have that.
We don't have that, we don't have these coordinates
or other types of vectors that are passed
to us that explain the state we're in, in that environment.
So we're gonna have to remove that
to make it more realistic.
And then what can we replace it with?
What do we see, or what do we do as a human
to get information?
Well, most of the time we see,
of course we use all of our senses,
but most of the information that we are getting
about the out the world around us
comes through our sight.
And that is why we are going to change that little arrow
which we had into a whole convolutional neural network.
So this is from our annex number two.
We've got the convolutional layer
and that's why it's important to be quite comfortable
with convolution, convolutional neural networks
and how they work.
So you've, if you've done our deep learning course
then you should be comfortable with that,
or you can just have a look at the annex number two.
We've got some very good intuition tutorials there.
So here we've got the convolution operation, which happens.
So we we're actually going to be looking
at this as an image.
So this is an image of, um, net environment.
And so the agent is actually looking at the environment.
So in this case, uh, not that he's like looking
from within there, he's like looking like
let's say he's playing this on a computer
and he can see this environment
and therefore he can see like where this figure
representing the agent is actually, actually is.
So he can see his whole environment
or whatever a human would see.
If it's an actual maze
and the human would see the maze from inside.
And so the agent should be able to see exactly
the same thing.
So whatever he sees is done through,
goes through a convolutional layer,
it goes through a full pulling layer,
it goes through flattening again,
you can find about out about more
about these different parts
of a convolutional neural network
in the annex.
And then after it's flattened,
then we have inputs which go into the neural network.
And this is way more realistic
because the agent has to use their sight and
or has to process images which the environment
is supplying to the agent
just as a human would be processing images.
And the beauty of this is not just
that it's more realistic and it's kind of like more
as a hu, the agent's actually more as a human would be.
But it allows us to process much more complex environments.
For instance, this is how we can play Doom or other games
like that because instead of just getting a vector
of information which like somebody would've created
for us in this environment, we can just hook
up our artificial intelligence to any environment which
as humans we would have vision of this environment.
So we as a human, when you're playing this game
you can see exactly this picture.
And that's exactly what the artificial neural network
or the agent would see now.
So in, in this part of the course when you're gonna
be programing the practical tutorials,
the agent will actually see this exact picture,
it'll see the pixels.
It will get this exact picture for all of the pixels
with this person, with this, with this gun,
with this face, with this percentage,
with everything exactly what we see here,
that's exactly what the agent will see.
Then it'll have to dissect that through
convolutional layer, pulling layer, flattening,
and then it'll go into a neural network.
And needless to say that the neural network
will actually be much more complex than that.
So let's replace it with something like this.
This is not much more complex.
This is, uh, looks a little bit more complex,
but in reality the neural networks
you're going to be working with
and creating with Alan are going to be quite
interesting and gonna be much more complex than this.
But as you can see already here, even if we just have
five inputs instead of two,
things become much more complex.
And here you can see we have many more actions
that the agent can take.
So in the game of Doom, turn left
turn right, look down, look up, run, shoot, reload.
Or you know, all those different actions that are possible
in first-person shooter like, like Doom.
And moreover, it doesn't have to be that you can
you can attach this agent to another type of game.
That's the beauty of it, that it then realizes
that it can now operate any kind
of environment that you attach it to.
Because as long as there's like a visual representation
of environment, of that environment
it's already got the whole infrastructure
the whole structure is ready to process that.
So that's what deep convolutional key learning is all about.
So we're taking it even to the next step.
We're adding convolutions into, or the convolutional layers
into our agent's brain now
and we're making it even more complex
and therefore we can, we're rewarded
with being able to solve even more complex challenges.
So I hope you're very excited about this.
This is gonna be an, an epic section
and we're going to create some amazing things
and I can't wait to see you on the next tutorial.
And until then, enjoy AI.