-: Hello and welcome back to the course on Deep Learning.
Now that we've seen neural networks in action
it's time for us to find out how they learn.
So let's dive straight into it.
There are two fundamentally different approaches
to getting a program to do what you want it to do.
One is hard coded coding where you actually
tell the program specific rules and what outcomes you want
and you just guide it throughout the whole way
and you account for all the possible options
that the program has to deal with.
On the other hand
you have neural networks where you create a facility
for the program to be able to understand what it needs to do
on its own.
So you basically create this neural network where
you provided inputs,
you tell it what you want as outputs
and then you let it figure everything out on its own.
Two fundamentally different approaches
and that is something to keep
in mind as we go through these tutorials.
Our goal is to create this network
which then learns on its own.
We, we are going to avoid trying to put in the rules.
And a good example that I can give you right now is....
this will come further in the course
but it's just a very visual example.
For instance, how do you distinguish
between a dog and a cat?
For on the left side on the approach that's depicted
on the left, you would program things in like the
the cat's ears have to be like this,
Look out for whiskers
look out for this type of nose
look out for this type of shape of face.
Look out for these colors.
You kind of, you'd describe all these things
and you'd have conditions like
if the ears are pointy, then cats
if the ears are slopping down, then possibly dog and so on.
On the other hand, for a neural network,
you just code the neural network.
So you code the architecture
and then you point the neural network at a folder
with all these cats and dogs
with images of cats and dogs which are already categorized.
And you tell it,
okay I've got some images of cats and dogs.
Go and learn what a cat is.
Go and learn what a dog is
and the neural network will
on its own understand everything it needs to understand.
And then further down, once it's trained up,
when you give it a new image of a cat or a dog,
it'll be able to understand what it was.
So there they are,
those are the two fundamentally different approaches
and today we're going to slowly start getting
into how that second approach works.
All right, so let's get straight to it.
Here we have a very basic neural network with a one layer.
This is called a single layer feed forward neural network
and it is also called a perceptron.
Now, before we proceed, one thing that we do need to
adjust is that output value.
Right now you can see that
it's just a Y we need to
put a Y hat in there.
And the reason for that is
usually Y stands for the actual value.
And that's what we're going to be using.
So Y is gonna be the actual value,
which we see in reality,
output value is the predicted value
by the algorithm by the neural network.
Y hat is the output value.
Basically that's the denomination for the output value.
And the perceptron was first invented in 1957
by Frank Rosenblatt
and his whole idea was to
create something that can actually learn and adjust itself.
And this is what we're going to be looking at now.
So we've got our perceptron.
let's see how our perceptron learns.
So let's say we have some input values
that have been supplied to the perceptron
and or basically to our neural network.
Then the activation functions applied,
we have an output
and now we're going to plot the output on a chart.
So there it is our output Y hat.
Now what we need to do is in order to be able to learn
we need to compare the output value
to the actual value
that we want the neural network to get, right?
And that is the value Y
And so if we plot it here
you'll see that there's a bit of a difference.
Now we are going to calculate a function
called the cost function
is calculated as one half
of the square difference
between the actual value and output value.
Now there are many ways you can come up with cost function.
There are many different cost functions that you can use.
This is probably the most commonly used cost function
and why it is specifically this function we use
We'll find out further down when we're talking
about gradient descent, but for now
we're just going to agree that this is the cost function.
And basically what the cost function is telling us is,
what is the error that you have in your prediction?
And our goal is to minimize the cost function
because the the lower the cost function,
the closer the Y hat is to Y. Okay?
So as long as we agree on that, let's proceed.
So basically from here what happens is
there's our cost function.
And from here what happens is now we are going to,
once we've compared,
now we're going to feed this information
back into the neural network.
So there we go.
There's the information going back into the neural network
and it goes to the weights and the weights get updated.
Basically, the only thing that we have control of
in this very simple neural network
are the weights,
W one
W two,
all the way to WM
And our goal is to minimize the cost function.
So all we can do is update the weight.
So we update the weights,
tweak them a little bit
and how exactly we'll find out further down.
But for now, we, we agree that we update the weights
and then we continue.
So, but here I've put up this screenshot of the data
just to make one point very clear that right now
throughout this whole experiment,
everything we're doing right now,
we're dealing with just the one row.
So we're dealing with,
we have a data set of one row
where we have, for instance
we're dealing with how long you study
like the variable that we're predicting is
what results you're gonna get on an exam.
And the independent variables that we have is,
how many hours did you study for?
How many hours did you sleep?
and what did you get on the quiz in the mid-semester?
So in the middle of the semester, there's a quiz
what percentage did you get there?
So based on those variables
we're trying to predict what score you'll get for the exam.
And in exam, the 93%, that's the actual value.
So that's Y.
So we feed these three values
into our neural network again
for the second time now
and then going to be comparing the result to Y.
So let's see how this works.
We feed these values into the neural network
everything gets adjusted and weights get adjusted.
So as you can see, let's do this again.
We're going to feed the values again.
The point here is that we're feeding in these same values.
So we only have one row
We're trying to, we're training on one row
This is because this is just a very simple basic example.
Then we'll see what happens when there's more rows.
So again, we feed these rows in
our cost functions get adjusted.
As you can see, everything happens along those lines again.
So as you can see, every time our Y hat is changing
because we've tweaked the weights,
our Y hat is changing
our cost function's changing
let's have a look again.
So we feed those in.
Y hat is changing
cost function is changing.
We get information back,
feed back to the weights
so that the weights get adjusted again.
We feed in the same values every time.
Everything gets adjusted,
goes back to the weights
and one more time,
we feed in.
Okay?
And another time.
So we've adjusted the weights
we feed in the information
and there we go.
So now this time the Y hat is equal
to Y cost functional zero.
Usually you won't get cost function equal to zero.
But this is a very simple example.
So hopefully all that made sense.
Every time we feed in exactly that same row
because just in this case we're just dealing
with that one row,
into our neural network
where then the values get supplied by the weights,
the activation functions applied.
We get Y hat
Y hat is compared to Y.
Then we see how the cost function has changed.
Feed back that information back
into the neural network
and adjust the weights again.
And then we repeat the same process again
with the same exact row.
We're trying to minimize that cost function.
So up until now we've been dealing with just that one row.
Let's see what happens when you have multiple rows.
So here's the full data set.
We have eight rows of
how many hours you slept?
or maybe these are different students
in taking the same exam,
how many others hours they studied?
how many hours they slept before the exam?
what to get on the quiz and their final result on the test.
And as you can see here on the left
I've got eight of these perceptrons.
Actually they are all the same perceptron.
So this is also important to understand.
I just multiplied it
or like duplicated eight times just
so that we can
conceptually understand.
But the important thing here,
it's the same neural network.
We're going to be feeding these
into one same neural network.
So let's go.
Let's get started.
So one epoch
as you'll hear (indistinct) mentioning,
one epoch is when we go
through a whole data set
and we train our neural network on all of these rows
So let's go, let's get started.
So there's our first row
and there's Y hat for the first row.
There's the second row
there's Y hat for the second row.
So again, it's being fed
into the same neural network every time.
I've just copied them several times
so we can visually see how this is happening.
Then again, it's happening again.
That's third row, fourth row
there's our Y hat for the fourth row and so on.
Basically then we get the same values
for the remaining four rows as well.
So every time we just feed
in a row into our neural network,
we get a value.
Then we compare to the actual values.
So there are the actual values.
So for every single row we have an actual value.
And now based on all of these differences
between Y hat and Y
we can calculate the cost function
which is the sum of all of those squared differences
between Y hat and Y
and all of that is halved.
And there's our cost function.
And basically now what we do
after we have the full cost function
we go back and we update the weights.
We update W one, W two, W three.
And the important thing to remember here is
that all of these perceptrons,
all of these neural networks
is actually one neural network.
So there's not eight of them, there's just one.
And when we update the weights,
we're going to update the weights
in that one neural network.
So basically the weights are gonna be the same
for all of the rows.
So it's not the case that every row has its own weights.
Now all the rows share the weights.
And so that's why we looked at the cost function
which is the sum of the square differences.
And then we updated the weights.
And now from here, that was just one iteration.
Next, we're going to run this whole thing again.
We're going to feed every single row
into the neural network,
find out our cost function
and do this whole process again.
So just as we saw previously
where we had just one row
and we were doing everything
again and again, again, again
same thing here.
But now we're gonna be doing it for
eight rows or 800 rows or 8,000 rows,
however many rows you have in your data set.
You do this process
and then you calculate the cost function.
And the goal here is to minimize the cost function
and to get,
as soon as you found the minimum of the cost function,
that is your final neural network.
That means your weights have been adjusted
and you have found the optimal weights
for this data set that you're training on a
and you're ready to proceed
to the testing phase or to the application phase.
And this whole process is called backpropagation.
So some additional reading that you might want to do
for the cost function,
and I know we we just talked about one
and they are many different ones.
A good article is located on Cross Validated.
It's called,
A list of cost functions used in neural networks
alongside applications.
So the URL's there,
but you can just Google for that exact search term
or search phrase
and you'll that this one will be the first one that pops up.
It's actually got some good examples
and application or use cases for different cost functions.
So if you're interested to learn more about cost functions
check out this article.
And on that note, I hope you enjoy today's tutorial.
I look forward to seeing you next time.
Until then, enjoy Deep Learning.