-: Hello and welcome back to the course on deep learning.
Today we're talking about RELU,
which is rectified linear units.
And this is an additional step
on top of our convolution step.
So it's not a separate big step, it's a small step.
It's step one B basically.
And what is going on here?
Well, we have our input image
we have our convolution layer, which we've discussed,
and then on top of that, we're going to apply,
wait for it, our favorite, rectifier function.
And you're really familiar with the rectifier function
from the previous section on artificial neural networks.
And in our, so sometimes
authors or instructors separate
the convolution and the rectifier as two separate steps.
In our examples
we are just going to consider them just one big step.
First the convolution, then the rectifier.
And the reason why we're applying the rectifier
is because we want to increase non-linearity in our image
or in our network, in our commercial neural network.
And a rectifier acts as that filter,
or acts as that function, which breaks up linearity.
And the reason why we want to increase non-linearity
in our network is
because images themselves are highly non-linear,
especially if you're recognizing different objects
next to each other or just on backgrounds
and stuff like that.
Like the image is going to have lots of non-linear elements.
And the transition between pixels,
adjacent pixels is often gonna be non-linear.
That's, you know, because there's borders,
there's different colors,
there's different elements in your images.
And, but at the same time
when we're applying mathematical operations
such as convolution and running this feature
detection to create our feature maps
we risk that we might create something linear
and therefore we need to break up the linearity.
So let's have a look at an example.
Here is a image, an original image.
Now when we apply a feature detector to this image
we get something like this.
So you can see here
that black is negative, white is positive values.
Well, when you apply a feature detector to a
proper image, which has not just zeros and ones
but has lots of different values, and you apply
as we saw previously
future detects can have negative values in themselves.
Sometimes you'll get negative values, and here or there
black ones are negative, white ones are positive.
And what a rectified linear unit function does
is it removes all the black, right?
Anything below zero turns into zero.
And so from this, it turns into this, right?
And so it's
it's pretty hard to see what exactly is the benefit
in terms of breaking up linearity.
I'll try to explain, I'll try to show an example
on this image, but at the end of the day,
this is a very mathematical concept and we'd have to go
into a lot of math to really explain what is going on.
But let's try, let's have a look.
So for instance
let's look at this building here, right?
So this is a building on its own.
Then you can see this shadow, this black part
this shadow over here.
Well, you can see that it's white,
the reflection of the light,
and then it's a gray, and then it gets darker
and then it gets darker again, right?
So, and when we take it out, we take out that black part.
So think of it in terms of linearity, right?
So it looks like when you go from white to gray
the next step would be black, right?
The next step would be black.
It's a linear progression from bright to dark.
And therefore this is kind of like a linear situation.
When you take out the black, you break up the linearity.
Let's try another one.
Let's have a look here.
And, and at the same time
it's still that same building, right?
It's not like you are
blending two buildings
into each other, but that is secondary.
The main point is breaking up the linearity.
So let's have a look here.
Same thing.
So you see white, gray, black, gray, white.
And when you break it up
you don't have that anymore, right?
You don't have that progression,
a gradual progression that you just have
like an abrupt change.
And that helps introduce non-linearity into your image.
So it is a very rough explanation, very kind of like
on the fingers explanation rather than technical.
But hopefully it kind of helps you understand
a bit better what we are talking about here.
So here again, you can see white, gray is a better example.
Even see bright, darker, darker, darker,
darker, darker darker.
So this part looks like it's linear.
Then you break it up like that.
Again, so this is a very rough explanation.
It's not absolutely perfect, but at least it gives you
some idea of what's going on.
But if you'd like to learn more,
there's a good paper as always.
There's always a paper.
This one is by CCJ Core from the University of California
and it's called Understanding Convolutional Neural
Networks Who Have a Mathematical Model.
And basically there he answers two questions
and you need to just look at the first one.
And the question is, why a non-linear activation function
is essential at the filter output
of all intermediate layers.
So that kind of explains it in a bit more detail,
both in terms of intuition
and mostly in terms of mathematics.
So that's an interesting paper where you can get some more
additional information on this topic.
And if you really want to dig in
and explore some cool stuff here
then there's another paper
that you might be interested in.
It's called Delving Deep into Rectifier,
Surpassing Human Level Performance
on Image and Net Classification.
And here the authors Kiming, Head, and others
from Microsoft Research, they propose a different type
of rectified linear unit function.
They propose the parametric rectified linear unit function
which you see here on the right.
And they argue that it delivers better results
without sacrificing performance.
So interesting read
if you'd like to get a bit more into this topic.
And that's all for today.
The RELU layer is pretty simple,
pretty straightforward, just applying the rectify function.
And I look forward to seeing you next time.
Until then, enjoy deep learning.