Hello everyone, and welcome to this session on error Sanctioning.
The error sanctioning method or the loss function we shall be using in this section is going to be the
binary cross entropy loss.
And right here we have this binary cross entropy loss formula.
So um here we have negative y log y or y chapel uh minus one minus y chapel log of one minus y chapel.
To understand this formula, we'll start by um taking this um log plot into consideration.
So our log function is this plot we have right here.
Or it could be, um, explained using this plot right here where the, you see, as we approach zero,
as we're going towards zero, the log approaches negative infinity in the section.
Our values for y will fall under the range zero one.
And so with that we'll consider that we have a Y because here if we we could consider that our y.
Um chapel is what the model predicts.
So y chapel is or the model's prediction.
And then y itself is the actual um or expected prediction.
So here we have model's prediction.
Take that off.
We have model's model's prediction.
There we go.
And then right below or just here we have the actual prediction.
So now here we have um actual prediction.
So this is what we expect or expected prediction.
So that's it.
We have Y chapel and we have y.
Now when you or let's suppose that we have um or the model predicts let's say this is what the model
predicts.
And then this is the actual.
So here we have model and then we have the actual prediction.
So let's suppose that the model predicts zero an output of zero.
And ah the actual prediction is the label zero.
In that case you would have um y I let's let me take this off.
Let's, um, take this off here and have that to be Y and y chapel.
So here we have y chapel, which is what the model predicts.
And then this is what we expected to predict it.
So here we have zero and zero.
So we have zero.
But because we're multiplying this zero by uh very large negative number, all here multiplied by a
very large negative number, we're going to have output of zero.
And then on this other side we have one -0 which is one times the log of one -0 which is a log of one.
But when you look at this log like you get back to this, um, plot, right here, you have the log
of one which is zero.
And so because log of one is zero we have one times zero.
And so overall we have zero.
So we find that the error, the error at the end or the loss is zero.
And this makes sense because we want to have a loss function which is such that when our model predicts
exactly as uh, or the same output as the actual prediction, then we should have uh, an error of zero
or loss of zero.
Now let's suppose that we have um, zero and then one.
In that case you would, you would have 0 or 1 right here.
So this is going to be one.
This is one times a large negative number is um here we have log of um y or chapel which is log of zero.
It's a large negative number as we have seen already.
So one times a large negative number.
So um, a large negative number.
So we'll have negative um, let's call it l let's call this L uh, here we have negative l and then
here we have one -0 which is one times the log of one minus one which is the log of zero.
So we have uh one times a large negative number again which leads to, um, us getting um, l so here
we have l so overall we're going to have negative l minus L, so we will have uh, negative two L.
So we have a very large negative number.
So.
We are penalizing the model for predicting the wrong outputs.
Actually, when you have this negative, this negative times this negative will give a plus here.
So this negative times this gives plus.
And here we have minus.
So we have minus here.
And then we here we have one times log of zero which is negative l.
So here we have negative l and l minus minus l is two l.
So we have a large positive number.
But what's important to note is the fact that we are penalizing the model for the wrong prediction.
Now if we have one and we have zero we would have something similar.
Then if we have one and we have one, then we should have a zero, because we can now try out this binary
cross entropy loss method on or in TensorFlow with a simple example.
So let's say we have y true and there we go.
We have y true.
Let's give it some values.
Let's say we have zero.
We have one we have zero and then we have zero.
And then we have y pred widespread.
Let's say we have zero.
We have one, we have zero and then we have zero.
There we go.
So we have our binary cross entropy loss which is equal.
This um TensorFlow Keras, um binary cross entropy, binary cross entropy.
There we go.
And then now we could simply call that.
So let's print out BCE which takes in the y true.
And then the y pred.
We run that and let's see what we obtain.
Getting an error cannot convert 0.0 to integer tensor of type int 32.
Well, our y pred should be a float.
So let's have this point and then run that again.
And there we go.
You see we have an error of zero.
Which makes sense because the y is the same as the y.
True.
Now let's suppose that we have 0.8.
We have um instead of zero we have 0.8 instead of one we have 0.2 instead of zero.
We have say zero point um nine.
And then instead of zero we have let's say one, we run that again and then we'll find that the loss
now must have increased as compared to when we really predicted all the values.
Now let's go closer to the actual values.
Let's reduce this, let's increase this and then let's reduce this.
And then let's say this is zero.
We run that again.
And you see now that compared to the five we just had, now we'll have a smaller value though not zero
but um a value approaching zero.
So you see we have 0.1, which makes sense because our predictions are now much closer to a y true values.
And so that's it.
We understand the binary cross entropy loss.
Now we will dive into compiling our model.
So we just need to have model compile.
And this compilation will consist of specifying the optimizer.
The optimizer which is Adam optimizer.
Uh we have learning rate learning rate equal say 0.01.
Then we have um, the loss which we just defined.
Um, this loss right here, which we've just defined is the binary cross entropy loss.
So we have binary, um, cross entropy.
Getting back to the documentation, you would find that we have this from logits, um, parameter right
here.
If the output value or values range between negative infinity and positive infinity, then we want to
use the from.
Or we want to set the from logits to be true.
But if it's a probability that if it lies between 0 and 1, then um, we will have from largest to be
false.
Now if you get back to our model, you would find that we have the sigmoid activation.
And the sigmoid activation, um, makes or ensures that we have values ranging between 0 and 1.
And so because of that we have our from logit set to false and which is the default value for the from
logits argument.
We also have the labels model.
You could check out documentation float in the range zero and one where when when we have zero there's
no smoothing.
Um that's it by default is zero.
So there's no smoothing.
And when it's greater than zero, we compute the loss between the predicted labels and a smaller version
of the true labels.
So where the smoothing squeezes the label towards 0.5.
So larger value of label smoothing corresponds to heavier smoothing.
Now if we get back to the code, what will actually or what they're trying to say here is if you have
the label smoothing set to say 0.1, then by just having 0.1 year and year, 0.9 and 0.1, and then
let's say zero and running that we see we have a loss of 0.35.
Well this is zero point.
One.
This is 0.9.
This is 0.1.
Let's say we have now 0.1.
We should have a loss.
A small, an even smaller loss.
And this is because instead of comparing zero or instead of comparing zero, we have here with 0.1,
we're now comparing 0.1 with this 0.1.
And that's because we specify the label smoothing value to be 0.1.
So you could see again that having all this the exact same for y pred and y true doesn't yield 0 or
0 anymore.
If you go back to zero and you run this, you should have zero.
Let's get back to 0.1 and then modify this y pred.
So you see when you have um zero we should have um an output of zero.
So what we're getting here is for this output zero going back to 0.1, you would find that when is when
you increase zero or when you move 0 to 0.1 and then 1 to 0.9.
And here 0 to 0.1 that you have a smaller output for the loss.
And so that's it for this section.
In the next section we'll dive into training our model.