Hello, everyone, and welcome to this new and exciting session in which we are going to build our own
Yolo like model.
So from the paper we had already seen that we have this initial conv layers which are pre-trained on
the imagenet dataset such that they could be used to extract very useful features from our input images.
And then this conv layers were followed by two fully connected layers which were designed in order to
adapt to our problem of object detection.
Now, given that we do not want to train this backbone here from scratch on the ImageNet dataset, we
will use an already pre-trained backbone, which is the Resnet 50.
Again, we have the output dimension defined as number of classes plus five times B from the paper.
B is given to be two.
We've seen this already and then five because we have the probability of obtaining an object.
And then for the remaining four we have the bounding box.
So we have two of this bounding box predictions.
That's why here we have two times five, ten plus the number of classes, which in our case is equal
to 20.
Now we define this number of filters to be 512.
So that's it.
From here we have our full complete model.
You could take this off.
You see that we have, um, our pre-trained resnet, which is our backbone or our base model.
And then we follow this up with several conv layers similar to what we have here in the paper.
And then we have the global average pooling, which is what is given here in the paper.
Now one thing we should note about the average pooling is the fact that when we have inputs, let's
say we have this seven by seven, then by, let's say five.
So we have one channel, um, two, three, four and then five.
So let's suppose we have seven by seven by five input.
Now after going through the global average pooling, what we will have here is the averaging of each
and every value or let's say pixel in each and every channel.
So for this channel, for example, we will have one representative value.
For this channel, we will have one representative value, which is the average.
So for this we will have the average, this will have the average and this would have the average.
So we average all these values here.
And the problem with this is information about object position is lost.
And so instead of using this, um, average pooling is preferable to use the flattening.
And so what we'll do is we'll take this off from here and then we will have flatten.
Okay.
So once we have that, just as with the paper you see here, we have the fully connected layer, which
is this dense layer right here, and then this other fully connected layer.
So we have that and then we reshape.
Now, this should be actually split.
Let's take all this here, copy and paste.
So split by split by split or split by split by output dimension.
So seven by seven by 30.
So this is now our model.
You can see we have a total of 53 million parameters.
Um, 30 trainable and 23 non trainable.
That's from our, um, Resnet.
We are now going to get into the Yolo loss.
So here we have this Yolo loss method which takes in our y true and y pred where this is our target
y.
And then this is our predictions.
Now the Yolo loss we have here will be an implementation of what we already discussed from the paper.
And so it's made of four different parts.
The first part is for the coordinates.
The next part is for the object, the next no object.
And then finally we have the classification.
So we have this four different parts.
Now we are going to start with this first part here.
We're going to start with the object.
So in this case, we shall penalize the model for not detecting an object at a particular cell.
And that's why you see here we have this.
One obj, which denotes if the object appears in cell I and this j right here.
Notice that here we don't have a j here we have this J denotes the fact that the j bounding box predictor
in cell.
I is responsible for that prediction.
Now, if we take a look at this figure right here where we have split this image, which is actually
224 by 224 into seven different parts.
So it's basically now seven by seven grid cell image.
And each and every grid cell, as we had seen, has its own predictions.
Now, if we specifically break this grid cell right here, so let's pick this one.
As you see, we've picked this.
Now we have two outputs or two possible outputs.
We have the Y.
True.
Y true.
And then we have the Y predicted by the model.
Now, you'll notice that if you count this, it's going to be a total of 13 and this is going to be
a total of 18 as we've seen already.
This first year represents whether we have an object or not.
The next four, which is all in green, is the position of the object in the image.
And then this other eight specify the class of that object.
Now, before we move on, you should note that when we first recorded this section on the Yolo loss,
we were working with a dataset with just eight classes.
But since we are now dealing with a Pascal data set with 20 classes, our Y true will look instead like
this.
So now we're going to have this additional 12 values here and this additional 12 values here.
So instead of having 13, 18 as with eight classes now we are going to have 25 and and 30.
So it's like 13 plus 12 and then, um, 18 plus 12.
Nonetheless, this doesn't change much on our Yolo loss method.
So wherever you have.
Eight classes you should consider in our specific case of the Pascal VOC dataset, we're dealing with
20 classes.
Now, we had also seen previously that although we have this Y true, which has this 13 different values
for each cell, this y bread has actually 18 values because we have two possibilities or two possible
positions of the object.
We have this first possible position.
Let's call this B1 and then we have this other possible bounding box which we will call B2.
So we had already seen previously that we set B to two because we have two possible bounding boxes.
And we also saw that even though our model or even though our data was designed such that the outputs
were seven by seven by 13, the model outputs seven by seven by 18 outputs, as you could see here.
And so when we have a loss function like this, where we are given the Y true and the Y pred, where
the Y true is in fact seven by seven by 13, and the Y pred is seven by seven by 18.
Since we want to start with the object part of our loss function, which is this part right here, our
focus should only be on these two grid cells, this grid cell and this other grid cell right here.
So we want to focus only on this grid cell and this because these are the only two grid cells where
we have an object located.
Now, the way we shall select this programmatically is by gathering all those grid cells where the target
is equal to one.
Now, our target here is simply y true, where we've selected only this object score.
So if you look at y true here and if we pick this object score that coincides with this one we have
here.
So in the case where we have an object, we will obviously have a one for the Y.
True.
And so when we say we are going to pick all the different cells of widespread where we have a one at
this y true level, that's this first for this first value of Y.
True.
It means that we will now have y pred, which is going to be all this different cells where we have
actual objects and then we're also going to do the same for y target or y.
True.
So here we have y pred extract and then we have y target extract which is simply taking, getting all
the positions where we have the objects.
But this time around for y.
True.
So now we have y pred where there are objects and we have y true.
Where there are objects.
And with this we could now focus on only this two cells, this cell and this cell.
Now let's take this off and run this so that you could see what this produces.
So we run that.
Let's, let's print this out.
So let's print out y pred extract and let's print out y target extract.
Well, we could also print out target, so.
Well, let's just print out this, this word right here.
So let's print.
Um, tf let's copy this.
We'll get this.
So let's paste this out here.
There we go.
And that should be fine.
So let's run this.
And then now we will test with some inputs.
Now the inputs we are going to be using will simply be the exact same coordinates we got from the dataset.
So here you have this and then you have this where this here corresponds to, um, this, this car,
this vehicle right here.
And then this one corresponds to this other vehicle.
Now, we had seen already how we could use generate output method to produce our Y true or our data
set value for Y or the output.
So once we have Y true, we are going to um, add an extra dimension.
Now we have y true already.
Then for y bread we will just generate this random values.
And then what we'll do now is for this specific values or for this specific cells, that's one, four
and three two.
We are going to put its own values.
You'll notice that here we have 18 different values where here we have the probability of having an
object.
We have the position one, two, three, four.
There we go.
We have the position, the probability of having an object.
We have a position.
One, two, three, four.
This is one, two, three, four.
Here.
And then the for this rest here we have the class or the different class probabilities.
Now, we could do the same for this other one.
We have this and then we have one, two, three, four.
Okay, so there we go.
So this is what we have as our y pred.
So we supposing that the model predicts this and then this is our Y true.
Remember we had already seen that this y true here would produce our dataset outputs.
So now let's run this, let's run this and then run this and then see what we get.
Okay, so there we go.
You can see, um, from here we have all the different positions where the target value is equal to
one.
So that's what we printed out here.
And you could see clearly that we have, um, one, four and three two.
Now the zero here is simply because this is the first batch value.
Now that's it.
We have this one, four, three, two for the next.
We have widespread extract.
So want to extract only those values where we fall in this grid cells.
Now you will notice one thing that you have 0.90.470.31, which coincides exactly with this.
And then we have this other 10.30.01, which coincides exactly with this.
And so clearly we are only focusing on the model's outputs where we have actual objects from our data
set.
So we've picked this here and we've picked this, and then now we're ready to compare what the model
produces at this position and what was expected, which is this Y true.
Now, so this is y pred and then this is y true.
We're comparing only at this two cells.
See here we have two.
Two.
Now, what if we suppose that we have only one object?
So if we have only one object, what we'll do is we'll take this one off.
Let's run this again.
And you'll see now that we only have one which is picked.
So let's get back and run that.
And there we go.
So you see, we've we've picked this.
We've picked this cell and this other cell, and we're ready now, as this described in the paper,
to compare the different probability scores with one another.
So we just simply have to subtract this, and that will be good for this, um, part of our loss.
Now, if we only had a single bounding box prediction like here, if we didn't have this.
So let's suppose we had only this and the Y true.
Only this.
Then what we'll do is we'll take this one minus whatever value we have here, and then we'll subtract.
So if y pred is one and y true is one, we'll just have one minus one, we'll subtract that and then
we square it exactly as we have in the paper right here.
You see, we have, um, the object score minus the target object score and then that squared.
And from here we add this to all the other, um, positions or all the other objects at different grid
cells.
That's why we have this summation.
Anyway, we seen this already.
So, um, getting back here now that we actually have two bounding box predictions, as you could see,
we have B1 and B2.
What we'll do is we'll take whatever value we have here.
And then for this, let's, let's wipe this off and wipe this off.
Let's say this is a dash, this is a dash and then this is a dash or some value.
Let's, let's call this lambda and we'll call this lambda.
What we'll do is we'll take this one minus.
One of these that either lambda one.
Let's if this is lambda one and this is lambda two.
So we'll have one minus lambda one or we'll have one minus lambda two.
And the way we shall pick between lambda one and lambda two is by looking at this coordinates right
here, this positions.
So if this bounding box we have here is closer to this other bounding box, that's a true bounding box.
Then we'll pick lambda one.
But if lambda two's bounding box, that's this is closer to this true bounding box, then we'll pick
lambda two.
So that's how we do the picking and the way we would compare this bounding box with this and then this
other bounding box with this is by making use of the IOU scores.
So let's take this off and suppose that we have this input image.
Let's take all this off to we have this input image and then we have this bounding box.
So let's say this is the true bounding box here.
This is our true bounding box.
And then we have this bounding box B one, which is something like this.
So let's say here we have a lambda one and we have this coordinates.
So we produce something like this.
Let's change this color.
Let's say we have something like this, see, say we have something like this.
So this is what we're getting from this one here.
And then we have some other bounding box where we have maybe say something like this.
Now in this case, because this year let's get back.
So we differentiate between the two bounding boxes.
Let's redraw this here and then we suppose we had this.
Okay.
So because this green bounding box is closer to the red bounding box, we are going to pick Lambda one
and simply we are going to take this into consideration.
We're not going to take B two into consideration when computing our objectness part of our loss.
And so getting back to the code, at this point, we have to subtract 0.9
with one that we have here or would subtract.
Um, this is 12340.8 and one.
So we have, we actually have this our this and the choice between this and this will depend on this
coordinates and this coordinates.
But then our IOU method is designed such that it actually takes in pixel values.
So what that means is we could design our IOU method, which will take the center and the width and
the height of these two boxes.
Let's say we have this this box and this other box or this rectangle.
And so the rectangle we take the center of this, its width, its height, we take this other one's
center, its width, its height.
And based on that, we compute the IOU score.
But the problem we have in here is that what we get in here is not the pixel values.
What we have here is this.
Pre-processed.
Values which we had seen already.
And so if we have the center here, you could read that from here.
If we had the center, which is at position, let's say 43, that's 43.
Well, let's let's fix this to this.
We have 46 140.
So we have this center, which is at around 46, 140.
Now we have well, 46.
Take this off.
We have 46
140.
If we focus on only this first coordinate here, then we would have 46 modulo 32, which will give us
12 or rather 14.
Let's get back.
This will give us 14.
And then now this 14 here will move to the next step.
14 divided by 32 would give us 14 divided by 32 will give us 0.44.
So we would have 0.44.
And this is a percentage occupied in this grid cell.
Remember, we to obtain this coordinate or to obtain this value from the original center value of x,
we would have to make sure that we get this position of the center with respect to this grid cells center,
which is right here.
So this distance from year to year is about 0.44 as we've just calculated.
And that's exactly what Yolo takes in.
But since now we need to get back to this original values from these values we use to train because
we need to be able to compute the IOU scores between a given box and some other box.
Then what we'll do is now reverse this process.
So we are going to go from what we get here, like for example, this value back to this value.
46.
Now if you're wondering why we pick 32 right here, you should note that this image is too 24 by 224.
And if we divide 224 by seven, you would have 32.
So each cell, this design from year to year is 32.
From year to year is 32, and so on and so forth.
So if from here to this point here is 44 or rather 46, then if you want to get this remaining distance,
you just need to do 46 modulo 32 to have this remaining distance in here.
And then you divide this distance by 32 to obtain the fraction of occupied by this distance in this
full grid cell.
So with that, we are going to also repeat the same process for this height coordinate.
So this is the center X and center Y.
We will take 140 modulo 32.
We obtain this value divided by 32.
We now get this fraction occupied from this year to the center.
We've gotten this already.
This is 0.44.
We could also get this Y.
And so that's how we we obtain this fraction as we have here.
Anyway, we've seen this already, but now what we're trying to do is get back from this to a value
like 46 or from this to a value like 140.
Now, it should be noted that for the width and the height is going to be quite easy because remember,
when designing this or when obtaining this, what we had simply done was we took the distance or we
took the width of this and divide it by the image width.
So if we take this and divide by the image width, then to obtain the original width, all we need to
do is multiply by the image width.
So if we had originally, let's say, let's say this width was originally 20, for example, then to
obtain what we have here, this width and this height we take to 2020 divided by 224, we get some value,
let's say approximately this is like one and 11, something like that.
So it's like one on 11 or let's say approximately 0.11.
Okay.
So if we have 0.11 then now to obtain 20 from this, we just take 0.11 times 224.
So that will give us back 20.
So that's how we reverse this.
It's easier compared to this where we need to go through all this two steps before getting back to this
original values.
And again, the reason why I want to get this original values is simply because we want to be able to
compute the IOU between a given bounding box and this two bounding boxes here so we can make the choice
of whether to say one minus lambda 1 or 1 minus.
Under two.
Now let's dive into how we will move from 0.44.
0.375.
It's actually 0.375 to 46 and 140.
So the first thing we'll do is we'll simply multiply this one four by 32.
Now one will become 32 and four will become 128.
And this will take us from this origin right up to this position.
So this is where we will be found At this point here, at this point, you have from year to year is
32.
We've already seen that each grid cell here has a size of 32.
So from year to year is 32.
From year to year is 128 because we have 32, 64, 96 and then 128.
So you see we're already getting close to the, um, center of our object.
And so that's why we have this scalar right here.
We have this, um, re scalar which is gotten by multiplying those values where the target equal one
by 32.
So all the positions where the target equal one, we multiply by 32.
And so now let's run this and you see that we have 32, 128, and for 32 we have 9664.
So for this other object here, for this object, we have, um, one, two, three, that's 96 and then
64 is one two.
So we have this position, we now left to get to the center.
Center should be around this year anyway.
We've taken this first step.
Now the next thing we want to do is get rid of this batch here, this batch dimension.
We don't need that.
So we'll just have 32, one, 28, 96, 64, and then we'll add two zeros for the width and the height.
So here is width and height while this is X center and y center.
Now to do this, we are going to simply take off the batch.
So you see, we start from one.
We take off the batch dimension.
We do not consider the zeroth index.
We take that off and then we add two zeros.
So here we just add this two zeros.
Now the number of lines we will have will depend on the length of our scalar.
If our scalar has length of two, as in this case, then we'll have two by two.
Um, a two by two matrix which we are going to concatenate with this rescaled outputs.
So essentially we had this year we had this 32.
Let's rewrite that when we take only this first index or from the first index to the end, we have 32
128 and then we have um, 96, 64 and then we concatenate that with a matrix which is two by two where
we fill all those with zeros.
And so now we have this here which represents x, y and this height or width and height.
So the next thing we'll do is we are going to take this target coordinates now that we take from 1 to
5.
So it's essentially X, Y, W and H and then we'll multiply by 32, 32 to 20 4 to 24.
Now let's explain why we choose to multiply by others.
So let's suppose that you have, for example, 0.44, you know that this is 0.44 right here and then
you have 0.375 for Y for the center now taking 0.44 and multiplying by 32 will give you this distance
from year to year in pixels.
So if you have 0.4, four times 32, you should have 14.
Now remember that to have 0.44, you had 14 divided by 32.
And so now to get back 14, you simply have to do, um, 0.4, four times 32.
And that's exactly what we do here.
You have this 32 times 0.44, and then you have 32 times 0.375.
So once we have this first two, now you have the width, the height and the width where you have the
values multiplied by 224 and 224, which makes sense because as we've seen before, to obtain the width
and the height, we divided by 224 And so now to get back to the original width and height, we need
to multiply by 224.
So we essentially have in this year multiplied.
By the coordinates or by bounding box coordinates.
Now, the reason why we are repeating here is simply because we could have several objects.
Now in case we have two objects, so we'll just repeat this twice and carry out the same calculation
for the two different objects.
And this repeats here where we specify the length of the scalar, the length of the scalar is equal
to.
So this permits us to repeat twice.
So that's it for the target where we've taken this value, this values, because this is one, this
is zero one, two, three, four.
And so we go when we say 1 to 5, we're taking one, two, three, four.
So we're taking this, this, this and this multiplying by 32 for this, multiplying by 32 for this,
multiplying by two, 24 for this and this by 224.
And that's how we obtain this distance from year to year in terms of pixels.
So from year to year we get the distance.
And then now we could also repeat the same process for the other two predictions we have for the prediction
one, and then we have the prediction.
Two for prediction.
One, we simply do the same, you see here.
But the difference is widespread and yes, widespread unlike here, which is why Target, we still have
the same calculation.
And note how here we go from 6 to 10 because this is 1 to 5 and then this is 6 to 10 for bounding box
two.
So that's the only difference we have here.
And then we would get the part one and part two.
So at this point, for this object here, we note the distance from the origin right up to the nearest
cell, which is this distance in this case is 32.
We had seen that already.
And then we also know this distance from here to the center of the object in terms of pixels.
And that's what we just calculated.
And that gives us about 15.21.
So we have now 32 and we have 15.21.
So it means we have the center or the distance from here to the center in terms of pixels.
Now for this 128, you see we're going to go from here to here.
We know that this is 128 and we also have this distance from year to year, which is ten.
So now we could add this up for the width and the height.
We have simply taken what we had and then multiplied by 224 and that's what we've had.
So we don't need to do any modifications here.
Now you'll notice now that because we have this, we could add this with this, this with this zero,
with this, zero with this, and then we could get our output now, which will be this width and the
or rather this center with respect to this origin and the width and the height with respect to the full
image.
Now, to make it very clear, let's let's consider we had only one year, so let's take off this prediction
and then take off the sort of prediction.
And suppose that we had only one object.
So you could see on that very clearly.
You see here we have, um, 96, 64.
Well, this should be on the order because we took this one off.
Um, we should instead take this year off.
Oops.
This should be the one taken off.
Okay, so let's run this again.
And there we go.
So you see here we have 32 120 800.
We have 15, ten, 28, 52 where we will take now 32 plus 15, and then we take 128 plus ten, we take
zero plus this zero plus that.
And that gives us the the width and the height and also the center.
And then for the predictions, we we will take this, plus this, and then we take this.
Plus this.
Take this, plus this, this, plus this.
And that will be it.
So finally, we'll be able to obtain the bounding boxes in terms of pixels for the target, for and
for the two predictions.
So finally now we are going to take what we had here and then add with this other one target, this
of scalar one and target of scalar two.
So let's run this so you could see the outputs.
Uh, there we go.
You see, we have now 47.
We've taken the 32, plus that around 15 and we've had 47, 138 this, this, we've had this, this,
this, this and then this.
So we now have this bounding boxes in terms of pixels.
So the next step we want to follow is actually compare this box with this and then compare this box
with this and then see which of these two are actually closer to this.
Now, one thing we did while designing this bread was to ensure that the first one, this one was,
is going to be the closer one, because you could see here 4747 they are actually almost the same um,
values.
So that when testing you will see that this one is closer.
So to compare this, we will need the IOU score.
And to compute this IOU, let's consider this simple example where we have two boxes, this B1 and B2.
Now the IOU, as we saw already, is simply the intersection over the union.
So we have to compute this is going to be this here intersection divided by the union.
So it's essentially, um, this region as we've seen already, divided by all this, including this,
um, intersection region.
So it is, um, this divided by this now starting with this intersection, what will need will be this
year, this coordinate year, and also this we need this position and this year.
So the way we're going to get this point is by starting by getting this point, this point here, this
point, this point and this point.
But remember that we actually dealing with the center and the width and the height.
So we are not having the x min, y, min x, max, y max and x min min max y max values by pass in year.
What we pass in here is what we actually receive.
There is the center, the width and the height.
Now the first thing we have to do is to convert this coordinates given to us in this form of center
width and height to one in which we have this x mean y, mean and this x max, y max.
And to do that, we have this piece of code right here.
Now, first thing you have to note is the fact that we have, um, zero one, two, three where this
represents our X center, y center width and height.
Now, if, if I'm giving this year X center, um, y center to obtain this position, X mean y mean
it suffices to take, for example, this x C, which is this distance from year to year and then subtract
it or take away half of the width from it because from year to year is a width and then from year to
year is half of the width.
And if we take x minus half of the width, then we get back to this position.
And then if we take y C minus half of the height, then we get this position here.
So at the end of the day we get back to the origin obviously of this specific, um, box.
So if we want to get back to the origin of the specific box which happens to be X mean y mean I need
to take x minus half of the width and y, c minus half of the height.
And so you see here we have this is x, c this is x.
We've seen this already minus half of the width.
The width is to see the width, half of the width that is C width divided by two and then x, y, c
which is this minus half of the height C3 here.
So that's what we do to have X mean y mean.
And now if we want to have x max, y max, then we have to take x plus half of the width and then y
c plus half of the height.
And that's what we do here.
Just simply replace and put the plus and that's fine.
So that's how this box is.
Now, uh, this box one is converted to box one where we now have X mean y, mean x max max.
And we do the same for this second box.
Now, once we have this, the next step will be to get this coordinates here.
Now it happens that if we want to get this X mean because remember this is the intersection now this
intersection forms another box now to get this year it suffices to compare this X and Y mean and this
other x mean y mean.
So we take the x mean mean of be one and the x mean y mean of B two and look for the maximum between
the two.
And is this maximum or the one which is rightmost because our origin, our origin is at the top left
corner.
So increasing is this direction increasing, is this direction decreasing?
Is this direction and this direction.
So when we say maximum of this and this, then we're looking for the one which is in the rightmost direction.
So when we're comparing here, you see we've taken the first two and the first two here.
So we're comparing the Xs and X mean Y means.
Actually.
So when we compare the X and Y, I mean, we get the one which is maximum and that's the one which will
play the role of the mean of this intersection rectangle, which we formed here.
And then for the x max, max, we need instead a minimum.
So we need the one between these two, between this and this, we need a one which is to the which is
leftmost.
So that's why we instead take the minimum and we compare the last two indices.
That's X, max, Y and Max, as you could see.
And then that's how we obtain this position and this position for this intersection rectangle.
Now, once we've gotten that, that is x min min x max max for this intersection, the next thing we
want to do is actually compute the width and the height so that we could multiply and get the area of
this year.
Now to get the width and the height is quite simple.
We just need to take we need to subtract this simple because here you have let's, let's write that
again.
We have X mean Y min x max y max.
So you just take x m that's x max minus x and then you multiply by y max minus Y.
That's all.
So this is the width and then this is the height.
Take this minus this times this, minus this.
That's all.
So that's what you see which is actually done here.
You see, we've done the subtraction from here, we've done the subtraction.
And then we now multiply this two and that's how we get the area, the intersection area.
Now once we have the intersection area, the next thing we want to do is obtain the union area.
So we take the box one, you see two is the width and box or one again the height, multiply the width
times height.
We have this area for the box to width times height.
We have this area now we add this two areas up.
You see here, we add this two areas up.
We remove the intersection because we want to get only this and not we do not want to add this twice
because when you calculate this area, you already have this.
And when you see calculate the area, you still have this.
So you want to take off the intersection because it's going to be computed twice.
And then we now have the full union.
And so now we have intersection divided by union and that's it.
So once we have the intersection divided by union, now we will be able to compare this with this and
this.
And so now in order to compare this different boxes, we would have the target box compared with Pred.
The second prediction and we also have the target box compared with the first prediction box.
Well, before doing this, we could actually print this two out separately.
So let's print this one out.
Let's do TF print.
Um, second, second box that's comparing the target with the second box.
Let's have that there and then we'll compare the target with the first box.
So let's print now first box.
There we go.
We have now print one.
Okay, so that's it.
We'll run this before having this.
So you better understand what we're doing here.
So let's run this.
Um, let's take this off.
We could take this off and then run this.
You see that with the second box, we have 0.16.
And with the first box, we have 0.93, which makes much sense because when you look at this predictions,
the first box looks much more similar to the target as compared to the second box, which is this one.
So it makes more sense that the first box has a higher IOU score.
Now, applying the TensorFlow math grater method, we will be able to have this boolean output, which
tells us whether this second prediction is greater than the first prediction or the IOU between the
target and the second position is greater than the IOU between the first and the target.
So, um, given that in this case, for example, this IOU between the first and the target is greater
than the IOU between the two and the target, then this will output false.
And since this will output false, casting this to an integer will produce an output of zero, and the
zero will simply mean that between the two options.
That's um, the first output and the second output.
This is actually the first output.
And then this is the second output or second bounding box between the first and the second bounding
box.
This is the one which has, which is closer to the target.
And that's exactly what we want to have here.
So.
There we go.
We have this output zero.
Now let's include this other example.
So now we suppose that we have two objects, this object and this object.
By adding this, we've added this other one here, which happens to be this object.
Then we'll uncomment this part and then see what we have for the mask.
So there we go.
You see, we have zero one meaning that for the first object that is this object here.
It is this second box, which is this one here, which is closer to the object or to the target.
And then for the second box, that for the second object, here it is this first, which is closer.
Now let's let's interchange this, let's take this off here and paste out here and then take this from
here and then paste out here and see what we get.
Take this off and add a comma right here.
So now, because this one here will be closer, we should have this one being picked.
So we should have one one.
Now let's run this and see what we get.
You see, we have one one.
And you could see that from here.
The second boxes have higher IOU scores compared to the first boxes.
Okay, so now we have the bounding boxes which are closer to the target.
Like in this case, we know that B1B1 is closer to the target as compared to be two.
So the next thing we need to do now is get Lambda one and lambda two, and then from this lambda one,
lambda two, choose um, lambda one.
Now let's start by getting lambda one.
Lambda two is going to be quite easy.
All we need to do is pick this zero value and this fifth value.
Remember, this is zero one, two, three, four and then five.
So this is a fifth.
So that's why you see, we pick this.
This is the probability of having an object for B two, and then this is that for B one.
In fact, this is Lambda two and then this is Lambda one.
So we concatenate this and then we rearrange this by transposing.
So let's run this and see what we get.
There we go.
As you can see, we have 0.9 and 0.8.
That's it.
And then we have 0.3 and 0.98.
And so what we'll do now is based on the mask, we'll say, okay, for the first value, which is zero,
it means that we are going to take this one instead of this.
So we're going to take this probability instead and then move it to the next position or move it to
the next object.
We are going to select the value number, this first index or this first or the second value.
So we'll skip zero and then we will pick one.
So for the first one, this is our lambda lambda one we picked, and then for this other one, lambda
two will be picked.
Remember that if we, um, change this positions, then we will pick Lambda two for this one and lambda
two for this one because this will be one one.
So since this is zero one, we'll pick lambda one for this first object.
And then since this is one here, we'll pick lambda two for this other object.
So now to do that programmatically, we are going to gather for all this, um, lambda values.
That is probabilities here based on the mask.
So doing that you'll see that we'll be able to pick those values of lambda corresponding to this bounding
boxes with the higher scores with respect to the targets.
So we've seen how when given this, let's let's, let's take a single example.
So let's comment this again.
We'll comment this and this run that.
We'll see how we're able to take this two bounding boxes that this bounding box and this other bounding
box.
And then from this pick, the one with the higher IOU score and hence pick the probability which we
are going to use to subtract from one.
So now we know that we will have one -0.9.
Now let's reverse this.
Let's reverse this again.
Um, we'll reverse this, take this off and then we paste this out here and then we take this off.
Then we paste this here.
Okay, so let's run this and then see what we get.
You see, now we have 0.8 that we've picked this other one instead because this bounding box is having
a higher, higher score with respect to the target as compared to this other bounding box.
So this is how we are going to pick the bounding box which we are going to use for computing the objectness
part of our loss.
Okay.
So finally, now we just need to compute the difference between what we have here that's in this case,
0.8, for example.
Let's, let's get back.
So we have what we had originally, which is going to be 0.9.
So different between this.
0.9 and one.
So you see here we have once, so you could see the length is based on the scalar.
So since we have only one object, this is going to be one.
So one -0.9 and that will be it.
Now, this difference method we have here has been defined is basically the square of y minus x.
So we take to subtract, find the square as defined the paper, and then we have this reduced sum method
to get a single value.
So that's it.
We run all this and we could now print out our um, object loss.
So we have print object loss.
There we go.
Let's run that.
And you see what we have.
0.01.
You see that?
That's quite small because 0.9 is close to 0.1.
Now let's just modify this and say, okay, let's say 0.09.
So that's going to be having a higher loss.
You see that we have higher loss now.
That's it.
So that's what we do now.
You see that no matter what we do here, even if we put here 1.0, we'll never have a loss of zero here.
And that's because it's this one which is used to compute that loss.
We are now going to move to the No objectness part of our loss, which is this here.
And it's kind of similar to this calculation.
But the difference is that now we focusing on those regions where we do not have objects.
So unlike here where we focused on this here, this cell where there's this object and this other cell,
what we'll do now is we'll be focusing on all the other cells except for this and this one and this
one.
So as you could see, we get all the Y parts.
We gather all this predictions where the target is zero.
That is simply where there is no object.
So let's print this out.
Let's print out this one different positions where we have no object and the corresponding predictions.
Here we have tf prints y pred extract.
So this is seven by seven, meaning that we have 49 different cells right here.
Now two cells have objects and the remaining 47 do not have objects.
So let's run this and see what we get.
And there we go.
Yes, what we have.
So you could see clearly from here that we have all these different cells and let's do print so we could
get its shape.
Um, there we go.
Let's run that.
You can see this actually 48 now.
Um, that's because.
Okay, so that's because we had only one object.
So if we have two objects, two objects.
Oops.
Let's get back.
Um, we have two objects.
So let's run this again and see what we get.
Okay, so you see now we have 47 cells where there is no object.
So that tells you that, um, out of the 49, we have two where there is an object.
And then this y pred extract here now contains all this, um, score and bounding boxes.
You could see from here that this is 47 by ten.
So let's get to the bottom.
You see 47 by ten, because for each and every one of this, we could take this off from here.
We could make this connection from, um, a cell where there is no object.
So this cell, for example, here.
We have this ten different, um, predictions and then now we will make use of this to compute the loss.
So obviously the no objectness part of our loss will be computed using this lambda and this lambda.
So we'll take this one or rather zero because we expect it to be zero.
So this time around our Y true will be zero because when there is no object like yours, zero.
So this is going to be zero minus lambda.
Here, minus lambda one.
Um, square plus zero minus lambda two square.
So the idea here is to ensure that those probabilities equal zero when there is no object and equal
one when there is an object.
So getting back to the code, you see that we break this up into two parts.
This like getting the Lambda one or competing with the lambda one, and this is like competing with
the lambda two.
So unlike with the object, um, part of the loss where we're trying to pick which of this was responsible
for the prediction here, we just simply take this minus this, plus this, minus this.
So, um, our target is obviously zeros, unlike where our target was once.
So now we have zeros and then we compute the difference between our Y target here and the y pred.
Now we pick the zero for this one, and then we pick this five for this other box right here.
So that's it.
We simply, um, find the difference or compute this difference using this method which we've defined
already right here.
And then we sum those two up to obtain our no object loss.
So that's it, let's print out or let's take off this here and then print out our no object loss.
We have no object loss.
There we go.
You see, we have 110.
Now we'll move to this next part.
That's for the classification where we focus on the objects class.
You'll see that we are going to only compute this or get this loss for cells where we have an object
here only where we have an object and we do not care about which of the bounding boxes is responsible
because we actually focusing on only the classes.
So instead of I here we have just I because we are not focused on choosing or we are not interested
in choosing any specific bounding box, given that it doesn't really matter since we're focusing on
classes.
So getting back here, we have our object class loss, we have the predictions and the target.
Now, if you check from here, you remember that the target will start from five because this is zero
one, two, three, four and then five.
But for the predictions, it will start from ten.
So you see here we have zero, one, two, three, four, five, six, seven, eight, nine and then
ten.
So we start from ten right here.
So this starts from five and this starts from ten.
Now we go from ten to the last and then we go from five to the last and then we'll simply compute the
difference between this and this.
So that's it.
We also make sure that where there's an object.
So now you have this difference and you obtain your class loss, you see that we obtain a class loss
of 5.47.
Now we get to the last part of our loss, which is that involving the coordinates, which itself is
broken up into two sub parts.
This first part is just for the center and then this other is for the width and height.
Now this part is more similar compared to this object, this part of the loss, simply because here
we are going to focus only on cells where we have an object and only on those bounding boxes which are
responsible for the prediction.
So again, we are going to gather all our predictions here.
We gather all our predictions where the target is equal to one.
So simply where we have objects.
Then.
Now we combine the centers.
So we see we go from 1 to 3 and then from 6 to 8.
Reason being that this year, because this is zero, this is one, this is two.
So this year represents our centers.
And then when you have here, this is actually, um, six seven and this is representing the other centers.
So this one is the center, this two center.
So that's why you see we take this and this and stack them up to form our center joined then similar
to what we had done already with the Objectness loss, we are going to only pick a given center based
on whether it's the bounding box which is responsible or not for the prediction.
And again, we're going to use this mask.
Remember, we had seen this already previously with the Objectness loss.
So we had this already seen so exact same process we're following here.
We just want to make sure we're picking the bounding box which is responsible for the prediction.
And since we've computed this mask already, that's what we're going to do.
Now, let's, let's print out, um, our center joined and then let's print out the center pred So we
see that we actually pick out only some bounding boxes from the two choices we have center print.
You see here that for the first object, which is this, we have this option and we have this option
now because for the first object, this is the first or the zeroth index that's responsible, you see
here.
And then for the second object, we have this option and this option.
But because this is this one that responsible or the second bounding box or the or the first index that's
responsible, we actually pick this.
So you see that this one is discarded and this one, too, is discarded.
So we focus only on this.
Now for the target, we simply just pick out this one and two.
So that's it.
So we pick this.
Obviously, going from 1 to 3 is simply taking one and then taking two.
So we pick this and then now we compare with whichever one of this is responsible for the prediction
and comparing that is simply applying our difference method.
So now that we're done with the center, we we finished with the center that's actually this part here.
We're now going to move to the width and the height.
So here is the exact same thing with just the difference that we're picking the width and the height.
So now instead of one two, we're going to pick three, four.
So that way you see we go from 3 to 5 and then here we pick eight, ten.
Well, okay, this is a prediction.
So instead of picking this, this, we pick three, four.
So we have this.
Um, let's pick this.
We have this three, four and then eight, ten.
So that's what we do here.
We stack them up, we carry out the selection, and then we also get the target.
So we take this three, four, see that, and then we compute the difference.
Now remember that when computing this difference, we have to make or take the square root.
So you see here we have square root.
And since the square root takes in only positive numbers, we make sure to compute the square root of
the absolute values.
So that is it pred and size target.
And from here we've gotten the center loss.
We've also gotten the size loss.
This now forms our box loss and that's it for all the different loss functions.
We're now simply going to add them up now before adding up.
Remember from the paper we had seen that lambda coordinate is going to be five and lambda no object
is going to be 0.5.
We have seen this already from here.
We have this lambda coordinate and this lambda no object.
So that's it.
We have this.
We make sure when adding this up, we take this into consideration.
So let's run this and then, well, let's print out the loss.
So let's print out the loss.
There we go.
And that should be fine.
We are then going to define our model checkpoint where our file path is this here.
Then we're going to save only the weights.
We're going to monitor the validation loss.
Um, we're going to obviously save the model which produces the minimum or the smallest validation loss
and that's it.
We save the base, we save the best weights only.
So we run that, and then now we move to the scheduling here.
If the number of epochs is less than 40.
So the first 40 epochs we use, um, a learning rate of one times ten to the negative three between
40 and 80.
We use a learning rate of five times ten to the negative four.
And then after that we use a learning rate of one times ten to the negative four.
So that's it.
We compile our model and then we start with the training.
Now after training for a few epochs, you'll notice that the model starts to overfit.
And so in the next section we are going to treat our use several techniques to help solve this or resolve
the problem of overfitting.
Now we've been training for over 20 epochs and you could see clearly from the loss and the validation
or the train loss and the validation loss that our model starts performing well and at some point it
starts overfitting.
As you could see here, we have the training loss which keeps dropping right here.
And then the validation loss drops and then at some point starts increasing.
So clearly our model is overfitting.