This yellow paper published several years ago was one of the first to come up with a single neural network
which predicts bond and box and class probabilities directly from full images in one evaluation.
Now, back then, models are, to be precise.
Object detection models followed this kind of pipeline where we have a region proposal generator, a
feature extractor, and then a classification unit.
So you can look at this with a simple RCN model where we had the input image, the extract regions,
that's regions where the model thinks we could have images, where the model proposes the locations
of objects.
And then from your each proposed location is passed into this feature extractor which obviously extracts
features from those warped regions.
As you could see here, for example, this region.
And then we have the classifier which tells us whether this is a person, an airplane or say, a TV
monitor.
Now, with the yellow, as we were saying, we have a single network, so let's get your you see, we
do not have all that different stages in our pipelines here.
We just have our input image, a single neural network, and then we have the outputs.
See that there's also this additional non max oppression here, but we'll look at this shortly.
Now, the performance of the yellow is quite impressive or was quite impressive in terms of speed is
especially as now we can obtain speeds of up to 45 frames per second.
And with the yellow of the smaller version of the yellow, we obtain up to 155 frames per second while
achieving double the mean average precision of other real time detectors.
We'll look at the mean average precision subsequently.
This is the metric used generally in object detection, and so a high mean average precision means the
object detection model is performing better.
Another advantage of the YOLO to take note of is the fact that it reasons globally about an image.
So unlike the sliding windows and the region proposal based techniques, we looked at the original the
region propose proposal based techniques like the RCN here for the sliding windows.
The way it works is we have let's get this way, we have this image right here and then this window.
You see, you can you can look at this as a window.
So this window is slide that through the whole image, unlike here where we just simply pass in the
image as an input with a sliding window, you would have to take each window and pass into our neural
network.
And you do that while you're sliding through the full image.
So as we're saying, getting back to what we're saying here as compared to the sliding window and the
RCN kind of models, the YOLO performs better in the sense that it sees the entire image during training
and test time.
So it implicitly encodes contextual information about classes as well as the parents.
So you find that models like the RC and mistake background patches in an image for objects because it
can't see the larger context.
Okay.
Another point is the YOLO learns generalizable representation of objects.
So you'll see that here the YOLO, which was which wasn't trained on this kind of paintings, performs
quite well.
You see the YOLO was was trained on on this particular version of YOLO was trained on the Pascal view
C dataset.
But when we test this on this image paintings, you see that it does well.
So the YOLO models compared to others, learns more generalizable representations of objects.
Let's now go in depth and see how the YOLO algorithm works first and first.
Remember that if you have this kind of model, see our YOLO model, which you could have your.
There we go.
You see, this YOLO model has several calf layers and then completes with this connector layer.
So we have the feature extractor and the classifier unit right here.
Anyways, we are not going to get into this.
Now let's just consider that we have this model.
And then we have inputs like this one.
See, these are our input and then we have some output right here.
Now, this output obviously is meant to be a bound and box.
So here we will we could draw this bounding box for this woman here.
So this shows that or this bounding box is for this this person.
And then let's take this off at the stroke.
Let's change that color.
So here we have this your this, this person detected.
And then we also have this other person right here.
So we'll have someone like this.
See that?
So we have now this two detections, one for this woman and and this other one for this other person.
And so we're going to build a model, which is this one which takes in these kinds of input output pairs
and then learns to get those inputs and predict the the outputs correctly, such that when given a new
input, it could tell us where every object is located and what type of object it is precisely.
Now, that said, the actual outputs wouldn't be this image.
With this bounding box, the actual output will be different from what we are seeing here.
Now, the way this outputs are created is using some sort of encoding system where we have this input
which is broken up into some grid cells.
So right here, take the pen.
So right here, you see we have let's suppose that we have this 224 by 224 input image, that's this
image here, 224 by 224 this one year.
And then we break this up into several grid cells.
So this is a single grid cell, another one, and so on and so forth.
Now each grid cellular, given that we have seven grid cells, so we have seven by seven output, see
that we have one, two, three, four, five, six, seven, one, two, three, four, five, six,
seven.
So that's it.
And now each of this is going to be 32 by 32, so it's like 32 by 32 patches.
So we take each patch here, combine them to form 224 by 224 or better still we take the 224 image and
break it up into 32 by 32 grid cells.
See that now once we have this ready or once we want, once we have some somehow broken up our image
into this grid cells, we are then going to encode the outputs based on the locations of the center
of our bounding boxes right here.
So you see we have we're going to redraw this bounding boxes so that you see that clearly.
Let's increase this to say five.
Okay, So here we are having this bounding box right here.
See this bounding box?
And then it has a center at about this position here.
See that position?
If we if we want to locate this here, it falls about this around this year.
So you see, it's about this.
Now, for this other person, we have another bonding box like this.
See that sort of bounding box?
And the center is about this, about around the child's nose.
So it's around your see that?
So what we're going to do now is we're going to have each and every one of these let's change this color.
We're going to have each and every one of this sorry, We're going to have every one of this year.
Having certain values.
Now in the case where the so like this one let's let's make this bit more transparent the case where
a cell like this one, let's do 20 so you could see that better.
So are we going to have as we're saying, and the case where a cell like this one which doesn't contain
an object, then we'll say, okay, this first value will be a zero.
You see that this first value will be zero.
Let's get back.
And here our first value is zero.
So for this cell, the cell, your first value is zero.
Now, for this order cell, your where the there is an object there.
See, this value will be a one.
You see that?
So each and every cellular, each and every cellular has or takes certain values based on whether there
is an object or not.
Now, as you can see, this cell, you have a zero, the zero, in fact, all the zero and except this
tool which we put in red, which will take values of one.
And this is simply because it happens that the centers of the bounding boxes fall in those cells.
So that's the first step.
Now, once we have this first step, the next thing we want to do is we want to locate the exact position
of our images.
So the first thing is, we want to know whether there's, first of all, an image that's by encoding
it like this.
The next thing we want to know is the exact position.
Now, this exact position obviously depends on the kind of the way we want to represent our bounding
boxes.
Now, we could represent our bounding boxes by specifying X mean X mean y mean and x max y max.
With this kind of representation, an object like this maybe right here, or let's say a person right
here will have this of values, or we'll make use of these values to locate this person.
So we'll make use of this point here, which is X mean, y mean.
And this other point your which is x max y max with respect to the origin, which is the top left corner.
So this is our origin right here.
You see that?
So we go X steps and Y steps downward.
Then your X steps to the right, Y steps downward to locate this person.
Now, once we once we've done this, we could we, we could just put this out here so we could say,
okay, we're creating our output.
Remember, our aim here is to create our outputs.
So we are creating our outputs.
We know for every cell or for every grid cell where objects are located.
That's it then.
Now, to get the bounding boxes, we could make use of this.
But what's important to note here is the notation used by the authors of the YOLO v one paper was instead
of X center y center, then the width and the height c that of the bounding box.
Obviously none of the image.
So here instead of having making use of x mean y mean x max y max.
We make use of x center, which is the center y center.
So we'll go x.
Y.
And then we look for the width of this box, the width and also the height of the box.
So let's get back.
So that that's basically how we do that.
And then there's another spatial encoding which is done.
And what they actually do is for the width and the height of the bounding box, they're going to divide
this by big W, where big W is the of the width of the whole image.
So if our image is 224 by 224 we'll take this width.
Let's suppose that the suite is say 160, so we'll take 160 divided by 224 and get our width and then
for the height we'll do the same thing.
So if we have the height of say 200, we'll take 200 by 224 and we get that.
So it's h divided by the height of the whole image.
See that?
Now once we have that, we, we here for this x y, see, we're going to do something similar, but
not dividing by the whole width and also not dividing by the height of the image.
What we are going to do here is we are going to have this x c with respect to its specific grid cell.
Now, let's explain.
Let's pull this way so you could see that clear.
So we suppose in that we have this here and we've already seen how to get for the width and the height.
We take that divide by the total width and for the height divided by the total height, total width
and total height in our example is 224.
Now for the x and Y.
See here, for example, we have this, our x, y, c is for these other objects, our x, y, z.
Let's consider the example of let's consider this example here.
So what we're saying is we are not going to take this respect to the full image.
Instead, we're going to say we're going to take this grid cell and suppose that this distance is one
and we take this and suppose that this distance towards one.
Now, our origin here is this.
See, this points our origin.
Now, if we're for this cell, this will be our origin and this will be our distance one.
And here's our distance one.
Now, if this is our distance one, this distance one, then this point here, let's say this point
here will be a fraction of one basically.
So it be a value between zero and one.
Now, if we take this distance, we could approximate this to be about zero point, say, five six.
And then this distance is about, say, 0.7.
So in this case x c will be this distance and y c will be this distance.
In that case, we'll have 0.56 and 0.7.
So that's it.
Now, once we have that, the next thing we'll do is we'll just simply put that out here.
So we'll have that x, z.
With respect to the grid cell, we write that G and then we'll have Y, C, G, That's it.
So here we now know how to obtain these values.
Let's change the color back.
So here we would have say 0.7.
0.0.56 is 0.7.
So that's how we have this.
Now, once we once we get this, remember we already have one year, so the next one we have is x,
c, g for the next part.
If we have a data set where we have say 20 classes like the Pascal velocity is set with a cocoa data
set where we have 80 classes, what we would have from your C we've had whether the object is there
or not, if the object is there, we want to get this location now from the location.
We want to know what object exactly is found there.
So yeah, we suppose that we have 20 classes.
C Now, initially every one year is zero, so we have 20.
We're just going to align 20 zeros.
So basically we have this, this 20 zeros here.
Now if the person even in our classes, we decide that the person occupies the of third position in
a list of classes.
So we could, we could have a class or let's just let's just check out the list of Pascal C classes.
Whereas you can see here we have the 20 different classes and then if we count this, we have one,
two, three, four, five, six, seven, eight, nine, ten, 11, 12, 13, 14, 15.
Notice how this person is the 15th position.
15, 16, 17, 18, 19, 20.
Okay, so we have the person at that 15th position and so.
This means that when encoding our image here or when encoding our output, we would have to take that
into consideration.
Will simply do.
One, 2345, six, seven, eight, nine, ten, 11, 12, 13, 14, 15.
So you're at this position will change this and put all one instead of the zero which all the other
classes get.
So we take this off and then we have a one right here.
See that?
So we have 15, 16, 17, 18, 19, 20.
Now if you have another data set with fewer classes, say eight classes, this will be of length eight.
Now we've seen how for each and every one of these grid cells, which actually sell 49 of them because
we have seven by seven, we have this encoding that's so what our model sees or what our model gets
as output isn't this, but instead this kind of output.
So now we are going to prepare our data set so that we have an input image, which is this, and then
an output label which is going to be this label right here.
And then the model is going to struggle to be able to correctly produce these kinds of outputs.
And so as we've just discussed, our output will be of shape.
This output here will be our shape seven by seven by 20.
See number of classes 20 plus five.
Just say five plus 20.
Which is in fact seven by seven by 25.
So this is what our label will look like.
So when we get our input and our outputs, I want to get the input and the bounding boxes with the specific
classes.
We're going to create this kind of outputs from the labels we get depending on our data set.
Nonetheless, when we zoom into this model right here, we notice that we take in some input and then
at the level of the output, we have seven by seven by 30 instead of seven by seven by 25.
Now, the reason why we have this is simple.
We have the first position.
Let's reduce this.
We have the first position which tells us whether there is an object or not.
So that's one.
And then the next position gives us the location with the x, c, y, c, w, and H.
So here we have the position.
We have the position, so we have x, y, w, h.
Now the next ones give us the classes or tells us what class our object belongs to.
So yeah, we would have a series of zeros.
At some point we will have a one and then we have your zeros.
But the length of this year is 20.
Now 20 plus five is 25, obviously, which is less than 30.
Now, to understand why this is actually 30.
Oops.
To understand why this is actually 30.
What we'll do is let's take this off.
What we'll do is we'll suppose that we have for every cell two boxes responsible for locating the object.
So instead of having this year only this particular box.
So we suppose that this is a box.
Let's take the pen.
We suppose that this year.
This year is a box.
This one is a box.
Instead of having only this box, we are going to make two boxes responsible for locating the object.
Remember, this is this premise us to locate the object because first of all, it tells us whether the
object is found in the cell or not.
And here it tells us it gives us the exact coordinates of the objects.
This one is for the classes separate.
So the next thing we'll do is we'll take this and multiply it by two.
So let's shift this again.
We have this and then we paste that out.
See, now, if we have the number of boxes to be three, then we'll just multiply twice.
Then the yellow view on paper they use, they use B to B they considered B to be equal to.
You could check that out your B equal to.
So that said, we see we now have two boxes responsible for locating the object in that image.
So when designing the labels, we could design with just one because we know that this does a correct
answer.
But what the model will predict will be two values.
You see, it will predict these two.
We predict this and predict this.
Now, the reason why we're doing this is because we want that one of these boxes or any one of these
boxes should be more specialized, depending on the size of the object.
Now, as you've seen here, we're dealing with relatively large objects with respect to the to the image,
actually.
But what if we have an image of a person where the person is say, just a very sad we have a very,
very small person to image ratio like this where we will have a bound and box which will be small compared
to the whole image.
In that case, what we want is for the model of these boxes to be specialized such that maybe this first
box will will learn to detect the smaller objects while this one learns to detect the larger objects.
So that's it.
That's how we construct this output.
So we understand how to construct the labels and then how to construct the model outputs.
Remember, we have to update this weight such that the difference between the labels and the model output,
let's say oh, is minimized.
Getting back to the paper, we see the exact structure of the YOLO model.
So right here we start with some kind of layer conv layer max full layer conf layer max spool.
We have several layers, then the max spool, then we have this order of layers this year, times for
the max spool called layers then.
Layers.
Then finally, we have the fully connected layers for classification.
So this is for feature extraction this year, feature extraction.
And then this is for classification.
Now, for the training, what they do is they pre train this model on image net with 1000 classes.
But note that this pre training is for the problem of classification.
So it's a visual classification problem and then the pretend this model for over a week here and achieve
top 5% accuracy of 88%.
And then from this model they add four convolutional layers and two fully connected layers.
We randomly initialized widths.
So while going going during the training for the object detection, we have weights from the previous
training that's from the pre trained weights from the image net and then the add some the add for the
actually spoke of four.
So it should be this and this one this one this last two year.
So they add up for conv layers and to connect fully connected layers which have been randomly initialized.
As I said, you're now following the example.
Okay.
Does it detection often requires fine grained visual information, So we increase the input resolution
of the network from two two to 20 to 24 by 224 to 4, 48 by 448.
So what they did was they trained on 224 by 224 images and then add detection time.
We used 448 by 448 images.
Now they also use a linear activation function for the final layer and all the layers use the following
key rail so the user value for the final and then the key value for the all other layers as the activation
function.
Now, as I recall for that we have our yellow, which is simply this What the rail does is all values.
All values it takes, which are less, less than zero, are sent to zero, and all values get around
zero, I maintained.
So if you pass in a value like three value to the value, you get back three.
But if you pass, say -0.5, -0.5, what you will get will be zero because all negative values are sent
to zero and all positive values are maintained.
Basically is you remain x if x is greater than or equal zero and you go to zero if x is less than zero.
Now for the final layer, or rather for all the other.
This is for the final layer.
This is what I use for the final layer.
Now, for all the other layers, they use the key value for the liquid value.
What we have is not this year, not a straight horizontal line, but instead something like this.
See something a bit slanted.
And then here is still maintain the positive.
Still maintain.
So we still have still X.
It still remains x4x greater than or equal zero.
But for this one it goes to 0.1 x4x less than zero.
So for all negative values, we will have 0.1 x, So it means our gradient here, this gradient is going
to be 0.1.
So the better border too much.
If we don't understand the notion of gradient anyway, we have that we have this.
It means that if we if we send a value like say -0.5 now what we get will be -0.5 times 0.1 and normal
zero.
So that is the difference.
And this is the activation function used everywhere in the model except for the last layer.
So that's what that's what's defined right here.
Now, the last function that you're the users, the sum squared error.
So that's the simple loss function they use.
And that's why at the beginning the speak of
we frame object detection as a regression problem.
So that's it.
So it's like a simple regression problem.
Actually, that is the sum square of the difference between the model output and the expected output,
which is a levels.
Now that we understand globally how the models build and the training process, let's get to look in
depth into this loss function.
So right here we have this loss function and then we're supposing that we have this labels here.
So you're all levels.
And then here's what the model predicts.
Remember, we have seven by seven, seven by seven cells, grid cells by 34.
The models predictions, whereas the levels are seven by seven by 25.
See that?
So we have this first five for the location, this location, this location.
We have to add this one year.
Oops.
Let's change the color back to red.
So you see that clearer.
So we have this location.
Here we go.
We have this location here.
That's it.
We have this five, this five, and we have the classes here.
We also have the classes and then we have this five right here.
Now, the way we obtain the loss is we break it up into several parts.
We'll look at this part first.
We'll start with this part.
Now, this part is just basically adding the parts up.
The first part.
The next part.
This part.
And this part.
For this first part.
Here's this.
It punishes the model when it makes errors with respect to whether there is an object in a particular
grid cell or not.
So if we have for this grid cell one as a label, we expect the model to predict a one year or and a
one right here.
So this means that what will happen here is we are going to go through we see the sum from I equals
zero to X squared square here.
In our case, seven.
So x squared is seven squared, which is 49.
So we go through each and every grid cellular, which is logical.
We go to each and every one of these.
So we go to each and every one of those 49 different grid cells, and then we'll calculate the difference.
See this?
We have c i, c I, Chapo, that's C minus Chapo squared.
So we have this year minus this square, and then we add all those up.
Now, also notice that there's a double sum in your the reason why we have this double sum is because
we are actually going to take this minus this, plus this, minus this C that, take this minus this,
plus this, minus this.
If we had, say, five or three boxes, then we'll have three of this.
We have one, two and we add another one before the classes.
So in that case we will go three times.
So hopefully that's clear.
But one thing you notice is this this notation here one one of obj I or obj is actually object one of
object.
So you notice this notation right here.
Now this notation, let's get back and just circle that out here so it's clear.
So this is, this is it right here.
You notice this.
And what is here in the paper is this one obj I notation denotes if an object appears in cell I and
this one obj ij because this is actually ij not i ij denotes that the j bounding box does.
Here we have two bounding boxes, so either this bounding box or this bounding box predicted predictor
in cell I so I is the cell clearly see that I goes to 49 and then j goes to two.
If we if we have this, should I equal 1 to 49 and then this should be equal 1 to 2.
This means that there's a slight error notation right here.
Anyways, we understand that we're going from 1 to 49 and then we're going from 1 to 2 because we basically
go into each cell year and we also go into each and every one of these boxes.
Now getting back to our one IJ or one obj IJ notation, we're saying that we've already seen that this
denotes that the j bounding box predict all in cell.
I given cell is responsible for that prediction.
You see that Now, what does this mean?
It means that if a particular like your if a particular box, if this box is not responsible for the
prediction, then we are not going to include it when computing this error.
Now, how do we know whether this box or this box is responsible for the prediction?
The way we get this is simple.
Let's pull this to the right or let's reduce that so we could get more space.
So what happens is let's suppose we have this image and then we have one object here and we have another
object here.
Then we have some bounding boxes.
So we have this bounding box and we have this other bounding box.
Now, for this for this year we have a particular cell.
Let's suppose our we've broken this up and then we have a given grid cell like this one which is responsible
for predicting this object.
Now, the first box you see, this first box will predict maybe this bounding box, and then the next
box will predict maybe this bounding box.
Now, when we say it will predict this, this will predict this.
On in box and the other pretty This is actually because they have different values for x, y, c, w,
h, c.
This quadruple here is different from this other quadruplets.
And because they are different, it means that obviously the bounding box that you will get will be
different.
And because those these two bounding boxes you get will be different.
It means that you can now compare which of these two is closest to the actual bounding box, which is
this.
So this is the actual and this is what the model predicts.
So we comparing this to they are competing for which of them is closest to the actual.
So let's suppose that the actual is, is something like this suppose that actually something like this
here.
So we have something like this.
Okay.
So in that case it's clear that this one let's look for a neutral color.
Now, it's clear that this because we remember we are having this one, this box competing with this
black box, competing with this black box.
So this is B one and B two to compare competing.
But the blue box is the actual one.
B, we just call that B, So we're going to compare the difference between B one and B and the difference
between B two and B now the one which is which resembles B the most, that's the one which has the least.
The, the smaller difference will be the one responsible, as I said here for that prediction.
You see that?
So in our case here, it's clear that B one is responsible.
So for this particular case, because for a different grid cell, we may have B two responsible for
whatever grid cell you may have again, B two or B one.
It just depends on on what?
On the difference between the the bound and box by that box specific box and the actual bounding box.
Now, that said, another question you may ask yourself is how do we compare this bounding boxes?
Now, the way we compare this bounding boxes is by using the IOU score.
So if we have two bounding boxes like this, we have these two bounding boxes and then we have let's
see this one.
So we have this other box here.
And then we also have this one, something like this.
If we are to compare the how close this one, this pair, this pair of boxes is compared to how close
this other Paris.
You see clearly that this pair is closer or simply put, they're more closer to each other as compared
to this other pair right here.
Now, the way we look at this is we compute the area between the two boxes.
So that's it.
You look for this area of the intersection.
So this is the intersection.
And then so we have your we call this IOU IOU.
Actually, let's just put it right here.
IOU actually stands for intersection over union intersection divided, which is equal.
The intersection, we'll call it intersection divided by the union.
So if you take, for example, this intersection here and divide by the area, this area plus this area,
then that's actually including the this intersection.
Basically, this is this is the intersection.
And then let's change this.
So you could see Clara and this is the union.
See this?
This is our union.
That's it.
So that's our union.
And this is our intersection.
So we take this area divided by all this area, and we get the IOU score.
We're going to repeat the same process for this one where this is our union.
And you could see clearly that this one would have a higher IOU compared to this one.
And so this is how we compute or we know which of the boxes is responsible for that prediction.
Now, getting back here, let's get back to our last function.
As we're saying.
We're going to have that if this box, for example, if it happens that this box be one year, this
box here is responsible for the prediction, then we would have this difference times one, C times
one.
Now, if this box is not responsible, then we have times zero.
So this is not going to be considered when we compute in this last.
See that?
That's it.
So it's true.
We sum into the two boxes, but actually we're going to take only or consider only one when calculating
this difference.
For this one, we're not going we're going to submit it.
Now we move on to the next.
This other one year computes or permits us to have grid cells or permits us to correctly predict when
there is no object.
Notice here, this is no object.
Here is object.
So what we have in here is for the cells where we have an object like you see, where the label we have
the cell and the cell we're going to use this year where there is no object, we're going to use instead
this one year.
And you're basically when there is no object, we're just going to take the the output or the value
we have here, minus the value we have here then plus the value we have here, minus the value we have
here.
Now, the next I also know that we have this lambda no object Now in the paper, the talk a little bit
more about this year.
We have to remedy this.
First of all, let's understand this.
Yeah.
To say they use the sum squared error because it is easy to optimize.
However, it does not perfectly aligned with our goal of maximizing the average position.
It weighs localization error equally with classification error, which may not be ideal.
Also, in every image, many grid cells do not contain any object, so this pushes the confidence course
of those cells towards zero, often overpowering the gradients from cells that do not contain objects.
Now this can lead to model instability, causing training to diverge early on.
So to remedy this, they increase the loss from the bounding box, coordinate predictions, and decrease
the loss from the confidence predictions.
For boxes that don't contain objects, we use a parameter.
We use the two parameters.
Lambda coordinates this for the the positioning and lambda no objects for when we have no objects to
accomplish this.
So we set lambda coordinates with five and lambda no objects to 0.5.
So as we're saying, this lambda no object here, 0.5 and lambda coordinate is five as they have given
us.
Right.
Year.
Now, what we can deduce from this, from looking at these formulas here is that the model will be punished
more severely if it has or if a particular grid cell was meant to predict an object.
And it didn't predict that as compared to when it doesn't have an object and it didn't predict that
correctly.
So for the object, we have more punishment as compared to this one, because this lambda no object
is 0.5.
Now, for the coordinates, it receives highest punishment here because as we as we have lambda coordinate
equal five for the classes, it's still equal one.
So here we have 10.51 and five five.
Now, getting back here, we have this one for the classes.
Basically, what we have here is we have this condition.
So this is one object or object of eye.
This one was object or one object.
So notice that this is now I.
Now, basically what we have in is we calculate this difference only when that grid cell has an object.
So if, like in the level year, if we have a one like for these two grid cells, then we'll go ahead
and compute this difference.
But in the case where we have no object, like in the cell and the cell and the cell and the cell or
all of the cells except this two, then we wouldn't get into this.
So we'll just keep that note.
Also, we have this.
Oh, sorry, we have we actually wouldn't get into this.
So here we have just one.
Now, this what we do here is similar to what we're going to do for the coordinates.
With the coordinates, we have the same process where if there is no object, we wouldn't go ahead to
compute this difference.
So like here, if this is zero, then we'll skip this.
Now, if it is one, then we'll go ahead and compute the difference between this and this.
And that's what we have here.
Now, this is the x minus x bar squared, plus the x, y, minus y bar squares, basically this x minus
this square plus this minus this squared.
And then here and also notice that this this two is squared and then it's added up.
And then here we have the square root of the width minus the square root of the order width or the predicted
width.
All of that squared plus the square root of the height minus the square root of the predicted height,
all of that squared.
And then they add this up and multiply by lambda coordinates.
But it should be noted that we are only going to compute this if we happen to fall in a box which is
raised possible for that prediction.
So if this box is responsible for the prediction, it means that we are not going to make we're not
going to compute this for this box right here.
We're not going to compare this and this.
We're going to do this and this because this box is responsible.
And we've seen already what it means by a box being responsible for the prediction.
Getting back to the paper.
Let's get back up here.
Now, what we have also is that the sum square arrow are equally waste errors in large boxes and small
boxes.
So the error metric should reflect that small deviations and large boxes matter less than in small boxes.
To partially address this, we predict the square root of the bounding box width and height instead
of the width and height directly.
So what you're saying is, if we have this couple here, there's two boxes and then we have this deviation
or we have the difference, which we're trying to compute for the loss.
And then we have also this smaller boxes here with this similar difference.
So let's let's try to have something similar to that.
We have something like this hope similar enough so we have something like this with the initial method
we will have W minus W bar or sharper or tilde squared plus the height minus the height bar square where
this is W is what the label expects and the W by is what the model predicts.
So the same, the same for the H and the H bar.
Now what they want is that if we have this difference here or better.
So let's say that what is going on with this here is that.
If we have this year these two boxes, this difference because is equal difference.
You see, the difference is the same.
If we have this year, then let's say the difference is five.
Let's let's suppose that the difference is five year difference five that's the width and the height
difference is five.
Then we'll have five squared plus five squared as 50.
Now for this small box here, this is five and then here is five.
Then we also have 50.
But this is not what we want.
The reason why we don't want this is because this kind of difference for smaller boxes is more important
than this difference for this bigger boxes.
So it's just like you you supposing that you have say say you have a loaf of bread like this.
Suppose we have a loaf of bread and then you have this part your which you can have control.
Now compared to a case where you have the smaller loaf of bread, you will find that cutting off this
part here is like current of practically a third of my loaf.
Whereas for this case this was like cutting out, say, 1/10 of my loaf.
So so clearly from here the loss is less felt as compared to this other one.
And so to, as I say in the paper, to remedy the situation, to add the square root here.
Now let's add the square root and see this difference.
Now, the square root of let's say let's say this was 30 and this was 25, so we had five and then there
was 30 and 25.
The square root of 30 now is let's say this is 30 with 30 and 44 for this other one.
So this one has with high 30 and so the one has with height 25.
So this gave us the difference, which was five, and that's how we had this different here.
Now when we when we take now the square root of 30, let's take the square root of 30, the square root
of 30, that's 5.47.
So 514, seven.
Now minus the square root of 25 will be 0.4 7c0 .47, which when you square let's compute that directly,
which when you are going to square it will give you approximately 0.22 C So you have 0.22 plus 0.22.
Now that will give us
0.440.44
for this to be good boxes.
Now for the smaller boxes, let's suppose that this one was say because we want to have a difference
of five, we could say ten and five.
So here we had ten by ten and here we have five by five.
In that case we will have square root of ten minus square root of five.
And now when you compute that, you will have 0.84.
So what you mean here is when you take the square root of ten, minus the square root of five and you
square it, it gives you 0.840.84 plus 0.84.
We give you about 1.6.
Let's see, 1.6.
Anyway, that's already much greater than 0.44.
It means that the model is penalised more now for making this error as compared to making this error
so that now solve the problem of where the model would have penalised or the model would have been penalised
in the same way for this same difference, even when the size of the size difference between these two
boxes is quite considerable.
Now that said, from here the train for over 135 epochs on training and validation data sets from Pascal
Vos and two in 2007 and 2012.
We test with testing on 2012.
We also include the VOC 2007 testing for training, test data for training.
Then through our training we use a set of 64 momentum of 0.9 and decay.
Thus weight decay of 0.0005.
The ratio was as follows For the first epochs, we slowly raise the learning rate from 10 to -3, so
ten zero negative two.
So started with a.
Relatively lower lending rate, slowly increasing it.
Because if we start at a high learning rate, the model often diverges due to unstable gradients.
So we continue training with ten to the negative two.
That's after going from this slowly increase to 10 to -2.
And then you continue training with this tensor negative two for 75 epochs, then density negative three
for 30 epochs.
So after this 75 epochs they drop two tensor negative three and then finally drop against tensor negative
four for 30 epochs.
That makes now 75 plus 30.
That's 105 plus 3135 epochs OC.
So to avoid overfitting we use drop out and extensive data augmentation drop out layer with rate 0.5
after the first connected layer prevents quad adaptation between the layers.
For this augmentation, they introduce random scaling and translations of up to 20% of the original
image size.
We also randomly adjust the exposure and saturation of the image by up to a factor of 1.5 in the HSV
color space.
Now, given that after the detection has been made, let's get back to the top.
That's after or let's say after the model has been trained, we will get detections like this.
Let's increase this would get detections like this.
So we might have many more detections than expected.
So we're going to apply the non max suppression algorithm to remove those cells or rather to remove
those bounding boxes which are repeated around a certain region and focus only on bounding boxes which
have the highest probability scores.
So like this one, you see, the thickness here signifies the probability score, the probability of
an object being that location.
See that?
So that's why we left with this after the non max suppression.
Now, the way this normal suppression algorithm works is as such, we have after the person, after
the model has been trained, we pass this input image and we may get predictions like this.
Now let's, let's suppose that the this one, this, these two whites have the highest probabilities.
And then we also have some other predictions.
We have maybe say this other prediction here.
That's it We have see another prediction around your.
Something like this.
Now, what we're going to do is we are going to consider that for a particular object.
Let's say this this bounding box right here for a particular bonding box, we look at this probability
and compare with that of the bonding box around it.
And obviously to know what a bonding box surrounds or is very close to this bonding box, we will look
at an IOU.
So if we fix the IOU to a threshold of 0.5, it means that if we're taking this box, for example,
constructing this box, then any box with an IOU, that is any box is close enough to this here such
that is IOU is greater than 0.5, meaning that they are very close, then we are going to remove that
bounding box.
So it means that we're going to take this off because they are already very close.
So you see that now you could you could play around with this value, meaning that you could take,
say, 0.2 or even 0.7, depending on your the data set you're working with now.
So what we're saying is, because these two are very close to each other, we take that off.
Now, obviously they must be placed in the same object.
Now, if we have another box like this one, if we had another box like this one, see, suppose we
had another box like this one, then this box would not be taken off because the IOU is less than 0.5.
So when the oh is greater than 0.5, we know that they are very close to each other.
We compared your probabilities.
The one with the highest probability is going to win, the other one is going to be taken off, hence
the term non max suppression.
So if you are not, you're not a max, we're going to suppress you.
So we suppress all of that.
And you see this one is left now for this year.
We're going to say, okay, this one has the highest probability.
We're not going to compare this with this because obviously the IOU will be less than 0.5.
Now, we're going to take this one and this one we're going to compare this to the IOU is going to be
greater than 0.5.
And so you have we're going to remove this other one.
So this one here will be taken off.
You see, this one is still left.
So that will be it will be left now with these three predictions anyway, generally when training our
model, we aim to even be able to avoid the non max oppression as as a whole and other variants have
been developed to try to reduce that dependence on the Nomex oppression.
Although you'll have variants like the yellow nine 9000 you love to you v three you love for you also
have yellow v5uxur which performs even better than this yellow v one.
We discuss and write your so you have some tables which compare to other methods.
See, you could always look review this, see your fast r c and then you have the yellow.
You see that it performs better than the than the yellow, but this one is faster than the fast rc.
N.
We also have this comparison table for different objects.
See four different objects.
We see the the position for this different objects and we compare it with this different methods.
Here we have the yellow.
Then you could get down here, see the recall, see that we've already had a tutorial on the precision
and recall, so you should be able to understand this.
Or if you're new to that, you could check out our previous videos here.
This is some quantitative results in the VOC 2007 Picasso and people are data sets so that it performs
best on the P on this view C 2007.
It performs best on the Picasso performance based on the people art data set.
Let's get back here.
Oh, okay.
This is fast.
RC And while you're comparing with the with a, with a simple.
Arsene Okay, so that's it.
We see that with the Picasso and the people Art it performs outperforms other methods like Arsene,
meaning that it's, he has marginalization, capacities as compared to other methods, techniques.
Here we have some limitations of the yellow usually imposes strong spatial constraints and bounding
boxes.
So since each grid cell only predicts two boxes and can only have one class.
So this means that let's get back here.
This means that if we have.
Uh uh, if we had a person who was, say, standing just behind, right behind the person here, it
would have been difficult to predict this person and this other person.
And also in the case where we have images, where the objects are quite small, so we have some very
small objects and packed up like this, this Yo-Yo Ma algorithm or this model, we find it difficult
in detecting each and every one of them.
Getting back to the paper, our model struggles with small objects that appear in groups such as flocks
of of birds.
Since our model learns to predict bounding boxes from data, it struggles to generalize to objects in
the new unusual aspect ratio or configuration.
So this has been trained on the past couple of data set where the objects have certain aspect ratios.
Now I may turn it out.
You have a different data set where the aspect ratio is different by aspect ratio, we simply meaning
the width to height ratio.
So this ratio right here, this aspect ratio can be say uh two by five.
If we take in height by with the yellow is going to be like say three by two.
So what, what goes on here is you've trained this on past couple of years, he did a set where you
have a specific ask or a general kind of aspect ratio or aspect ratios, but when this is taken to different
images where the aspect ratios aren't similar to that of the past called VOC, then the past, the model
finds it difficult or struggles to generalize in such situations.
Finally, we went Wild Train while we train on a loss function that approximates detection, performance
or loss function trace errors, the same small bounding boxes versus large bound bounding boxes and
say that the main source of error is incorrect localization.
That said, we're done with this review of the yellow paper.
The next section, we are going to build this yellow from scratch.