In this video tutorial we will look at Non-maximum suppression.
Non-maximum suppression is a technique used to remove redundant or overlapping bounding boxes that may
arise from object detection algorithm.
For example, let's take this input image of car.
So after doing object detection using yolo v9 algorithm, we get this output image.
Like.
You can see that we have a car in this image.
And after doing object detection using yolo v9 algorithm, we see multiple bounding boxes are being
drawn around this car.
So like you can see that we have four different overlapping bounding boxes around car.
But there should be one bounding box around car.
So there should be a bounding box around car which has the highest confidence score.
Not there should be overlapping bounding boxes around car.
So to remove this redundant or overlapping bounding boxes, we use a Non-gm expression.
So after applying non-mixed suppression, like you can see that we have only one bounding boxes on one
bounding box around car, and all the overlapping bounding boxes or redundant bounding boxes have been
revoked.
And we have the bounding box around car which has the highest confidence score.
So our apply using NMS or after applying NMS on the redundant bounding boxes or overlapping bounding
boxes are removed that may that we may get from the object detection algorithm.
So I have divided this complete tutorial into five different steps.
I will be showing you.
I will be showing showing you the Google Colab notebook where I will show you how you can implement
Non-expression.
And before we go towards the Google Colab notebook part, let's discuss the steps which we I will be
implementing uh, in this tutorial to implement Non-maximum suppression.
In the step number one, we will do object detection on image.
So I will I, I have an input image of a person and a dog.
So we will first do object detection on that image.
So after doing object detection on the image we will get the bounding box coordinates for each object
in the image.
So we will have multiple bounding box coordinates for a single, uh, for each object in the image,
like if we have person and dog in the image.
So I will have the multiple bounding box coordinates for person.
And I will also have multiple bounding box coordinate coordinates for dog as well.
So in the step number two, after we get the bounding box coordinates for in the step number one.
In the step number two, I will sort the bounding box coordinates based on the confidence score.
So.
After getting the bounding box coordinates, I will sort the bounding box coordinates based on the confidence
score.
So I will just create a list and the bounding box coordinates which have the highest confidence score
will be at first in the list, and the bounding box coordinates, which have the least confidence score
will be at last in the list.
So.
So the bounding box coordinates, which have the highest confidence score will be at the top of the
list, and the bounding box coordinates, which have the least confidence score.
I will be at the last of the list, so I will just create a list.
Uh, considering the, uh, I will create a list of the bounding box coordinates for triggering the
confidence score.
And the the confidence score will be at, uh, in that descending order.
So the, uh, as the higher bounding box coordinates with the with the higher confidence score will
be at the start of the list.
And the bounding box coordinates with the lowest confidence score will be at the last of the list.
So after we get the book, after we have sorted the bounding box coordinates based on the confidence
score in the step number three, I will start the from the bounding box coordinates with the highest
confidence score.
So I will just pick the bounding box coordinates which have the highest confidence score.
Okay.
And I will compare those bounding box coordinates with all the other bounding boxes of the same class.
So for example, if I have a person in the image so I if I get multiple bounding box coordinates or
overlapping bounding box coordinates.
So I will just pick the bounding box coordinates which have the highest confidence score.
And I will do compare those bounding box coordinates with the overlapping bounding box coordinates for
the person class.
Okay, so if we compare the bounding box coordinates for the person class which have the highest confidence
score with the overlapping bounding box coordinates of the same person class.
So we will do the comparison based on the intersection over union.
So to to calculate a metric or to to do the comparison we will be using intersection over union approach.
So if the intersection over union of the two bounding boxes which we are comparing over here.
So if the intersection over union of the two bounding boxes over the overlap ratio is above a certain
threshold.
So for example, if I have to find a IOU threshold of 0.5 okay.
So I have to find the IOU threshold of 0.5.
And if I am doing the comparison of the bounding box coordinates with the highest confidence score,
and with the other bounding box coordinates which have less confidence score.
So while doing comparison, if I get IOU value as 0.6 okay, so for example a while doing comparison,
I get intersection over union value as 0.6 and I have to find the IOU threshold as 0.5.
So I am getting IOU score of 0.6 and I have to find the IOU, uh, threshold as 0.5.
So I am getting a I score above the IOU threshold.
So the bounding box with the lower confidence score will be removed.
So if we get the IOU value intersection over union value of the two bounding boxes above the threshold,
then the bounding box with the low confidence score will be removed.
And we will repeat this process until all the bounding box gets examined.
So let me show you how intersection over union works.
So intersection over union tells us the percentage of overlap between the ground truth bounding box
and the predicting prediction bounding box.
But in the case of non-max suppression, the definition of intersection over union changes a bit.
So in case of non-maximum suppression, we find intersection over union between two prediction bounding
boxes.
For example, let's take this image of a bird.
You can see the bird over here.
So like you can see we have two bounding boxes like the this bounding box within the red color has a
confidence score of 0.8, and the bounding box in the blue color as a confidence score of 0.6.
Okay.
So the.
Term bounding box in the red color, has a confidence score of 0.8, and the bounding box in the blue
color has a confidence score of 0.6.
So we calculate the IOU intersection over union for these.
Of two bounding boxes.
Okay, so I'm just calculating the intersection over union for these two bounding boxes.
And I get the intersection over union values 0.96.
And here I have defined the IOU threshold as 0.5.
So this IOU value intersection over union value which I am getting is 0.96.
So this value is greater than the IOU threshold which I have defined over here.
So what will happen.
So the bounding box with the low confidence board.
Like you can see that uh, the blue bounding box has a confidence score of 0.6.
So the blue bounding box which has a low confidence score will be removed.
And this red color bounding box will be retained.
Okay.
So let's take another example.
So you can see we have uh the bird over here as well.
And we have the uh bond bounding box in red color which have the confidence score of 0.8.
And over here the we have the bounding box in blue color which have a confidence score of 0.2.
Okay.
And I am just getting an IOU threshold of 0.2 or I or I can say I'm getting intersection over union
value as 0.22.
So.
Like you can see over here we are getting intersection over union values 0.22.
So we do a comparison of red color bounding box with the blue color bounding box using intersection
over union.
And I get the intersection over union value as 0.22.
And my IOU threshold is being defined as 0.5.
So in this case, like you can see that my intersection over union values 0.22 is less than the threshold.
So this bounding box will not be removed.
So over here you can see that we have IOU threshold of 0.5.
And my intersection over union value is higher than the threshold.
So the bounding box with the low confidence score will be removed.
So the blue color bounding box has a low confidence score than that red color bounding box.
So the blue color bounding box will be removed.
In this case, while doing the comparison of these two bounding boxes, I am getting IOU as 0.22 while
my threshold is being defined as 0.5.
So I am getting a low IOU than the IOU threshold, so this bounding boxes will be retained and no bounding
box will be removed over here.
So this is how we can create the intersection over union.
Or you can say that intersection over union.
It can be represented in mathematical form as uh, the union's intersection size.
Like you can see we have two bounding boxes.
This is the box one.
This is the box two.
So intersection size of box one and box two and divided by the union size of box one and box two.
So this is how we get grid intersection over union in mathematical form or intersection over union in
mathematical form can be represented as uh, intersection over union or intersection size of bounding
box one and bounding box two, and the union size of the bounding box one and the bounding box two.
Okay.
Or you can say that we can calculate intersection over union by dividing intersection area.
Over the Union area.
So we divide intersection area over the Union area.
So to calculate IOU, uh we can divide intersection area over the union area okay.
So here we have the intersection size of bounding box one and the bounding box two.
And here we have the union size of the bounding box one and the bounding box two.
So to calculate intersection over union you can divide intersection area divided by the union area.
So that's you can see that we have two bounding boxes.
Uh like you can see this is the bounding box one.
And this is the bounding box two.
Like you can see over here.
So IOU you can see IOU is equal to 0.0 okay.
So like you can see that uh, if the two bounding boxes are very far away from each other.
So you can see that IOU is 0.0, but you can see over here if the two bounding boxes intersect with
each other, we have IOU as 0.08.
And if the two bounding boxes, uh, intersect a bit more with each other than the IOU values increases.
If the bounding box two bounding boxes intersect, uh, more closely with each other than the IOU value
rises to 0.43.
And if the two bounding boxes completely overlap with each other, then we have a perfect score of 1.0.
So this is the some conclusions which we get from what we have understand from till now.
So if we stack the select the R2 threshold, uh, value higher, like I have said, the higher threshold
value to 0.5.
And if you set the threshold value as 0.6 or 0.7, so what will happen?
This will result in fewer bounding boxes being suppressed or removed, which may lead to more overlapping
bounding boxes being retained.
So now if you just see over here, I have set the threshold of 0.5 and I am getting the IOU score of
0.22.
So as my IOU value is less than the threshold.
So.
This morning box with low confidence score is not removed.
So if I have set the threshold is equal to 0.1.
So if I have set this value of I with threshold as 0.1, so my IOU will be greater than this original
threshold, then I will be removing this bounding box with low confidence score okay.
So if we set the higher R threshold value, this will result in fewer bounding box will be removed and
this will result in more overlapping bounding box being retained okay.
Similarly, if I set a lower track threshold, uh, this will uh be more aggressive in removing overlapping
bounding boxes.
So if I set the threshold a low value of higher threshold like 0.1.
So this will result in removing of all the overlapping bounding boxes.
Or you can say that our model will be more aggressive in removing the overlapping bounding boxes, which
may lead to fewer bounding boxes but more accurate object detections.
So this is the conclusion which we draw from what we have understand till now.
So let's move towards the Colab notebook where I will be showing how we can implement non-maximum suppression.
So here is the Google Colab notebook that I have prepared.
So like you can see that here we have a image that in the image we have a person and a dog.
So after doing object detection for example using yolo v9 algorithm, we get multiple bounding boxes
around person as well as we get multiple bounding boxes around that dog as well.
So then we apply non-maximum suppression.
And after applying non-maximum suppression, you can see over here we get only a single bounding box
around dog and only one bounding box around person as well.
Okay, we are using non-maximum suppression.
We have removed all the redundant and or overlapping bounding boxes from the dog as well as from the
person.
Both.
So now let's get started with it.
So in the step number one I will be importing all this required libraries uh, in cv2 so that uh I'm
using OpenCV Python library so that I can do pre-processing on the image.
Then we have uh, Matplot library so that I can show you the input or output image in the Colab notebook.
And similarly we also require image library to show you the input or output image in the Google Colab
notebook.
Okay.
So I am just running this stand so that I can import all these required libraries.
So I have just placed this like you can see this image or this input image in my Google Drive.
So I'm directly downloading this input image from my Google Drive directly into this Google Colab notebook.
So.
If I just run this cell here, pass the Google Drive link of this input image and like you can see over
here, now we have the input image over here.
I have downloaded this from my Google drive and using Image library uh image library which we have imported.
I will show you the input image over here.
You just need to copy the path from here and add this path over here.
And you just run this cell, uh, you will be able to show I see the input image in the Google Colab
notebook.
So you can see now this is our input image okay.
So now, uh.
I will be downloading the bounding box coordinates, confidence score and class name for the person
as well as for the dog in this image.
Okay.
So I have placed the dot txt file on my Google drive which contains the bounding box coordinates, confidence
score, uh, and class name for each of this object.
Like we have two objects like person and a dog.
So I will be downloading a txt dot txt file which will contain the bounding box coordinates, confidence
score and class name for each object in the image.
Okay, so.
So as I showed you in the image we will have, we will have multiple bounding boxes for each object
in this image.
Okay.
So I have now downloaded the predictions dot txt file which you can see over here.
Okay so now I will be reading the predictions dot txt file line by line.
So for this I have created this function create dash predictions.
So we will read, uh, that data from the dot txt file line by line.
And here we have the results.
So now you can see that the first four represents the bounding box coordinates.
This is the class name.
The fifth uh or an item represents the class name.
And the last one represents the confidence score.
So the first four are the bounding box coordinates, the fifth represents the class name.
And the last one which is the sixth is the confidence score.
Like you can see that we have two objects in the image, uh, person and a dog.
But you can see that we are getting multiple bounding box coordinates for each object in this image.
Okay.
So now you can see that, uh, this requires some pre-processing as well.
So now here I am just doing pre-processing and just converting this bounding box coordinates, last
name and confidence score in the required format that we require so that we can get rid, uh, we can
draw the bounding boxes around these objects, uh, using OpenCV, Python, and uh, we can add assign
colors to each of the bounding boxes so that we just need to, uh, convert this bounding box coordinates
into a proper format.
Okay.
So I've just created a function by the name process prediction, so that I can just do the complete
pre-processing and convert this bounding box coordinates into a proper required format.
So now you can see over here, uh, I've just converted this bounding box coordinates confidence score
and class name into this proper format.
Like you can see that, uh, this is the first sub list which I have.
Uh, the first four represents the bounding box coordinates.
This is the class name and this is the confidence port.
And we have two objects in the image.
And we are getting multiple bounding box coordinates for each object in the image.
So I'm using OpenCV, Python.
I'm just reading this input image now, and just converting this image from BGR to RGB so that I can
show you this image using matplotlib.
So to display an image using matplotlib, we need to convert the image from BGR to RGB.
So when OpenCV Python reads an image, it reads in the form of BGR blue, green, red.
And to display any image using matplotlib, we need to convert the image into RGB.
Okay, so OpenCV reads in the form of BGR and to display using a matplotlib, we need to convert it
into RGB.
So now you can see this uh is the input image we have.
So now here is our color map.
So now.
Over here.
Like you can see.
Uh, I'm just drawing the bounding boxes around each of the objects in the image.
So zero represents the person class and one represents the dog class.
So for a dog class, you can have an orange color.
And for the person class you can have a yellow color.
Okay.
So I'm just drawing this, uh, color over here.
Okay, so now you can see that I have, uh.
Uh.
Drawn these bounding box partners over here.
Like, you can see that, um, we have, uh, round two bounding boxes around dog, and we have around
three different bounding boxes around the person.
So this is the output which you get from the object detection algorithm.
Like you can see that we have multiple bounding boxes, uh, around dog as well as we have multiple
bounding boxes around the person as well.
Okay.
So now we need to calculate norm expression.
To calculate norm expression uh or we need to have IOU as well.
Like we need to uh uh have a function for the intersection over union as well.
So why we are implementing norm expression.
The reason for implementing norm expression is that and that we need to remove all the redundant or
overlapping bounding boxes.
And we need to have only one bounding box for a single object.
And that bounding box will be the bounding box, which has the highest confidence score among the overlapping
bounding boxes.
So now I will just first create a function to calculate intersection over union.
So like you can see over here if we have two bounding boxes this is the box one.
And this is the box two okay.
So we will be calculating uh intersection over union uh for two bounding boxes like this is a box one
and this is a box two.
And these are the bounding box coordinates.
So, uh, on the top left corner we have the, uh, x one, y one coordinates.
And in the bottom right corner we have the X2Y2 coordinates.
So here you can see we are just calculating intersection over union.
And here you can see that.
What each of the bombing groups were like.
You can see over here we are separating the x1, y1, x2, y2.
Like you can see that for the box one this is the x1 y1 and this is the x2 y2.
Value for the box two.
This is the x1 y1 and this is the x2 y2 value.
So you can see we have just all these values over here.
Then we just uh, as I told you, that intersection over union formula is, uh, intersection area divided
by the Union area.
So I'll you Vonda is intersection area divided by the Union area.
So here we are just calculating the area of intersection by uh by multiplying width with the height.
So calculate the area of intersection with uh multiply width by by y the height of the bounding box.
Then here we have the.
Area of union.
So to calculate the area of union, we calculate the box area or we add up the box area with the box
one area with the box two area.
And we subtract the area of intersection which is this okay.
So in the end we divide to calculate area we divide the area of intersection by the area of Union okay.
So now after I've just created a function, I intersection over union.
Then we will start calculating Non-maximum suppression in the non-maximum suppression.
Okay, so let me just run this cell as well.
Okay.
Yep.
So I'll just run this cell as well.
So now this is our main function which is non-maximum suppression.
So here I have this pass the confidence threshold.
So we will accept all the bounding box partners which have a confidence or score above 0.10.
So here I have just set the confidence threshold limit as 0.10, which means I will accept all the bounding
boxes which have a confidence score above 0.10.
Similarly, I have also defined the threshold of 0.10.
So.
Uh, if I when we get grade intersection over union.
If I lose you, uh, if our IOU is above the IOU threshold, then we will remove the bounding box,
which has the low confidence for, as I told in the slide.
So I have set the threshold value as 0.10.
So here you can see that I am just getting IOU.
So if my IOU score is above the IOU threshold, like you can see if my IOU score is greater than the
I o threshold which I have defined 0.10, then then bounding box will be removed.
Um, okay.
So the bounding box which which has the low confidence score will be removed.
So in the first step, uh, here you can see I've just created a function non-maximum suppression.
In the first step we are doing sorting.
So we are just sorting out the bounding box coordinates, which, uh, are considering their, uh,
confidence score.
So I am just creating a list.
And the bounding box which has the highest confidence score will be at the start of the list.
And the bounding boxes, which have the lower confidence score will be at the last of the list.
So I am just creating a list and I am sorting, uh, the list based on the confidence score.
So I am just creating a list, uh, which contains the bounding box score, uh, which contains the
bounding box coordinates for each detection.
Like you can see this input image.
So you can see we have uh two bounding boxes for dog and we have three dog bounding boxes for person.
So I will be just creating a list which contains the bounding box coordinates.
And the bounding box coordinates will be sorted like based on their confidence score.
So the bounding box coordinates, which has the, uh.
I is confident squad will be at the start of the list, and the bounding box coordinates which have
the low confidence score are which are the least confidence score will be at the last of the list.
So I will be just uh uh, creating a list where I will be just, uh, placing the bounding box coordinates
so you can see that here I have defined empty list by the name filter dash boxes.
So okay.
So here you can see that I am performing the sorting.
So in the fifth, like I told you that, uh.
The first four indexes in the list, uh represents the uh bounding box coordinates, the fifth represents
the uh class name and the last represents the confidence score.
Okay, so if I just get rid, uh, considering the python.
So this is 012345.
So this is the fifth index okay.
So like, you can see that in the last, uh, if our, uh, confidence score is greater than the confidence
threshold, like you can see over here, then the, uh, which we have defined 0.10, then you will
be up, uh, appending the bounding box coordinates into this, uh, filter boxes list.
This is the empty list which I have created over here.
And I will be adding the bounding box coordinates for each of the bounding box into this list, and
the bounding box coordinates will be added based on their confidence score.
So the bounding box coordinates which have the highest confidence score will, uh, will be at the start
of the list.
And the bounding box coordinates, which have the lowest confidence score will be at the last of the
list.
Okay, so I'm just, uh, doing the count over here.
You can skip this as well.
So if we have if the length of the, uh, this list is greater than zero, then we will just take dot
pop zero.
So we will just pick the first index.
Like you can say that we will first the first sub list from the list.
So we will be just picking up, uh, the first sub list from the list okay.
So the first sub list will be uh, first sub list will be the sub list which has the uh, which has
the bounding box coordinates, which has the highest confidence score.
So I'm just picking up the first sub list from the my list.
Okay, so then I'm just checking if the, uh, if I just pick the first sub list, and then I will be,
uh, comparing that sub list, uh, with the other uh uh uh list as well.
So the first sub list will have the bounding box coordinates.
And these are the bounding box coordinates, which has the highest confidence score.
So I am just taking the bounding box coordinates with the highest confidence score.
And I am just comparing the bounding box coordinates with the highest confidence score.
Will with the, with the all the other bounding boxes which has the lowest confidence score for the
same class.
So then we calculate the uh intersection over union.
And if I just get the IOU value above this IOU threshold, then the bounding box coordinates with the
lowest confidence score will be removed.
Okay.
So let's run this cell over here.
So now you can see that after blank non-expression, we get this.
Okay, so now you can see that we have just getting the, uh, best fit bounding box coordinates for
each of the subject for the person as well as for the dog.
So all the overlapping bounding box coordinates have been removed and we are just getting the best fit
coordinate reporting box coordinates for that dog, as well as for the person.
Like you can see over here.
There is one bounding box for the dog, and there is one bounding box for the person as well.
And these are the bounding box coordinates which have the highest point transfer among the overlapping
bounding boxes.
So these bounding boxes have the highest score among the overlapping bounding boxes.
So that's all from this tutorial.
Thank you for watching.