Either.
And welcome to this new and exciting session in which we shall be looking at different strategies to
reduce overfitting.
And in the YOLO v one paper, some strategies were underlined to avoid overfitting they use drop out
and extensive data augmentation.
Now a drop out layer with rate 0.5 after the first connected layer prevents co adaptation between layers.
And so you see that after this fully connected layer, we are going to have the drop out and we're going
to give it parameter 0.5.
Then for the data augmentation, the authors introduce random scaling and translations of up to 20%
of the original image size.
Then they also randomly adjust the exposure and saturation of the image by a factor of 1.5 in the HSV
color space.
So that said, we are going to break up our data augmentation strategies into two main categories as
the very first category will entail modifying the pixel values without modifying the positions of the
different objects.
So we could have something like this.
Let's click an edit right here, and then let's try to say brighten up the image.
You see, we could modify this like this.
So we go from this initial image to this by playing around the brightness, playing around with colorization
and so on and so forth.
So this first category, as we've said already, entails just modifying the different pixel values without
any change in position of any object we have here.
And then for the second category, we could go from this image to this one where you see that this flipping
has made this object position to go from here to this position right here and now.
In the first case where we just modify, for example, the image brightness, there is little or no
updates made to our existing code base.
But when we have to modify the image such that the bounding boxes have to be changed or the positions
of the bounding boxes have to be changed like this one here, or this one which will go from year to
year, this one year which goes from year to this other one.
It means that we are now updating this bounding boxes and so we would have to write some extra code
for all these different modifications.
Now, nonetheless, it turns out that when we work with a library like Album notations take this off.
When we work with album annotations, all changes made in the positions of the bounding boxes are carried
out automatically.
So you could see here we have this input image with this dock, this tennis ball and this cut.
And then after going through some transformation, like here we see we have some transformation on this
image where first of all, the image is flipped.
So you see the dock moves to this other position and then the image also appears zoomed in.
So you see that this cut, for example, is now not as complete or we don't have the complete cut as
we had in this original image.
And so now we have completely different bounding boxes.
This one, for example, becomes this, this tennis ball becomes this, this docks bounding box becomes
this.
So it becomes you see that it becomes larger as compared to the inputs.
And so with implementations, as we're saying, you have this input bounding boxes like you could see
for the dock at this position, 23, 74 and 295 388 is automatically converted to one 4969 like you
see your 295 381 So you're you just need to define your transformations and then argumentation.
Make sure you have the right bounding boxes as output.
So now diving into the code as you might have seen in some previous sessions, we are going to import
albumin patients.
So that's the import of our team implementation.
We now move on to integrate our transform.
You see right here we have this different transforms.
The first thing we'll do is to resize our image.
So it's 224 by 224 and then we'll apply a random crop.
Now this random crop is applied such that the output image will have a height or right.
Width line between 202 24 and a high line between 202 24.
So we could just have this height or let's say height -20, or we could just say 0.9% of the height.
So we have that.
We go from that to and then here we also have 90% of the width.
So we have 0.9 of the width.
Take that off and we go here we have the width.
Okay.
So what we're saying here is, as we've had already, we want to randomly crop the image.
So we have the image Now our new image will have a width which will fall in this range and the height
which will fall in this range and then one half of probability of 0.5 of applying this transformation.
So if you want this transformation to always be applied, then you could always set, always apply to
true.
If not, you will just have p all the probability of applying to set to 0.5 or maybe even 0.2 or say
0.8, it just depends on you.
So that's it.
Now for the next we have this random scaling here.
We specify the skill and limit.
We have the interpolation type and then again we have this probability.
Then, yeah, we have the horizontal flip.
So we're going to apply horizontal flip and probability set to 0.5.
Now, finally, because after doing this random crop, we will have an image which is not 224.
By 224 we actually resizes back to two, 24 by 224.
So that's it.
That's all you need.
That is all our transformations here which we pass in this compost method.
So it's basically a list made of this, transform this, transform this, transform this and this.
Now, one additional term we're going to pass in this compost method is this bounding box parameters.
And the reason why we need to pass this is simply because, unlike with the image classification, with
the object detection, we have bounding boxes which are going to be modified.
So here we specify this bounding box params.
So take to consideration the kinds of boxes we're dealing with.
And here you notice that we specify a format yolo.
Now getting back to your documentation, we actually have three formats.
We have the Pascal Vos format implementations, format, cocoa format.
Here you'll format Pascal Vos augmentations.
Now it turns out that in our specific case we actually dealing with the format not just because we built
in a YOLO model, but because the way we've normalized our inputs or process our inputs is such that
we have our x center, y center width and height representing our bounding boxes.
So if we had instead X mean y, mean with height would have picked this.
So it's not, it's not, it's not because of the name, although it's actually coincides with the fact
that we're building a YOLO model.
But then as we said, we have an X enter Y center with height.
And again, this is normalized notice here is normalized.
So this year is divided by the width, this divided by the height, this is divided by the width and
then this divided by a height.
Remember, this is the width and the height of the specific bone and box, which happens to be exactly
what we have seen when we're doing this here.
Remember, we took the X mean Y mean we obtained the X center and then we divide it by the width to
the Y mean y Max, open the y center divided by the height to the width, divided by the total width
to the height.
Divided by the total height.
So it's the YOLO format we actually using right here.
Now again gets into the code.
You see we have the format YOLO specified.
Now here we have this mean area set to 25 and this mean visibility is set to 0.1.
Now, to understand the concept of the mean area, consider you have this input right here.
And then after carrying out the transforms, what you have is say this in this output here.
So we have this output where the area of the this box is 4344 pixels.
This is actually the cocoa format.
So you find that the bounding boxes will be different from the kind of bounding boxes we have.
Nonetheless, the error as set after transformation is 4344 pixels, so clearly it's quite small compared
to the 23,892 pixels we had already.
On this 132 times 181 computation we had here.
So after transforming this, we obtain this right here.
But if we set the mean area to 4500, it means that any box, less than 4500 is going to be omitted.
And so that's why you see that when we specify this mean area to be 4500, this box here disappears.
So if you don't set anything, the box remains.
But if you set this, then the box is going to disappear.
And then we also have the mean visibility.
So let's take this here.
If we set the mean visibility to, say 0.3, then if the output box, which is this box divided by the
initial box, which is this is less than or gives us a ratio less than 0.3, it means that box is going
to be omitted.
And so right here, if you take this this area, which is 6888 divided by 24,108, from this original
box here, 24,108, you would have 0.286, which is less than 0.3.
And so when you see you see when it says 0.3, this box up here disappears.
So that's it.
That's that's the idea behind the mean area and the mean visibility.
Now we can actually leave those out.
So let's just take this off and that should be fine.
So this is it.
We have our transform so we could run this.
There we go.
We have our org argument method, which takes in the image, takes in the bounding boxes, and then
all it does is it passes the image and the bounding boxes into our transforms here.
And then we obtain the transformed image and the transformed bound and boxes.
So as we are seeing here, we go from this image bounding box pair to this transformed image transformed
bounding box pair.
So let's get back to the code.
We could run this and then we have our our processed data method which makes use of this TensorFlow
non pi function because actually here these are non TensorFlow pairs which we are calling especially
here when you use make use of abominations, it's made up of computations and non pi.
So we make use of this method right here in order to integrate that in our data pipeline.
So here we specify the function arguments, the inputs image and bounding boxes, the output tensor
here this one's this two year the floats actually floated too.
So that's it.
So we run this and we create our train dataset so we could visualize that.
You see, for example, here we have this image, let's write that image out so we could see it.
We had output one and output two.
Let's check that out.
See here we have this or no modification was made.
So in order to be sure that we make some change, what we could do is we could make sure, for example,
let's let's comment this random scaling and then let's carry out let's make sure the flipping is always
done.
So we have always
true set, always apply set to true.
There we go.
So we run that, run that again and we have output one and then output two, which has been flipped.
Now one thing you would notice is also the fact that it's bound and boxes are changed.
So let's copy this from here and then let's piss it out just here so we could see that.
Okay, so this is what we have before and this is what we have after.
Now you see that this actually makes sense, because when you have this original image, when you have
this original image where you have something like this for the bounding box, oops.
Well, all of that, we have someone like this.
And then when you flip it, you see when you flip it, what actually changes here will only be this
X center.
So if your center was around this, let's say, centers around this, then flipping the X Center changes
position slightly.
And that's what we would notice here.
You notice how does X Center change this position?
Just a little bit.
But for the Y center, it doesn't really change.
The width and the height remains the same.
Now, let's play around with this here.
So let's let's have this random crop now.
Let's do this.
Let's add this to always apply.
It always apply to true.
So we'll have that.
Oops, We will have always apply.
We set it to true.
Run this and check out our output.
So yeah, we'd have.
Okay, we should have.
Let's run this here to have out two.
So we have out one.
And then we have our two.
Okay.
Now what we have is this.
And then this, you see, appears somehow zoomed in.
Anyways, this example.
Now it shows us that we have Let's take this off.
We have this image which has this bottom box here, something like this and this.
And maybe the center around here.
And then now it's modified and you see the height gets modified.
And although not too much, actually, the width doesn't change.
Well, it changes just a little bit.
Now, let's change this example so that we could see this clearly.
This example isn't very demonstrative of this transformation process.
So we'll take maybe the second or instead the a skip.
So here we have Skip here.
Yeah.
Then we break.
Hopefully the second has maybe many more objects or is a better example.
Actually, let's check this out.
Okay.
So after flipping, we expect to have something which is looking different from this.
Okay, so that's it.
Let's take this off for work with this example.
Now let's run this again.
We have this year, we run this.
There we go.
We have that.
We have our output, which we obtain from skipping, and then we get back here and then we also skip.
So we skip and there we go to and then from here we break, we run that and then see what we get.
Now, you could see from here we have our output one.
Let's also check out our output to.
There we go.
We have our one and then our two exactly as we expect.
So that now this this example is much different from what we had before and should be a better way to
demonstrate what goes on in our mutations.
So right here we have this input boxes.
Let's copy that.
We have the input boxes, we're going to paste it just here, our input boxes.
Let's take that off.
There we go.
We have this output boxes.
Copy that.
And then peace right here.
Now, before we move on, notice the fact that the classes are the same.
So class 18 and yes, class 18.
And if you get right to the top, you.
Well, we use classes, so let's let's say classes.
Let's get 18 and see what we get.
It should be trained from here.
We have well, yeah, we have classes.
Classes 18.
There we go.
We see we have train.
Okay, so that is it.
Let's take this off now.
Let's check this out.
We have our art.
One could see from here.
It makes sense that the center is about 27% of the full width.
So this distance is about 27% of all this distance.
So that's it, then?
This distance, too, it's about 36% of all this distance.
And then we have the width.
The width, which is about 0.540, 54% of the total image width.
Let's change the color.
That's about 54% of the total width.
And then this is about 71% of the total height.
Now, after flipping, you see that this has to change.
Let's drag this now, drag this, this way.
And we have something like this.
Let's take this off.
You'll see that this now the center or this distance here, this distance from year to year is about
73% of the full image.
See that X center changes and then the for the for the height, it doesn't really change much or changes
very little.
See that almost the same, the width remains practically the same and then the height remains also almost
the same.
Well, it's the idea here is to show that after going through this augmentations transforms augmentations,
permits us to obtain this output bound in boxes which match up with a transformed image.
Now, don't forget to make sure you change this back from always apply to probabilities of 0.5.
So that's it.
The next set of transformations would make would be with TensorFlow and we make use of TensorFlow image.
So yeah, we have this random brightness, random contrast.
We're going to leave out the random crop for obvious reasons.
Remember, if you have to do this random crop, it means you would have to write the code which permits
you also modify the bounding boxes because when you carry out a random crop, the bounding boxes actually
change.
So that's why we're making use of ABU mutations.
Since it makes life, it makes life much more easy.
And then we use this, we use this, we carry out no, we're not carrying this out.
We carry out.
No, not this.
We carry out random new random saturation.
Okay, So this we're going to make use of You could see that you're the code.
We have brightness, saturation, contrast you and then we find out we carry out this clipping by value
to make sure all the values lie between zero and 255.
Now, you could always feel free to comment on comment.
Any one of this right here.
So that said, we again going to carry out this preprocessing.
See, we have that.
And then remember, this is for the training and then this is for the validation.
So that's it.
We carry out this mapping, we batch pre fetch and then you could check out your outputs right here.
So let's say out one, out two, let's check out our three.
Here we go.
We have our three.
Well, this should be let's go to Skip.
Let's keep this skip two and break.
Okay.
So let's run that again and see what we get out.
One out, two and out, three out, one out, two and out three.
Well, since this was already batched, we will maintain the take.
So we'll take the first.
We'll maintain this, and then, yeah, we'll pick out one.
Run that and there we go.
We still have this so this should be two.
Let's run that.
We have our one, our two and then our three, our one, our two and our three.
Okay, so you see that this now appears much darker as compared to this one.
So that's it.
We have seen how to carry out this different transformations or augmentations.
But before we go on with the training again, one slight modification will make is we'll replace this
resin at 50 with the efficient net be one, then we'll go ahead and compile the model and restart the
training.
There we go.
Train has begun and after training for several epochs years where we obtain.
You can see here that the loss, the loss, the training loss and the validation loss both keep dropping
to see that they all keep dropping up to where we have this hundred and 23 around this year.
So up to around this, our loss keeps dropping and then somehow increases slightly and stabilizes around
128.
So that's why at this point we had to stop the training and get the weights which produced the lowest
validation loss.
And that's it for this section.
In the next section, we are going to test out this model.