Hello, everyone, and welcome to this new session in which we're going to implement the CAD Mix data
augmentation strategy with tensor flow to the CAT Mix data augmentation strategy, though based on the
fact that we are combining two different samples is different from that of the mix up.
Instead, with a cut mix data augmentation, we are going to take a random patch from one of the samples
and attach to the other sample while modifying the labels accordingly.
We've looked at how to implement data augmentation with TensorFlow and also how to implement more advanced
data augmentation strategies like the mix up in the section.
We'll look at this cut mix data augmentation strategy here.
If we suppose that we have these two images, Image one and image two, we are going to randomly crop
a part of this image.
So just like you can see in this output here, the randomly cropped the section from this image and
then attach that to this other image such that what you have in the output is this one with this patch.
So this is how this cut mix data augmentation is implemented.
If we take this example where we have this cut and this dog right here will try to randomly crop apart
from this dog and then attach that pad at the same position on this cat image here to do this cropping
operation, we get to TensorFlow image.
Let's have this.
We have this TensorFlow image and then we have the crop to bounding box.
So we click on this and then we have this definition.
We see the arguments, we pass in the image, and then we are going to see, oh, we're going to specify
the offset height, offset wheat, target height and target wheat.
Now, let's explain what all this means.
If you have an image like this one, just as it's giving you the offset height is a vertical coordinate
of the top left corner of the bounding box in the image.
And so this means that if we randomly select, for example, this box right here, let's suppose we
randomly selected this box and our our offset height will be this distance that is will our reference
is this top left corner and our offset height will be this distance here.
This distance and our offset width will be this other distance.
So that's how we we have this offset height and the offset width.
And then the target height is the height of the bounding box.
So here we have this bounding boxes height and target wheat the width of the bounding box.
So once you provide this, it will be able to automatically crop out this zone from the image.
So coming back to the code, we are going to add this other extra subplot.
Let's have this other subplot we have.
Let's pass this code first.
Could you just copy this out?
We have this and then we have the subplot.
Third position, what we'll be doing is we're not going to be having this.
We're going to have image, let's call that image three.
And then what we'll be doing here is we'll be making use of this method.
So we have this method and then we'll specify this offset height, offset width, target height, target
width.
Let's suppose that our offset height is, let's say 20.
So here we have 20, 20, say 15, so we have 2015.
And then let's suppose this target height is 100 hundred and you're say 98.
Okay, so we have this specified and then without passing the image, the image will be using as image
to this image here and that will be it.
So here we take this off and then we could simply show this parses here and that's fine.
Now we could take this off.
Okay, so we have this.
We can now plot this out and see what we get.
Okay.
As you could see, what we obtain here is the correct is a cropping of this zone, as you could see.
You see that we have something around this.
We crop out this zone.
So actually it takes more of this.
So it's something like this.
Now we could modify this, like we could shift the height and the width so we could actually maintain
the height, but shift the weight so that we could get more off the dog.
So let's do just that.
We shift the width.
Let's say we we take 100 to we run that and here is what we get.
You see that we get the docs face this time around.
And basically this is how we crop out a region from this image.
Now, once we've done this cropping, we want to create another image which is made of only this crop
while the remaining zones are actually left out.
Now to do that, we'll make use of this other method, which is the path to bounding box.
So yeah, we have this image PA two bounding box, we'll copy it out and you see how it works.
So here we have this image path two bounding box.
We're going to create another plot.
So let's, let's increase this number of plots.
We have four and year four and year four, and let's pick this out first.
So we have that.
We copied this pieces out and then create this foot plot.
So we have this foot plot and now what we'll do is just copy this here and paste it out.
So here we, we paste this out and then let's get back.
And then what we pass in as image now is this cropped image.
So let's, let's actually copy this.
Let's say we have this crop, let's call it a crop, and then we have let's take this off.
We have the crop pays it out.
So yeah, we have the crop and now we're going to take in the crop now.
So after let's look at this.
So after we've all, we've had an error before, so let's do this so you could see better.
Okay.
So actually what we're seeing here is we have this pattern to be done and then we're taking this image.
And then what we want to pad it all, like stamped on another image which contains only zero pixels.
So let's look at that.
We have that, We have this pad, we pass in the crop, we have the offset height and width, and then
the target height has to be given.
So here we're going to put in the image size because we want this to be padded on an image with this
dimensions.
So we have the image size there.
Now we could run this and see what we get.
We have this error.
Let's modify that quickly.
So yeah, we have this taken off and then we have 2100 or 100 OC.
So let's run that and we should have some reasonable offset with the gain.
OC We should take this off.
Okay, So let's look at this and there we go.
As you could see, we have exactly what we expect.
You see that we have this somehow the same image, but we've taken out only this crop.
So that's what we actually want.
We want to be able to take all of this crop and then take this like this crop and then add it with this
image here.
So what we want to do is want to take this image now and then add it with this image so that we could
be able to create our data augmentation pipeline.
Now, we could we could call this image four.
So let's let's have this as an image for equal this year.
Let's take this off image for we paste that out.
And then once we pay that out, we could.
Now let's also do another subplot.
So we keep on doing the subplots.
Let's copy this and paste it here.
That's fine.
We take the crop as we have the the aurora.
We have this image four plus our initial image image one.
So let's have that and then we plug it out.
Okay, We get in this plot, but it isn't very clear.
So let's let's increase that figure size.
Let's add the figure size here and that's it.
So we run that again.
And then let's look at this now.
Claire Okay, here's what we get.
So as you could see here, we have this patch, like it looks like some is working, but there is a
problem as we have some sort of mixture of this and the initial image.
Now this is logical since when you do this addition for this for this black region, you just have only
the cut.
But for this region you still have the part of this cat image.
So what we need to do is we need to remove this part such that when you take this and add to this,
it just fits in like a puzzle then to crop out just this part from this image, we are going to use
the same process we've used for this dark image right here.
We simply copy out what we had already here.
So we had this cropping.
I could copy this out and piece it out here.
We run that and here is what we get.
So you see, we have this crop again.
Now we'll take this crop and then back to bounding box as we had done here previously.
So let's copy this.
And then what do we have?
We have the seventh and then we're going to pass in this crop.
Kat So we passing the crop.
Kat And that's it.
But we'll print out this image, so let's call this image five and here we have image five.
We run that and this is actually what we get.
Now.
The aim is for us to be able to take this and subtract, like take this and subtract.
This image from eight such that I will be left with the full cat.
Without this portion.
This portion right here.
So that said, if just here you do image one minus this image five, you would get this answer.
So as you could see, you have this whole cat without this portion.
And this is exactly what we want.
Now that we have this part, we could add it up with what we wanted initially here.
So we add it out with this image four.
So here we have plus image four and we get the response and there we go.
So we've completed this process of cutting out this portion from your and then fit it in on this image
one.
We now take off this other parts or take this off.
And there we go.
We now have the crop, the image four, crop cut, image five, and then our final image.
We copy out this mix up code we had done previously.
So we get back to this.
You see now, we could easily integrate this since we have image one and image two.
We just left with the labels.
Let's just copy this out.
Cut that.
And then here we take this off.
Pays it out.
There we go.
And we now have the output image.
We now modify this variable name, such that crop is crop one.
So here we have crop one instead of crop.
And then yeah, we have crop two.
So everywhere we meet crop crop cut, we have crop two, here we have crop one and then this image two
is maintained is from here we have image two, then the image four we could turn this to pad one, So
let's call this pad one.
Yeah, we have pad one, but this is actually pad two since we're working with the image two.
So let's call this pad two and crop two.
We have the crop two.
Then from this we have crop one and your image one.
Yeah, we have crop one and here we have this image five will call it pad one.
Then coming right here.
Image five, image five is pad one and then image four, pad two.
So we have that done.
Now we could focus on how to get this bounding boxes.
So yeah, we just picked this bounding box, but how do we get this bounding box?
To get an answer to this question, we are going to make reference to this formulas which we're given
in the paper right here.
We are told that our X is drawn from the uniform distribution which takes parameters zero and the width,
and then our Y is drawn from the uniform distribution, which takes parameter zero and the height.
Now if we consider this image and say this box randomly picked, then our X is a center that is this
distance from this that is based on this origin actually.
So we have this vertical distance to this center, which is our Y, and then this horizontal distance,
which R, which is our X, So that's how we obtain our Y and our X, we should draw from the uniform
distribution.
Now to obtain our W and our H, we have this formula right here.
RW equal w w the width of the image.
So we have w times the square root of one minus lambda.
Recall how we obtained lambda with a mix up data augmentation strategy.
So is this exactly the same way we get lambda?
So here we have w times one square root of one minus lambda r h equal the h the height of the image,
Times Square root of one minus lambda two.
Now note that when you multiply this two, that is, if you take our W times our H, you will have the
numerator right here.
Let's have this our w times our h gives you this numerator and then if you multiply these two you will
have w h times the square root of one minus lambda.
So if you multiply these times this, it gives you w h these times, this square root of one minus lambda
times the square root of one minus lambda, and you have the square root of one minus lambda.
Okay, so this is equal our w r h.
Now we'll see how they obtain this formula.
You see here that since you have RW rh you could divide your by w h and divide year by w h.
This goes away and we left with this.
Obviously, if I have the square root of X times the square root of X, then it is equal x since I'm
having the square root of x squared.
Now the square comes with the square root, it gives you x.
So in this case our x is one minus lambda.
So this is equal one minus lambda.
And that's how they obtained this relationship right here.
Now, that said, we have our H or we've taken this off.
Let's go one step back.
We have our H, Let's take this.
We have.
Sorry our W and then we have our H.
So this is our W the width and then yours are our H.
The height.
Our height.
But recall that what we have to pass into the method which permits us crop out the zone, for example,
is actually this point right here, the top left corner of this bounding box.
So we are not going to use the center, but instead it's top left corner.
Now how do we get the top left corner to to to leave from the center to the top left corner?
What we need to take into consideration the fact that where we notice this width, for example, now,
if we know this distance and we we have this distance from this to this point here, we're supposed
to the center, then we could obviously get this distance here.
We could get this distance.
To obtain this distance, we need to take all this.
Minus just this portion.
So obtain this potion right here.
Now, to obtain this portion is easy because we already know the width are a RW.
So all we need to do is divide this by two.
If we divide RW by two, then we have this distance.
And since we already have this distance our X, then we can obtain the coordinate, the x coordinate
for this point right here.
Now the next thing to do is to find our H, and we're going to use sorry, we're going to find our Y,
and we're going to use the same method we did to find our X from base in the top left corner for our
Y.
We know this distance.
Now, do we know this distance?
Yes, we know this distance because this is half of the total height since we are found at the center.
Now, if we have this distance, then we could find this distance because this distance plus this distance
gives us this distance.
And so to get this distance, we need to take this distance minus this distance, to obtain this distance.
And if we have this distance, then we have the y coordinate of this point right here.
Recall that the reason why we go into all this is simply because the methods of by TensorFlow consider
that the bounding box coordinates are given based on the step left on this top left corner right here.
So now we define this function box, which takes in the lambda, takes in lambda.
And then what this does is it makes use of this uniform distribution to obtain our X, our Y, and then
makes use of lambda to obtain rw rh.
Now, getting back to our uniform distribution, we could copy this right here and then put this in
our code.
Let's for now, let's keep the functional side.
Let's just have this.
So here we have our uniform distribution.
Recall our low is zero.
Let's take this back.
Our low is zero and our high is the width.
So we have your M size and we have that OC.
So we've defined this and then we have our W or rather our X, that's our x.
Now we want to have our Y, let's have it, our x, our Y, and then we could simply copy this out.
There we go.
We have this, let's print this out.
So we see what we get.
Print out our X, for example.
There we go.
We have our x m size not defined.
We started a notebook.
So let's get back to we defining this in size.
We run it again.
And this time around everything looks fine.
Now we could draw a sample from this distribution, so let's have sample.
We draw a single sample and see what we get.
You see, we have that.
We could take this zero element and there we go.
So now we're able to draw a sample from our distribution.
We could do the same for our Y.
So here basically we have this, and then what we want to do is to ensure that this coordinates are
integers.
So we could cast this.
We can see that we have the D type equal int to two.
That's fine.
Copy this out and paste it here.
So we have that and then we cast this to now we have our X are Y, let's just do the sampling directly.
So we've taken this and then we do the sampling right here.
Sample, take a single sample, take zero elements and that's fine.
We do the same year, pays it out and everything looks okay.
So yeah, we could now print out, we could now print out our X and our Y.
Okay, it looks great.
We have now our random, our X and our Y go run again.
So you could see the response.
Now we have to obtain in size times the square root of one minus lambda.
So we suppose that we have lambda, we actually going to use the same method we had used previously.
So suppose here we have our lambda and then right here to obtain our w we have the size times tf match
square root of one minus lambda OC one minus lambda.
Okay, that's fine.
Now we have our h, our height, our H same.
Size one minus lambda.
That's okay.
So now we have our H, our RW and our H, our X and our Y.
What we'll have to do is to modify this our x so we will have our x equal our x minus the width divided
by two.
Now we want to have a whole number, so we have that minus that divided by two.
But the width is the M size.
So we want to have this in size.
There we go.
Now, if you don't get why we're using this, just get back to this image to obtain this distance.
That's to obtain this coordinate.
This top left corner will simply take this distance to the center, minus the width divided by two,
and we do the same for the height.
So that's it.
Now we're doing that.
We have our x equal x minus M side divided by two.
We repeat the same.
And then we have it for Y.
Now, this is actually this width divided by two, the width of the box, not the another width of the
whole image.
So we're making an error here.
Let's get back to this.
We're making use of this width, actually, not the width of the whole image.
So let's get back and we'll put this code after this.
So because we're going to be making use of this, our H and our W, now we have here this is our w rw.
That's fine.
Okay.
Now, yeah, we're going to make use of our H So we have the rh, r y, our Y, okay, so now we have
our X and our Y, we could now print this again and then also print out our W and our H.
We get in this error.
We're told that in this computation right here, the RW is meant to be an inch that was passed as a
float since after this computation here we would have that as a float.
Now we could, we could always print out our w dx type.
Let's run that and see what we get.
You see, we have a float, so we'll modify that and cast it to ensure that we have an int.
There we go.
We have D type equal.
Let's just copy this to F instead of two.
So we paste this out here and paste it out this way to that cast and that's fine.
So now we expect to have the expected response and there we go.
You see we have our X, we have our Y, we have our W, and we have our H.
When we run our we run the cell several times you will notice that you will have some negative values
popping up from time to time.
And this actually happens when, for example, we could have our center at this level, but our weight
is so large that it doesn't fit in the image.
So it goes in a negative direction.
And this kind of situations are, say, in this kind of situation, you could have our box going out
of the image.
And so we have to ensure that each time we create in this box, we limit it to the image.
So this box now let's take this off.
This box now will be this one.
And this auto box right here will be this box.
So we redraw this box, but it wouldn't go out of the image.
Our aim is to ensure that we take this pan off.
Okay, so that's it.
We are now going to keep this values programmatically by using the clip by value method.
So here we have to clip by value.
There we go.
We have that.
And then once we pass this in, we're going to specify the range.
So we're making sure that the values always fall in this range zero to the size.
So that's fine.
Piece that out.
And we have this clip by value right here.
So that's it for our X are Y.
And now after running, you should have only values fall in that range after clipping.
What we have is if we had say, initially this box right here, we now have this box.
So this is what we get after clipping.
Of course we get the coordinates of the top left corner, but the width and this particular case has
changed.
In another situation we could have.
US, for example, if we have this in this case, the height changes because we don't we no longer have
this height, but now this new height, in a situation where we had a box which was like this, for
example here, but the height and the width will have to change.
And so based on on this new modifications, we have to ensure that we actually pass the right width
and the right height.
And so what we could do is make sure that once we have the center and we subtract the width or we subtract
half of the wheat and half of the height, to obtain this, we call we had the center, we had x minus
the width divided by two to obtain the x coordinate of this point and then y minus the height divided
by two to obtain the Y coordinate of this point.
Now what we'll do is we'll obtain this coordinate right here.
But while implementing the clip function and so this means that if we had a box like normally centered
in the image like this, and then we have this would obtain this point top left corner, and then we'll
obtain this point bottom right corner just by using X plus W divided by two and Y plus W divided by
two.
So this point is of has coordinate x plus W divided by two and y plus W divided by two.
And that said after clipping would have just this now.
But what's interesting about the method of clipping after getting this is that now we know exactly where
this bottom left is found or rather this bottom right is found.
And if we know if we have this cut in it and we have this cut in it, then we could recalculate the
width and the height to recalculate the width.
And the height suffices to take, for example, for the width.
We take this points here, the x axis minus this x axis right here or minus x axis at this point, because
this point and this for the x remain constant, whereas for the Y axis, this point and this also remain
constant.
So that said, all we need to do now is to take this Y minus this Y, so obtain the height and then
this x minus this x to obtain the width.
So based on that, we will re copy this out and paste.
So yeah, we have x button right x bottom right and y y bottom right.
All we need to do is to change this and add a plus.
And here we add a plus, we clipping by value.
So we'll always ensure that it's found in the image then also now to obtain this final RW.
So our RW now is equal to our y bottom, right?
Minus our our Y, and then the r h will be equal our Y instead.
This is actually X and here is x class H That's W sorry.
So we have y be R minus our Y.
Okay, So now we've modified the way we calculate this based on the fact that the box may be out of
the image and that now it has some modifications to be done on the width and the height.
Now the next step we have to take is if this r w is equal equals zero, then we have to make sure that
our W becomes one and then you're we repeat the same process.
If our H equals zero, then our H becomes one.
So that is it.
Now we have our X, our Y, our W, and our H.
That said, let's now create our box method.
We have our box method right here.
Let's have that.
And this is what we return.
So we return this.
But note that here we taken the height, the offset height, offset width, target height, target width.
And so this means that what we have to output in this method right here has to be our Y.
R.
X and then RH.
RW.
So there we go.
We return that and normally everything should be fine.
So we have this box method defined.
We run that, we get in this error, we should pass in our lambda.
Let's go down.
We have here lambda.
Okay, so we've passed in lambda and that's fine.
Now next thing to do is get back to our mix up and then we like we did before with this lambda, we
actually could get this.
We, we passed in the lambda so we could take this off from here.
So we run this again and then we'll define this lambda creation just right here we have Lambda.
Let's go back, we have lambda and then now we have the box.
So this box is going to produce this output we need here.
Let's call it our Y and that other in the documentation are y rh are w equal box of lambda.
So now we have box of lambda and then we have this outputs here.
Now instead of passing this, we have our Y, our X, our H, and our W, Let's take this off those
boxes which we fixed initially.
We have this.
Yeah, we have our Y and our X scroll down.
That's our X.
Okay.
Now we repeat the same process here.
We just simply copy this out and paste here, paste it out.
Then also we have our Y, our X again.
So our y, our x, let's take that off.
Okay, So that's what we have now and the mix up seems to be fine.
So we could, we could get back, run this method and then run our mix up method again.
Now as usual, we are going to mix up these two data sets, so it suffices to run this.
And then for this one we are going to instead of using the mix up.
So yeah, we commend this part and then we have the map and cut mix.
So here we have the cut mix method, we run this and everything should be fine.
We get this error.
Let's check on how we call this.
Oh, we call this mix up.
Still, this should be cut.
Mix can mix.
We run that.
Fine.
Let's get back to this here and run it again.
Okay, that looks fine.
Train data set.
Everything looks fine now.
We'll try to plot out some values.
So you see clearly what this looks like.
Let's create this new code, sell down your PCs out, and we are going to show this image from our training
data.
And there we go.
We could notice this patch could run this again.
And this time around we even have a bigger patch.
So that's it.
We've seen how to come up with this data augmentation strategy.
But yeah, we had to do do with the label.
So I think if we print out this label, let's print out a level here.
See, we print out the label, we just get an all zeros.
Anyways, let's go ahead and implement this section for the label.
And to do that, if you could recall this formula, you have one minus lambda equals this.
So this means that lambda is equal one minus rw r h divided by w h And the reason why we need to do
this again is simply because the RH like at this level where we had this RH and RW modifications, all
those clippings, we have to ensure that when creating the label, that condition is verified.
So let's get back, let's check on this formula.
Lambda Equal one minus RH.
RW So here again we have Lambda equal.
RH Times our W or RW.
RH Anyway, we have this RW RH then divided by w H So here we have MX size M size times.
M size.
Okay, Now this is one minus all the.
Then we apply the same formula for the mix up.
We have this lambda we've got now lambda.
We now pass this.
We have level one and level two.
Okay, so there we go.
We have our cut mix and then let's get back.
Actually, let's, we will have to re run this run our cut mix and then for our training data would
have to re run again.
So we'll have to run these cells again.
We run this and then recreate our training data set.
We're getting this error meaning that Lambda is of type float 64.
We get back to this and then we do some casting to send this make this float 32 OC That said, we run
our cut mix cut makes run we we run this.
There we go.
Yours can mix we run this again and this is fine now.
So we have our training data set and now we'll go ahead and visualize it.
We have this double knowns right here.
So it's preferable for us to actually we run this from this point.
Okay, So we run that from that point and then we get back to this.
We run this one.
That's fine.
We can run this and this should be OC now.
Okay, that's fine.
Now we have everything in tact, so that's fine.
We can now visualize our data so you can see we have our patch and then unlike before where we had at
zero or one, now we have values between zero and one as our levels.
So let's now run this model compiling and training.
After the training is completed, we obtain this results.
We see here how this accuracy.
Let's scroll down here we have this accuracy, which doesn't really change much.
So it's just around this 4648 and how is this 50%?
So it's around this 45 to 50% range.
And the last two doesn't really change much.
Now, this is due to the fact that the model is getting confused.
And the reason why this model gets confused is because if you have, say, this uninfected cell right
here and this prototype cell, it's actually this portion which permits the model, know that the cell
is parasitized.
And so when you come and crop this part right here and then attach it to this cell, you're at this
position, the actual part of this parasitized cell image which makes this parasitized isn't taken into
consideration.
And so the model gets confused as now it doesn't really know how to differentiate between and uninfected
and parasitized.
So again, this cut mix data augmentation strategy isn't adapted for our model, though it could be
applied in many other problems.
We thank you for getting around up to this point and see you next time.