Hello, everyone, and welcome to this new section in which we'll delve deep into implementing data
augmentation with TensorFlow two.
The first method we'll use in implementing data augmentation will be using this TensorFlow image model,
which is made of this different functions.
Now, this is just a few of those functions and you could check out the rest in the documentation.
So we'll be able to do stuff like adjusting brightness, contrast, gamma adjustments, saturation adjustment,
flipping left, right, flipping up, down and rotation.
Using this TF image is known to be a more flexible way of implementing data augmentation as we could
alter an input image with all this different functions given to us.
The next method we'll use will implement in this course is by working with the Keras layers.
And although we're limited by the number of data augmentation layers made available to us, this method
permits us to carry out data augmentation more efficiently and hence speeds up the training process.
Now, to find a balance between these two methods, we could implement our own custom keras layers.
And that's exactly what we're going to do in this section.
We'll now look at how to implement data augmentation with TensorFlow.
So we're going to get into TensorFlow and images.
So here we have TF.
Here is layers.
We actually in Keras closes up and TF dot image actually.
So we have the image with all these methods which we could use in data augmentation.
Recall that in order to do data augmentation, we're basically modifying the images while keeping the
labels fixed.
So that said, as you could see here we have this several methods.
Let's check out on this overview.
And we should be able to get this list of methods.
So here we have the list of methods and we have the categories.
So here we have image adjustments.
Working with bounding boxes here is for object detection.
Where you have to get into this here is cropping, flipping and decoding and encoding.
So this is what we have now.
If we get back to the top, we will have this adjustments.
Um, okay, so we have this resize which we've seen already and then we have this adjustments, we could
adjust the brightness, contrast, gamma hue, jpeg quality, saturation, brightness, contrast, hue,
saturation and per image standardization.
Now notice that these ones here are random, so we are adjusting our image randomly.
As you can see, we have random brightness, whereas here we actually adjusted this brightness with
some fixed parameters.
So if we click on this adjust brightness right here, you see we have this image and we have this fixed
delta, which we choose.
Whereas if it's random, let's get back.
If it's random, that will be fixed randomly.
So we click on this random brightness.
You see we are just giving a max delta because we just all we need to pass in here is a range and then
we are going to randomly pick a delta in that range.
So that's it for this image adjustments.
We also have this cropping so we could crop out some parts of the image, we could crop out the central
crop, we could crop and resize.
So this means if we have this image, so we have this image and it's 224 by 224 and then we want to
do a crop, we could do a center crop like this.
We could take this center crop.
Sorry for that.
We have this center crop and we have this new image right here, which is this center.
Now, after getting this image, let's suppose we have now this smaller image of shape, say a 150 by
150.
And then what we could do now is to resize this so that we get this shape which we need to pass into
the model so we could resize this.
Hence the reason why we're using the crop or crop by resize crop and resize right here.
And you could as well just crop and then do the resize manually.
So that's it.
We have flipping, rotating, so flipping we have an image, we flip it left, right, up, down or
random randomly.
So it's basically the same thing.
But this time around we're doing the randomly rotation and.
And the transposition of the image.
So as you could see, TensorFlow gives us all these methods which we could use in modifying our images
and hence augmenting our data.
So here we will define this method, visualize.
So you could see this clearly.
We have the original image and then we have the augmented image.
Now in here we have subplot.
We want just one line, two columns, and this will occupy the first position in that in that two column
space.
So here now we have this image show which takes in the original and then we repeat this.
So here we have this second position and then we have the augmented augmented.
Okay, So that's our visualized method.
Now what we'll do is we are going to get the original image.
So let's say we have this original image which is equal, an element which we take from our data set.
So we have train data set.
And then let's just pick one element.
So we have this, and then we take just one element from our data set.
Now that's it.
And this should output the label.
So we should have the label and there we go.
So we have the original image here, and then we now work with the augmented image.
We get this augmented image and to obtain this augmented image, we are going to modify this original
image.
To do this modification, we are going to make use of this methods which we've seen already.
So here, let's pick out this flip left, right.
We'll click on this flip left, right and see what we get.
So all we need to do is just pass the image.
So let's copy this.
And then what we have here is our flip left, right and our image which has been passed.
So that's fine.
We have original image.
So that's it.
Now we have this augmented image, we run that and then we could go ahead, visualize this augmentation.
So here we have original image and augmented image.
Okay, so we run this and see what we get.
So that's it.
You see that now in your data or in your data set, you will not only get this image right here, you
will not only get this one, but you would get this and this.
So this means our data set now will be multiplied by two.
Let's check on other augmentation strategies.
Let's get back.
Um, here we go.
We have flip up down this similar to what we've seen already.
Okay, let's check out let's do this random flip up down and then let's click on this and then rotation
by 90 degrees.
So here we have this image similar and then they could specify the seed.
So here we have the random flip up down, let's run that.
And then here we have random flip up, down.
Okay, so we run that again.
We have our mental image and then we visualize.
You see that it happens, that it's exactly the same image which is outputted, Let's run this again.
And now all we get is insert this.
So notice how the difference between these two images.
So that's it.
The next will be this route 90 degrees.
You could click on that.
You could always feel free to click and then understand what it actually means.
Like, yeah, we told this rotate image counterclockwise by 90 degrees.
Now let's take this off.
We know clockwise is this direction, so rotating counterclockwise means we'll be rotating it in this
direction by 90 degrees.
So that's it.
Let's take this off.
And then we have Route 90.
We run that and there we go.
Okay.
So as you can see, we've rotated this by 90 degrees and the anti-clockwise direction.
We can also try out this adjust brightness and random saturation.
So let's get back to this notebook.
We have your adjust adjust brightness and we run this visualize.
We have this error missing positional argument.
So let's get back to this and understand how it's used.
We check and adjust brightness and there we go.
We need to pass in a delta, so we have to pass in this delta.
Now, this delta should be in the range negative one one.
And as you could see when you add this delta, it basically just adds up to each and every pixel we
have.
So this one turns to 1.1, this 11 turns to 11.1 and so on and so forth.
And we told you that this delta is a scalar and it's an amount to add to the pixel values.
So let's go ahead and add this delta.
Let's have that.
We add the delta and we have 0.1.
So we run that again and this should be fine.
Okay, so that's fine.
We now visualize and that's what we get.
You will notice that there's some difference with this.
Like this one appears brighter than this and let's go ahead and increase this.
Let's take 0.8 and then have that.
Okay.
So you see, we are able to modify this brightness.
We could also include the random saturation, random saturation, and we have the original image.
Let's get back to the documentation for random saturation and see what we need to pass in.
So there we go.
We have this image we pass in the lower and the upper limit.
So here, um, let's find this lower upper limit.
We told you we're going to get an error if the upper is less than the lower, which is logical.
Lower should be less than the upper.
And then if the lower is less than zero.
So we have to ensure that we're dealing with values greater than or equal to zero.
So getting back here, we have the saturation that and then we have lower, um, let's say two and then
upper let's say 12, take this off, run again and visualize.
Okay, so that's fine.
So you see, we've added some random saturation and this is what we get.
We could obviously reduce this.
We could make this fall in the range of zero one and here is what we get.
Okay.
We now check out on cropping.
Let's check out this central crop here.
We have central crop, central crop and the original image.
So that's what we have.
Let's make sure we understand exactly how it's meant to be used.
So here we have this and then there's this central fraction.
So you have to explain how this works.
Now, where X is the central 50% of the image and it is a float which lies between 0 and 1.
As you could see here, 0.5 has been peaked, means that we're going to pick 50% of the image, but
this portion of the image we're going to pick has to be centralized as it has to be surrounding the
center.
So that said, we will understand how that works.
We can now put that, let's say, 0.3, run that and see what we get.
Okay, So you see, we get this portion right here.
Now, what if we expand this to like, say, 0.8?
We should we expect to have some black regions now.
Run that.
And you see we have some black regions, so it's actually centralized.
Let's take this off.
You see, it's actually centralized.
And we the reason why we have something like this is actually carrying out like this and something like
this.
So this is what we get in here.
Something like this.
Okay, so this is what we get when we do this central crop.
Now, the way we're going to integrate this augmentation in our data pipeline is going to be similar
to the way we did with the resize rescale.
So just like we use this map method, we are going to reuse this same method for our augmentation here.
Um, let's suppose we are going to, let's add this code and then define our AUGMENT method.
So we have this, let's put this down and okay, so here we, let's just have here augment.
We have this augment method, image label.
And then what we'll do is we'll take the image and then for that image we're going to have, um, the
rotate 90 degrees of that image.
Then next we are going to have adjust saturation and then we'll pass in the image and specify the saturation
factor.
So here we have our saturation factor equal, say 0.3.
Now you could always feel free to visualize this so you don't get to fully augment your data.
That said, here we could have this saturate always and copy that and we just flip left, right.
Let's come back to this.
We have this adjust saturation.
We could check that out and let's have it here.
Okay, we have that.
So we run this and visualize to see exactly what it's going to look like.
We modify that image to original image and then run it again.
Okay, so this is what we get.
Now, that's fine.
We can also include the flip left, right.
So here we have image, um, tf dot image, flip left, right.
And then what we do is we have, we can return the image and the label, so that's fine.
Then we could also just simply include this resize, rescaling, this augment.
So here we have uh, image and the label equal resize rescale, which takes in the image and the level.
So we resize and rescale before doing the augmentation.
So that's it.
Now we have this augment method defined.
We could now instead of doing this here.
So here we have augment.
Now another thing we want to do before doing the map is actually shuffling.
So we'll modify the order in which we're doing things.
Now here we have this train data set, we could have that and then we have the train data set right
here.
Dot Shuffle.
Dot match and dot prefetch.
So yeah, that's what we have.
And then we just include the mapping right here.
So instead of doing the mapping before the shuffling, we'll just do it after the shuffling.
And there we go.
Let's now have this, basically this with AUGMENT.
So we just have that and then there we go.
We have this map.
Okay, So we now have that for the training and then we can close this up.
So that's fine.
We repeat the same process for the validation and the testing.
So here we're going to modify it just as we did with the training we have here, our validation and
then we include this map right here, map, resize, rescale.
Now notice how we are not augmenting the validation and test datasets.
So here we have the validation and then we now do the same for the testing anyway, for the testing
we had, we only did the we only use the map.
That is, we only did the resize rescale.
So that's fine.
We will now take this off train validation.
That's off.
That's fine.
Okay, so here we have our test data, we have our train data and then we have our validation data.
At this point, everything is set up.
We could now rerun ourselves.
So we start out with training.
So that's fine.
We'll come back, get back to this.
We have our data, we run all the cells and our model created.
So yeah, we're going to work only with this sequential API and skip out all this other methods of creating
models.
Once we're done with that, we get right up to this level.
Okay, so now we get into training.
That's fine.
I'll compile the model and we wouldn't use any callbacks for now.
Take that off and start training.
We're now done with training and we'll see the results right here.
So as you could see, the model performs very poorly.
And we'll try to understand why the model doesn't perform as well as it performed without the data augmentation.
Now, if you can recall, though, what we did for the data augmentation was that we rotated the image
by 90 degrees, anti-clockwise.
We adjusted the image, the image is saturation, and then we flip the image left, right.
But taking a look at the kind of data set we're dealing with, this saturation shouldn't be a great
idea.
And this is simply because if we look at each and every one of these, what differentiates a parasitized
cell like this one and this uninfected cell is this patches we have right here.
So you'll notice that with the parasitized cell we have, we generally have this kind of patches, whereas
with the uninfected cell, we practically have no patch.
And with this, whereas here you see, you could see this patches and so rotating or flipping left,
right, wouldn't change much about that.
So that is acceptable.
Now the problem with modifying the contrast is when you modify the contrast, you tend to make the parasitized
cells and the uninfected cells less differentiable.
And so it isn't a good idea in this case to modify the contrast or saturation of the image.
Let's take this example so we could clearly understand why this saturation data augmentation strategy
isn't a good idea.
So right here what we do is we just copy out this and then paste here.
We take out just two elements from our data set, and then here we have this one, four and two I plus
one, and we have the show From here we create another subplot.
There we go.
We have that.
And this one now is a saturation.
So TF image dot adjust saturation.
That's it.
And then we're passing the image and that's what we have.
So here we have this first subplot and this other subplot.
We have that two now, or we could run this and see what we get.
I'm missing the positional argument.
That is the 0.3, the the saturation rate.
So here we have 0.3.
We're using exactly what we used previously.
So here we have that and this is what we have and this example we have both in uninfected cells.
So we wouldn't actually see much difference.
It's true, this is modified, this two is modified, but there isn't much difference.
Let's take a parasitized cell.
Let's rerun this again.
Hopefully we get a parasitized cell.
Okay.
As you can see here, um, let's take this off.
As you could see in this parasitized cell, what could clearly show the model that this is a parasite
cell is this patch we have right here.
But when we adjust the saturation, it turns into this.
And you notice that this parasitized cell now looks more like an uninfected cell.
So the model isn't able to differentiate between these two clearly.
So we see clearly that using the adjust saturation data augmentation strategy isn't a great idea.
Let's comment this part and that's fine.
Now we're going to modify our strategy.
So just right here, what we had previously, we were going to take this off.
So we comment this and then we'll retrain our model.
And so here we retrain our model.
After modifying our data augmentation strategy, we run this again.
And after training, as you could see, we get this results.
Now we check out this loss.
This is what we get for the loss.
And if we look at the accuracy, we see that it looks more like what we had previously, though we still
have that maximum validation accuracy of about 94% and the train accuracy of about 99% right here.
And though the data augmentation strategy hasn't closed this gap, we had between the training and validation,
accuracy would see in other sections how the augmentation will be very instrumental in helping reduce
this gap between the training and the validation scores.
So far, we've been applying data augmentation by using the image methods.
Now the other way we could apply data augmentation is by directly making use of Keras layers.
So you come right here.
TF dot keras layers.
You could see.
And then you'd notice that among those layers we have this layers which permit us do modifications on
images, like with this random contrast which permits us randomly adjust contrast during training.
Here we have the definition of this layer, this random contrast layer, and we have this attribute
right here.
Now, apart from this random contrast, we have the random crop, we have the random flip, and you
could always get all this definitions get back to, yeah, let's have this flip.
Now the mode horizontal and vertical.
So we're going to flip it both horizontally and vertically.
We could also have other modes like horizontal.
That's kind of a left right flip.
We have vertical kind of up down flip.
We have horizontal and vertical, which is a mixture of the two and the default is horizontal vertical
as we've just seen right here.
We also have this random rotation which you could see random translation, random zoom, so you could
randomly zoom the data you're training on and you also have this resizing.
So we looked at resizing previously, but not actually on the data augmentation.
So we did resize using TF image.
Now you could also do this resizing via this resizing layer.
Now here you have the height and the width as we had similarly to what we had in the image.
We could scroll down and check on others.
We don't really find anyone here.
We could scroll back up and that's it.
Anyway, what's important to note is that you could create or you could implement data augmentation
via two main ways using the TF image and using the Keras layers.
Now we are going to look at how to implement this data augmentation strategy we had previously via the
Keras layers.
So here we had this rotation, so we had rotation and we had the flip left, right.
So we'll get back to the documentation.
And then what do we have here?
We have this random flip right here and then we're going to select the mode to horizontal since we're
doing left right flip.
So we'll copy this out.
We have that layer, random flip.
Let's put this just below.
So this is the augmentation.
This is augmentation using TF image.
So TF image augment and then this is augmentation using keras layers.
So tf Keras layers augment.
Okay, so here we have our augment.
It takes the image and the label, but since we actually built in Keras layers, we are not going to
take this this way.
We're going to have augment our layers.
It's just a way of differentiating between these two augment and augment layers, which is a sequential
model.
So we have the sequential all we're going to make use of the sequential API of the Keras dot sequential.
And then just right in here we are going to specify all this different steps.
And so we copied this out from the documentation.
Let's just paste it out here.
We have this random flip, so we're going to use this random flip right here.
Let's take this.
Let's import random flip from the layers.
So we have your keras layers.
There we go.
We will import random flip.
Now, we could have this.
We have random flip.
Okay, so we have the random flip imported.
And then the other one is that rotation.
Let's get back.
And we have here, um, rotation.
Let's check on that.
We have, let's scroll up and let's just do a quick search.
So here we have rotation.
Okay?
So here we have random rotation.
And don't forget we rotate it 90 degrees in the anti-clockwise direction.
So here we have to be careful to make sure we re-implement that same kind of rotation we had seen before.
Now here we have this definition.
We have a factor fill mode, reflect interpolation by linear, the seed for reproducibility fill value.
And that's it.
Now let's check down here and understand better this factor.
Now this factor we have here is actually this tuple, and this tuple is going to represent our range.
To better understand this range, you should note that it's like each and every one of these represents
a fraction of two pi.
Now what is pi?
Pi equals 180 degrees.
So two pi is 360 degrees.
So we're looking for a fraction of 360 degrees.
Now recall that we're going anticlockwise by 90 degrees.
Like we're just trying to re do the exact same.
Thing we have with the TF image.
So here we are going to look for a way to get 90 degrees from this 360.
And to do that we need to divide this by four.
In other words, it means we're going to get 25% of this 360 and 25% of 360 is 90 degrees.
So 90 degrees not percent.
So that said, we want to get 25% and that's fine.
Now, from what is given to us here, this is we represent as a single float.
This value is used for both the upper and the lower bound.
For instance, this results in an output in an output rotation by a random amount in this range.
Um, a positive value means rotating counterclockwise, while a negative means rotating clockwise.
Now we're interested in rotating counter clockwise or anti-clockwise.
Thus we're actually interested in rotating.
This way we call it was this.
This is clockwise, so we're not interested in this.
We're interested in this anti-clockwise 90 degrees rotation.
So we take positive values and then we will try to ensure that it lies in that given range.
Now, given that we are actually picking just random numbers and unlike the Route 90 method we are seeing,
which was somehow fixed, yeah, what we do is we could just pick a range between, say, 90 degrees
and say 90.1 degrees or any value very close to 90.
So that said, recall is going to be positive.
Unlike uh, if it was clockwise, it will be negative.
So here we're going to pick the 0.25 because we want to get 25% of this 360.
So we have 0.25.
That's positive, 0.25 up to say 0.2, let's say 501.
So let's pick out this range.
So we'll work in this range.
So that will be around this 90 degree that we actually looking for.
Now that said, since we're taking 0.250.205201, what we will have here is a 25% of two pi, which
is 90 degrees.
So it's going to be translating it to 90 degrees and a value very close to 90 degrees.
So it will be like 90 point a value very close or 0.00 whatever degrees.
So it's going to be around this.
So we pick this up and that's fine.
Now we ensure it's anti-clockwise because we've picked it to be positive.
Now the film mode and the fill values will just be the default values, even the interpolation tool.
So that said, we get back to the code, let's copy this out right here, run the rotation, copy this
out, get back to the code.
Let's include random rotation here so we have random rotation.
Okay, random rotation.
That's fine.
We could now run this.
Let's run this cell and then get back to our method.
Or this layer we're trying to construct right here.
So here we have the random flip and then the random rotation.
Now we started in this order method with the rotation.
So here we're going to redo that.
Now we have this random rotation, random rotation, okay?
And then we have this factor.
So we just do exactly what we just agreed to put.
So we have the factor a tuple take 0.25 and 0.32501.
Okay.
Now we have that.
We just for the rest we take the default value.
So we don't need to put those values.
So that's fine.
We've done with this first layer.
The next layer will be this random flip.
So we just put this random flip, let's copy this, copy that and then paste it out here.
Now we have our random flip.
We've decided on the mode, Take that off and that's fine.
Now this is horizontal, so let's have this as horizontal.
Okay, so we have this random flip and that and there we go.
We have our layer, our sequential layer.
Okay, now let's take this off and that's fine.
So we have this augmented layers or this augment layers which we run this time around.
Horizontal is not defined.
Now let's use the small case, uh, horizontal and run again.
Horizontal, still not defined.
Let's pass this as a string horizontal run again.
Then we could redefine an augment method.
Let's call this method, augment layer, and then this takes in the image image layer.
Sorry, labels.
And then what we'll do is we're going to return this augment layers which we just created, and then
passing the image specify that training is equal to true if that's okay.
And then we send out the labels.
So here we're going to take this augment layers, that's we're going to take the image and pass into
this layer right here.
It's going to rotate the image and then into this other layer which is going to flip that image.
So we get our output, which has been augmented and we have our labels, we run this, run this, and
then we get to this here where we have our AUGMENT and we have augment layer.
So we call the augment layer method.
Right here we call this method.
That's fine, we run that and everything is okay.
Now after running this training, we have this error, so it shows that we have an error at the level
of the shapes.
Now let's get back the problem here is we forgot to do this resizing before the data augmentation so
we could copy this out and then put out this resizing.
Here we have the image and labels or label, let's say image and label.
We have that and then we resize and rescale.
After resizing and rescaling, we now do the augmentation.
So we run this again and then we run this training.
As you can see, all is well.
We will now look at another way of creating of doing data augmentation.
Before doing that, we should also note that instead of doing this, resize rescale using the TF image
as we had done here, that is using this TF image resize, and then by dividing by 255 as we just did,
we could also use the resizing and rescaling layers.
So right here we're going to include those layers.
Here we have resizing and then rescale.
We run that and then just here we're going to include that resize rescale.
So here we say resize, rescale layers.
Okay, so now we have this, our sequential API and then we always start with resizing.
So we have the resizing.
Take this off and have this.
Yeah, we're going to put our image size.
We have M size M size.
Okay.
And then we do the rescaling.
So we have rescaling take this off and we do the rescaling.
So here we have this taken off one divided by 255 Okay.
So now we have this layers which are responsible for resizing and rescaling, uh, resizing and rescaling.
So we'll see how we'll replace this method in doing the resizing scale.
So that said, let's get back to this augment.
We had augment layer right here.
After getting this image, we do resize, rescale and layer.
So we do the resize scale on the image before doing the augmentation.
Let's take this off now and this should be fine now since we've built this resize rescale layers, we
could replace this here, resize rescale layers even for the test set and then we do same for this.
This already fine augment layer as well.
We've seen already for the validation we have resized rescale layers.
Okay.
So that's fine.
We run this, run the validation run train and that's fine.
We then train our model and see what we get.
So that's it.
Our model is training.
Everything is working well.
Now let's pause this and then check out this other interesting way of doing this data augmentation.
So as of now, we've seen how to do data augmentation by using TF image and then by building this layers.
Now we'll see the importance of building these layers.
But before looking at this importance, it's important to note this, and that is the fact that when
you're working with a TF image, you have more flexibility.
As with those layers, you are kind of like restrained to what you could do with data augmentation.
Though this layers already have many common data augmentation strategies like the rotation, the zoom,
the crop, the flipping, and the others.
Nonetheless, when you're working with a TF image, you will see that you have many more operations
you could do.
Like if you open up this TF image right here, you see you have many more methods and.
You could do many more stuff with this.
Anyway, getting back to our code, we have to note that the advantage of this layers is that you could
embed this into the model itself.
Now let's explain.
Supposing you have let's take this off.
Supposing you have your training pipeline here, so you have that TF data.
You do the shuffling, you do the mapping, you do prefetching caching and everything you need to do,
like with your training data.
So you are using your data.
And then next, after doing all this resizing and scaling and everything, you now pass this into your
model.
So you pass this into your model and then you train your model and have all your results.
Now, when you want to test this model, you will actually still need to do resizing and rescaling.
So you still need to ensure that you resize your image and then you also ensure that you rescale your
image before passing to the model.
Now, if you recall like, let's, let's go down, let's get down here, you would see under the testing
section, let's look at this under the testing section.
That will make use of this test data and this test data, as we have seen from here, was already resized
and rescaled.
So if you check here, you see we have done some resizing and rescaling right here.
Now, what if you have to take this this your model, which you have trained into another setup and
obviously in that new setup you will still need to resize and rescale because just like here, before
testing your model, you had to resize and rescale.
So what if we input this resizing and rescaling in the model?
So if we embed this resize and rescale in the model, it means that no matter where we go with this
model, we don't need to resize and rescale all we need to do now we need we will take this off or rather
I will take this and put it in the model.
So all we need to do now is just to pass in your data without the resizing or rescaling and the job
is done.
So now we're going to see how to do this with this augment or rather with the resize rescale layers
which we've just seen.
We also do with the AUGMENT, but the AUGMENT isn't very useful since in testing time we don't make
use of the AUGMENT or what's more important here to note is the resizing and rescaling.
Although if you can include this, augment layers into your model and make the training faster.
Anyways, let's go ahead and see how this is done.
So firstly, you have the layers which you've built, you've had the augment layers and then you've
had the resize rescale layers which we have had here.
So all we need to do is to get to this model right here and then after the input layer you just have
the resize, rescale layers and that's it.
And then you have augment, augment layers.
So here you have resize rescale layers and augment layers and that's all.
So with this you don't need to do any resizing or scaling on your input because this is going to be
done or this our layers, which are now part of your model.
And then here too, you don't really need to specify this image size.
You could now have this as known and then you have that.
So you could put in any image and then the resizing and rescaling will be done in that model so that
you could put this model or use this model in any environment.
And all you need to do is to pass in the image and you have your corresponding output.
Now we've modified this model.
We need to also modify the way we working with our testing and training and validation datasets.
So let's get back to this right here.
We have the down here, we have this augment, augment, and then we have this test data set.
So with the test data set now, we do not need to do this again.
So we just comment this because we don't need to resize and rescale anymore.
Now with the validation you see, with the validation, we do not need to do this mapping anymore.
You see, we take this off.
We don't need to do this resizing and rescaling.
And then with the training we'll get back to the training.
Um, the training.
We do not need to do this augment because we've included this already.
So all we need to do now is just to shuffle, put this in batches and do the profession.
So that's it.
So we have this, okay, we can now rerun this and see how the training goes.
We run this training.
As you can see, we have this very long trace.
And with all this with this message right here, which isn't very readable.
So what we could do is generally when you're trying to get or understand an error and you don't seem
to have an idea of what's going on, you could run the model eagerly.
So we use the eager mode, so run eagerly equal True.
And here you see when we run this again, you would have a shorter error message or more readable error
message.
So you check this out and you see you just have the shorter error message.
It should be noted that TensorFlow operates in two main modes that is the graph mode and the eager mode.
So as you could see here, when you don't specify this, you run the training in the graph mode.
The graph mode is actually faster and more efficient way of training our model, whereas the eager mode
is slower but more easily debuggable.
So when you have errors like this, you advise to run in this eager mode and when everything is okay,
you could take this off or set this simply to false.
But by default, this is not is not true.
So by default, we are not in eager mode while doing this training.
Now that said, let's scroll down and check on our error.
And what it reads is we cannot batch tensors with different shapes and components.
Zero First element had the shape and the element one had this other shape.
So we try to we understand from this that since we doing the rescaling in the model that is in here,
we actually doing the rescaling, the resizing and rescaling in here we have this batching.
That's what we had here.
We had doing this batching on elements which do not have exactly the same shape.
And so to avoid this, we will just fix this batch size to be equal one.
So we will work with batches of one and work with batches of one.
We run that and we run our model again.
We run this and that's fine.
So you notice that the training starts and everything now works well.
Those are going to be slower since we're no more working with batches of 32, but now our batches of
one.
And also to ensure that this goes even faster.
Let's take this off.
So let's pause this and then let's restart training with the graph mode.
So we run this and run this.
And this should now be faster than what we had previously.
Now training is going on.
We could pause with the training and see how it's going to be very easy now with this model to run inference,
to test this.
We've had this cell here, the cell image which will run inference on.
So we have the cell image and then we'll make use of OpenCV to read this image.
So here we have our image equals Cv2 Imread M read this image.
So and yeah, you could start by printing out image.
So you would have that.
Not defined.
Let's import cv2.
Um.
Right here.
Okay, so we go ahead and import cv2.
We can close this and we could import that just here.
So we import Cv2.
That's OpenCV.
We run that and then we now rerun the cell.
OpenCV actually stands for open computer Vision.
So this is a very popular image processing library, which normally you cannot do image processing or
open or computer vision today without really working with this.
So that's how with OpenCV you could read an image.
So image read and then here we go.
We have the let's print out the shape so you see that it matches up with what we expect.
We have that.
Okay, so here is the shape of our image.
And now we are ready to let's add a batch.
So here we're going to have image equal TF dot expand, add dreams, and then we have the image and
then we specify the zeroth axis.
So we want to add the batch dimension and then we print out this image.
So here we have our image.
Okay, that's fine.
Let's print out the shape.
So we should.
We should see that.
Okay, We have this here.
And now you see that we're going to pass this into our model directly.
So here we're just going to do, uh, net model dot predict image, and that's it.
So all you need to do is just do this.
We don't need to divide by 255, we don't need to resize and the model is going to understand.
Now note that we are passing this image or this input, which is 265 by two, 62 by three, and our
model is going to automatically resize this because we've included resizing in the model.
So let's run this now and see what we get and that's fine.
So it tells us that zero, that is, it's parasitic, but which is actually wrong.
But this is normal because we have not really trained the model for a long period of time.
And clearly from here, our results show that we should expect very poor results.
Why is this disturbing?
Let's scroll down.
Okay.
So let's scroll this way and then scroll back up.
Okay, So you see that here we have this poor accuracy 48%.
If we train for longer, we could get to the 99%, which we had already.
So this is actually done to show how easy it's going to be for you to run inference on this model anytime
you want to.
In every in in any environment you find yourself.
So far we've seen two methods of creating, of implementing data augmentation with TensorFlow within
the TF image.
And then we've seen the tf.keras.
Now with the Tf.keras is actually interesting when you want your pre-processing that is, say resize
and rescaling and data augmentation to be in the form of layers and the TF image is more flexible to
work with.
Now we are going to look at a way in which we could build a custom keras layer based on the TF image
operations.
So recall that when we're working with a TF image, we had this route 90 degrees, which was right here,
and when we wanted to work with those layers, we had to play around with this to give us that same
route, 90 degrees effect.
Now what we will do is in order to create a layer which takes to create this route 90 degrees keras
layer and not just use this random rotation because it's not exactly this.
We are going to define a class which inherits from Keras layers.
So right here we have this class.
Which will call Route 90.
Let's call it Route 90.
And then it inherits from layer we have in it.
There we go.
Self.
Super.
There we go.
Dot init.
And that's fine.
So yeah, we have this route 90 which inherits from layer and then we'll define the call method.
So we'll define the operation which is going to be carried out in this route 90 layer which we're about
to create.
Yeah, we have self and then we have the input image.
So what we simply want to do here is we want to have well, we want to return the TF dot image, the
Route 90.
So we're carrying out the same operation.
But now we converted this into a layer.
So here we take that and then we have image.
So that's what we output.
Now that sounds fine.
So here we've had this Route 90, which we've created.
We run this and then instead of just random rotation, we are going to use this route 90.
So here we have Route 90.
Take this off, not in its past and that's it.
So that's it.
We have our Route 90 now and we are not long.
We are no longer using the the random rotation layer.
So that's it.
We get back and then run the cells and get to training.
And as you can see, everything just works fine.
And we'll see how to get the best of both worlds.
And now we have the flexibility of TF image and the speed and portability of Keras layers.
Before we move ahead, there is this point which needs to be clarified.
So we've already seen that when we have a single sample like this one with data augmentation, we're
able to produce different samples.
In this case, we have the rotated version of this.
In this case, we have another version in which we've modified the saturation.
And this we have this version where we've flipped this horizontally.
That said, in practice we are not going to feed or we're not going to create, let's say this three
other extra samples.
What we do is if we suppose we want to implement three augmentation strategies, that is rotation,
saturation and flipping, for example, then for one epoch, let's have this here.
For one epoch we may decide that we do not want to carry out rotation and then from here we may decide
to adjust its saturation.
So let's have from this we have its saturation adjusted and then in the last step we will decide that
we do not want to flip.
So at the end we have this input and then we have this as our output.
Now in another epoch you will have this.
Instead, you may decide that you want to rotate.
So you may decide that you want to have something like this.
Let's rotate this by 90 degrees again here.
This is random.
So we randomly rotating this.
We decide at random whether we want to rotate this at 90 degrees or not.
And then once we have this, we may decide that we do not want to adjust the saturation.
And then finally here we have flipping.
So let's flip this.
Um, there we go.
We flip this and we have this as output.
So in this case, this is what we pass in the model.
In another case, we may decide that we still want to rotate.
So we may decide, I want to have this rotation.
Um, then, um.
Still no saturation, but here we may want to maintain this, so we may still have this.
You see, now we do not flip this.
And this is instead what this example or this model sees.
Now, in another case, we may, we may have the input itself.
That's this input.
Um, it gets passed this no rotation gets passed this, no saturation gets passed this, no flipping.
And this is what the model sees.
So we can see that the model has now seen four different examples of this same input without us necessarily
having to create this four different examples separately before starting to train.
And so this helps us carry out the data augmentation much more efficiently.
That said, when we get back to the code, you see that that's what we actually did.
Um, yeah, we decided to.
Or we may rotate or not.
We may flip this or not.
And if we are to flip, it's going to be horizontally.
And then here we may modify the contrast or not.
So these are all augmentations which are randomly carried out during training.
Now for the other alternative, which is the TensorFlow image, we need to make sure that this rotation
here is carried out in a way that sometimes we have the image rotated and some other times the image
remains intact.
Now get into documentation.
You'll see here we have this Route 90, which takes in the image.
And this K right here.
Now this K is simply a scalar integer tensor and the number and it's actually the number of times the
images, the image or the images are rotated by 90 degrees.
And so here we're going to define this K, which is a random number which could either be a zero or
a one.
And that's it for this rotation.
Now, getting back to documentation, you could check the same folder, adjust saturation.
Yes, actually stateless random saturation.
So you click on this as usual.
We pass in the image and then we have this lower and upper bounds, which are those of the random saturation
factor.
So let's take this here.
Basically this, we copy this out here.
So instead of this, we would have this.
Well, let's just let's just put this out here.
Um, we have image which is equal this.
Okay, so we're taking the image and then we could have, um, let's say 0.3 and then 0.5.
Okay, so that's it.
We are going to have this.
And then now for the flipping left, right, we get back again here.
Here you have random flip left, right.
And then we copy this out, see?
So we're going to randomly decide whether we're going to flip the image or not.
Let's paste this out here.
Um, instead of this, we have this now.
There we go.
Take this off.
And that's it.
So now we have this augmented method which randomly selects which augmentation strategies we are going
to use for a given instance.
And this permits us to carry out augmentation much more efficiently.
Now, let's do the same here.
Let's copy this and paste out right here.
So here we have this.