Hello, everyone, and welcome to this new section in which we will see how to implement data augmentation
using a specialized data augmentation library called Albumentations.
We will see how to use Albumentations with TensorFlow and also PyTorch, which will permit us to see
how easy it is for us to integrate augmentation with just any library.
Note that this session was inspired by a question posed by one of us.
Feel free to always ask questions as this will permit us better discourse.
We shall be looking at the Albumentations tool, which is a specialized data augmentation library.
Albumentations is a Python library for fast and flexible image augmentations.
It efficiently implements a rich variety of image transform operations that is optimized for performance
and does so while providing a concise yet powerful image augmentation interface for different computer
vision tasks, including object classification, segmentation and detection.
Now we'll look at why we need a dedicated library like Albumentations here.
In the documentation, they argue that if you have to do this kind of visual data augmentation, there
are a host of libraries like TensorFlow, which we've seen already, which do this kind of data augmentation.
But then when carrying out data augmentation with different types like, say, object detection, like
in this case, or let's take this example here, which is more illustrative.
You have this image as input and this bounding boxes which will permit you build a model for object
detection.
But then if you want to do data augmentation and then you want to augment this data by applying cropping,
like as you could see here, notice how this image right here has been cropped like what we have now
is something like this part or rather this part actually.
Now notice you have cropped out this whole image and you left only with this.
I think it goes right up.
So it should be something like this.
Um.
Yeah, something like this.
Okay.
So you have you've cropped this image and it's part of your data augmentation pipeline.
So let's take this off and this off.
As we were saying, you crop this image and here's what you get now with the usual data augmentation
methods are working with the usual libraries.
What you have to do is after doing this cropping, you would have to manually modify each and every
bounding box you find.
Here.
You see this bounding boxes right here.
All this ones are not taken into consideration because it's not part of this crop.
So you have to modify each and every bounding box you find here.
This red brown boxes right here.
And this is because this bounding boxes are got they get their positions with respect to this origin
right here or in this case with respect to this origin.
And so when you do this cropping, the obviously these positions change.
Now instead of doing this manually, albumentations permits you to get this restructuring of this bounding
boxes automatically without having to manually modify them.
And apart from object detection, another very common use cases in image segmentation, we have this
original image which has been transformed into this one, and then the image has a mask.
So you are trying to segment the different parts of this image.
And so after you've applied this rotation, you now get augmentation to apply the same rotation in the
outputs.
Another reason why using Albumentations is advantageous is the fact that it has this declarative definition
of the augmentation pipeline and provides a unified interface.
So basically this is what it takes to build our augmentation pipeline.
Now notice how with this we can also include this probabilities.
So you see here probability 0.3 and this probability is 0.5.
Now stating that this brightness contrast augmentation will get a probability of 0.3 simply means that
you are going to apply random brightness.
Contrast three times for every ten images you process.
And this means that if you have a data set of let's say, let's say we have a data set of 10 or 1000
images, we have this data set of 10,000 images, then what our model will see are the probability of
our model.
Getting the original images is in this case of P equals 0.30.7.
So we have a probability of it is most probable that we will get 7000 out of this 10,000.
From the original dataset, which our model sees, and then we have 3000, which is going to be augmented.
Then you also have this horizontal flip, so you have the 0.5.
So this means that you could pass in an image and then there is no bright.
Brightness contrast and there is no horizontal flip.
Same as you could pass in an image and you have the random brightness contrast and you don't have this
or you may not have this and you have this or you may not.
We've seen this already or you may have this and have this, but with this random crop, you will always
have it because you're there's no probability specified.
Then also it's advantageous to make use of albumentations because it has been rigorously tested.
As you can see, it has been battle tested, used widely in the industry.
Deep learning research, machine learning competitions and open source projects, high performance,
diverse set of supported augmentations, extensibility and rigorous testing.
We also have here this list of transforms and the supported targets.
As you could see, this list is broken up into two parts.
We have the pixel level transforms and we have the spatial level transform.
So if you want to get more information for each and every one of these transforms, it suffices just
to click on this.
So you just you could say, for example, let's pick this sharpen right here, click on the sharpen,
and you have here the arguments and the description.
Let's take another example from the spatial level transforms.
Here we could take the vertical flip.
So we click on this vertical flip.
You see we have this float probability of applying the transform default 0.5.
Now note that if you're dealing with a very large dataset, that is, if your initial data set is very
large, then you could use probabilities between 0.1 and 0.3.
Reason being that since your dataset is already large, it doesn't need data augmentation that much.
Now if you're dealing with a small, medium sized dataset, you could use probabilities between 0.4
to 0.5.
Nonetheless, you could always pick whatever value depending on how it affects your model performance.
Getting back to the code, we're going to make use of this example TensorFlow Data Augmentation pipeline
built with Albumentations.
So here we have the transforms and then we have the augmentation function.
Then data pre-processing.
And finally, integration with TensorFlow datasets.
Now, based on the kind of dataset we're dealing with, we have to be very careful in the dataset augmentation
strategies we're going to be using.
So like, yeah, we could use this random rotate because rotation doesn't wipe off those sections which
contain information that permits us to differentiate between a parasitized and an uninfected cell.
Getting back to our random rotate 90 you have here this argument a float which is a probability of applying
this transform default 0.5 so we could make use of this random rotate right here.
Then just below we have other different rotations.
We have the random rotate 90 apply, which rotates the image a certain number of times.
We have the geometric rotate where we can select the angle and which will do the rotation.
Then we have this save.
Rotate would avoid this kind of data augmentation strategies like the cropping because yeah, you could
crop out information which permits us differentiate between the parasitized and uninfected cells.
You also have this resize here which we will use in, um, pre-processing our images.
We have the vertical flip and the horizontal flip, which we are going to use.
So here we have horizontal and vertical flip.
Let's add that here.
We can also just a flip.
We also have this random grid shuffle which we could make use of this randomly shuffled grid cells in
an image, meaning that if we have this kind of image and then we've picked a grid size of three by
three, we could break this up this way, you break this up, and then you have this three by three
grid cell, and then you simply randomly shuffle this position.
So this one can end up here, this ends up here, this ends up here, and so on and so forth.
You could also have this random brightness contrast.
So with this one, we will take this all and then paste it right here.
Random brightness Contrast.
The next one, let's take this sharpen.
So that's it.
You could use other data augmentation strategies you want.
Always make sure that you visualize the outputs to better understand exactly what you're using.
Now we define our transforms.
We have our transforms, albumentations compose.
And then we have this list which is made of the different augmentations.
Yeah, we just copied this out.
Let's have this copied.
But before this, let's do a resize.
So here we could have a dot resize and then we specify the image size and that's it.
So we specify the image size.
We're going for the horizontal and vertical flip, so let's copy this out.
There we go.
You have that horizontal vertical flip for this random brightness contrast.
We're going to use the default parameters.
Sharpen.
We're going to use the default parameters and that's it.
We now have our transforms so we can run the cell.
We're getting this error.
And this is because right here we have to have this.
So it's album annotation.
This is going from album annotation.
We run this again.
We're getting some errors.
Our meditation has no attribute sharpen.
Even when we comment this one, we will also get this error.
Augmentation has no attribute random grid shuffle, so we'll just comment this too and then run that
again.
Okay.
That's fine.
That looks fine.
We can also implement this album meditation one off with this one off, either the vertical flip or
the horizontal flip.
So let's take this out of this here and put it right here.
We have one off horizontal flip or the vertical flip.
Now we're going to have this here.
And then so we have this list actually, we're going to create this list and it's going to be made of
this two transformations, which is the vertical and the horizontal flip.
Let's have this here and close this.
Okay, So we have defined that one off and then we could also specify a probability.
So let's take P equals 0.3, for example.
And there we go.
This probability here actually defines whether the one off will be applied or not.
And so we have that.
We can run this again, and that's fine.
Drawing inspiration from this method given in the documentation, we are going to create this, our
own method of album end, which takes in an image.
Creates this dictionary, fits this information in the transforms which we've created here and then
normalizes the image.
So this is what we do with this album meant method.
From here we could have our training dataset similar to what we've been doing already.
But the difference is that instead of this, we have now processed data.
Getting back to the documentation in this process data here we have let's copy out this process data
and this process data.
What we're actually doing.
Let's add this cell here and this process data.
We're taking, the image, we're taking the label, we're taking the image size, and then the actually
modify this image and have this and then take this label and pass it to the output since the label remains
unchanged.
Now, for this image size, we wouldn't need this so we could take that off.
We're just taking the image and the label size and then as this input we pass in the image or the tensor
out is going to be of data type.
Float 32.
The function is going to be Arc album.
And so ARC album meant is our function and then we are going to make use of this TensorFlow numpy function.
Now get into the documentation.
We see it wraps a python function and uses it as a TensorFlow operation.
That said, we could still work in the graph mode even though we are having this python code right here.
And this is because our TensorFlow is going to convert everything that goes on in your as a TensorFlow
operation.
So we could run this.
We have the processed data that looks fine.
We could run this too.
And then finally we have our train data set.
So we run our train data set batch size not defined.
We should have run this here.
Okay, We run this cell and then getting back, we run this again.
We should be fine now.
Okay, so here we have train data set.
We could look at that and that looks fine.
We can quickly visualize our data set.
We have the image, the label, and then we have an element pick from our data set.
So now let's show this image, show em we run that, we get this error because we are dealing with batches
of 32.
So let's just pick one of these elements.
Let's run that now and that should be fine.
Okay, so here is what we obtain.
Then we could have many more plots.
So we have this figure we define the figure size for I in range 1 to 32 one to have plot that subplot
subplot eight for and then I and then plot that show our image.
I okay we run that.
And here's what we get.
So we have this.
And now what we could do is let's let's get this cut out, which we have here in the documentation,
specify a number of holes, maximum height, size, maximum width, size.
The fuel value always apply false and the probability.
So when we apply this, you're going to see exactly or better understand all this different arguments
right here.
So let's go ahead and apply cut out, which should be more visible as compared to the other transformations
like the rotations that we did.
So here let's have this year, let's you see how easy it is now to integrate any data augmentation strategy
you have.
So here we have a URL and paste that a cut out and then we take the default parameters.
We have the cut out, let's run that again.
And what we need to do because the train data set has been modified.
So we need to like get the initial train data set we had after getting from here.
Let's get back down and then check this out.
Okay, so we have done the Transform album and run this, run this, and now we could visualize, okay,
yeah, the cut out hasn't been implemented.
That is, obviously there's a probability of 0.5.
So it's possible that we don't have cut out here.
Now, if we look at this, you see clearly that in some parts of this let's increase the size, let's
say 15.
Okay.
As we were saying, if you look at this, you'll notice that in some parts or in some images, we have
this portions which have been cut out.
So you could see, for example, this one, you have this part which is cut out and then getting back
to the documentation.
We have a default number of holes.
Eight and we could copy all this out and then modify it so you see how this could change the kind of
output we get.
Because here you see when there is cut out, we have one, two, three, four, five.
Surely you have the other holes or cut out regions in the black spot so you can identify them.
And then here you have one, two, three, four.
You see, you even have this looks like a cut out.
Anyway, that's the idea.
You specify the number of cut out or number of regions you want to cut out.
You also specify the value is going to take.
So like in this case, when you specify zero, it simply means you're going to be having this black
spots here.
So let's have that, um, back here.
Okay, so let's put this in here and then you have number of holes.
So that's it.
You could modify this and observe exactly what goes on to increase the size of the cut out region.
You could simply specify the values here, and then you have the fuel value.
You have this boolean always apply and then you have the probability.
So if you want to always have cut out, you could simply send this probability to one and that's how
it works.
So we've seen how this works.
We could go ahead and train our model using this augmented data, but before training, let's take out
the cut out as it was just meant to show you how this works.
Okay, so let's rerun this again.
We run this and then we visualize.
As you can see now, there is no cut out region.
So that's as expected.
That said, we'll move forward to training our model again.
So we run that training process.
So here are the results we get.
After training for several epochs, you'll notice that the accuracy doesn't go up to 99% as it used
to be before.
And also with this validation accuracy, we even get better results here.
As a result, we get after training for several epochs.
Notice how here the accuracy doesn't get as high as it used to be before.
So the accuracy we are getting now is the highest value of accuracy we get is like 94.5% and the validation
accuracy we get in here, it's about that.
Like the highest we get in here is like 94.48%.
And this is the accuracy versus epoch plot we now get.
Also, after checking on Albumentations GitHub page, we found this solution to this problems we are
getting here where we weren't able to make use of the cut out or rather we were able to make use of
the sharpen and the random grid shuffle.
So here when we do this install, which you have here and then uncomment the sections, you should now
have this working.