Hello, everyone, and welcome to this new session in which we'll treat mixed sample data augmentation.
Previously, we saw how to do data augmentation on a single image like this or this one.
And now we'll learn how to create new samples based on a combination of mixture of different images
or different samples from our dataset.
And more specifically, will treat the mix up data augmentation strategy, where we'll pick two samples
from the data set.
We're going to mix up the samples, and then we're going to include this strategy in our TF data pipeline.
Up to this point, we've implemented data augmentation strategies, which involve modifying input samples
like this one.
So we could take this input sample, we could do a zoom, we could do a center crop, we could rotate,
we could translate and do many other stuff with this in order to augment our already existing data.
Now, in this section, we'll look at another data augmentation strategy, which is known as mix up.
Now, mix up doesn't only involve just one sample, as we had seen previously with mix up, We are going
to make use of this two samples instead of just one.
Mix them up and then produce this output sample.
And if you look very carefully, you'll notice that this image contains this one.
You could see here, you could see it carved out, this dog carved out like this you see here, and
then this other dog right here.
So we also have this one, which is also carved out like this.
So we could see the mixture of this two to form one input.
So we have this output image which you could define as x prime, which is a mixture or a combination
of this image x one and x two.
So yeah, let's have x one and x two.
But this actually happens to be a weighted addition.
That is, we have a certain factor, Lambda, which is a value between zero and one and drawn from the
beta distribution such that we have x prime equals lambda times x one plus one minus lambda times x
to this means if lambda equals 0.5, then we'll have 0.5 x one plus 0.5 x two lambda equals 0.3.
For example, we have 0.3 x one plus 0.7 x two.
So some sort of weighted addition.
Now we've created this new input.
It's logical that we need to modify the labels because this doesn't belong to either or.
You cannot really say that this belongs to this class or this class.
It actually is a mixture of both classes.
So unlike previously, when we do data augmentation on this image, we are going to maintain the label
because it still remains this particular class.
But when we mix classes up together, our images from two different classes together like this, we
need to modify the level.
And so that said, we have a new label Y Prime equal lambda and you guessed that right Y one plus one
minus lambda y two.
And from here we can now dive into the code and implement this mix up that augmentation strategy.
Recall we said that Lambda was to be drawn from a beta distribution, but if you come to the documentation
we'll be using so far where we have this TensorFlow that ARG API docs, you wouldn't really find this
beta distribution.
So you are advised to look out in this one year.
So go to tensorflow dot org slash probability.
Instead it's here you would find this distribution.
So here we have this TensorFlow probabilities and then you have TensorFlow distributions.
When you click here, you could find many probability distributions, including this beta distribution,
which we want to work.
And now, now just right here you could see the beta distribution.
You have the definition and then notice how we have this two parameters which we must pass.
So the beta distribution is defined over zero one this interval zero one using parameters, concentration
one a.k.a alpha and concentration zero a beta.
So we have the parameters, alpha and beta to pass right here.
Now, if you look up in this mix up paper, you would see that the.
It is all for the youths like you're from.
This experiments is 0.2 and then sometimes you use 0.42.
But most times the test had on 0.2.
So its this parameter will be using and then getting back here.
All we need to do now is copy this out.
So we copy this, get back to our code and then we have the mix up down here.
So here we have this mix up, mix up the term mutation.
Let's reduce this and then we reduce this part so you wouldn't run the cells because we're not going
to make use of this now.
So yeah, we have this mix up.
Let's test out this, we run it and we have this error name.
Construction is not defined.
Anyway, we could get back here and then import TensorFlow probability.
So let's import TensorFlow probability and then put that as TFP.
We run that and then get back to our mix up.
Okay, so here we have this mix up.
Let's take this off and let's take all this off.
Actually, 0.20.2.
Okay.
So we got that from the paper.
That's understood.
And then we have lambda.
So here we have lambda.
If we spell it this way, we have the keyword, the python keyword.
So let's just keep that simple and just have this lambda spelt or spell wrongly.
And now we'll do lambda plus print out lambda lambda dot sample.
So we take, pick out one sample from that beta distribution and we run that and here is what we get.
So you see, if you run this again, you obviously get different outputs and all this drawn from the
beta distribution and in the range zero one.
So you're I'm just going to do this, pick out this zero item and then we have this output.
We could also put this nom py to get it.
See you have this now so you know not having a tensor, but anyway, we prefer to use it as a tensor
and we'll explain why we'll it's preferable to work with tenses in the function we're trying to build.
From here, we just simply apply the formula.
So we'll have the output image, which is equal lambda times, the image one plus one minus lambda times
the image two.
So that's it for the image would repeat the same process for the labeling.
Then using open CV, we'll test this on this two images which we've added here we have this dog and
this cat image right here.
So we have this tool.
We're going to test this on it.
Let's take this off right here.
We have we're going to read the images.
So we have image one.
We're going to do the reading in read, and then we'll do the same for him, too.
So.
Right.
Yeah.
We could print out image the shape and label that shape.
Or rather, let's just print out a label so we have that.
We get in this error.
Let's get back up and re correct this.
So this actually lambda, let's modify a lambda lambda equals this.
So let's have that out.
So we have that right.
We run this again and require broadcast table shapes, which happens because we haven't yet resized
this, because here we have image.
If we print out the shape of image one and image two before doing this operation, you see there are
two different shapes.
So we have to ensure that they are about the same.
You see, there are two different.
Now let's resize this with CV to resize resize and then we specify the shape.
So let's have your size in size M size.
Okay, we have that done.
We just copy this out.
Paste Out here we have size and then CV to resize.
Okay, so we've read, we've resized and that should be fine.
Now we run that again, same size, not defined.
Let's have that to be defined here.
We actually defined this previously, but we haven't run those previous cells since we started a notebook.
So that's why it isn't recognizing the size.
Okay, now level one or define this looks great already for this levels.
Let's say we have level one.
So yeah, we have level one equals zero and then level to equal one.
So we have just this two levels.
Okay, we run that again and this is what we get.
You see, we have this output image and then we have this final level and which happens to be near zero.
No one from here.
We could plot this out.
PLT m show and then we pass in the image and normalize this.
We run that and this is what we get.
So you see, we have a mix up of this two images now that we have succeeded to do this, let's make
this part of our TensorFlow pipeline so we could take all this out now and then define this mix up method.
We have this level.
Okay, we'll take this off.
Okay.
We're going to define this method.
Let's call it mix up.
And the way this method works is we're going to take in our data.
So our trained data set here, we have one and then train data set two.
So we have this two data sets which contain the same elements by which I've been shuffled so that we
could have this kind of mix up.
Then we can now make this data sets available which will do the mix up on.
So let's add some code, take this up and then yeah, we have our first train data set.
We have train data set one, which is actually our train dataset we have built already.
And so we're getting our train data set right from here.
We've run this already, we get back, be careful, we are not running this, but we may make use of
one or two methods from this.
So we have that and then we get back to this.
So here we have trained this set.
We do some shuffling, we specify the buffer size, and then we also specify that we're going to reshuffle
after each iteration.
We just copy this and pace out here to have our trained data set to.
Now, once we have this train that is set to, we now have our train, let's call this mix dataset or
mixed.
Dataset.
We have our mixed data set and then we make use of the zip method.
And with the zip we are going to pass in the train dataset one and then the train dataset two.
Now, if you could remember, we had an error when we pass into images which had two different shapes.
So we have to ensure that we do some preprocessing before doing the mix up.
So that said, after doing the shuffling, we could do the preprocessing.
So let's say pre process, let's get back up where we define this pre processing and we had your okay,
so we had this level of data augmentation, we had preprocessing, although we had inserted this in
our augment, but it's practically this resize rescale method here.
So we could run this and I'll be fine.
And then we just do resize rescale so we wouldn't call it appropriate processing.
Again, we just have resize rescale as we've done already.
We do the same mapping here.
We have resize and rescale.
Okay, So now we shuffle, we resize and rescale, and then we have our dieter, which is now a combination
of dataset one and dataset two.
This gif was gotten from Giphy.com.
Now we have this mixed dataset formed.
We run the cell.
That's fine.
And then in here, let's take this off.
We've run this already, so we have that.
And then yeah, we're going to take in image one.
So image one and label one, label one.
We have that this tuples, we have image two and then label two.
Let's close this up.
Okay, so we have that and then we get this from the train dataset, one dataset, one and the train
dataset two.
So that's how we get this.
We have image one, level one, image two, label two.
The image one that we had here, we done this again, we have Lambda, we get lambda, we get the image,
we have the label, and then we have our output.
So we return image and label.
So this is all about the mix up.
Now we run the cell and then we create this other new cell.
Yeah, we have this error, should have this, and then we create this new cell right here.
Then we pass out from what we had done already.
And then here we have our mixed data set.
So we now have this mixed dataset.
We shuffle again, we do the mapping with the augment layer, we do batching and perfection.
But since our AUGMENT is no more the previous augmented layer we had, yeah, we have now the mix up.
So yeah, we replaced that with mix up and that should be fine.
We have the train, we could do the same for the validation.
We get an error spelling error train data set.
Let's get back to this train dataset.
Okay.
So yeah, we have the train data set, we run that, run this again, we have here input y of small
operation has type in 64 that doesn't match the type flow 32 of argument X.
So this is where we multiplying the lambda by the labels.
So clearly we have the lambda which is a float and labels which are ints.
So yeah, we're just going to cast this.
So we have this casting, we specify the D type float 32 that's fine.
And then right here we do same casting, specify the D type again and that's it.
Let's run this again and see what we get is he works fine.
We have this one is also note that yeah, we were trying to experiment and change this.
So let's get back to the point to run that again.
Okay, So now we have our training data and let's see here we have the batch and then the image and
then the batch and then the label.
So that's it.
We've now created this data set, which happens to be a mixed dataset.
Then from here we are going to prepare our validation data set is actually the same as what we had already.
So you could just simply run this previous cell right here.
This cell lines data loading, just simply run this, run this and you should be fine instead of doing
this augmented layer.
We meant to just do resize rescale.
We just have to resize and rescale our validation data.
We don't really need to shuffle.
We could take that off.
We could take the fashion off, and that's fine.
So we run our validation and then check it out here.
So you.
You have a validation theater.
Now, the reason why we have this is because we run this twice.
So let's re initialize this right here.
Let's get back to the splits.
We create a trained theater and validation data.
Okay, let's run this again here.
Now, everything should be fine.
Okay, so you have that.
You have the batch dimension, and that's good.
So we now get back to training and make sure everything is okay.
We rerun this and everything is now fine.
So we're now set to train our model.
So we run our sequential API right here.
We run this, we then compile our model, and then we could get ready to train the model, have this
poor results.
Reason being that the mix up data augmentation strategy isn't adapted to the data set we're working
with.
Even if the mix up, the documentation strategy we've just applied wasn't very helpful for this particular
problem.
Is important to note that this mix up strategy could be used in many other problems and we that will
come to the end of this section.
Thank you for getting up to this point and see you next time.