We've just looked at the section on corrective measures.
Now we're going to dive into TensorFlow data sets.
So right here in the documentation you click open on TensorFlow data.
Click on the overview.
Um to have an idea of what TensorFlow data sets are all about and also why they're important or why
we should generally want to use them.
The TensorFlow data API, as we shall see throughout this course, will permit us to build complex data
processing pipelines in a very efficient manner.
This API would permit us, um, work with large amounts of data, and also be able to perform complex
computations on this data effortlessly.
The first method, which we shall explore in our TensorFlow data API is from slices method.
So let's just copy this out.
We get back to our code.
Then we change this to train.
So this is our train data set.
And now we have um our from Tensor slices method which is actually going to take our tuple which will
contain the x train and the y train.
So that's going to be our train data set.
Now we could do for I in train data set.
Um let's um print out I well let's, let's break once we're done with the first sample.
So that's it.
You see we have this input and then we have the output.
Let's say I j I j and then print out I and then also print out j.
So you see that we have the I, we have this I.
And then we also have the j.
That's the, the output.
Now um getting back here you would find that it's very easy to um, do stuff like shuffling, um, batching
and also prefetching, which we shall look at shortly.
Now let's get back to documentation.
We could scroll down and here we have the shuffle method.
You see that this, um, shuffle method actually takes in a buffer size, um, a seed, because you
want to be able to reproduce a particular setting of your data set.
So if you want to shuffle your data in a way that, uh, when you shove reshuffle it again, you have
exact same shuffling order.
So that's it.
And then you have the reshuffle after each iteration.
So you want that each time you iterate on your data, you want to maybe reshuffle again.
Anyways, that's it.
We have this, uh, definitions here.
It randomly reshuffle or randomly shuffle the elements of this data set.
And the data set fills a buffer with buffer size elements.
So here we told that for instance if your data set contains 10,000 elements but the buffer size is set
to 1000, then the shuffle will initially select a random element from only the first 1000 elements
in the buffer.
So when you set the buffer size to 1000, what you're saying is, uh, want to shuffle from the first
1000 elements, and once an element is selected, it's placed in the buffer is replaced with the next
um or 1,001st element.
So maintaining the 1000 element buffer.
So here what this essentially is that if you have a data set with say ten elements and a buffer size
of let's say three.
So let's create this uh, ten element data set 12345678 um 910.
So here we have eight 910.
So we have our ten elements um data set.
And if we have a buffer size of three then the shuffling will be done such that the first element will
be picked from either one, 2 or 3.
Once we pick an element, let's say we randomly pick out two.
If we randomly pick out two, then what this means is we are going to take this off, and then we will
now pick between one, three and four.
So that our buffer size, um, remains three, because now we've picked out two.
And so we're left with one, three and four.
Now let's randomly pick an element.
We pick four.
So this is no more in our buffer.
We now move on to five.
And then we could randomly pick out whatever other element and so on and so forth right up to the end.
So that's essentially what um, is being described here in the, in the documentation.
Um, reshuffle each iteration controls whether the shuffle order should be different for each epoch.
So, um, after every epoch you could decide whether you want to reshuffle or not.
So getting back to the code, we have our train um, data set.
Well we already had this.
So we'll just say train data set.
Um, equal train data set train data set shuffle.
And then we provide a buffer size.
We could take buffer size of for example um let's say 16 of a buffer size of 16.
And then we have the batch.
So let's specify let's take this off.
We have batch and then we have batch batch um size.
We could also have your buffer size.
Anyways let's just put this let's just um have this buffer buffer size.
And then we could define this above.
So now we have the buffer size.
We have the buffer size which is set to 16.
And we have the batch size which we could set to 64.
So now we have our buffer size set and the batch size set.
Another thing we could do is um prefetching.
Getting back to documentation we have here that uh, most um data set input pipeline should end with
a call to prefetch.
This allows later elements to be prepared while the current element is being processed.
This um, often involves or rather, it often improves latency and throughput at the cost of using additional
memory to store the prefetched elements.
So.
So here we told that, uh, this prefetching has a buffer size.
So we have this buffer size argument and it represents the number, the maximum number of elements that
will be buffered when prefetching.
And if the value of um, TensorFlow data auto tune is used, then the buffer size is dynamically tuned.
So we'd like to use this.
So to understand the concept of prefetching, um, let's suppose that this red is for loading the data.
And this blue is for um training.
So let's say we load a batch and then we train and then we load again.
Let's copy this.
We load let's drag this.
This way we load.
And then after after loading you see after loading we go ahead with the training.
There we go.
And then, um, let's say we load again.
So let's copy this.
Let's say we load again and we go ahead again and, um, train on that loaded data.
The whole point of prefetching is that while you're training the model, you could be loading the data
set.
So instead of having to always wait for the model training to complete before loading, you could instead,
um, do this, um, simultaneously.
So let's, um, take this off so you could have this here.
So now you finish, you're done with, um, um, loading.
You start training and you're also loading.
So once you're done with this training, you see you could now you can now pick up from here or.
Well, you could pick up from here because you must complete the loading so you could pick up from here.
Let's take this off.
And then once you obviously while you're training again with this, you could also, um, start loading.
So we could start loading from here.
And let's take this off and then we could pick up from here.
Let's take this off.
We could pick up from here and start training.
So you see that at the end.
We've taken up uh, for the first scenario.
Let's let's reduce this.
Um, for the first scenario, we we took a time.
Let's say we took a time, t this time we took a time t to complete.
But in the second scenario, we've taken a time of about, um, two divided by three t.
And so that's why here we just do prefetch prefetch.
And then we have TensorFlow data auto auto train.
So that's it.
We run that and there we go.
We have our um train data set.
Um that's fine.
We we repeat the same for the validation and the testing.
So take this off.
We have um train off we have Val and then here we have Val, then here we also have Val.
And instead of train we have Val.
So train here we have Val.
We do the same for the test.
So let's run this and then do the same for the test data set.
We have test test and take this off.
We have test and yeah test.
So that's it for the um train validation and test data set on this.
You will notice that the shape is now different.
Let's let's actually print out just the shapes.
So let's say I shape and then J shape.
Run that and see what we get.
You see now we have 64 by eight and 64 by one.
And that's simply because our batch size was set to 64.
Now we're not going to change anything uh the level of the model.
So we still have our model.
Let's rerun that.
So we initialize the or, we reinitialize the parameters, we recompile.
And then here we're going to replace all this with train data set.
So we'll just say train data set.
And that's fine.
And then for the validation now we have val data set.
So take this off and rerun that again.
And our training should all be fine.
So you see we we go on with the training as usual.
And we could check out our plots.
And now the evaluation will be on the Val data set.
So let's take this off.
Run that again.
Yeah it's going to be test um data set.
Test data set.
Take that off.
Run that again.
And that should be fine.
So you see we've replaced um, our initial um data pipeline with now the TensorFlow data API.
Um, pipeline.