Hello, everyone, and welcome to this new session in which we'll trade dataset versioning with weights
and biases.
If you've already used Git for code versioning, then you should note that this is kind of similar to
what git does as here we have this original editor.
Let's take this off.
We have this original data which is then pre processed into this data year and then this other pre processed
data or this pre process version is again pre processed to give us this one.
And then finally we have this last step or this last preprocessing step which gives us this data set
version which happens to be an augmented dataset version.
So this shows us that if at any point in time we are satisfied with this preprocessing, which was done
right here to produce this dataset version, we could simply pre process from or making use of this
data set version right here.
And so this greatly simplifies our dataset management when working in our different machine learning
project.
Just as good parameters do code versioning, weighting biases gives us the possibility of doing dataset
versioning and model versioning.
And this can be done using weights and biases, artifacts which help us save and organize machine learning
data sets throughout a project's lifecycle.
We are going to start with the data set versioning and the most common ways in which weights and biases
artifacts have been used for data.
Versioning are to a version that is seamlessly prepackaged data splits like training and validation
and test sets, iteratively refine data sets, juggle multiple data sets, and finally visualize and
shared data workflow before getting into seeing how artifacts could be used in data set versioning.
Let's look at this simple example.
In order to obtain the malaria data set, we have to start by loading this data.
We have this data loader which is represented by the square.
And now once we've loaded this data, we have our data set.
So let's say we have this original dataset right here.
We now go ahead to split this data.
So from here we can split up this data and then have the three different parts.
We have the training data, the validation data and the test data.
At this point, all three data sets and are being passed in this pre processing units right here.
So we pre process the train, pre process the validation and pre process the test and this gives us
an output here.
So for the train we have a pre processed training data here.
We have this pre processed training data, we have this pre processed validation data and then here
we have also this pre processed test data.
Again at this point we are now going to carry out augmentation on those pre processed training data.
This Peters means pre processed this pre processed training data and the pre processed validation data,
pre processed testing data.
So we have that and then we pass this through this augmentation process.
So we have this process with raw will be that of carrying out data augmentation on this data set to
produce another data set, which is actually now an augmented version of this.
So here we have pre processed training data.
Now we have augmented training data and this is kind of like the lifecycle of our data set in this particular
problem in which we're working on margin that this original dataset contains mislabeled examples.
In that case, what you want to do is now to carry out this labeling correctly, such that we have this
data set here which now has been cleaned.
Overall, this leads to a higher level of accountability as when working in a team and let's say you've
done some modification, let's say you have cleaned the data.
All the people in your team can now view this cleaned dataset and decide whether to modify it, deleted
or keep using it.
So if we suppose that the team accepts this clean data set and everyone is happy with this newly cleaned
dataset, we now see that instead of passing this directly into the split, we will now pass but this
one into the split.
So we'll have something like this.
We'll go this way, Let's take this off.
In that case, we will have to go this way.
We go this way.
And then into the speech and then talking about data set versioning for each and every dataset we've
created here.
Like for this one, this one, this, this, this, this, this or this one, we have different versions.
So we could have, for example, this PVD here, that is the pre processed validation data and it could
have a version zero or version zero.
So we have this version.
All right.
Yeah.
And then later on you may modify this preprocessing and it leads us to have another version of this
validation data.
So you could have another version, version one and the version version two and the version you could
say version best you can have version latest.
And so on and so forth.
And a good thing is when waiting buyers or when using Western buyers to start this dinner, or when
we use the Western bias artifacts, the data is stored in such a way that if we have data in this original
data set, which is exactly the same as what we have in this pre processed validation data, then that
data wouldn't be duplicated.
Now, it's true that if we pre process data, obviously we'll have all the data set, all the elements
of the dataset changing.
Now let's take this example in for the clean data set.
So here we've had we have this original data set, let's suppose we have say we have ten, we have ten
different examples, different samples in this original data set.
And then after cleaning our dataset, only two of these samples have been modified.
So we have eight unmodified and two modified.
So we have modified.
We have changed the levels of these two samples here.
What Wilson bias will do is it will ensure that this eight year aren't duplicated, that it wouldn't
create extra space for those eight others which haven't changed from this previous original data set
here.
And so in that case, we're going to only store this to new samples here while weights and biases keeps
track of the fact that this other eight samples haven't been modified and so don't necessarily need
to occupy extra space in the storage unit, which Wilson biases makes available for us for free.
That said, we could look at these different processes right here, the processes in the square boxes
as one DB runs, while this difference forms and even the versions which are data takes are the artifacts.
And so we could consider this here, where this represents the artifacts and there's the runs.
Also, we see that the artifacts are connected together by these different runs.
So this two artifacts, this training data and the original data set has been connected together by
this pleats run and then this to Peter Z and train data are connected together by the preprocessing
run.
Getting back to the documentation, we'll see how we create this artifact here, which is called the
new data set of type raw data.
And then this is created within this run right here.
So we see how we create this run and we'll specify the project, my project.
And once we create this artifact, we're going to add data into it.
Now, one thing you could do with one TB is you could simply add a whole directory.
So we're supposing that all your data is a given directory, and so you just all you need to do is specify
this path and then you make this data part of this one.
DB Artifacts, which we've called my data.
And then finally you log this artifact to one TB.
Let's copy out this sample code, paste it out here.
We have that sample code and then we could get started with our dataset versioning.
Now we are going to put this in a width statement, so we have your width one DB in need, and then
we specify the project project we're working on, which is this malaria detection project, the entity
learn and that's it.
So we have this year and then we are going to create our original leader.
So we have original data.
There we go.
We are going to create this artifact.
Actually, we have that original data one TB artifact new this type raw data.
Now, to check out the different arguments we could pass in your get back to documentation, let's check
out your and then you could scroll down.
So yeah, you have this references.
Let's reduce this and have this.
Clearly we have references on the references.
You have this Python library and then you have one TB artifact.
So you could click on this and what do you get?
You have this documentation right here.
So we have the different arguments, name type, description, matter editor incremental and use us
Now note that this for optional.
So that's why in the example we just had a name and a type we'll now add up this here we have name and
then type and we have description would have made a detour.
So let's have this description.
Write your description simply we could say malaria dataset or TensorFlow TensorFlow malaria dataset.
So that's it for the description is actually a string.
And then we have this matter data, which is this dictionary which contains information related.
All decent.
So, yeah, we define this dictionary.
We are going to start, for example, the source.
So we could have your source and then we say TF data set.
Okay, so we have that source.
We could also add other information from this home page.
So let's even copy out this description here and then paste it out here.
So in the place of this description, we have that and there we go.
That's fine.
Now we'll check out on the home page.
We have home page source code.
Let's copy this out and paste out here.
And now we have all this necessary matter did our information.
So here is it.
We've created this artifact.
We then paste out this code, which we had seen previously, and which permits us load the malaria dataset
from TensorFlow dataset, and then we'll save this dataset in the non PI compressed format.
So with our original data, which is this one year, this artifact, that new file, new file, so unlike
here where we add directory here, we add in our we created this new file which now contains this data
set.
So with original data, the new file, we'll call it original original data dot np, z.
So that's our file name.
And then the mode is going to be a write mode.
And we have this as file.
So we have this artifact, we add this new file with a file name, and then we save this file while
putting in the appropriate content.
So right here we have NP save Z, That's our compressed format called Here we have and Z.
And then yeah, we have this file and then what we pass in is our data set, which is this one.
And so at this point we've written our information or this data set in this artifact, and then we're
now ready to log this.
So here we have run log artifacts this basically this year.
So let's, let's do this.
Let's take this off.
We have there we go.
We have run the lock artifact.
And then what we're passing in here is original data.
Okay, so that's it.
So now we've seen how to create this run.
And in this run we have this artifacts and then we put in the information, in the artifacts.
Now, one thing we could do is put all this in a method, so we'll define the method load original data.
So we load the original data and simply that's it.
So let's send this one step and that's fine.
There we go.
So we have that load original data method defined.
Let's add this code so and then we can call it right here.
So we could call load or regional theater and there we go.
So let's get back to this diagram we had previously.
You'll see that in this diagram.
If we take off, if we don't consider this path that is information flows this way and then gets pleats
and so on and so forth.
So here what we have is we've had this load original data method which takes into consideration this
to here we have this run, which is the data loader.
So we have a little run.
And then what it does is it recuperates this original data set from TensorFlow data sets and then produces
an artifact, which is this one now.
So the artifact we have just created here, this artifact original data is this which we we had done
previously here.
We then run the cell and load the original data.
We get this output, and fortunately, this process has failed.
Let's check out the reason why this field cannot convert a tensor of D type variance to a non array.
Now, since this variable dataset we have here is of the type variant, what we're going to do is we're
going to take out each and every element of this dataset.
And save it in a directory.
So from here, we are going to copy out this code.
Let's copy all this code and paste it out here back.
And now what we have in here is we are going to go through this data set.
So we have for the data set of our data for data in data set.
Now we have a list here and then we could pick out the zero elements.
So pick out the zero element.
And then for that we are going to create this folder data set right here.
So let's have this new folder and call it data set.
So we have that new folder data set in that folder data set, we are going to put in this different
tenses which are going to be stored in this non p compressed format.
So that's it.
We have this will modify the name.
So let's say we have malaria, malaria data set and then we give it a number.
So let's have that plus we'll give it a number plus this.
So yeah, now we have this K, we've initialized K and then we're going to continually be increment
in this case.
So here we have now malaria data set, we have k k, and then we have this extension right here.
So that's how we're going to save this file.
And then the data we're going to be saving is going to be this.
So we have data which we're going to be saving in each and every one of this file.
Now, let's test this out and let's put out a break here so we see exactly what we're getting.
We run the cell original data not defined what I need to do open here.
So we're going to open this file and then save that.
Let's run this again.
We check out now this directory.
We have your data set and there we go.
You see, we have this information here saved.
Now let's run this for all the data set.
So let's take out the break and then run this.
Let's print out K and then like after a thousand steps.
So if K modulo 1000, 1000 equals zero, we should print out K.
Okay, so let's run again.
We have now all this data locked.
Let's take off this one now and then get back to documentation and you have my data directory.
So you're passing the directory right there.
So from here we have now instead original data dot add directory, and then we're going to pass in the
directory, which in this case is data set.
So we have that data set past.
We don't really need to have this again, so we could take that off.
Okay, so that's it.
Everything looks fine.
Let's now run this.
All right here we run that and run this.
And this time around, the artifact is loaded successfully.
So now we are going to check out in our dashboard, and here you'll see that we have this raw data,
new data set, and then you have this versions, two different versions of this new data set.
We have version zero and version one.
Now you click on this version one, you should be able to have this.
Let's check out the overview.
You see, you have this data set, malaria data set on all this.
In fact, there's a mirror data we put in already.
So that's it.
We have this API.
We're going to come back to this actually, because to use this artifact, we're making use of this
API.
Then we check out on meta data.
We have this metadata we should be locked in or we should put in in the notebook.
We have the files.
You see those different files here, this directory you see you have the root and then you have basically
all those files.
Now you have this graph view, which for now is very simple.
So here you have a run, the runs actually this run here.
So this run.
And then we have this artifact which has been created.
Now this artifact contains the raw data.
And so we could click on Explode and basically that's it.
Yeah.
You have the name of this run Vivid water name of the run, and then the data is this new data set and
it's version one.
And then the next thing you want to do is to move to the next step.
That is be able to split this data set into this order and to this three different parts.
Now, what we could do other than this is process this before doing the splitting so we don't have to
do this preprocessing tries.
So instead of having this let's take this off here, we will modify this search that here is what we
now obtain.
So coming back to this, we have this run and this artifact, which are all is set is basically what
we have here.
And then the next step will be to reprocess this.
So we'll do the preprocessing on our data set and then produce this pre processed data.
And then from here we'll now do the splitting.
We have this run which was row will split our data into the train validation and testing and then for
the train we'll have another run which role will be to do.
Augmentation on our training data.
So, in fact, in summary, this is what we want to achieve.
Like when we're done with all this, we want to have a graph view which looks like this.
We now go straight forward into the preprocessing.
Is copied.
We get back to the code.
That's fine.
We paste this out here.
We then put this in a statement and then run this code.
And here's where we get.
We've now downloaded this 27,000 files.
We check out your.
And we should have this artifacts, new data set, version one.
And see, we have all those files which we are locked in previously.
So now this means that the next time you want to work on this dataset, you don't really need to come
and run this year.
Like we don't need to run this again so we don't have to do this again.
All we need to do now is just to use this artifact, which Wilson biases starts for us.
And so now that we have this.
We'll get back here and then we do this resize rescale.
Practically our preprocessing here is resizing and rescaling.
So we are going to do a resize and a rescale of all images.
You can copy this and then get back to where we actually have loaded our artifacts.
When we print this out, we have this path to our different files.
So we're going to have your artifacts.
We can check out all those files here.
And then right now we're going to create this other one.
So let's copy this out and then have the space that here.
Now, here we have this pre process, pre process the data.
So this is this method.
We're going to be defining one to be in neat project entity as well.
We've created this new run and then we have now this pre processed data, pre processed data which is
now this new artifact and the name is pre processed, pre processed data set and then the type is pre
processed data we have our pre processed dieter OC description will say oh pre processed version of
the malaria data set of the malaria data set.
Let's take this off And then for the data editor, we could let's take, let's take all this off, let's
have this taken off.
But you could always put information above the middle data.
We are now going to go through each and every file we have in this directory here.
So we have four file in, let's say for F for F in OS does list DA of this directory here.
Let's copy this from your So we're going to list, we're going to create a list from this directory
and we go through this list.
So we go to this list and then for each and every file in this list, what we'll be doing is open up
that file.
So here you have artifact directory, which we've just defined, and then we have this modified directory
plus F to specify the current file, and then we read that file as file.
So from here now, once we've read this as file, we have x, y outputs.
We call this output.
So for each file, since each file we had has an X and Y, we'll take this from here and then we have
this load file.
Now, once we have this, we're going to have X or better still x full.
That's X data.
So let's, let's call this x data set or data set there is set X, so we have our data set, x dot append.
We're going to create this as a list.
So we have your data set, X a list, and then we have data set Y and not a list.
So that's it.
We've created this tool lists and then we want to take each and every element and appended to this dataset
X and dataset Y respectively.
Here we have for X and then we have data set Y that happened y.
Now recall what we have to process as this is our preprocessing method.
So here we have this resize.
Let's make sure we run this resize for skill and it takes in the image, it takes an X basically.
So here we have resize for skill instead of person x will do resize rescale and then we pass an x.
Now we ensure that we define this in size.
Let's have this size defined here in size equal to 24 rounded again, that should be fine.
We need to also specify the.
Baghdad.
We allow people here.
So we have this argument, which is turn to true.
You could always check out in the documentation, write your.
And then from your.
We are not going to take this directly.
Yeah.
What we get is n p z array.
And then to obtain x and Y we have x and Y, which is equal np z array.
We get the number array and we get the values.
So that's what we do to get this.
You could always check our documentation to understand how all this and payload and all of this work.
Now once you have X and Y, this is what we pass in here.
So from here we can run this.
This looks fine.
We have our data set and that's okay.
But before running, we have to ensure that we convert this now into TensorFlow data set.
So let's get this done by having this called data set.
And we have to have theater data set from tensor slices.
And then what are we going to pass in here is dataset X and dataset Y.
Now we're going to be saving this as a file, so we're going to take out this and we have this portion
of this code right here, which is going from documentation.
So here we have with artifact.
The artifact here is pre processed, it's pre processed data.
So with pre processed later, the new file will specify the file name.
Let's call this pre processed data set as file.
We then save this with non pi we have np that save Z.
The compressed format we specify the file and the data to be saved.
In this case our data is this data set right here, which is this TensorFlow data set this little okay
we can now do log artifact pre processed dataset or other pre processed data.
Then before running we are going to take out just a part of all this data set.
So we'll take out only a thousand elements.
And the reason why we're doing this is because we do not have enough memory to store the two 22,000
520,500 different data points as a single variable.
So we'll have that for now.
Let's have this 1000 elements so you could see how this is done.
Now, if you have a real world problem where you have, say, 100,000 different elements or 100,000
different data points, then you could break them up into simpler parts.
So yeah, we have that 1000 we speak, we take out just 1000, and then one thing we'll do is copy out
this part here.
We'll copy out this part and then include it in this run.
So before, just before this.
So we include this.
You're in this run.
And the fact that you include this in this run, we'll link up this new data set with our pre processed
data set.
Now, that said, we could run this and then run the next cell while we get is this error saying cannot
convert a 10th of D type variance to a non piri.
So here we are having this tensor of D type variance and we're trying to store it as a non pi array
and to solve this problem.
Now what we'll do is we'll just ignore this and then save the dataset X and Y as this list we have dataset
X and it is set Y, so let's run this now and pre process our data.
We now have successfully logged this to our artifact.
Let's get to our dashboard one DB dashboard.
Let's refresh this page here.
You'll see your artifacts.
You could click on artifacts, module detection artifacts, and then you could select from this.
So this is what we had previously, this new data set under this raw data.
Now, our pre process data, we have this pre processed data set and this most recent version that we
go, you could check out the files.
You see this file right here, metadata API, which you could use now to do or carry out all operations.
The overview grab view this all grab view right here.
Now let's exploit this graph view.
Okay.
We have exploited this graph view.
Let's zoom.
Okay.
Now you see that?
Let's drag this one here.
And then what you'll notice is we have this first part which has to do with the creation or rather with
the loading of our initial original dataset.
And then once we load this original dataset, the next thing we did was to now pre process this dataset.
Now we carry out all the runs previously.
That's why you have this.
You don't really need to take this into consideration from here.
We just continue from this point here, we copy out this appears.
We have that copy and then we get back to our code.
Where are we going to start?
Now, with the data splitting spaces out here, you see that we're going to have We've already had two
artifacts created.
One was original data, the other the pre process data.
The next will be the train data, validation data and test data.
And so that's why we call this section the data splitting.
We'll again copy out this part from the processing data and paste here.
Now we have this, let's go ahead and take this off from here and then replace this one here.
We have this artifact, okay?
We have this artifact now, which is the ref.
We're using our pre processed data set.
And then from here we have three artifacts which we are going to create.
We have train data.
We can just call this train data set train data set type essays, pre processed data description, training,
data set, and then the artifact directory, we could get it from this year.
So when we create this artifact, we're going to get this artifact directory.
For now, let's just create the other artifacts so we copy this out train data, we have validation
data, and then the test data to obtain the artifact directory.
We are going to run this.
We have this output here.
Click on this artifact and see.
We have this processed data set which has been loaded and we have our prep data set the NP Z file here.
So this year we copy this path and then at the place of this artifact directory, we're going to place
this path.
Let's take this off.
We have this path now which has been placed.
We have now the artifact.
Let's call this artifact file and then let's paste this out here, take this off and there we go.
We've now taken all this off.
Okay, so we have this here, and then we're going to have our artifact file instead.
Feistier artifact file.
So we read that as file, and then we're going to load this file, allow Pico and get the array.
Then at this point, we define the trade and split vowels.
Split and split From here, we're going to paste this out.
We have our train array because we're trying to create this three different arrays.
We have train array, which goes takes values from zero to the train split.
We define a data length, data length, which is the length of our array.
Zero.
Now, recall that the array, the array we have in this array is made of.
This is a list actually made of two parts the X and the Y.
This x has a length of 1000, the Y length of 1000.
That's why we're doing array the zero index of array, and then we're taking this length.
So are we going to get the length to a value of 1000?
And once we have this data length now we're saying that we're picking out those X, we're picking out
this X right here, and then we take in values from 0 to 80% of the total length.
And so we have trains split, which is 0.8 times data.
Len So that's what we have.
And then we repeat the same for this Y, then for the validation array, we're going to start from this
train splits, we're going to start from here times that.
So from the exit index with respect to the total data set, that's where multiplying by the data length,
we're going to go from your right up to the trans plate plus the validation split.
So now we're going to from 0.8 to 0.9 and we have that set.
We'll just repeat this for the validation for the Y right here.
So let's copy this out, We'll copy this out and then paste this year.
That looks fine.
We have that pace that we have it year and this year this looks fine.
Let's repeat the same process for the test array.
Copy this and paste out your test array.
We're going to go from transplant plus vial split right up to the end.
So we're going from 0.8 plus 0.1.
That is 0.9 times total data length from 900th value right to the end.
We just copy this again out here.
Copy that out and then paste this for the y value.
So we have this here, take this off, paste it, and then we go right to the end.
So that looks fine Again.
We have now our twin array validation array and test array.
We're now set to write this information in our artifacts.
So here we have with pre processed data.
This is instead now we trained here color artifact.
We just create a US data train data validator and test data.
So we train data.
The new file we have our we call this train data set p z mode WB save the file and then we are saving
train.
What we're saving is actually just train array.
So we just saving this train array and then we repeat the same process for the validation and the testing.
Okay, we have that validation validation that's test.
And then yeah, we have test validation.
Okay, we have okay, here we have validation and then yeah, two we have test then now we log our different
artifacts.
So we not only log in this one artifact, but the three different artifacts.
So take this back then.
Yeah, we log, train ed, train data, validator and test beta.
Okay, we have that set.
We could run this.
Let's let's change it to split data.
We run this cell here.
Everything looks fine, and then we move on to split our data.
Split data?
We run that cell and wait for the response here, the output we get.
Now, the reason why we're having this is because we must have integers in our floats at this level.
So when we have this indices, we have to convert this all into integers.
So we have this int here.
There we go.
INT And from here we now run this cell again and then split all data.
The data has now been split successfully.
Let's get back to our dashboard right here.
We have all one DB dashboard artifacts.
Let's refresh this page.
As you could see, we have raw data, pre process data, and now we have the test data, validation
data and our training data.
Let's click upon this training data so you could look at a file here matter dealer or API.
We can always make use of this in creating order artifacts.
And let's get to the graph view and this graph view.
Now we'll be able to see the link between this original data set the process data and the train data
set.
Here's what we get.
Let's click on Explode and you could see this clearly now.
So yeah, you see, you have this original data set, this artifact here we have this run which produces
this data preprocessing artifact, and then we have this run which produces this train data set validation
data set and test data set.
So let's draw this take this off.
We have this path here.
You see, we have this here.
We take this, this and this.
Let's copy out this code right here and then start with the data augmentation.
So from here, we now download the train data set.
We could check this out.
Our artifacts, you should have train data set right here.
There we go.
We have our train data set.
We can now copy this part.
Let's copy this path and pace here.
Now we could click on this one to be finished to stop that run, and that should be fine.
Now let's go ahead and copy out this part of the code which was used for preprocessing.
So yeah, we had this year pre process data, we copy this and then there we go, let's paste.
This year we have our AUGMENT data and then project artifacts.
We're going to use this artifacts here.
So let's come right here and then get this path.
Okay?
So we're going to get that.
And then replace this one with this up on part or the pad to the train data set.
The space is out and this should be fine.
There's actually a file.
So we have artifact file.
Now we have this take this off pre processed data instead of pre processed data, we're going to use
augmented arc meant that leader want to be artifact, augmented data set, augmented data set type,
let's say pre processed data and augmented version, augmented version.
Of the malaria train data set.
So everything looks fine for now.
We have that.
And then here, let's get back to this and then copy out this year and then paste this right here.
So we get in this from our training data, obviously.
There we go.
And everything looks fine.
Next thing to do is do the actual documentation and then log the data set to our artifact.
So let's take this back and then we take this off.
Now we have this.
We open up this artifact file, right?
Your artifact file.
We open that up, we obtain our array.
Let's call that array then for for images or for image in the array.
Our array or we've taken out x.
So for image an X before moving on, we're going to create this to list your dataset X.
And then we have that OC, we have dataset X, and then dataset X append our meant of whatever image
we want to pass in.
Let's send this one step and that's fine.
So for all images, we are going to do this augmentation.
Then after this we have dataset Y, which is simply the unchanged labels we've had already.
So from here now we have all that is X and our dataset Y.
Then we can now run the salt and then we run the augment data we have here, augment data and they should
be fine.
We obtain this error pre processed data not defined.
So let's check back here and we see that this should be augmented data instead.
So let's change this.
And we have augmented data.
There we go.
We run this cells and see what we get.
Now, those artifacts have been logged successfully.
We could get to one DB and check this out here we have this and then we click on augmented data set.
So loading the artifacts.
Now that's loaded.
Let's click on this and check out on this graph.
So let's explode.
And then this is what we have now.
So you see again that we have this path, we have our data, we separate this or split this into train
validation and testing you here you have the training and validation and testing.
And then after the training we have this run which converts this training data into an augmented data,
which is this one right here.
And so at this point, we have different versions of our data which we could make use of depending on
our needs.
Thank you for getting right up to this point and see you next time.