Hello, everyone, and welcome to this new and exciting session in which we shall focus on preparing
our Pascal VOC dataset using the TensorFlow dataset pipeline.
So here on Kaggle we have this Pascal Vos.
He did a set which is made available by one hand China, and it's made up of this five different directories
that is annotations, image sets, JPEG images, segmentation class and segmentation object.
Nonetheless, we shall be making use of the JPEG images and annotations for our object detection problem.
And now get into the code.
We are going to start by installing Kaggle.
We are going to copy this Kaggle, the JSON file into this directory which we just created.
Now note that this cargo, the JSON file as we've seen already, is gotten from our Kaggle account.
So you get this from your Google account and you copy out your and then now after this copy, you change
the access mode of the file and then we can start with the data set downloading.
Now to download this or to have that command, you just simply scroll like this, get your copy API
command and you paste it out here.
So what we have here is simply what we've copied.
So that's it.
That's how we download this data set.
So we run that and we download the dataset.
Now, once we download that, we'll go ahead to unzip the content of that dataset into our dataset directory.
Right now we have our data set.
There we go, we have our data set, we pick out this one here and you see we have the annotations,
image sets, JPEG, segmentation, class and segmentation object.
You open up this and this.
Okay, so that's it.
Now what we'll do is we are going to define some variables here.
We have our trained images, which is simply this path to this JPEG images.
We have our trained maps, which is simply the path to the annotations.
So that's it.
And then we have our classes.
So Pascal, you see data set takes in 20 classes from airplane right up to TV monitor.
Then we have be set to two.
Now, to understand the significance of this, be remember from the paper that we had seen that the
image is divided into an SSE by SW grid and for each grid cell and each grid cell predicts B bounding
boxes Now yeah the defined as 2b7 and then B to be two.
So that's this number of bounding boxes which we are considering to be B and that's exactly what we
have here.
So that said, we have a number of classes which is simply from year 20.
We have the image height and width which is considered to be 224.
We have the split size SSE, which is 224 divided by 22.
That is equal seven.
We've just seen that this is actually SSE in the paper.
We have a number of epochs or 100 learning rate defined, although we have some sort of learning rescheduling.
So we could take this off and here we could say 135, then we have a batch size of 32.
So that's it.
We define all this and then we move to the processing of our annotations.
Now, given that this annotations are essentially this XML files we have here, what we are going to
be using is this element tree from this XML package, which we'll use to parse this XML data right here.
So diving into the code, we could see here we have the file name which is passed into this pass method
from which we obtain a tree.
Then from the tree we could get its root.
Once we obtain the root, we can now get the tree size.
See, we have root dot fine and we specify size.
And then once we have this tree size, we could get the height of a specific image and its width.
So here we have width, width and height.
What is how we obtain this tool from the size stack So we could also obtain the depth from Europe.
Let's copy this and paste out here.
And then let's say we want to get the depth here.
We specify depth and then you see we obtained the text.
So let's run that and then get the depth.
We run that and then let's have pre process XML and then the file name, it's train maps.
Actually it's actually the file path train maps.
Plus we have this file.
Or
7020700033
XML.
Okay, so it's actually this exact file here.
See this?
If you look up here or if you look at this here file name is actually in file name.
So let's run this and then see what we get.
There we go.
We should have.
Okay, so we have 366 for the height.
So here we have 500 for the width.
And then we have three for the depth.
We've converted all this into floats.
Okay, so now we done with obtaining the images width and height, which are all in this size tag.
Let's now move on to obtaining the different bounding boxes of the different objects.
So you see here we had route dot find size.
That's because we have a single size for that image.
Now here you see we have route dot, find all objects and that's because we will have many or we could
have many objects for a single image.
You're again single size, but you're getting all a possibility of having more than one object.
So that's why we specify find all.
So we are going to find all objects, meaning that we are going to get into each and every object tag
we have here.
You see, this is an object.
This is another object.
If we scroll down, we'll see we have another object.
So essentially in this image, we have one, two, three objects.
Okay, So we have this three different objects.
And now for each and every bounding box in this object, like this is our object here.
If we pick out a specific object.
So here we pick out this object.
For example, we have we go through each and every bounding box in this object and we take its mean.
Does X mean y, mean x, max and Y and max?
So that's exactly what we do here.
You see we have bounding box dot find now X mean and then we convert that to text.
We have y mean text x max and y max.
And at the end of this we now convert this into a float.
So that's how we obtain x mean y, mean x max and y max.
Now you'll notice that we have a break here, and the reason why we want to have this is because for
a particular object we just need a single band and box.
So if we have other bounding boxes.
We are not going to take those into consideration.
Okay.
So that said, now what we'll do is we're going to print out X mean y, mean x, max, y max and yeah,
x, max, max.
Okay, so let's print this out for each and every object.
Now let's run this and see what we get as we expect.
You see, we have nine 107 for 99 to 63.
And that's exactly what we have here.
We have for 21, 200 for 82 to 26.
And then finally we have this.
So here are our three different objects.
Now, what if we try out another different image?
So let's change this to 32.
We run that and see what we get.
Yes, it's on XML files.
Here's 32 instead of 33.
You see, now we have actually four different objects.
And so we have this four different bounding boxes, which you could see here.
We have object, object, object and object.
Now, if you consider this image here, you see we put police course on this point.
We have you could read from here we have 24 188.
So it matches up with this here, see this 26 189 And then here we have 46 to 40 matches are with 44
to 38.
Now, in order for us to make it easier when working with the yellow encoders, what we are going to
do is we are going to get the center of this band and box.
So the center should be around this point here.
Now that center is about 35 215.
So here we have 30.
Oops, let's get a pan.
So here is about 35 to 115.
And then now we've got in the center, we could also get the width.
The width is about 18 and then the height, the height is about 238 -189.
That's 49.
So we now have the center, which is this, we have the width and then we have the height.
And then what we'll do is we'll divide all this by the total width and total height of the image.
So for 35 we take 35 divided by the total width.
The total width of this is 500.
So we have 35 divided by 500 and then we'll have 215 divided by four.
Well, this 215 is divided by the height that is 281 because this, this year is our x coordinate and
then this is our Y coordinate.
So this is respect to the width and then this respect to the height.
So this is divided by 230 281.
Okay.
So we take this divided by 500 and then this divided by 281, then we have the width, which is 18 divided
by 500, and then we have the height 49 divided by to 81.
And so instead of having X mean y min, x max, y max, we have the center which is divided by oh,
which is normalized.
And then we have the width and the height, which are also normalized.
Now, putting this in the form of code, once we're done with getting a specific bond and box, we could
go ahead and obtain the class name.
So you see we have, as usual or make use of our object tree and then we find the name here.
Here we have name and here airplane.
Here's our plane, yours person, yours person.
We're not going to be interested in the polls or what is truncated or not or what is difficult or not.
We're just interested in the name and this bound and box, just as we had seen already.
That said, we have our class name.
From this class name, we could create this class dictionary, which we'll use to convert the different
class names into a specific integer.
So what the simply means is we're going to convert airplane to zero, bicycle to one bird to two, boat
to three, and so on and so forth.
So this is zero one, two, three, four, five, six, seven, eight, nine, ten.
Should be ten.
Yeah, 11, 12, 13, 14.
Okay.
So we have airplane which is zero and person which is 14.
Okay, let's take note of that.
Now getting back here, you see we're going to make use of this dictionary where we simply have a class
and that's converted into an integer.
So that's quite straightforward.
And then now we have our bond and box, which is essentially X mean plus x max divided by two.
That is, we get the center, this is the center, and then we divide by the width.
So that's it.
We have x min plus x max divided by two times width is essentially the center divided by the width.
And then we have the Y center that's women plus Y and max divided by two.
That's the center and then divided by the height.
And then for the width we want x max minus x mean because to obtain the width, to obtain this width
you simply take this, minus this.
To obtain the height, we take this, minus this, and that's it.
So that's how we obtain the width and the height.
So here we have x max minus x mean then divided by the width, and then we have x, y, max minus y,
mean divided by the height.
And then we have our class, which is going to be an integer instead of, say, person or airplane.
Then once we are done with this bonding box, we now store this in the bond boxes list.
So let's create your bond and bond and boxes list.
There we go.
We have that bond and boxes list.
And then now we will return.
Oops, We'll return bond in boxes.
So that's it.
Let's return that.
And then there we go.
So let's run this and then see what we get.
Now, first thing you can notice is that we have our four bounding boxes.
Now, take note of the fact that we have the classes zero zero and 14 million that we have airplane
and person, which matches exactly what we expect.
And then when we get back to this image, you see 35 divide by 502, 50 divided by 281 we should have
35 divided by 500 and then 215 divided by 281.
So you see we have 0.07 and 0.76.
Okay.
So does it make sense?
And then for the width.
For the width, we had 18 divided by 549 divided by 281.
So yeah, we have 18 divided by 500.
And then 200 divided by 281 0.04 or 0.036 and 0.170.71.
This should be seven one.
Oh, let's get back here.
It's actually 49 divided by 281 and not 200.
So this would be 49 because the height is 49.
So we divide that and then you see we should have .17.
Okay, So that makes sense.
So that is it.
We have our encoded bounding boxes and now we're ready to produce our outputs based on what was described
in the paper.
So in the paper we saw that our output will be this seven by seven by 30 tensor, where each and every
cell we have here, each and every one of those 49 different cells, because we have seven times seven,
that 49 will take values depending on whether they have an object or not.
Now, for a show like this one that's actually matching up with this one where there is no object,
we will take values like, oh, would have a value of zero for the object ness, meaning that there
is no object.
And then for the positioning, we have this here that's four zeros.
And then for the class, because there is no object, we will have all zeros.
Now we have 20 classes, so we'll go from zero.
I will have 20 of this zeros.
And there we go.
So you see, we have 20 of this zeros.
Now we're going to have the same for each and every grid cell where we do not have an object.
Now, it should be noted that a grid cell like this one, for example, this one here, let's change
the color, this grid cellular.
Let's take this well, let's get back.
This grid cell here has actually no object because we consider a grid cell to have an object.
If the center of that object is in the grid cell.
Now, although we have the wing of the plane in this cell, given that the center of this plane isn't
in this cell, we do not consider that this cell has an object.
So here would have exact same values we have here.
So there is no object like this is zero one, two, three.
This is zero one.
So we go 012, 301.
So here would have this exact same values, all zeros.
And with this, with this, with this, this and all this other cells.
Now, what we left with will be this cell which contains the center of this object, this cell which
contains the center of this object, this cell which contains the center and this cell.
So we have four cells which contain the center or the centers of this four different objects, while
the other cells contain no object.
Now, we shall focus only on this one here so you understand how this outputs are generated based on
the band and boxes.
And this length.
This length here is essentially the number of bounding boxes.
So it's got it could be gotten from this bounding boxes.
So here we have bounding boxes, which is in this case here for this object is actually what we have
here that is with normalized value such that we have the Center X Center normalized, Y center normalized,
we have the width normalized, the height normalized.
And we have this class now, as we saw in the paper, this isn't exactly what we want.
So what we want is a value which tells us the position of the object with respect to that specific grid
cell.
So if we take off this year, let's take this off and then we specify the center.
The center here is around this position.
Your your source center will find that base.
Now, on the paper, this position here has to be encoded so that we have this value based of this origin.
So it's based off this origin because we have a grid cellular, which is this one.
So we have this grid cell.
It's actually the same grid cell we have here.
So it's base of this and not base of this origin of the whole image.
Remember, the image has this origin and this grid cell has its own origin.
Now, let's shift.
This isn't very clear.
Let's shift this so that we have this full.
Okay.
So you see clearly now the origin of the image, which is this and the origin of this actually this
point here and the origin of the grid cell, which is this point.
And then we have the center of the image, which is around this year, so centers around this.
So the idea now is to obtain this distance from.
This year from well, from this origin to this point, and that's it.
So let's get this distance and this distance.
So that's it.
We need to get the distance from year to year and the distance from year to year.
As of now, we have our center normalized with respect to the whole image.
And if we want to normalize this now with respect to a specific grid cell, we need to take this value
and multiply it by the number of grid cells we have.
So given that we have we have 0.071 will take that and multiply by seven.
So we have 0.07 times seven, which will give us 0.49 and then we'll take 0.750.75 times seven, which
will give us about 5.25.
Now what this means is that the distance from here to this that's in this horizontal direction is 0.49.
That's approximately 0.5.
And this makes sense because the distance from here does this origin to where we have the center here
of this object is approximately half of the distance from here to the full cell and then the distance
from here, this origin going in the horizontal and the vertical direction to this center is approximately
0.25.
So we go about 0.5 and then here about 0.25.
Nonetheless, after multiplying year that 0.07 times seven, we have 0.49 that 0.5.
That makes sense.
We have we also multiply 0.75 times seven.
That gives us 5.25.
But this distance is actually only 0.25.
So what we'll do is we'll take 5.25 modulo
one and will obtain 0.25 because the distance from this origin to this center, the center, this is
the center of the object is 0.5 in this direction, horizontal and 0.25 in the vertical direction.
Now, let's make this bigger so you could see that even clearer.
What we're saying is we have a center which is around this, and then we have this origin here.
The distance from year to year is about 0.5 of our cell.
And then the distance from here downward up to the center.
This distance is about 0.25 of our cell.
And we've seen that to compute this automatically, what we need to do is get this already normalized
values from here.
We already have this normalized values.
We multiply them by seven and then we compute the modulo of those where we find the the output from
your model of one to obtain this distances.
So let's do 0.49 modulo one, you should give you 0.49 and then 5.25 modulo one.
It gives you 0.25.
So you see that now we have this center with respect to this grid cells origin, and that's exactly
what we had in the paper.
Now, diving into the code, you'll see that we are going to create this output level, which is essentially
going to be a seven by seven by number of classes plus five.
That's 25.
The number of classes is 20 by 25 tensor.
And then we are going to go through each and every bounding box we have here and then put in the values
corresponding to the specific cells.
So again, we are seeing that for all these different cells here we have all zeros.
But for this cell, this cell, the cell and this cell, we have non zero values or not all values are
zeros.
So let's concentrate on this one as we said already.
So we have here for being range length.
That's for being the range of the number of bounding boxes.
We say that length is the number of bounding boxes.
We have the bound and box.
That specific bounding box this zero year is simply this.
So this is X center, the center we multiply by the split size, multiply by seven.
So just like taking 0.07 times seven, that will give you 0.49.
So this is actually 0.49.
If we are dealing with this bounding box here.
And then for this next one, it's essentially 5.25.
So this is this times seven will give you 5.25.
Now, one other thing we need to do is we need to pick that specific or the specific grid cell to pick
the specific grid cell out of all the 49 grid cells, because we have seven by seven here, this one,
two, three, four, five, six, seven.
One, two, three, four, five, six, seven.
So we have 49 options.
We need to pick only one option, and that's what we are doing right here.
So to do that, what we need to do here is we take this 0.49.
And then we convert it into an integer.
Or we simply round this down.
So around in 0.49 down we give you zero.
And then round in 5.25 down we give you five.
So it simply means that we are going in the x direction or the direction we add the zero position,
but in the Y direction we go to the field position.
So this is zero one, two, three, four, five.
And then the acceleration, we still add zero.
So that's how we we get this here.
And then so we have we said zero five and then the output level zero five.
Just like remember we had this output which is seven by seven by 25, this is one, two, three, four,
five, six and this other one seven.
And then here we have seven.
So we have seven by seven by 25.
Let's add this year we have this, this, this one, two, three, four, five, six.
And then we have seven.
So essentially what we're saying here is zero five, that's zero.
And then 1012, three, four, five at this position right here.
For the first five values.
For its first five values we would have one once signifies that we have an object.
Then for the positions you see, we have 0.5449 modulo one, which is going to give us 0.49.
Then we will have 5.25 modulo one, which will give us 0.25.
So now we will find its position with respect to this cell.
And then we have the width, we have the width here, which is this bounding box too, because this
is 012.
So this is the width.
But remember from the paper, it's of what we need to have here was a width of this bounding box with
respect to the whole image.
So what we have as value here will stay unmodified.
So that's it.
0.0360.036.
And then for the next value is 0.170.17.
So that's how we obtain the first five values.
You see, we assign this values we have right here now for the classes we have from five right up to
25.
So we have from those five plus this.
We assigned a value of one.
Now to understand why we what we are doing here, remember that this bounding box is or this bounding
box of value or index value for ticks in this value.
14.
So what we are saying is add the position five plus 14 we want to assign a value one.
Now remember that we have this first five values which tells us the object ness or which gives us the
object score, which gives us a position, and then the remaining 20 values would tell us the class
of the object.
So after listing this, we also have the class, but we have 20 zeros, see?
So again, we have this object ness right here.
We have object ness, we have the bounding boxes, and then we have the 20 zeros.
Now, when we say five plus 14, it means that we'll get to the 19th position.
This is five +01, two, three, four, five, six, seven, eight, nine, ten, 11, 12, 13, 14.
So at this position here, we are going to take this off and then replace that with a one.
So here we have a one meaning that this year has an object.
This is bound in box and that object is of class person.
So you see here we assign this value one and that's it.
We output or we return the output level.
Recall that we've just done this for this object, but we have four different objects.
That's four, in fact, four different bounding boxes.
So we'll do for this, we'll do for this, we do for this and we do for this.
That said, at this point, we could take all this off and then carry out some testing.
So we have this pre process XML which outputs this here.
So we could do generate generates output which takes in pre well as copy this here, just copy this
paste our year, the output of this is this bounding boxes and then scroll so we could see that clearly
and then we have the length of that output.
Okay, so that's it.
We have this output and we have the length of the output.
Let's run this and see what we get.
This isn't defined.
Let's run this.
And there we go.
So we should have something reasonable from this.
You see, we have this output.
And if you check this out, you see seven by seven by 25.
And then now what we could do is we could say, okay, one two, get
000.
So zero zero, let's run that, see what we get.
Zero zero.
Does this sell right here?
You see, all is values, all the values of this.
So let's take all this off.
All the values of this seller are zeros, which makes sense.
Now let's do 0505.
We run that.
You see, we have exactly what we expect here.
We have 0.49.
We have 0.31, meaning that the distance from year to the center is about 0.31 year.
We have 0.036, we have 0.174 and then we have a one at this position.
And that as a person.
Now if you go 012, well, this is 012 understand the direction and 012 in the vertical direction.
So this is two two, let's do two two and see what we have.
We have two.
Let's get back to two.
We should have an object there.
See, we have an object.
So we have we have one we have is bounding box.
And then notice how this is this class.
Remember the aeroplane.
The aeroplane, as we had here, was the very first class.
So it makes sense that we have a one at this very first position.
So that makes sense.
So that's it.
We see how we could generate our output from this data set.
We've been given all this XML files we have for each and every image.
Before we move on, we are going to do some slight modifications on the code.
So here we no longer need this length.
We have the bound and boxes.
Then we are going to create this non pi array instead of the TensorFlow variable we had before.
So exact same shape as before.
Still our output label.
And then we'll get the length directly from this bound and boxes.
So we have for being range length of the bounding boxes then to account for the fact that we are going
to have this computations bashed.
We are going to add this three dots right here.
So we have now our grid X computed as similar to what we've just seen, and then we have our grid y
computed still our I J garden.
And then here again we add the three dots to account for the batch computations.
And so that's it, this exact same code.
And then now we're going to convert this non pi array into a tensor.
So we use the convert to tensor method in TensorFlow.
So that's it.
We run this and we still have our same output.
Now to ensure that our validation set doesn't get mixed up with the training, we are going to define
this set of 64 images, which will be our validation set, which will make up our validation set.
There we go.
We have this valid list and then the next thing we'll do is we are going to copy, as you see here,
we're going to copy or rather we are going to move this files into this two directories.
So you're if you open this up now, you'll see we have we've created this directory valid JPEG images,
which is this, and we've created a directory of our annotations, which is this other one which contain
the validation set images and annotations.
And so that's it.
So now we're going to create this different lists.
Here we have the image patch and the XML pads.
We have the validation image pads and the validation XML pads.
Let's run this.
You see, we have 17,061 files for the training and then 64 for the validation, and that's it.
Again, you should know that this training images has already been defined here.
We've defined training images right here.
We've also defined Val images and Val maps.
So that's where we get all this from and that's how we obtain all these different paths.
Now from here, we're going to create the TensorFlow data sets, that is the trained data set and the
validation data set, which is essentially going to be made of these different paths which we've just
created.
So we make use of this from tensor slides, this method, and we put the image pads and the XML pads
and the validation pads and.
Validation XML patch.
So let's run this and then we could visualize our validation data.
So here you see we have this path here that is our image path.
And then you see we have this other path, which is our XML file.
So we have the image and its corresponding XML file.
Then now that we have our image pads and our XML pads already making up our data set from this tool,
we could obtain the image and the bounding boxes for the image.
All we need to do is pass in the image path.
In this read file method, then decode that red file and then we go ahead and resize and cast the image.
So let's take this off.
We actually do the casting here, so no need doing that before resizing.
So as we're saying, we resize and then we carry out a casting and we obtain our image from this image
path.
Now for the bottom boxes, remember we looked at this pre processed XML method which was already explained
here.
We looked at this method which takes in our XML path or our file or file name and then outputs the bounding
boxes.
So that's exactly what we're doing here.
And so that's it.
Now, given that this method does a pre process, XML method isn't made of only TensorFlow operations
will need to make use of this non PI function method where we are going to pass in our function here.
We specify the input, which is the path that's our XML path.
And then we also specify the data type of our output tensor, which in this case is flow 32.
So this lets let's say this is XML path and here we have XML path.
Okay, so now we have the path and we have the method.
We can now obtain the boxes.
And then now we have our trained data set which is going to be redesigned such that we have this year
we have this path's, the image path and the XML path which gets in here and then outputs the images
on the boxes.
So you see train data set, we map and then we specify the method, which is a get image and bounding
boxes.
Thus we get the images and the bounding boxes from the image path and the XML path.
So now our training data set is no longer going to give us the the image path and the XML path, but
it's going to give us the image itself and the bound and boxes.
So let's run this and then see what we get.
There we go.
You can see we have this image and then we have this corresponding amount and box.
In this case, we have just a single bounding box.
Let's go ahead and write this.
Then we check this out here.
There we go.
We have this output you could see.
So you see we have this output here and it's showing that we have an error plane.
So if you check this out, you see we have the bounding box here and then we have the class.
The class is zero.
So if we scroll to the top, you find that the class error plane was here.
So this is the zero class, which makes sense.
Now, let's try out with some others.
Well, most of those ones here have only one object, but there is this one at this eighteenths position.
Let's keep, let's keep that and then a break.
Then we run this again and then see what we get.
So this, this particular image has several objects so we could check that out, run this.
Okay, so you see here we have now many more objects.
Here.
We have many, many people.
Actually you have 14 and this happens to be person.
So if you if you if you have here, here we have this.
Well, okay.
So if you do classes.
Well, the the list is classes.
So let's just do classes 14 and we get the exact class, we have the classes 14 And you should see you
should have person.
Okay, so you see we have one, two, three, four, five, six, seven, eight persons.
And if you get here, you see, we should have one, two, three, four, five, six, seven, eight.
Exactly what we expect.
And then we have ten, eight, eight.
Well, this eight, this eight here and then after we check out ten, eight is cheer.
So maybe it's this cheer.
Okay, it is twice.
So we have a cheer here.
And we also have another cheer here and then ten is let's check that out.
Should be dining table.
Okay.
See, we have this dining table.
We have dining table, we have cheer, cheer and then eight persons.
So now we have the image and it's bound in boxes.
The next thing we want to do is have the image and it's output levels.
Remember, with our generate output method, which we had seen already here, it takes in the output
boxes and outputs the labels.
So let's get right here and then you see we have this pre processed method which takes in again this.
MH And it's bound in boxes.
And then right here we are going to output the image.
Here, we just simply output an image.
But for the bottom boxes, we need to convert this into output labels.
So we make use of generate output method, which itself takes in the bottom boxes and then we specify
the data type of the output tensor.
Again, we're using this non pi function method right here because this generate output isn't made of
only TensorFlow operations.
So that's it.
We'll run this.
And then there we go.
Our final steps now will be to batch our data set and then implement pre fetching.
So let's run that and then we have our training and validation data sets which have been prepared.