After preparing the data, the next step will be to dive into modeling.
First things first, we shall start by explaining the formal model which we are going to use, and then
we will dive into model creation with TensorFlow.
The formal model was first developed in this paper entitled Simple and Efficient Design for Semantic
Segmentation with Transformers.
That was in 2021.
And this model, which you could see right here.
Essentially.
It's made of two parts, that is the encoder section and the decoder section.
Now talking about the encoder and decoder sections, we could get back to 2015.
That's six years before the former paper, uh, where the unit paper was published.
The unit architecture itself is made of the encoder and the decoder block.
So here on the left of the U, you have this encoder block.
Let's make that transparent.
So you can see that we have this encoder block.
And then to the right we have the decoder block.
Let's change the color.
There we go okay.
So we have this encoder and then the decoder blocks which make up our U-Net architecture.
Now unlike with image classification where our output will be some vector and in which each position
in that vector will be a specific class which will expect our model to pick from all the remaining classes.
Here we have an output which is similar to the input.
That is, we're going from an input image of shape height by width by three three.
Here represents the number of channels to an output of shape height by width by one.
Or sometimes we could just consider this to be height by width.
So um, the important point to note here is the fact that the kind of output in the case of semantic
segmentation is different from that of image classification.
And hence a U-Net architecture makes a lot more sense here.
It should also be noted that in practice this is generally H divided by four and weight divided by four,
because it's much more computationally expensive to have to generate or reproduce this output with the
exact same dimensions as the input.
So here we have um h and four uh weight divided by four.
So this uh figure will look somewhat like this instead.
So we have this input and then this output.
Getting back to our U-Net architecture in this blocks we have right here, we have the usual ConvNet
layers.
And then we have the downsampling by the maxpool layers we have here Maxpool.
So essentially we are making use of convnets and maxpool to downsample the inputs.
And then in the order turn as this downward, turn this upward turn, we have again the ConvNet layers
and the upsampling layers.
Then in between this downward and upward turns we crop out parts of our features in this blocks.
You could notice that.
Or you could take note of that with the dotted lines.
And then we concatenate this to the blocks in the upward turn.
We suppose that you already have an idea of how the conv layers work, but let's take this simple example
which will show how, um, a simple conv layer works, and also how the upsample layer, which is the
conv 2d transpose layer works, which can be used for the upsampling.
So let's suppose we have this example.
Then in this case, uh, if this is a kernel.
So let's say this is a kernel.
And then this is an input feature.
In this case two we have an input feature.
And then here we have the kernel.
So we will take this input kernel and multiply by this part of our input.
So this is two by two.
Here's two by two.
We take two times zero four times zero all of that zero.
Then one times one plus one six times six.
That's 36 plus one.
So we have 37.
So at this point we have 37.
And then we are going to take this off.
We're going to take this off or we're going to just shift this.
We shift this.
You see, we now multiply four times zero plus six times zero.
That's zero.
Six times one, that's six plus four times six.
That's 24.
So six plus, um, 2430.
We have 30.
We just have your 30.
And then for the next we'll just move this again.
So let's move that, move this and then we multiply again.
We have zero times one plus zero times six.
That's still zero.
And then one times one that's one plus six times six that's 37.
So again here we're going to have 37.
So we see the output we are obtaining.
And then we will shift this again.
And multiply six.
This is six times zero plus four times zero plus six times one plus four times six.
So that's 30.
So again here we have 30.
So that's fine.
So now we have this output.
And there we go.
So that's how the the the conv layer works for the conv transpose layer which could be used for upsampling.
It works um slightly different way.
So again we have this kernel.
Then we have this patch or this part we take five times zero plus five times zero.
We select this patch and multiply by all these different values here.
That's for all the different positions.
And now we have this zero.
Here we have zero.
Five times one is five.
There we go.
Um, zero times six.
Or rather, five times six is 30.
So that's what we obtain.
And then we'll move this patch again.
I will move this position again.
So we go to this.
Here we have two.
There we go.
We have now two times zero.
But note that we are not going to do two times zero.
Um two times zero.
Then put it out this way.
We're not going to have two times zero like this.
Two times one and then two times six.
This way what we're going to have instead is we are going to have this frontier values which are going
to be added to this other frontier values.
So it's going to be a combination of those values.
So let's have this this way.
You see it's going to be something like this.
So this is super post.
Let's just put it out.
Um this way okay.
So that's how that's how it's going to look like.
And then for the for the next, let's shift this position to for this next one we have one times zero.
Let's write that out.
We have let's change the color.
So you could see that clearly the green.
And there we go.
So we have one times zero zero.
Again we have zero zero.
So we'll have um zero zero.
See we added that zero again.
And then one times one is one and then one times six is or rather.
Yeah one times six is six.
So we have six.
And then for this last position we will take this and move to zero.
Well for zero obviously all values will be zero.
So we'll just pick another color.
Let's pick um say black.
Let's pick black.
And then there we go.
We have we would have take that black.
There we go.
We have zero here we have zero here we have zero.
And then here we have zero.
Okay.
So now this is this final result.
Then from here we could let's drag this up so you could see it clearer.
From here we just have this output matrix which is going to be um let's take this pan.
Our output matrix now is going to be zero.
So here we have zero.
We have zero, we have zero.
We have five plus zero.
That's five.
And then we have 30 plus one plus zero plus zero.
That's 31.
Then we have 12 plus zero.
That is 12.
Then we have one, then we have six, then we have um zero.
So that's it.
And so now we see clearly how we are able to leave from an input, which is um, two by two to an output,
which is three by three.
So that's a simple upsampler at this point.
We're still in 2015 with a U-Net model.
Now we move to, um, 2021 with our former model and with the former model, we'll still use the encoder
and decoder as we've seen already.
But now instead of our blocks made of conv layers, we have the transformers.
The former architecture came with five major methods, which made it the state of the art in semantic
segmentation back in 2021.
The very first of these is the hierarchical feature representation.
Now, as we had seen with the U-Net, we already had this hierarchical representation.
But then with the Vit that's the vision transformer.
This is the transformer which precedes other transformers like the former.
We didn't have this hierarchical structure.
And so what the format did was borrowed this hierarchical structure from other models like the unit.
The reason why the authors decided to work with this hierarchical structure is because when we represent
features hierarchically at higher resolutions, we obtain or we are able to extract more coarse features
as compared to the lower resolutions, where we tend to extract more the finer features.
And so with this, we mimic the kind of feature extraction mechanism we had already with the convolutional
neural networks.
Now that said, let's take this off and see how in practice, um, the patch merging is done.
Now, it should be noted that with the vit shift this with the vit.
Um, if you have an input like this one where you have eight pixels by eight, you could count them.
We actually having 12345678 pixels by eight pixels.
Now, um, for the input, let's get to the paper.
Here's a white paper.
As you can see, we have um, in this case it was actually a 16 by 16.
So what we did here was we flattened these patches, uh, and here, um, if we have it, then we would
simply take, um, this one and then this next one lesson said do it this way.
So it's simpler.
So we take like this and we flatten it out this way.
Copy this other one.
Um, let's cut that and paste.
There we go right up to where we have, um, all this flattened out.
And so at the end we are obtaining this output feature which is one by one, because obviously here
you have one by one, but it has a certain number of channels.
In this case, since we have 64 pixels then we will have 64 channels.
But if this input was eight by eight by three, that is if we had three channels, then the output will
be one by one by 64 times three.
In a way, you understand that this is how we were able to flatten out the patches.
Now we're going to do something similar with the SEQ former, where we are going to regroup this different
pixels together.
So let's um, take this off.
Let's just put it up here.
So you see that clearly.
Um, eight by eight by 1 to 1 by one by 64 times three.
Now, as we're saying, one to go from as you see in the paper we had, uh, we have the original height
by width by, let's say three.
And now we want to obtain height divided by four by width divided by four by a channel, let's say C1.
So getting back here, given that we have eight by eight, it means that we are going to obtain two
by two because eight divided by four is two.
Now obviously in in practice it's not actually eight by eight like the inputs will generally be let's
say 512 by 512.
And so it makes sense that, um, you could go up to let's say height by 22 by 32 or width by 32.
So getting back here our output is going to be two by two.
So we'll make sure that this this part let's take off this eight from here.
We'll make sure that this part this this quarter you see let's copy that quarter.
Um we put it out here.
Then we flatten this out.
So again, just like with this we just simply flatten this out.
So this first quarter we flatten that out.
Let's let's break this up with a pen.
So you see that clearly.
So we have this like that.
There we go.
So this first part we take it here and then we flatten it out.
Then once we flatten out like this we move to this next quarter.
We take all this and then flatten out again.
So what we just do is simply just copy this and paste okay.
So let's copy that and paste.
Maybe we should change the color so it looks different.
There we go.
Um let's copy this color code.
So we just um recolor this part here okay.
So paste it out and there we go okay.
So that's fine.
And then this other zone for this other zone we have speak that let's pick this red or let's pick yellow
okay.
So we pick that copy the color code.
And then we're going to do the same with this.
So paste out.
You see this top um left quarter here.
Is it in blue.
Then this um top right quarter here.
Is it in orange.
We have this other quarter yellow and this other quarter green.
Um, if we join this together, this should be below.
So let's put this around about this, um, make it this way.
Okay.
That's fine.
So, you see, if we bring all this together, um, now you have this output, which is two, you see,
because here you have two pixels by two.
So this is it.
We have two by two, and then we have a certain number of channels.
And that's exactly the way the authors did to, um, downsample the inputs or the input features.
The next method, which was implemented in the former paper was the overlapped patch merging.
Now with the overlap patch merging is simply the same as the patch merging, which we've just seen,
but with the difference that this time around is overlapped.
So unlike what we just seen, which is actually, um, non overlapping with the overlapped patch merging,
we'll do something a little bit different.
So before we took this quarter, we took this quarter this next quarter this quarter and this other
quarter.
And then we produce this output.
Now instead what we'll do is we are going to overlap.
That is get into, uh, the other quarters while trying to produce this output.
So this overlap now.
So we will make use of all this part to produce the first, um, this first part of our um, output.
And then for the next we just shift this.
So let's shift this tool.
We'll move this this way.
You see, it takes in part of, um, this quadrant, this quadrant, this quadrant and this quadrant,
and this ensures that we, we have local continuity between the different patches, unlike with the
non overlapping, um, patch merging.
So for the next one we just simply drag we pull this.
There we go.
Let's pull it this way.
So you see for this other part we just do this.
And then for this other part that's the bottom left.
For the bottom left we'll simply do it this way okay.
So that's it.
So that's uh the overlapping patch merging.
And then we'll move to the next where we have the efficient self-attention.
The efficient self-attention, as the name goes, provides a more efficient way of carrying out the
self-attention.
Now, the self-attention, as um, from the original attention is all you need.
Paper.
This is the transformer.
Paper um, can be shown with this formula.
So essentially the computational cost of this operation care by CT is big O of N square.
And the reason why we have big O of n square is simply because if QR that's our query is given to be
um, an n by e matrix, and then we have k which is also an n by E matrix.
Then multiplying this by k would be multiplying this by an e by n matrix.
And so you end up with an n by n matrix.
This means that um, if you have a very large value for n, then you would end up with a very large
matrix, which is also very expensive to compute.
Now that said, in the domain of computer vision, values of n are pretty large.
So um, n in our case will be height by width.
So we'll be um h times w.
So if you have a height of, let's say 512 and a width of 512, then you'll be having 512 times 512,
which is um, pretty large.
And so the the role of this efficient self-attention method is to.
See how to compute this.
Shocked.
Um, much more efficiently.
So let's take this off and show you how, um, this actually is done.
So what we will have here is we're going to take this, um, h times w and divide by uh, a factor r,
so if it's divided by R then let's, let's um just copy it and put, let's take this from here, copy
it.
And put this way we have h times w divided by r.
And then this c will be multiplied by r.
So we can we will be able to reshape it.
Now.
Now note that if you try to reshape if you just divide this by R and maintain C, you cannot reshape.
So that's why you need to divide by r and then multiply by r.
So you will be able to reshape.
Now that said what this means is you're going to have um this this this side reduced because now this
is the h times w divided by r.
Let's divide it here.
We have divided by r.
So this side is going to be reduced.
See it's going to be reduced divided by r.
And then this other side is going to be increased because now you've multiplied by c.
So let's multiply here by c by by r.
Sorry.
So we multiply by r and that's increased.
Now from this point what next we're going to do is we are going to um simply project this to C.
So with a linear layer we can obtain C from um c times r.
So just like in the paper here you have on the paper um there we go.
We have this linear projection which takes C times R and projects it to C.
So just like as in the paper we're going to have, we're going to get back to our original shape, um
in this other direction.
So now we're going to have C instead of C by R.
Let's actually come and measure it here.
So you see it's actually exactly exactly C.
So you see that's C.
But h um times w divided by r still maintains.
So we can just copy this from here and then paste out here we have h times w h times w um divided by
r.
And then we have now c here.
So you see we have left from h times w by c to h times w divided by r by c.
And now um instead of multiplying so this, this um here had um times up.
Let's call this n this was an n by c.
So instead of multiplying n um by c with um n by c transpose, now you're going to multiply n divided
by r by c times n by c transpose.
And so we go from big O of n square in complexity to big O of n square divided by r as we have in the
paper.
So here's in the paper we go from from o n square to o n square divided by r.
And this is very important because um, as we've said already in computer vision, the value of n is
generally very large because we're dealing with high or with high values for the height and the width.
And then uh, the set R to 6464 one from 1 to 4.
So depending on the stage um, let's, let's increase this.
Depending on the stage, you, you will have different values for, for r copy and paste okay.
So depending on the stage you have different values for r.
So uh this makes the self-attention uh much more efficient.
There we go.
So that's it.
So at this point now we move on to the next method which is the mix feed forward network mix FM in the
paper.
The positional information or information about the position is added on to the the linear projection
of the flattened patches.
So here we have patch plus positional embedding.
You can see that uh you see here for example 012 up to nine to give us a position or to tell us or give
us information about the position of each and every patch.
It turns out that this way of getting information about position works fine when we have a fixed, um,
inputs.
But when we are in test mode and that the inputs we are getting are different from other, the shape
of the input is different from the shape of that which we use when we were training.
Then this way of storing or locating each and every patch doesn't work so well.
And so what the authors of the former paper did was they simply, um, encode positional information
using this three by three convolutional layer.
So that said, we've treated all the different blocks.
Which make up our encoder.
We can now look at the decoder path.
Now essentially we have a lightweight all MLP decoder.
And the reason why we need just a lightweight all MLP decoder without any transformers is because of
the hierarchical architecture we have at the level of the encoder, which permits us to have, uh,
an effective receptive field below.
In this figure, you could see the receptive fields of the deeplab version three plus and the farmers
compared.
We could see how even at stage four, the receptive field of this deeplab version three plus is still
pretty small compared to those of the former at the different stages.
And this is simply because of the hierarchical structure or architecture in at the level of the encoder.
And because we have this features rich information passed um, into the decoder.
We do not have, uh, we do not need to have a very complex decoder, um, architecture.
So at the level of the decoder, we just have this lightweight, um, all MLP decoder because the encoder
has already done a lot of work, uh, when it comes to extracting, um, the codes and the fine features
from the inputs.
And that said, you could see how with fewer parameters, the former models outperform the other models.
Um, as of 2021.