Hi there and welcome to this new and exciting session in which we are going to treat quantization our
training with TensorFlow.
Now, in some previous sections we started by explaining what quantization is all about and the advantages
of quantization models.
We also looked at different quantization methods and we looked at the relative advantages and disadvantages
of this different methods that said, we are going to see in this section how to quantized a full model
or just some layers which make up that model.
In TensorFlow, the spatial model which parameters carry out quantization is this tf mod, which stands
for tensor flow model optimization.
And so here we start by installing this TensorFlow model optimization model and then we'll import this
as tf mod.
Then since we want to do quantization, we could get in your class and that's it.
Here we have different methods and classes.
Let's get into this concise model right here.
As you can see, quantized accurate model with a default quantization implementation.
And so here we have to simply pass in this to quantized argument, which is the model to be quantized.
And then we should get a quantization aware model.
So you're for example, you see this model defined the sequential model, and then we also have this
functional model then to quantized this, we just call this method quantized model right here.
And then we pass in our model and we have our quantization, our model.
So that said, let's go ahead and implement this here We have our model in this case.
Let's let's start with our hugging face model.
So we have this our hugging face model, which we declared already.
And now let's say we want to have our quant aware hugging face.
Okay?
So we want this quantization, our hugging face model, and then we want to use TF mode dot quantiles
model.
So basically, let's copy this here and then paste it out in the code.
So we have that picks it out and then here we have our hugging face model.
So from this we're running, this should give us our quantization, our model.
Now we get an error quantization, a Ptfe Keras model inside another Ptfe Keras model is not supported.
So as of now, this isn't supported.
Now let's try out with the efficient net model though this should should be the same error because the
efficient net model let's get up here.
The definition for the efficient and model.
First of all, you can see the hugging face model.
You have this model in this model.
So that's the reason why that doesn't work.
And then we get to the efficient net model trans follow.
Okay, So we have this model right here, and we could see that we have this Keras model, which is
this backbone here in this model.
And so if we are to use this, we have to look for a way to break this backbone up into its different
layers.
But as of now, what we've been doing is just making use of this backbone as here we just have this
backbone.
And that was it.
We didn't actually break this model up into different layers.
Now, that said, let's copy out this efficient net model right here.
We'll have to take all this off.
And then now we are no longer making use of this input right here.
So wouldn't use this.
We use the backbones input directly.
So let's look at that.
We have this X.
So from here we have the backbones output which will get into this global average pooling layer.
So here we have backbone output.
That's it.
We have this output which gets into the global average pooling.
We have this x here, that's it, which now passes to this dense layer and then to the batch nom layer
and then to this dense layer such that we have an output right here.
Okay, So we have that.
Let's pass in this x values.
There we go.
And finally we have this.
Then from year now we create our model.
So it's our Pre-trained or Pre-trained model is Keras model and which takes inputs, the backbone input.
So now our our backbone input is our input and then our output.
It is simply this output right here.
So we have that output.
Okay.
So we have this set now and everything should work fine.
So let's run this here.
And what do you notice?
You notice that the Keras model, what we had previously as our Keras model, has now been broken up.
So you could see I think we should have just said here pre-trained functional model.
Let's call this functional model.
Okay, so let's.
Let's go back and run this order.
Sell your.
Let's get back.
Your Pre-trained model.
Okay, so let's run this again, this operating model and then so we could run this summaries down here
and you could see.
So you see here we have this model is exactly the same model we're dealing with.
So for this exact same model, what we want to do is just to paste this, to have the pre trained.
So this is a pre trained model or let's get a summary pre trained model Somali run that let's reduce
this so we could get into the space we have this and you see we still have this exact same total parameters
is here same number of parameters number of non trainable parameters exactly the same.
So it's basically the same thing.
But the difference here is we do not have this.
Let's open that up again.
We do not have pre trained summary.
We do not have this Keras model here, so we do not have this model right here.
And so because we don't have this now, it will be possible for us to make use of this method and quantiles
are full model.
So that's it.
We have this pre-trained model.
Now let's run this again.
So we have this pre trained, pre trained, functional model.
Let's run that.
Okay, we have our model set and now what we can do is we would run this now.
So let's run this again and see what we get.
So increase the size and there we go, we get another error.
This the same error actually.
Let's get back here.
This should be pre trained, functional.
So let's run this.
Now.
It's taken more time.
Hopefully everything shall work well.
Now, instead, we get this other error where we're told that this is killing is not all this.
This layer here is not supported.
And this is normal.
Since you're in this reskilling layer, we do not have any weights and so we are not going to be carrying
our quantization for such layers.
So what we can do now is instead of quantized, the whole model will select some layers we want to quantized.
So your there we go.
What we'll do is instead select some layers.
So this means that if we had let's define a simple model, so let's let's get back to the top.
And then we define, for example, this little net model without this resize scale.
So that's it.
Quite simple model.
We have that.
Now let's run this, let's run the cell and then oops, we get in an error.
So that's because we we took off the researcher skill.
I would not specify this year exact for an exact input size.
So let's run this again.
That's fine.
Now we have our Loonette model.
Let's do this little net model and run that.
We get another error.
Let's check that out.
This bash Norm is not supported, so you cannot count quantiles this batch nom layer.
So what we'll do is let's let's basically remove the batch nom layers, but later on we'll see how to
have to to, to, to quantiles only some layers.
So for now, let's just remove this batch.
Norm layers, batch num off, drop out.
Let's take the drop out to batch norm of and that's it.
So we have that.
Let's run this again.
Let's see what we get.
Okay, so that's fine.
You see, now we've been able to make this net model quantization aware and we have done this for the
whole or the full model.
Now, in cases like this here, this model, you're this efficient model where we have this backbone,
which is our pre-trained backbone, we cannot start taking off the normalization layer, for example,
here and taking off this risk killing which comes with the backbone and so on and so forth.
So what will instead do is we'll move layer by layer and select the layers which we want to actually
make quantization aware.
So that's basically what we'll do.
And so instead of proceeding as we did here, we just learn that that is quantized in the whole the
full model, we're going to go layer by layer.
So with that we could comment that section there and now take this model of now in order to quantiles,
only some layers of the model will make use of this quantized annotate layer method right here.
So we see again we have quantization gyrus, quantized energy layer, and this takes in the model to
annotate with some quantization configurations.
So you're what they explain is this function does not actually quantized a layer.
It is mainly merely used to specify that the layer should be quantized.
So you see it's there to specify that the layer should be quantized, and so the layer then gets quantized
accordingly when we do a quantized apply.
So this is the quantized apply method here, click open that and that should be it.
So let's get back.
Oh, let's, let's just get let's just look at this example here where you see this layer.
You see we have this model, but in this model want to quantized only this layer.
And so as you could see, we have quantized annotate layer.
And then once this is done, we do a quantized apply to get our quantization, our model, which here
is called quantized model.
So let's go ahead and see how to implement this with our pre-trained efficient model.
We will now define this method, apply quantization to the conv layers, which takes in a layer, and
then if that layer the name or rather if if this conf is in the layer name, we are going to carry out
the quantization.
So we're going to apply the quantization on the conv layers.
So here we have layer OC.
So in the case where we don't have that, we'll just return the layer itself.
So the layer remains unchanged, whereas conv layers will become quantization aware.
So we have this applied matter right here which will run.
There we go.
Now once we we have this method defined will make use of this clone model method right here to create
a new model, but one which takes into consideration a certain clone function.
We get back here and then we paste this out or we wouldn't making use of this in input tensor.
So we just make use of the clone function.
And our clone function here is this apply quantization to the conv layers.
So that's it.
Now you take this year, if you check out the errors, you'll see that wherever we have the conf layers
you see like this one, we have this conf here, we have this comb for the device, convolutions and
so on and so forth.
Now you could also include this for the expand and reduce layers, but let's just work with only this
comes.
So that's it.
We, you, you now understand how to pick out certain layers or how to leave out others from the quantization
awareness process.
So from here we have this apply right here.
And then we'll call this our quant aware efficient net so that we have this quant, our efficient net,
and then we run this, uh, not this model.
This model here has to be our, our pre-trained model.
So it's our pre-trained model.
We run that again, and now this should be fine.
Okay, so we have our quantiles, our model, which is now quantization, where and you notice that
when we do quant our efficient net summary, we should get something slightly different from what we
used to get in.
We haven't this now.
Oh this is, this should be func so we should have func model.
Let's run that again and run now.
Let's run this, let's get back here.
And you see we have this quant aware efficient net and you will not notice that.
Let's get back to the top.
You notice that wherever we have this conv layers, C wherever we have the conf layers, we now instead
have this quantized annotate.
So as you scroll, you wouldn't see a kind of layer, but instead we have the quantiles annotate.
So that's it.
Years, years.
Because this didn't have there's no conf in this name.
So we could, as we said before, we could include this as e expanding as it reduce layers.
So that's it.
We, we now have this quantized annotate layers which wasn't what we had before making this model quantization.
I wear some layers of this model.
Quantization are where.
So with that now we are done with the annotation and we're ready to make this actually quantization
aware.
So we call this quant our model.
This is quant quant aware.
Let's get this exact name right here.
Quant aware, efficient.
Okay.
So we call that quant aware efficient.
There we go.
That's it.
And now we have our quant, our model.
Let's run this again and then see what we get as summary.
And that's it is now quantization aware.
So we know we no longer having the the annotations but now some wrappers.
So your you see the, the layer name, but now we have this quant which is added to those layers.
Let's scroll down, check out on this.
You see we have this other ones here and so on and so forth.
So that's it.
We, we, we now have our quantization aware model and we're now ready to compile this model and train
it.
Like every regular model, we get back to our training right here.
And then at the level of this compile, we have your quant, our model, that's it.
This, this is a similar rate.
So that's it.
Okay, let's run this.
And then here we also have our quant aware model.
There we go.
So let's run this, We get in this resource exhausted error.
So I'm going to restart the session and hopefully everything should work fine.
There we go.
We started the session and now we able to train.
Anyways.
We see how to implement quantization our training with TensorFlow and in the next section we'll dive
into post training quantization.