We now move on to fine tuning that we've looked at transfer learning and now we get into fine tuning.
But before getting to fine tuning, what we would first convert this code, which was built with a sequential
API now into the functional API.
So yours are converted model is basically the same thing.
We have the input, we have the backbone which takes the input because this output global average pulling
dense layer by storm dense layer and this dense layer right here.
Then we have our fine tuned model right here.
Let's run this code cell and then we could view a summary which is meant to be identical to what we
had already with the Pre-trained model.
So here we have this 17.673 thousand.
See, this is 675.
It doesn't match with what we expect.
So let's get back here and we notice that we did not put this here.
So let's X we did not include a batch nom layer.
So let's run this again.
And then we now have this summary right here, which is exactly the same as that of our previously built
model with a sequential API.
And then now we want to fine tune our model.
That is all this layers which we have frozen that is not trained.
We now want to make them trainable.
So right here we'll get back and then we simply have backbone dot trainable and we set that to true.
Then here we are going to set this trainable to false.
Now, recall when we were building this Resnick 34 model right here, we had this trainable parameter
which we made use of.
You remember here we had training.
Sorry is not trainable.
Let's get back here.
That was training.
So we should have training.
You're not trainable is different from this one.
So take note of that.
This is trainable and this is training.
So when we set this training to false and we'll get back here, you will find that this batch Norm took
in this training parameter.
And the reason why we need this specially for the batch norm is simply because the batch norm works
differently during training and inference, and then during training the batch nom layer normalizes
its output using the mean and the standard deviation of the current batch of inputs.
Whereas doing in France, the batch norm layer normalizes its outputs using a moving average of the
mean and the standard deviation of the batches it has seen during the training.
So since at inference or when we fine tuning, we do not want the batch norm to take the current mean
or rather the mean and the standard deviation from the current batch of inputs.
We are instead going to compute this from what it saw during training.
And so that's why this training parameter is very important.
We have this training right here.
You see, it takes the input and the training where we could set training to be false for training mode
and training to be rather training to be true for training mode and training to be false for inference
or let's say, fine tuning.
Now, before we move on, you should know that this layer, the trainable set to false, is different
from set in training to false.
When you set a layer's trainable parameter to false, it simply means we do not want to update the weights
when training, But when we set training to false, it means we're working in inference mode.
In the case of the batch norm, this gamma and beta are trainable parameters.
And so when we say later trainable equals false, it means that they're not going to be updated during
training.
But on the other hand, this mean and variance aren't trainable parameters.
Instead, they are parameters which adapt to the training data.
And that is why when we are in France mode, that's when we set training to false.
We do not want to disrupt.
The mean and variance values garden during the training are based on the training inputs.
And so as we saw already, this mean and variance at inference mode will be simply the moving average
of the mean and standard deviation of the batches it has seen during training.
And so clearly these two concepts, though, that is setting the weights to parameters, not to be trainable.
And so clearly, the concept of setting these weights not to be trainable is different from that of
working in inference mode.
Nonetheless, it should be noted that in the case of the batch norm set and trainable to false on the
layer means that the layer will be subsequently run in inference mode.
Now, although we've seen that there's two errors that don't mean actually the same thing.
Now, also note here that setting trainable on a model containing all the layers will recursively set
the trainable value of all inner layers.
And if the value of the trainable attribute is changed after call and compile method on a model, the
new value doesn't take effect.
For this model onto compile is called again.
Okay, so that's it.
You could check out all the drop out here.
We have this drop out layer which doesn't have any trainable parameters, but remember that the way
the drop out works is that you have, let's say, some inputs.
Let's take this off.
You have some inputs, and then if you do pass to the drop out layer at the end, you have some of these
inputs which will be not considered.
So you have maybe this will proceed, but this not take into consideration.
Maybe this moves and then this not take into consideration.
So the drop out of 0.5 simply means that half of our inputs will be will move forward and the other
half will not be taken into consideration.
And so you see that at inference.
That is when we are actually trying to test our model.
We do not need to drop out some of this neurons right here.
And so generally the drop out also takes in this training parameter here where when we set training
to true, it means that we are in training mode and so we could actually drop out some of this values,
whereas when we set the training to false, then we are in inference mode and so we do nothing.
So we just allow the inputs to pass without any modifications.
And you could also see clearly from here that the layer that trainable doesn't really apply because
drop out doesn't have trainable parameters, whereas will, with this training, we could decide whether
it's true or false.
That is what training mode or inference mode.
So in fact, what we're saying is we have this model here, we have the backbone and we have the head
for classification.
We apply transfer learning by freezing all this, we freeze all our backbone.
So no, no parameter here is updated during training.
And then now we move on to fine tuning where in fine tuning, we want to update these parameters with
a very small learning rate.
And then we also want to avoid a situation where those mean invariant statistics which we have gotten
during the training process will be upset at during this fine tuning process.
And so the batch norm is kind of like a special layer where even during the fine tuning where we want
or where we have set the trainable to, to true, that is we want to update these weights during the
training.
We do not want to modify or upset the batch norms mean and variance.
And so we are going to set this training year to false.
So it still behaves as if it were in inference mode.
So getting back here, we have our training which has been set to false and then we could start training
our model again.
But one point to note here is we do the fine tuning on a pre trained model which has already been trained.
So what we do is we are now going to start training this model this way will start by training the pre
trained model.
So we start by having this backbone to be set to false and this training set to false.
So we're going to repeat the transfer learning process again before then applying fine tuning.
So each time you want to apply a fine tuning, make sure you have this set of false training set to
false and then you go ahead.
So yeah, let's run this again.
Here's our fine tuned model which achieves a best validation accuracy of about 70%.
Okay, now we're done with transfer learning.
We're now going to apply fine tuning to do this.
We're going to set this to true.
So all we need to do here is set this to true.
We're not going to run this again.
We use the same model.
And that's even the idea.
The idea is for us to start with the backbone, which is not trainable and then later on make it trainable,
or while now just simply recompile the model.
So don't forget to recompile this model to take into consideration the fact that some parts of the model
are now trainable.
So let's get back here, recompile the model and see what this gives us.
Now, as we start training, we notice that this validation accuracy here isn't looking like what we
expect, because before getting to the fine tuning, we already had a model with a validation accuracy
of about 70%.
But now we get in this 33%.
And the simple reason why this is so is because our learning rate here, we still maintain the same
learning rate instead of reducing it before the fine tuning.
So we'll have to stop this here.
Then we'll get back to this year, set this to to false.
So we'd have to start back the whole process set this to false.
And then we retrain the model.
We get this accuracy of about 70%.
Now we get back here and we set this to true.
So we set this to true.
So that's fine.
We run the cell back, but now trainable.
And then we are going to make sure this learning rate here, we divide it by 100.
So we're going to make use of very small learning rate.
Now, once we have this, we're going to run this again.
So we're going to compile our model and restart the training process train are completed.
The other results we get, you could see that the validation accuracy increases up to 72.2%.
So we make an extra gain of 2% for the validation accuracy after fine tuning our model.
And this makes sense since fine tuning permits us to squeeze out some extra juice from this backbone.
Since this time around, it's actually trainable.
And that said, we've just completed a section on trans.
Learning.
Thanks for getting round up to this point and see you on the next section.