Up to this point, we've been used to train our models with one fixed learning rate throughout the process
so we could fix our learning rate as we did to say 0.01.
And we use this same learning rate throughout our whole training process.
But it happens that if this learning rate is too large, then we risks diverging.
And if the learning rate is too small, it would take too long for our model to converge.
So let's consider this plot right here is actually a very simplified plot as actually what really happens
is way more complex than this.
So let's consider this plot and we have this.
And yeah we have the loss record in and the weights.
So yeah we have loss and then weights or parameters.
And our aim is actually to modify these weights such that the loss is minimized.
So our aim is to get all to get to this position right here.
Now we start with the case where we have a high learning rate.
So if you are dealing with a high learning rate, then it will be easier for our model to find its way
to this minimum position right here or to some position close to this minimum position, let's say at
this point here.
But the problem here is, oh, once it gets to this position, it could also very easily diverge from
it.
So you could very easily get back to another points around here and then repeat this kind of process
again where the model just kind of diverges.
So you could have something like this.
It could come down here and then get back to some point around this and so on and so forth.
So we may have this case where the model doesn't really converge because the learning rate is too high
and then for the small learning rates.
We may start training.
And then if, say, the model or if we find ourselves at this point here, that is with modify the weights
so that the loss value happens to be at at this point here, it becomes difficult for us to get to this
ultimate.
Are global minima Since the learning rate is too small and it changes, we are making very small changes,
so we may find ourselves just staying around this low call minima instead of going towards this global
minima, which is this.
And so to bring in a balance, what we could do is when we start the training process, we could use
a relatively high learning rate so that the model kind of approaches this global minima faster.
And then after a certain number of epochs, we start or we modify the learning rate so that it becomes
or it takes in very small values.
And since it now takes in small values, we now start taking up very small changes so that we can get
towards this global minima now without risking divergence.
Now, one way of doing this is by say you could fix let's say you can suppose that for the first ten
epochs you train your model are learning rate of C 0.1 and then from 10 to 20 you're going to train
a model, a learner of say, 0.01.
Let's say we're divide it by a factor of ten.
So we now move to 0.01 and the next ten, again, let's say we're training all this for 30 epochs and
then the next we go to 0.001.
So you could train your model and then after ten epochs, you restart the training by modifying the
learning rate in your optimizer.
And then again, you do this same year.
But now the problem with this is you always have to be there to ensure that after the training, that's
after the ten epochs, you modify this manually.
Now what if we're able to do this automatically?
As usual, this is made possible by TensorFlow callbacks with this callback.
That is the learning rate scheduler callback.
We could define a function which takes in the number of epochs and then modifies the learning rate based
on the current epoch or based on a mixture of the current epoch and some predefined function.
So as you could see here, we have the learning rate scheduler.
It takes a master schedule and then we could specify the verbosity.
Now, this is an example of this scheduler method, which has been defined here, here.
What it do is if the number of epochs is less than ten, then you're going to use this predefined learning
rate.
And then.
In the case where the number of epochs is greater than equal ten, then we start to reduce the learning
rate in an exponential manner.
So there we go.
Let's take this off.
Now, what we're saying here is we're modifying the learning rate such that after this we have in before
ten epochs, we have in this fixed learning rate.
That's it.
And then after this, we start to reduce this.
So the learning starts dropping as we continue with the training.
So now we don't really need to be monitoring the training manually because this callback will automatically
modify the learning rate for you.
We could simply copy out this example which has been given to us right here and then make use of it
in our training process.
So yeah, we have to include this text and then our code.
So yeah, we paste out this scheduler, let's do this.
And then for this one we have our learning rate schedule in.
Callback.
So learning rate scheduler.
And then we do this in part.
So yeah again we just have learning rate scheduler, we run that and then get back to this position
right here.
Let's do this one, two, three.
So that's fine.
So notice that here we have this other, so we have the callbacks learning rate scheduler and we have
since we logger is stopping learning the scheduler on other callbacks.
So that's fine.
Let's get back to that and then we are going to define.
So yeah, we've had the learning rate scheduler or this scheduler method, but we're yet to define our
scheduler callback.
So we have our, let's say scheduler, scheduler callback equals learning rate scheduler, and then
it takes in this schedule method right here.
So that's fine.
Notice how the scheduler method takes in the current epoch and the current epoch number and the learning
rate.
So here we have this given to us already, and so we'll modify this to take, say for example, three.
So after three epochs we're going to modify the learning rate and then we could print out the learning
rate so we could say, for example, the current learning all actually what we could do is let's take
this off, let's not have that.
Let's specify the verbosity to be equal one.
So yeah, we have the verbosity specified as one and then we'll take off this.
So you could also check out the CSV logger, the locked CSC file.
So you'll notice how throughout we've been logging this values since we just doing, we just adding
up or stacking up the values on the previous values we've had already.
So for now what we could do is take this off.
Now let's take this off the all this and then focus on just the scheduler.
So we have scheduler callback and that's fine.
Let's ensure that the cells have been run already, so that should be okay.
Now we have this arrow verbosity, unexpected keyword arguments.
Let's come back, get back to this here and then we have always verbose, so let's modify that as verbose
equal one.
We run that again.
That should be fine.
Now we could now get back to our training.
So we're expecting that below three epochs, let's say equal three epochs, we have a given learning
rate and then above that we have a learning rate which decreases exponentially.
So that's it is here that the training process starts.
We do not really need to print out any value because when we set our variables to one, it outputs this.
So here we have learning ratio set and learning rate to 0.00999.
That's practically 0.01, which is what we are given here in this learning resource is part of this
learning rate setting.
And then as time goes on, it's going to decrease this value and then always output the current learning
rate.
So as we carrying out this training process, we call that the aim of having to work with these kinds
of learning rate schedulers is that we want to actually get the best of both worlds.
So what we want is speed, because a very slow or very small learning rate doesn't assure speed.
And so we want speed and then we also want.
Stability when training.
So because we want this tool, we are going to use modify our lending rates such that we always get
this throughout our training process.
And so that's why whenever we're starting with training, we have high learning rates which could ensure
speed.
And then as soon as we we have trained for a given period of time, for a given number of epochs and
that we're trying to approach this global minimum right here.
We now seek for stability by reducing the learning rate so that we don't get to this point and have
to divert.
So if we want a more stable kind of training process, what we do is reduce this learning rate.
After the training is complete, here's what we get.
You could see that we have this learning rate, which is now modified after a given number of epochs.
So you see how the learning rates start decreasing as we go on with the training process.
And so that's how we implement learning rate scheduling.
But before moving on, let's check out this tutorial provided by the Amex Net Project Developers, which
talks of other different learning rate scheduling techniques.
So here we have this learning rate scheduling with warm up.
Yes, it's slanted triangular and as you could see, the learning rate actually falls between these
two values here.
So we have our learning rate between one and two.
So you could always define your max learning rate and then your mean learning rate.
So you could always have this.
And then with the warm up, what we do is we are going to increase this learning rate linearly up to
the max for a given number of epochs, and then once we attend this, once we get to this number of
epochs, we now start decreasing this learning rate.
So yeah, the decrease is linear, so it's a linear function we're using.
And then once it gets to the minimum learning rate, we just maintain that constant value right up to
the end.
So another Oh.
Learning rate scheduling technique we could use here is we have this linear increase that's the warm
up and then we decrease this exponentially right up to this minimum value.
Now, this technique of warm up was developed in this paper by Priya goyal et al, and they found that
having a smooth, linear warm up in the learning rate at the start of the training improved the stability
of the optimizer and led to better solutions.
Then from here we move on to the cosine learning scheduling method where there is this smooth decrease
in the learning rate, which is kind of resembling the cosine function.
Now this is what cosine function actually looks like.
So here we have this and that.
So here is our cosine function.
And then if you notice, you'll find that this portion actually looks similar to what we have here.
And now, after we get to a given number of epochs, we just maintain that.
So we have defined the mean and then the max.
And then once we get to the mean, we just maintain that mean throughout.
Then we also have this stepwise decay scheduling.
Let's take this off here.
You see how we start with the warm up.
So this warm up is kind of like a method used very much in practice.
And then from here we have this step wise reductions.
So we go in for this fixed learning rate.
After given a number of epochs, we now drop the learning rate and then we drop it and so on and so
forth.
So here is our step wise scheduling with warm up.
Then from here we have this cool down.
Where we follow the stepwise method, and then after a given number of epochs, we now reduce the learning
rate linearly.
Hence the term cooled down.
We also have this one cycle Shetland technique proposed by Leslie Smith and Nicole Toppin.
And is why it looks like you see that we increased as we have this warm up and then this linear decrease.
Once we get to this initial position, we also have again this linear decrease with a different slope
with a.
Smaller slope.
And then once we get to this minimum or this final minimum, we now just actually just maintain the
learning rate.
Now, finally, we have the cyclical scheduling methods originally proposed by Leslie and Smith.
The idea of cyclical increasing or cyclical increase in the degree and decrease in the learning rate
has been shown to give faster convergence and more optimal solutions.
So as you could see here, we are having the learning rates, which is going to be bouncing from the
minimum to the maximum, as you can see here.
So we have this linear increase to the max and then linear drop, linear increase in your drop and so
on and so forth.
And then finally we have this cyclical cosine online scheduling right here.
We see how we have the cosine online and then we have this full cyclical process as we go from the highest
to the lowest.
And and then this next cycle from the meet between the mean and the max, that's this right to the mean
learning rate.
And then we take up this final cycle from this quadrant of the total.
So we had 0.5, which is a quarter of two.
So we start from this and then right to the mean.
Now notice also that as we go into the cycle, the cycle lengths keep increasing.
So we move from this to this length and finally this length, based on the schedule you want to implement,
all you need to do is to modify this scheduler method right here.
The next callback we'll be looking at is that of model check pointing.
In this model checkpoint callback, we are able to save the model's weights at some frequency.
So all I previously were after training like with this, after training our model, we get to low or
rather we get to save the model like here.
So you we save the model or we save our weights.
After we're done with the training here, we could be doing this weight saving during the training process
and thanks to this model checkpoint callback.