In this lecture we are going to discuss stochastic gradient descent.
Previously we looked at regular gradient descent and how it can be used to optimize a function.
But as you may know in tensor flow 2.0 the gradient descent option is called S G D which is short for
us to castigate gradient descent.
So what makes it stochastic and how is it different from non stochastic gradient descent.
To understand this let's first think of a simpler problem.
Let's suppose we want to measure the average height of all humans in the entire world.
Now that would be quite a hefty task.
There are 7 billion people in the world and it would take a very long time to ask each person individually.
But what if instead you took just one thousand random people and took the average height of these 1000
people would you expect the average height of these 1000 to be similar to the average height of the
entire population.
The answer is yes.
It should be clear that the main advantage of asking only 1000 people instead of 7 billion people is
that it would take much less time.
OK so how is this relevant for deplaning.
If you recall in deep learning we have to calculate the cost.
And from this cost we have to calculate is gradient so that we can do a gradient descent.
The problem is this cost depends on the number of training samples in our dataset.
So imagine you are training on the famous image in that dataset which has about 1 million images.
This means that for each iteration of your training loop you have to add up all the errors for all 1
million samples.
That's going to take an extremely long time
but what if instead I took the average error over just 1000 images.
Of course they must be chosen randomly.
The same idea applies this average error is probably close to the average error over the entire dataset
and so if we do gradient descent using only these 1000 images we will probably end up taking a very
similar step.
Just with 1000 times less computation
in actuality we typically use smaller back sizes like 32 64 or 128.
The basic algorithm looks like this instead of just one loop over the epochs we have two nested loops
one over the epochs and one over the training set.
The idea is we are going to split up our data into batches and for each batch we are going to take one
step of gradient descent.
As you can see in this pseudocode we are using a batch size of 32 so the number of iterations of the
inner loop is n divided by 32.
We can specify the start index as I times thirty two and the end index as I plus 1 times 32.
So this will count from zero to thirty to thirty two to sixty four sixty four ninety six and so forth
on each epoch we will make one pass through the entire dataset.
It's important to randomize the data and each epoch because running through the same samples in the
same order each time could lead to undesirable patterns being learned
as an exercise.
You may want to try this out for yourself to prove that it is faster to set up your experiment.
Here's what you should do in the tensor flow 2.0 fit function.
You have an argument called that size.
You can take any of the examples from this course and explicitly set the back size to a small value
such as 32 and explicitly set it to the size of the entire dataset.
Whatever that may be from there.
Compare the time it takes for each method to converge to a minimum.
You should find that with batch gradient descent.
The training process converges much faster and in fact sometimes your dataset may be so large that doing
one step of gradient descent cannot even fit into memory.