In this lecture we are going to take a closer look at how the Gan code will work and hopefully iron
out a few details before we move on to the actual notebook.
This is another interesting model in this course because unlike all the other algorithms we looked at
now we're dealing with two neural networks simultaneously rather than just one.
So it's kind of an odd concept.
What I want to focus on in this lecture are a few overarching themes.
First we'll look at the architecture of each model.
Just for fun we'll use a few more advanced layers and activation functions that we didn't use earlier
in the course.
Second we'll look at how to train them.
This is the centrepiece of this section because unlike all the previous examples you saw in this chorus
this training loop will be very different from what you saw before
you may have assumed that because the generator is a kind of a. discriminator that they would have opposite
architectures.
What I mean by that is if the generator had hit and sizes of one hundred three hundred and seventy four
that the discriminator would have hit in sizes of seventy four three hundred and one hundred.
In fact this need not be the case in our example we'll make the generator a more powerful model than
the discriminator.
The intuition is binary classification is an easy task whereas generating images is a hard task for
the discriminator.
All you have to do is map the input to 0 or 1.
There aren't that many degrees of freedom for the generator.
You have to generate seven hundred eighty four different pixel values and they all have to be coherent.
So there are many degrees of freedom.
So that's the idea generating is hard but discriminating is easy.
So here's what we'll be using for the discriminator although you should feel free to play around and
design your own.
It's just a series of linear layers followed by leaky values.
Recall that a leaky Raju is just like a normal value except it has a small positive slope for values
less than zero.
This is an attempt to avoid the problem of dead neurons where the slope becomes zero and therefore the
gradient becomes zero and therefore no Eric and back propagate
here's what the generator looks like.
Again feel free to play around.
So in addition to leaky well using linear layers we'll also add back Norm to the mix.
We've also added a 10 H activation at the end which implies that we'll be working with images in the
range minus one plus one rather than zero to one experimentally.
Researchers have found that these changes tend to yield better results.
OK so here's the real meat of this code.
After we define the discriminator deep and the generator G we're going to define the loss and optimizer
the loss is just the usual binary cross entropy.
But we're going to have to separate optimizes one for each network.
You'll notice that when we instantiate the optimizes we only pass in the parameters that we want the
optimizer to update.
So when we call the optimizer dot step it's only going to update the parameters of the discriminator
when we call G optimize it out step.
It's only going to update the parameters in the generator
inside the training block.
We're going to start by instantiating constant arrays to store zeros and ones.
These will act as our targets.
So we don't have to keep creating new arrays on each iteration of the training loop inside the training
loop.
We're going to loop over the data loader.
Remember that we don't need the image labels so we can use an underscore to ignore them what we're going
to do now is two training blocks one for the discriminator and one for the generator.
This slide will focus on the discriminator.
Now one question you probably have is does it matter which order you train them in the discriminator
first or the generator first.
I would recommend you try it out as an experiment
so how do we train the discriminator.
First we start by passing real images into the discriminator.
That's just the outputs from the data loader.
From this we can calculate the loss using the ones target
for the fake images.
We start by generating a batch of random noise and then passing that through the generator.
From that we get fake images which we then pass through the discriminator.
And from that we get fake outputs.
We pass the fake outputs through the loss function using the zeros target
next.
We take the average of the two losses to get a single value.
We can optimize.
After this we would like to call d lost out backward and d optimize that step.
But before that we have to remember to zero the gradients
next.
Let's look at the generator training block.
One interesting technique discovered by researchers was that doing multiple steps of training for the
generator for each single step of the discriminator seemed to work well.
This goes along with a theme that the generator is harder to optimize than the discriminator.
Therefore it requires more training so to accomplish that we can just do a loop inside the loop.
We generate some fake images once again using a batch of latent noise and passing this through the generator.
Since this is for training the generator there is no need to use any real images.
We just passed the fake images through the discriminator and then passed that output through the loss.
Importantly this time we use the ones targeted to tell the lost function that we want the weights to
go towards these images being classified as real.
Next we zero the gradients and do a gradient descent step as usual.