In this lecture we are going to discuss a very popular topic in modern deep learning.
Gans Gans stands for generative adversarial network.
Yann Le -- who is one of the forefathers of Deep Learning described Ganz as the most interesting idea
in machine learning in the past 10 years.
By the way this is also who the famous Lean IT architecture of convolution on neural networks was named
after the CNN from which all other CNN's are based.
Ian Goodfellow the inventor of Ganz is now himself a prominent deep learning researcher.
My goal in this lecture is to introduce you to the fundamental theory behind again as you'll see that
although they sound complicated they are nothing but a new and interesting way of combining things we
already know how to build.
This is a common pattern in deep learning known networks themselves are nothing but a bunch of logistic
regressions chain together.
CNN and Oren Arnolds are nothing but neural networks with shared weights.
I like this approach because deep learning allows us to build tons of cool things with only a few simple
tools and some creativity.
First let's discuss what gains are even used for in the first place.
Since this may not be immediately obvious the main use case for Ganz is to generate data.
Ninety nine percent of the time.
This means generating images.
The reason gains have become so popular is because unlike the models that came before it gains are exceptionally
good at this.
Here is some example images produced by state of the art Ganz.
Believe it or not these are not real people but rather images generated by a computer
one thing you can do if you want to see a few more examples and observe for yourself the power of Ganz
is to check out a few Web sites that display such examples.
One Web is.
This person does not exist.
Dot com.
This one is simple.
You go to the page and it shows you a randomly generated face.
You can refresh the page to see more faces.
Another one which I really like is which faces real dot com.
This one is more like a game.
So you're shown two images and one is real.
One is fake and you have to pick the right one.
So check these out for yourself.
If you want to see more examples of how good gains are at generating realistic images
OK.
So what's the main idea behind gains gains or a system of two neural networks each of which has their
own objective the objective of the first known network is to generate images.
The objective of the second neural network is to discriminate between real and fake images.
This is all you need to understand how gangs get their name.
They are generative because you are generating images.
They are adversarial because you have two neural networks which oppose each other in an adversarial
manner.
They have opposite goals.
The goal of the first known network is to generate images that look real and the goal of the second
neural network is to detect real and fake images
intuitively.
You can think of Ganz like a counterfeiter and a shop owner.
The counterfeiter goes to the shop and tries to use fake cash to buy an item.
The shop owner has to detect whether the cash being given by customers is real or fake.
Perhaps at the beginning both the counterfeiter and the shop owner are not experts at what they do.
The counterfeiter is not that good at making fake bills but the shop owner is also not that good at
detecting fake bills.
However once the shop owner learns to detect the fakes the counterfeiter may then improve on his counterfeit
design and trick the shop owner once more.
After a while the shop owner may again realize he is being duped and learn to inspect the bills more
carefully to detect these improved fake bills.
The cycle continues as the counterfeiter gets better at tricking the shop owner and the shop owner gets
better at avoiding being tracked.
Of course these analogies only take us so far.
How do we actually build again.
Let's recall all the ingredients we need for a basic neural network that we want to train first.
We need a neuron that work that much is obvious.
Well in our case we actually have to neural networks in order to train the neural network.
We need an objective or a lost function.
This is the big question with Ganz.
What is the last function.
But once we have these two things everything else is nearly trivial.
We call a model that fit which does gradient descent and that's that
OK.
So let's dig a little deeper on what the lost function is.
And in fact for again we're going to have two lost functions one for the generator and one for the discriminator.
Let's first look at the discriminator since that's more in line with what we are used to already.
Since the discriminator must be able to tell the difference between real and fake images.
This is a binary classification problem for every input the discriminator can only predict one of two
categories real or fake.
As you know for binary classification the correct lost function is the binary cross entropy.
What about for the generator.
What is the last function.
This is what I call applying creativity to the basic tools that we already know about intuitively.
The basic idea is this we have our generator network in our generator network feeds into our discriminator
network.
From this perspective this is just one gigantic neural network.
What we can do is just like with transfer learning we can freeze the layers in the discriminator so
that when we run gradient descent only the generator network gets trained next.
And here is the ingenious part.
We want the generator to get better at producing real looking images.
And so our last function will remain the binary cross entropy except that we are going to switch the
labels.
In other words let's say for the discriminator real is one and a fake a zero.
We're going to pass in fake images but make the target one such that we're encouraging the discriminator
to identify these fake images as real but the discriminator weights are frozen so they won't change
when we do this only the generator weights are trained
here's what the loss would look like mathematically.
If you're interested here all the white hats are predictions on fake images.
Since the generator is now the input into the discriminator because we always want the target to be
one we only care about the first half of the binary cross entropy.
So essentially it's just the sum over the log of all the white hats.
Or as stated another way the negative log likelihood of the Y has
now that you understand the basics of how Gans work.
Let's look at some details that will help us in our implementation.
We know that neural networks must have both inputs and outputs for the discriminator.
This is easy to input as an image and the output is a prediction about whether the input image is real
or fake for the generator.
This will be a little strange.
We know that the output should be an image because its job is to generate images but what is its input.
In fact the input to the generator is nothing but noise.
What we are going to do is generate noise from a multi variant standard normal and the generator will
learn to map this noise to an image.
Mathematically what we are doing is saying we have some random vector z which comes from a standard
normal.
Let's say it has size a d equals 100 although this is a hyper parameter so you can choose what you like.
We call this one hundred dimensional space.
The latent space
in essence you can think of it as an imaginary space where the generator believes all the images of
digits live.
So maybe the ones are over here.
All the sevens are right beside it because seven's kind of look like ones near the Sevens or the nines
because nines look like sevens if you draw the loop very small so you can see how these images kind
of morph into one another when they are drawn in different ways.
So there's late space maps all the different kinds of images that you can have
and of course this is a source of confusion for both computers and humans.
When an image is right on the boundary between say 7 or 1 it's difficult for us to tell what digit it
should be.
Should it be a 7.
Or should it be a one.
In any case what the generator is doing is it's learning to associate each part of the latent space
with different images in a continuous manner.
Then later when we generate images we're just picking a random part of the space and generating an image
from whatever that part of the space represents.
You can think of the generator as kind of the reverse of a feature transformer or an embedding with
a feature transformer or an embedding or taking an input image or an input text or some other kind of
input and we're mapping it to a vector.
But for a generator and again we're doing the opposite.
We start with a vector and then we map that to an image
as always it's going to be helpful to look at some pseudocode before we move on to the actual implementation.
So here's how it's going to look.
So the first step is the load in our data set.
We're going to use amnesty for this example in a future courses.
You'll learn about more advanced types of games which will work on more complex types of data.
Next we're going to create our discriminator and generator networks.
These are just standard neural networks some dense layers some real use and so forth.
The trick is we're going to have to optimize is one that optimizes the parameters of the discriminator
and one that optimizes the parameters of the generator.
Conceptually you can imagine that the lost function is always applied to the output of the discriminator
only in either case we will use the binary cross entropy but with flipped labels when we train the discriminator
half of the data will be real images from the dataset and half of the images will be generated from
the generator.
These will use the true labels when we train the generator we will generate random noise pass that through
the generator then pass that through the discriminator and use a flipped label meaning that we say it's
a one even though it's really a zero we can conceptualize this chain of generator and discriminator
as a kind of combined model where only the generator weights our parameters although that's not exactly
how we'll look at code.
It's a good conceptual tool.
Once we've built our two networks we're going to run a gradient descent loop as usual inside the loop.
We're going to train the discriminator and generator in alternating order in order to train the discriminator
we'll grab a random sample of images from my dataset and a random sample of images from our generator.
Then we'll do one iteration of gradient descent by calling discriminator dot train on batch.
We do this for both the real and fake images then for generator training we'll do the following.
First of all generate a batch of noise samples then we'll call combines model dot train on batch passing
in the noise and ones for the target to do one iteration of gradient descent on the generator.
After several thousand iterations our generator should be good enough to produce images that look like
the M.A. data set.
As a final note for this lecture I want to mention that in terms of model evaluation we're going to
take a bit of a departure from what we normally do.
You recall that for most models what we're looking for after training is that the loss goes down and
the accuracy goes up.
This makes sense if you are doing classification or regression but now we are no longer doing classification
or regression.
Remember that in our case the generator and discriminator are constantly trying to one up each other.
And so what is interesting is that the loss functions for both the generator and discriminator should
look just like noise.
It should appear that if you only look at the loss per iteration that nothing is happening at all.
But in fact something profound is happening both the generator and discriminator are improving.
You don't see this in the Las per iteration because ideally both the generator and discriminator are
improving at the same rate.
Let's now summarize everything you learned in this lecture.
First you learn that gains are not just a single neuron that work but a system of tuner on that works
a generator and a discriminator.
The discriminator is responsible for telling the difference between a real and fake images.
The generator is responsible for creating realistic looking images both the generator and discriminator
learned in tandem.
The lost function for the discriminator is just the binary cross entropy because there are two classes
real and fake the lost function for the generator is not directly applied to the generator but rather
a combined model where the generator feeds into the discriminator.
This allows us to again use the binary cross entropy.
But there are two important details first we freeze the weights of the discriminator so that only the
generator is trained.
Second we flip the labels so that image is generated by the generator are labeled as real.
Next we learn that the input to the generator is nothing but noise sampled from the latent space the
latent space is where the in coatings for each digit lives.
You can think of this as the opposite of a feature transformation or an embedding and embedding or feature
transformation is like a mapping from an input image or an input word or any other kind of input into
a single vector.
The generator does the opposite.
It takes a vector which lives in the latent space or the embedding space and produces an image.
Lastly we learn that when training gets the last per iteration should not be informative in the sense
that it should not decrease over time as it does when we are training other models.
This is because both the discriminator and the generator should be learning at the same rate continually
helping each other improve.