Hi there and welcome to this new session in which we shall be treating the variational encoder.
And we shall see how it could be used in image generation.
In this first part, we shall dive deep into understanding the theory behind the variational encoder
and then the subsequent sections.
We shall practically implement a working variational auto encoder.
That said, we shall start with explaining or understanding this auto encoder and we'll make use of
this blog post by Jeremy Jordan.
Now to understand the auto encoder, we can break this word into two parts.
That's auto and we have ENCODE.
So essentially we have a system which self encodes itself.
Now, supposing you have an image like this one right here.
When we pass this into some encoder block, Let's let's have something like this.
We have some encoder block and then we could obtain this output vector.
Now, this output vector is six dimensional.
So we had six different positions.
And each of the position represent a specific characteristic of this image.
You could see a smile.
0.99 Skin tone 0.85.
Gender -0.73.
Beard 0.85.
Glasses 0.002.
Hair color 0.68.
So all these six values here, the six values we have here are characterizing our image.
So they encode information or information about this image is encoded in this vector or this vector
right here.
And then on the other hand, when we want to retrieve this encoded information, what we could now do
is we get a decoder which takes this encoded information and then reproduces this original image.
And so that's globally how we produce this kind of system, which could be used in image compression,
where we could take this image, encode it so that we have just this vector, then we could pass this
vector via some network.
And then on the other side of the network, we decode this vector such that we have the original image
apart from compression and other field where we could apply this kind of autoencoder network is in image
search.
So let's suppose that we have this image right here and we have this vector.
Now if we have another image of this same person here, so we have another image of the same person,
we call this image image B, and here we have image A, which produces a vector which we will call V
A, Then it means that in this six D vector space six, because we have six different values here,
it could be 128 D or whatever dimension we make it to be.
So as we're saying, we have this image B, which is the same image here, the image of this of this
person, but not necessarily the same image, but maybe some other image of the same person.
Then after encoding, after passing through an encoder, you have our six D vector VB, But because
it's a similar person or because it's the same person, we would expect this values to be similar.
And so v a will be close to VB, and if we have another person, let's say another person C this year
and we generate that person's V, C that's this encoded vector, then we would expect v A to be much
different from v c.
And so this means that in an image search scenario, we just pass this input, we obtain this vector,
and then we would compare the two vectors to see whether it belongs to the same person or not.
Now, it should also be noted that when training an autoencoder model where we have an image A and we
have a reconstructed image, a prime, then our aim here will be to minimize the difference between
A and a prime, so we could minimize a minus a prime.
Now it turns out that in image generation, to get better results, instead of dealing with discrete
values like what we had here, let's get back to this top.
You see here we had the a given value.
Let's take this off a given value for smile, for skin tone and so on.
And so forth.
So as we're saying, instead of having a fixed value for each and every one of these features, what
we'll do is we'll make use of a probability distribution.
So here instead of having a value, let's say this is -0.6.
We would have a probability distribution whose mean is at -0.6.
Well, this is this looks more like -0.5.
But here we suppose now we're going from 0.6 to this probability distribution, which means a -0.6 with
a given variance.
Now, for those of you who don't have a math background, what this essentially means is instead of
picking a value or picking the value zero -0.6, what we'll do is we'll pick.
Some random value within this range.
So instead of having -0.6, as we said, we're going to have a random value in this range and.
Values closest to -0.6 have a higher probability of being picked.
So instead of having this, we could now have -0.3 or -0.5 5 or -0.75 and so on and so forth.
So we have values which we can pick in this range.
Now, if we see this other example here, where we have zero.
Now turned to this probability distribution, we pick values in this range, -1 to 1.
So here the variance or the range of values which are in which we are allowed to pick a value is larger
than this other one.
But still, values are around this zero.
That's the mean around this zero have a higher probability of being picked.
So here you would have a higher chance of picking 0.1 instead of picking 0.9 to see that clearly here.
Let's say this is 0.1 at this Mac and then this is 0.9 around here.
You would find that if you link this up here, this 0.1, you see that this has a higher score, hence
a higher chance of being picked as compared to this one which has much lower chance of being picked.
So.
In a nutshell, instead of having this 0.5.
We now have a mean value, which is 0.5.
And a variance which shows us or better still, gives us the range of values for which we can pick the
specific value for a given feature.
And so as you could see, this one here.
Has a smaller variance as compared to this and is compared to this one.
And from this point we will define this mean as mu and the variance, which is some distance from here,
this distance as Sigma Square.
This probabilistic approach to generating a latent vector, which previously was this vector we had
here, scroll back up.
Previously it was this vector is now what leads us to the variational autoencoder.
So you see that here we have our input image.
It gets into the encoder which produces MU and Sigma Square.
Or let's just say Sigma Sigma is a standard deviation.
Sigma Square is a variance, so it produces mu and sigma.
And then using mu and sigma with our decoder, we are able to obtain our reconstructed output image.
Now note that in this case we would have one, two, three, four, five, six positions, so mu would
be this six z vector sigma will be another six d vector where mu one this first position here mu one
and sigma one represent the mean and the standard deviation for this distribution.
Now it should be noted that the main benefit of a variational autoencoder is that they are capable of
learning smooth latent state representations of the input data.
Now, to better understand that statement, let's consider this output generated by an auto encoder
and this other output generated by a variational auto encoder.
You will notice that as we go from one digit to another, like let's say we're going from 6 to 8 here.
You see here we have six.
Well, it looks very well like six.
So let's let's take this one, which looks already very well, like six.
This is six.
But here it's really confusing because we don't know exactly what this is.
Now, this looks more like eight, but not really very clear.
And then here we start getting eight and eight and well, this two doesn't look very clear.
But when you look at the output generated by the variational encoder or the variational auto encoder,
as we go from one digit to another, we can see here that we have an even much smoother transition.
And this is thanks to the fact that instead of working with discrete values at the level of our latent
vectors, we're going for a probabilistic approach with the variational auto encoder because we're going
in for this probabilistic approach.
The training of our variational auto encoder is no longer evident, and this is simply because during
the training we need to compute partial derivatives.
With respect to Z here, with respect to MU and partial derivative, with respect to MU and with respect
to Sigma.
But because the Z we have here is drawn from a normal distribution with mean mu and standard deviation
sigma, we won't be able to compute this partial derivative.
And so the idea now will be to convert this node that's here to one that is deterministic.
You can see here we have this key random node and then deterministic nodes.
So this one is deterministic.
That's fine.
This one fine Now.
Well, this is fine.
Now the idea will be to convert this into one which is deterministic such that we could compute this
partial derivatives and hence train the model such that we could update the encoder and the decoder
parameters.
And so now this idea of converting this node from one which is random to one which is deterministic,
is known as the reparameterization trick.
So instead of having this where we have well, let's take let's take this off, let's make it simple.
So we have this where we have the mean mu and the standard deviation.
Well, we'll pick any value at random in this range.
We are instead going to define this epsilon which is drawn from.
A normal distribution with mean zero.
So our mean now will always be zero and then the standard deviation will be one.
So we have negative one one.
So this epsilon here, as we've said, is drawn from this probability distribution.
And then to obtain Z, unlike here where we obtain Z randomly from values surrounding the mean here,
we'll do the mean plus the standard deviation times a random value which lies between or which surrounds
our zero.
And so now we could compute this partial derivative here, this with respect to Z and hence train our
variational autoencoder model.
The next and final point we'll look at in this section is the variational autoencoders loss Now from
the autoencoder or the variational autoencoder paper.
The authors break up this loss into two main parts.
The first part, let's take this off.
The first part is the reconstruction loss, and this other part acts as a regularizer for the reconstruction
loss.
We try to minimize the difference between x and x prime or x Chapo.
So we want that the input and the reconstructed input or the reconstructed output should be similar
here is denoted as this.
We try to minimize this.
And then for the reconstruction loss, we're computing the KL divergence between this distribution and
this other distribution.
Now to understand what this distributions actually signify, we can take a look at this figure.
So here we have this QR or this distribution Q of Z given X, which happens to be a learned distribution,
meaning that when we'll be training this encoder model right here, this is our encoder model, we'll
be training this encoder model.
We shall in fact be getting this distribution, which as we've said already, is a learned distribution.
Nonetheless, we do not want this learned distribution to be very much different from the distribution
P of Z.
And so that's why we are going to minimize the distance between this distribution and this or this learned
distribution and the true prior distribution P of Z.
Now it should be noted that this KL divergence here is a tool which permits us measure the distance
between two distributions.
And so if we could minimize this, if we could minimize this, then we'll reduce the distance between
this distribution and this distribution P of Z.
And getting back to the original paper, it should be noted that the reconstruction loss can be taken
as the mean square error.
While this here will be our regularizer.
And so this is what we obtain after computing the KL divergence between those two distributions.