1
00:00:11,640 --> 00:00:15,780
In this lecture we are going to look at how to implement again and code.

2
00:00:15,990 --> 00:00:19,050
This lecture is gonna walk you through a prepared code lab notebook.

3
00:00:19,050 --> 00:00:24,960
Although a very good exercise which I always recommend is once you know how this is done to try and

4
00:00:24,960 --> 00:00:31,050
recreate it yourself with as few references as possible as usual you can look at the title of the notebook

5
00:00:31,380 --> 00:00:33,880
to determine what notebook we are currently looking at.

6
00:00:34,910 --> 00:00:37,890
So at the top we have all our usual imports.

7
00:00:37,890 --> 00:00:43,470
We've also imported the Save image function from torch vision which is a useful utility for saving an

8
00:00:43,470 --> 00:00:47,210
entire batch of images to a single image file.

9
00:00:47,220 --> 00:00:54,280
This will let us check how our model progresses over each iteration.

10
00:00:54,320 --> 00:00:58,990
Next we define a set of transformations to perform on the data.

11
00:00:59,000 --> 00:01:04,160
Remember that researchers have found that using pixel values in the range minus one to plus one tends

12
00:01:04,160 --> 00:01:05,730
to work better.

13
00:01:05,750 --> 00:01:11,850
So this code might seem strange efforts but if you check the calculation you can verify that it works.

14
00:01:12,350 --> 00:01:19,100
First remember that the two tensor transformation already scales the image to be between 0 and 1.

15
00:01:19,100 --> 00:01:24,680
So if we take one subtract a mean of zero point five and divide by a standard deviation of zero point

16
00:01:24,680 --> 00:01:31,730
five we get zero point five divided by zero point five which is one if we take zero subtract the mean

17
00:01:31,730 --> 00:01:37,190
of zero point five and divide by a standard deviation of zero point five we'll get minus zero point

18
00:01:37,190 --> 00:01:40,370
five divided by zero point five which is minus one.

19
00:01:41,030 --> 00:01:43,820
Therefore we map 1 to 1 and 0 to minus 1.

20
00:01:43,820 --> 00:01:47,430
So this works.

21
00:01:48,040 --> 00:01:51,960
Next we load in the amnesty train dataset from torture vision.

22
00:01:52,000 --> 00:01:56,970
You've seen this before.

23
00:01:56,990 --> 00:01:59,480
Next we create a day loader from our data set

24
00:02:02,420 --> 00:02:04,640
next we create our discriminated network.

25
00:02:05,300 --> 00:02:17,980
So this is an A and end with layer sizes 780 for 512 256 and 1 and with leaky real you activations.

26
00:02:18,110 --> 00:02:20,690
Next we create our generator network.

27
00:02:20,690 --> 00:02:32,730
This is n n n with input dimension one hundred and hidden dimensions 256 512 1024 and 780 for.

28
00:02:33,090 --> 00:02:37,860
Notice how we've also added a batch final home after each leaky well you and that the neural network

29
00:02:37,920 --> 00:02:46,660
ends with a 10 each activation which maps the output values to be between minus 1 and plus 1.

30
00:02:46,700 --> 00:02:55,510
Next we set the device to be the GP you and we move both the discriminator and generator to the GP you.

31
00:02:55,690 --> 00:03:02,650
Next we create a loss and optimizer is for the loss we can use the binary cross entropy for both cases.

32
00:03:02,650 --> 00:03:08,890
The only thing that changes is the label for the optimizer as we make sure to pass in only the parameters

33
00:03:08,890 --> 00:03:10,540
for that specific network.

34
00:03:10,840 --> 00:03:17,170
So the D optimizer only optimizes the parameters in the D model and the G optimizer only optimizes the

35
00:03:17,170 --> 00:03:22,880
parameters in the G model.

36
00:03:23,030 --> 00:03:29,240
Next we have a convenience function for scaling images back from minus one plus one to zero one.

37
00:03:29,360 --> 00:03:35,300
This is useful for plotting and saving and so forth.

38
00:03:35,310 --> 00:03:46,030
Next we create a directory to store the images generated during training is called again images.

39
00:03:46,150 --> 00:03:47,770
Next we have the training area.

40
00:03:48,490 --> 00:03:52,440
So this will be slightly more complex than the code preparation section.

41
00:03:52,510 --> 00:03:58,540
Since I left out a few minor details but hopefully you understand the main idea and so these minor details

42
00:03:58,540 --> 00:04:00,390
are just that minor.

43
00:04:00,760 --> 00:04:06,540
Sort of start we're going to create one dimensional arrays just containing ones and zeros.

44
00:04:06,580 --> 00:04:09,560
These will be used as targets during training.

45
00:04:09,670 --> 00:04:14,330
The targets are constant values so instantiating these inside the loop would be wasteful.

46
00:04:16,930 --> 00:04:24,790
Next we create two lists to store the losses for the generator and discriminator.

47
00:04:24,800 --> 00:04:30,320
Next we enter a loop that will go for two hundred epochs and inside that loop we have another loop to

48
00:04:30,320 --> 00:04:36,430
go over the data loader notice we ignore the targets by using an underscore since they are not used

49
00:04:40,680 --> 00:04:41,480
inside the loop.

50
00:04:41,490 --> 00:04:45,000
The first thing we do is get in the backsides.

51
00:04:45,030 --> 00:04:51,150
Remember that since the backside may not divide evenly into the total number of samples the actual batch

52
00:04:51,150 --> 00:04:55,650
size may not be equal to the specified back size.

53
00:04:55,650 --> 00:05:00,590
Next we take the inputs and flatten them into an end by the matrix and move it to the jeep.

54
00:05:00,590 --> 00:05:00,930
You

55
00:05:04,470 --> 00:05:09,220
and now that we have N which is the current that size we can also index the ones array and the zeros

56
00:05:09,220 --> 00:05:11,710
array to get the correct size targets

57
00:05:16,570 --> 00:05:19,170
next we have the train discriminated block.

58
00:05:19,450 --> 00:05:25,060
The first thing we do is get the loss for the real images so we pass the inputs through the D model

59
00:05:25,210 --> 00:05:27,350
and call that real outputs.

60
00:05:27,430 --> 00:05:31,690
Then we pass real outputs into our criterion with ones as the target.

61
00:05:32,140 --> 00:05:37,490
We'll call this D loss real.

62
00:05:37,600 --> 00:05:40,420
Next we calculate the loss for the fake images.

63
00:05:40,600 --> 00:05:48,180
So we start by generating some noise which goes in a matrix of size and by 100 we move that to the GP.

64
00:05:48,490 --> 00:05:53,840
Next we create our fake images by passing this noise through the generator.

65
00:05:54,000 --> 00:05:59,130
Next we pass the fake images into the discriminator and we get the fake outputs.

66
00:05:59,250 --> 00:06:03,770
We pass the fake outputs into our criterion with zeros as the target.

67
00:06:03,960 --> 00:06:09,450
We'll call this loss D loss fake.

68
00:06:09,480 --> 00:06:15,970
Next we combine the losses by taking the average then we zero the gradients in both optimizations.

69
00:06:16,080 --> 00:06:21,570
Then we call the lost that backward and then D optimize it out step to do one step of gradient descent

70
00:06:21,930 --> 00:06:28,840
on the discriminator.

71
00:06:28,850 --> 00:06:31,490
Next we have the train generated block.

72
00:06:31,490 --> 00:06:37,040
As mentioned it's been found that training the generator more than once for each iteration is useful.

73
00:06:37,040 --> 00:06:40,740
So we do a for loop that goes twice inside the loop.

74
00:06:40,760 --> 00:06:46,620
We generate some random noise of size end by one hundred and move that to the GP you.

75
00:06:46,610 --> 00:06:51,830
Next we pass this to the G model to get a set of fake images.

76
00:06:51,840 --> 00:06:57,150
Next we pass the fake images through the D model to get a set of fake outputs.

77
00:06:57,150 --> 00:07:02,820
Next we pass the outputs through the criterion this time using ones as the targets.

78
00:07:02,820 --> 00:07:09,650
This is because we would like the discriminator to think these are real.

79
00:07:09,680 --> 00:07:12,180
Next we do a gradient descent step.

80
00:07:12,440 --> 00:07:14,130
First we zero the gradients.

81
00:07:14,180 --> 00:07:20,390
Then we call G lost out backward and then we call G optimize it out step and remember that even though

82
00:07:20,390 --> 00:07:27,410
the outputs that go into the lost function are outputs from the discriminator we are running key optimizer

83
00:07:27,590 --> 00:07:38,440
which only knows about the G models parameters so only the G models parameters are updated.

84
00:07:38,450 --> 00:07:41,390
Finally we save the losses to our list of losses.

85
00:07:41,480 --> 00:07:48,230
We print the epoch number and the losses and then at the end of each epoch we save the fake images that

86
00:07:48,230 --> 00:07:53,570
were last generated using the Save image function that we imported earlier.

87
00:07:53,570 --> 00:07:58,760
Note that before we save the images we have to scale them back to the correct range using the scale

88
00:07:58,760 --> 00:07:59,480
image function

89
00:08:03,720 --> 00:08:04,110
OK.

90
00:08:04,130 --> 00:08:06,270
So let's check out the lost period creation

91
00:08:11,420 --> 00:08:18,190
so as promised it's not the nice downward converging pattern that you're used to.

92
00:08:18,330 --> 00:08:26,540
Okay so let's look at the images we generated so here are the fake images after epoch 1.

93
00:08:27,160 --> 00:08:31,120
We can see that at this point they are already starting to resemble the MLS dataset

94
00:08:35,120 --> 00:08:43,150
after epoch 50 it pretty much looks like M.A. After epoch 100 it looks even more like an honest

95
00:08:46,430 --> 00:08:47,880
after epoch 150.

96
00:08:47,960 --> 00:08:56,920
It still looks like it missed maybe even a little better and after epoch 200 it still looks like amnesty.

97
00:08:56,990 --> 00:09:01,850
So as you can see the results are pretty much what we expect at the beginning.

98
00:09:01,850 --> 00:09:03,350
We didn't see anything at all.

99
00:09:03,410 --> 00:09:06,370
And by the end it looks pretty much like amnesty.

100
00:09:06,620 --> 00:09:11,900
The generator learns to generate real looking digits quickly so there's an aspect of diminishing returns

101
00:09:11,900 --> 00:09:12,670
here.

102
00:09:12,950 --> 00:09:16,090
After the first third of the way through things already look pretty good.