1
00:00:07,580 --> 00:00:08,810
Hi, welcome back.

2
00:00:08,840 --> 00:00:15,440
So in this lesson, lesson nine, which you will see here highlighted is we will take a look at filter

3
00:00:15,440 --> 00:00:18,500
visualization and filter activation visualization.

4
00:00:18,890 --> 00:00:22,880
So open that notebook, which I already have open here and let's get started.

5
00:00:23,420 --> 00:00:28,880
So firstly, in this lesson, we're only going to use cameras to visualize to do these visualizations

6
00:00:29,360 --> 00:00:31,460
simply because it's much easier than using PI two.

7
00:00:31,460 --> 00:00:37,310
It's not that PI is that much more difficult, however, because this lesson is more of a theoretical

8
00:00:37,310 --> 00:00:39,410
lesson to teach you how CNN has learned.

9
00:00:39,740 --> 00:00:44,630
It's not really something you would do much, and in practice, it's more of a way to reinforce what

10
00:00:44,630 --> 00:00:45,170
you've learnt.

11
00:00:45,800 --> 00:00:51,200
I will probably just stick with carrots because it is easier to easier to implement, and there are

12
00:00:51,200 --> 00:00:58,220
some cool Keros visualization libraries that allow us to to do some visualizations afterward, which

13
00:00:58,220 --> 00:01:02,720
you will see in the next code section section two, not Section nine, which is this one here.

14
00:01:03,170 --> 00:01:03,960
So let's see.

15
00:01:04,020 --> 00:01:04,850
Let's do this.

16
00:01:06,020 --> 00:01:06,950
Let's create that saw this.

17
00:01:06,950 --> 00:01:07,550
Listen to this.

18
00:01:08,060 --> 00:01:12,590
So to do this, firstly, we're going to have to train a basic CNN on the amnesty to set that someone

19
00:01:12,590 --> 00:01:12,980
will be.

20
00:01:13,370 --> 00:01:15,560
That's the one who's filters will be visualizing.

21
00:01:15,560 --> 00:01:16,940
So let's train that.

22
00:01:16,940 --> 00:01:18,320
So you've seen all of this before.

23
00:01:18,860 --> 00:01:23,930
We're just basically going to load of data sets here, run all the inspections, which we don't need

24
00:01:23,930 --> 00:01:27,140
to do, but just it's good practice here.

25
00:01:27,140 --> 00:01:33,020
We were going to reshape the data and due to pre processing and whatnot and coding of the data.

26
00:01:34,100 --> 00:01:35,480
So that's all done there.

27
00:01:36,410 --> 00:01:38,420
Next, we're going to define our model.

28
00:01:38,750 --> 00:01:43,610
So we defined the model architecture here and now we're going to train.

29
00:01:43,610 --> 00:01:46,340
So that's all trained for five 25 epochs.

30
00:01:46,340 --> 00:01:46,800
Let's do.

31
00:01:46,820 --> 00:01:52,910
The 10 will still get maybe close to 90 percent accuracy, I believe, but we'll find out shortly.

32
00:01:53,660 --> 00:01:55,460
So let's wait for this to train.

33
00:02:08,800 --> 00:02:11,530
OK, so we're finished reading and we got 90.

34
00:02:11,650 --> 00:02:15,230
We got 92 percent point ninety two point three percent.

35
00:02:15,230 --> 00:02:15,850
It's not too bad.

36
00:02:16,280 --> 00:02:18,400
Attorney box that was pretty quick to trim.

37
00:02:19,250 --> 00:02:22,440
So now let's start visualizing the filters.

38
00:02:22,450 --> 00:02:28,420
So firstly, let's take a look at let's look at earlier names here so we can do something where if we

39
00:02:28,420 --> 00:02:31,200
do it, layer top name in this list comprehension here.

40
00:02:31,210 --> 00:02:36,070
So we go through the model is we're able to access to lawyers like this here.

41
00:02:36,670 --> 00:02:39,940
So let's do that and you can get it to get the output here.

42
00:02:40,420 --> 00:02:45,370
Each one corresponds to this point to this object here at the memory address.

43
00:02:45,690 --> 00:02:47,500
Don't worry about what those numbers mean.

44
00:02:47,500 --> 00:02:48,520
They're not too scary.

45
00:02:49,690 --> 00:02:51,700
But importantly, this isn't what I wanted to show you.

46
00:02:51,820 --> 00:02:55,390
This dictionary has two layers in that CNN.

47
00:02:56,350 --> 00:02:57,760
So get two ships.

48
00:02:57,760 --> 00:03:00,970
Two, Let's get the ships of our kind of filters here.

49
00:03:01,570 --> 00:03:05,020
So to do that, what we do, we just loop through the model is here.

50
00:03:05,440 --> 00:03:09,640
And if it's a conveyor, we just continue here and we just print.

51
00:03:09,640 --> 00:03:15,910
We get we used leered dot, get weights to get the weights and the filters and biases and into the weeds

52
00:03:15,910 --> 00:03:19,600
and biases, and we just print delirium here as well.

53
00:03:20,080 --> 00:03:22,990
So in this particular, particularly if you're going to be using this here.

54
00:03:23,050 --> 00:03:28,420
This is probably some leftover kodaira by mistake, but just it's good to show you that we this is how

55
00:03:28,420 --> 00:03:32,440
we could actually get these things out of here if you wanted to get them in a usable way because this

56
00:03:32,440 --> 00:03:35,020
is going to keep overwriting as we go to the network.

57
00:03:35,530 --> 00:03:41,110
You'd probably want to store these in a list or the dictionary as well if you wanted, but a list is

58
00:03:41,110 --> 00:03:42,250
probably not practical.

59
00:03:42,910 --> 00:03:49,330
So but anyway, the point of this block of code is to show you the icon, filter size and the number

60
00:03:49,330 --> 00:03:50,110
of filters here.

61
00:03:50,500 --> 00:03:55,840
So you can see for the first layer, we have the tree by tree by one 32 filters.

62
00:03:56,260 --> 00:03:58,270
The next one, we have tributary with filters.

63
00:03:58,270 --> 00:04:04,150
We have 32 feature maps and 64 as well, 64, which is for this, for this one.

64
00:04:04,600 --> 00:04:08,440
So that's basically how we get the ship to the network.

65
00:04:08,830 --> 00:04:10,750
Let's run this week's show at work.

66
00:04:10,750 --> 00:04:11,110
So.

67
00:04:12,010 --> 00:04:15,790
So let's take a look at the widths of our first conflict.

68
00:04:15,820 --> 00:04:23,230
So using the did not get which function which you've seen before, we just access the first live network

69
00:04:23,560 --> 00:04:24,670
by pointing it to zero.

70
00:04:25,600 --> 00:04:26,550
We just get the weights here.

71
00:04:26,560 --> 00:04:30,280
So remember we can get the weight of weight or filters and biases here.

72
00:04:30,760 --> 00:04:32,830
So let's take a look at what that actually looks like.

73
00:04:33,430 --> 00:04:35,740
So remember, we're particular shape here.

74
00:04:35,790 --> 00:04:42,200
So the shape you'll see, it's where we're looking at tree by tree filters and we have 32 of those filters.

75
00:04:42,850 --> 00:04:44,020
So let's take a look at this.

76
00:04:45,310 --> 00:04:46,450
So what would miss?

77
00:04:46,450 --> 00:04:48,520
What's what's what's this messy output here?

78
00:04:48,550 --> 00:04:53,590
Well, basically, I'll tell you, well, this output is basically the filters.

79
00:04:54,040 --> 00:04:55,420
Here we have this.

80
00:04:55,450 --> 00:05:00,310
This is a consistent, multidimensional array has why there's so many brackets and these are the filter

81
00:05:00,310 --> 00:05:03,850
widths for the 32 filters that we have.

82
00:05:04,000 --> 00:05:05,290
So it's tree by tree.

83
00:05:05,830 --> 00:05:07,480
And then we have 32 weights here.

84
00:05:07,480 --> 00:05:10,960
So that's just the output of that's just to help you understand what you're looking at.

85
00:05:11,500 --> 00:05:15,370
And then the biases you remember, the biases are shared for the filters.

86
00:05:15,760 --> 00:05:19,300
So if we have 32 filters, we'll have 32 biases in that case.

87
00:05:21,010 --> 00:05:22,570
So we have all the biases here.

88
00:05:23,320 --> 00:05:29,950
So now, in order to visualize these filters, we have to normalize them or skill them between zero

89
00:05:29,950 --> 00:05:34,240
and one so that we can use MATLAB to properly plot them.

90
00:05:34,690 --> 00:05:37,300
So to do that, it's actually quite simple.

91
00:05:37,310 --> 00:05:43,630
But some Nampai functions here, so we just use filters dodman filters of Max to get the minimum and

92
00:05:43,630 --> 00:05:46,120
maximum value off or filter widths.

93
00:05:46,870 --> 00:05:48,940
And then we just skill them.

94
00:05:48,940 --> 00:05:52,510
Using this formula here to this ends up skewing all the values.

95
00:05:52,510 --> 00:05:56,950
So we just do some printing here just to show you what's going on.

96
00:05:56,980 --> 00:06:01,450
So you can see the min was minus point one four and Max was 0.3 six.

97
00:06:01,480 --> 00:06:03,520
At least that was when I ran it before.

98
00:06:03,520 --> 00:06:06,970
It might be different this time, and it is a little that's roughly the same.

99
00:06:07,810 --> 00:06:13,270
And anyway, after normalization, you can see we've now pushed this mean to zero and Broaddus Max all

100
00:06:13,270 --> 00:06:14,110
the way up to one.

101
00:06:14,590 --> 00:06:17,360
This will, I mean, this isn't absolutely necessary.

102
00:06:17,380 --> 00:06:21,390
However, it will produce better plots, better visualizations.

103
00:06:21,400 --> 00:06:25,250
So no, we can move on to visualizing our filters.

104
00:06:25,250 --> 00:06:28,330
So let's run this block of code and I'll run through it quickly.

105
00:06:29,080 --> 00:06:30,100
So what's happening here?

106
00:06:30,640 --> 00:06:35,900
Well, what we're doing here, we're going to plot the first few filters and set what we basically said

107
00:06:35,900 --> 00:06:36,770
the process initially.

108
00:06:36,770 --> 00:06:37,950
That's what a comment thing.

109
00:06:38,560 --> 00:06:42,160
And these are the filters we're putting and this is this IEX as well.

110
00:06:42,880 --> 00:06:45,970
So I access index that we loop through as well afterward.

111
00:06:48,280 --> 00:06:54,130
So for Renge and the filters and filters with a number of filters here 22.

112
00:06:54,820 --> 00:06:56,690
So we just get the filters here.

113
00:06:57,160 --> 00:07:01,700
They're extracting each year, which are all the values for that specific filter in the loop here.

114
00:07:01,720 --> 00:07:06,510
I could also use that I am not sure why I used AI separately.

115
00:07:06,550 --> 00:07:07,990
Maybe one way this one starts and.

116
00:07:08,070 --> 00:07:09,870
Zero, this one starts at one.

117
00:07:09,990 --> 00:07:16,190
That's why he says this for the subplot, plotting apologies for the confusion is just sometimes it

118
00:07:16,210 --> 00:07:22,680
happens to me quite a bit where I revisit code that I've done a month or even two days later sometimes,

119
00:07:22,680 --> 00:07:25,410
and I kind of forget what I did.

120
00:07:25,800 --> 00:07:31,140
So bear with me sometimes and apologies for that in advance if it happens again.

121
00:07:31,680 --> 00:07:33,850
So we just use the supplied function here.

122
00:07:34,290 --> 00:07:44,550
So the software function for 22 subplots, by its subplots by four, and then we have that index here

123
00:07:44,550 --> 00:07:49,500
so horrendous and will give us that output, which we just saw below.

124
00:07:49,980 --> 00:07:53,760
This was also the output I used in the slides, which you read about, which I hope you remember seeing.

125
00:07:54,240 --> 00:08:01,440
So what we're looking at here is the visualization of the two bits of the tree by tree filter.

126
00:08:01,440 --> 00:08:04,650
You can see one can clearly see three different pixels.

127
00:08:04,650 --> 00:08:06,480
So more apparent, maybe here.

128
00:08:06,890 --> 00:08:09,180
I'm not sure what it is, but it's roughly the same anyway.

129
00:08:09,690 --> 00:08:17,160
So you can see we have nine different tiles or squares or pixels actually in this filter, so you can

130
00:08:17,160 --> 00:08:20,070
see what these are, what the filters are looking for in the image.

131
00:08:20,220 --> 00:08:23,150
So this one, it looks a bit random.

132
00:08:23,160 --> 00:08:28,080
However, it is learning important things that correspond to different parts of digits.

133
00:08:28,230 --> 00:08:30,960
So that's what we're we're seeing here.

134
00:08:31,800 --> 00:08:34,200
So now let's take a look at filter activations.

135
00:08:34,200 --> 00:08:41,820
So if you remember correctly, filter activations are basically the feature map output we're looking

136
00:08:41,820 --> 00:08:42,030
at.

137
00:08:42,660 --> 00:08:46,380
So remember here we were looking at the filters themselves.

138
00:08:46,830 --> 00:08:53,700
But remember, after the first conv of the Liberty twenty eight by twenty eight input image?

139
00:08:54,210 --> 00:09:00,750
Well, that gives a feature map that is 26 by 26, and we have 32 of those feature maps to look at.

140
00:09:01,440 --> 00:09:05,010
So to do that, let's take a look at how we do filter activations here.

141
00:09:05,610 --> 00:09:07,590
Now, this can seem a bit confusing.

142
00:09:07,590 --> 00:09:12,810
And believe me, some of these filter visualization techniques actually do confuse a lot of researchers

143
00:09:12,810 --> 00:09:13,170
as well.

144
00:09:13,170 --> 00:09:17,580
So don't feel bad if you're confused what we're doing here.

145
00:09:17,590 --> 00:09:23,190
Firstly, we're extracting the outputs of the top seven layers here, so that's what we're doing in

146
00:09:23,190 --> 00:09:24,300
the first step here.

147
00:09:24,930 --> 00:09:28,690
Next, actually, the first two layers, I should say, not the first seven.

148
00:09:28,710 --> 00:09:29,340
Sorry about that.

149
00:09:30,000 --> 00:09:36,060
So next what we do here, we create a model that will return the outputs, given the model input.

150
00:09:36,360 --> 00:09:42,000
So that's what we used a model function from Carousel and we have the model input here.

151
00:09:42,000 --> 00:09:47,760
We model outputs, which are the outputs of these first two layers, and that's basically our activation

152
00:09:47,760 --> 00:09:48,080
model.

153
00:09:48,090 --> 00:09:52,050
So let's run that and let's take a look at the outputs.

154
00:09:52,290 --> 00:09:56,810
So it's as I said, we got the first two top layers, which are the first to convey is great.

155
00:09:57,630 --> 00:10:01,170
So now what we can do, we can take a random image here.

156
00:10:01,170 --> 00:10:06,690
So we're taking the twenty second technically 23rd image of our test dataset.

157
00:10:06,690 --> 00:10:07,650
We were shipping it.

158
00:10:07,800 --> 00:10:10,800
We just changed the image size and we're displaying it.

159
00:10:10,980 --> 00:10:14,310
So this is just to illustrate what the input image is.

160
00:10:14,610 --> 00:10:14,940
OK.

161
00:10:15,970 --> 00:10:21,270
Next, what we're going to do, we're going to get the output after the second filter.

162
00:10:21,270 --> 00:10:22,980
That's after the redo activation.

163
00:10:23,520 --> 00:10:26,580
So to do that, we take the activation model that we created here.

164
00:10:27,060 --> 00:10:28,650
We send this image tensor.

165
00:10:28,650 --> 00:10:33,600
That's this image here that we've reshaped into this correct form.

166
00:10:34,590 --> 00:10:39,150
I notice a lot of the little steps, but don't worry, at least all the code is here, so you can learn

167
00:10:39,150 --> 00:10:42,450
from watching this code if you want to spend some more time on it.

168
00:10:43,050 --> 00:10:48,750
So anyway, the activation model gives us the activations here, and we can see we can see how many

169
00:10:48,750 --> 00:10:50,840
activations we've extracted.

170
00:10:51,090 --> 00:10:55,080
So of basically this tells us how many activations we got.

171
00:10:55,080 --> 00:10:59,600
We got two sets of activations so we can take a look at what the first one.

172
00:10:59,610 --> 00:11:02,790
Remember the activations for this number?

173
00:11:02,790 --> 00:11:03,750
We're getting it for each layer.

174
00:11:04,140 --> 00:11:10,080
So the activations for this here can take a look at the first one is going to correspond to the output

175
00:11:10,080 --> 00:11:11,760
of the first convolutional layer.

176
00:11:12,180 --> 00:11:15,600
Remember, I set it up and is twenty six by twenty six and we have 32 filters.

177
00:11:16,080 --> 00:11:17,760
That's exactly what we expect to get.

178
00:11:18,600 --> 00:11:26,010
So what we're effectively doing here, we're just decomposing to CNN and its different layers and extracting

179
00:11:26,010 --> 00:11:30,240
the outputs midway as we forward propagate image truth.

180
00:11:30,930 --> 00:11:36,750
Next, we can look at a second because remember, we got two activation two types of activations types

181
00:11:36,750 --> 00:11:44,250
I should say the two groups or containers of activations, and each container contains this and that

182
00:11:44,580 --> 00:11:46,190
this is the second activation layer now.

183
00:11:46,200 --> 00:11:47,490
So let's take a look at it here.

184
00:11:48,150 --> 00:11:51,450
And just to recap, this is a good v knowledge.

185
00:11:51,810 --> 00:11:53,000
This is a model summary here.

186
00:11:53,010 --> 00:11:59,310
So what we what we've done, we basically extracted to filter activations freedom about feature map

187
00:11:59,310 --> 00:12:01,890
activations from the first layer here.

188
00:12:01,900 --> 00:12:06,450
That's why the output shape is that and this is a second one, and that's why the output shape is that.

189
00:12:06,810 --> 00:12:07,620
So hopefully.

190
00:12:08,230 --> 00:12:09,040
You're still with me.

191
00:12:09,310 --> 00:12:10,900
And then what do we do now?

192
00:12:11,470 --> 00:12:14,140
Let's take a look at the Fort Collins filter.

193
00:12:14,440 --> 00:12:21,550
Actually, I should say the fort to pick him up there should be a little bit better.

194
00:12:23,170 --> 00:12:27,630
And so we do that in the extreme because remember, it's sort of index zero one two.

195
00:12:27,640 --> 00:12:29,350
So the fourth one would be the tree.

196
00:12:30,130 --> 00:12:31,930
And basically, that's we just plotted.

197
00:12:31,930 --> 00:12:33,580
We plot everything had this one.

198
00:12:33,910 --> 00:12:35,560
This value isn't used.

199
00:12:36,010 --> 00:12:41,110
So we just use this one and we use the various color mapping.

200
00:12:41,290 --> 00:12:46,510
We can use a grey or one of the many of the others, but they're just as nice for this illustration.

201
00:12:47,770 --> 00:12:49,300
And there there we go.

202
00:12:49,660 --> 00:12:51,100
So what are we looking at?

203
00:12:51,370 --> 00:12:58,140
Well, this basically is the future map to the Fort Feature map in this layer.

204
00:12:58,750 --> 00:13:03,010
The first activation Cornelia, this see, it's 26 by 26.

205
00:13:03,010 --> 00:13:04,630
That's what that's what we're visualizing.

206
00:13:05,290 --> 00:13:09,010
This is the output of its of that feature map from that filter.

207
00:13:09,340 --> 00:13:10,330
So that's pretty cool.

208
00:13:11,050 --> 00:13:17,890
So we can take that same this little bit of code here that we've used and create a little loop or function

209
00:13:18,280 --> 00:13:26,770
that loops through all of the future maps and plots in a subplot or even a subplot grid, I should say.

210
00:13:27,370 --> 00:13:33,010
So let's run that function and run the code and we'll get a nice feature.

211
00:13:33,010 --> 00:13:35,630
Map activations, hopefully soon.

212
00:13:35,660 --> 00:13:36,340
Yeah, there we go.

213
00:13:36,940 --> 00:13:41,790
So you can see these are the feature maps from the first conflict.

214
00:13:42,010 --> 00:13:44,960
The outputs of them, that's when we input an image.

215
00:13:45,010 --> 00:13:45,370
Six.

216
00:13:45,370 --> 00:13:48,040
You can try different inputs to see what's what's going on.

217
00:13:48,580 --> 00:13:54,070
So you can see what the brighter with a more the brightness of these colors you see here.

218
00:13:54,070 --> 00:13:54,760
Like this one?

219
00:13:55,180 --> 00:14:00,130
This one is the most activated or perhaps one of these tree here or this one on top here.

220
00:14:00,700 --> 00:14:04,360
Two filters that correspond that activated the most upon seeing the number six.

221
00:14:05,350 --> 00:14:07,270
So hopefully this lesson was useful.

222
00:14:07,270 --> 00:14:08,530
I know it's a bit confusing.

223
00:14:08,920 --> 00:14:14,440
I don't want you to get too confused right now because this type of stuff isn't isn't a focus of the

224
00:14:14,440 --> 00:14:14,890
course.

225
00:14:15,430 --> 00:14:20,740
The focus of these lessons is to allow you to visualize what the CNN's are doing internally.

226
00:14:21,100 --> 00:14:25,690
This is also good for research students who want to have a deeper understanding of what their model

227
00:14:25,690 --> 00:14:29,920
is actually doing or what their filters are doing, so I left it in for that purpose.

228
00:14:30,340 --> 00:14:35,950
And but however, as a practitioner, you may not need to do this as as much unless you are trying to

229
00:14:35,950 --> 00:14:37,300
troubleshoot an issue.

230
00:14:37,630 --> 00:14:39,760
So we'll stop there for now.

231
00:14:39,760 --> 00:14:44,620
And then the next section, we'll take a look at filter and class maximization.

232
00:14:44,830 --> 00:14:47,270
So I'll see you in that section.

233
00:14:47,320 --> 00:14:47,770
Thank you.