1 00:00:00,600 --> 00:00:06,630 So in this lesson, what we're going to do, we're going to load the CNN amnesty model that we've trained 2 00:00:06,630 --> 00:00:11,280 in the previous lesson and what we'll do, we'll actually analyze its performance. 3 00:00:11,280 --> 00:00:16,680 We'll take a look and see which classes it's misclassifying and examine the confusion and classification 4 00:00:16,680 --> 00:00:17,220 reports. 5 00:00:17,760 --> 00:00:19,590 So let's open this notebook here. 6 00:00:19,680 --> 00:00:20,580 Not arbitrary. 7 00:00:21,150 --> 00:00:23,130 And you can see it's split into four blocks. 8 00:00:23,130 --> 00:00:26,460 First book is where we at our previous three and model and our data. 9 00:00:26,880 --> 00:00:29,250 So let's go ahead and do that. 10 00:00:29,610 --> 00:00:34,050 And I've uploaded that model that we trained in the previous lesson to my Google Google Drive. 11 00:00:34,410 --> 00:00:38,880 So now I can access it and download it quickly again within this notebook right here. 12 00:00:39,540 --> 00:00:44,180 Next, we need to load our data set so we load of data said like we did previously. 13 00:00:44,190 --> 00:00:45,420 I'm sorry, we load of model. 14 00:00:45,810 --> 00:00:46,560 Sorry for that. 15 00:00:47,190 --> 00:00:48,780 Like that was downloaded here. 16 00:00:48,810 --> 00:00:51,450 That's end this simple scene and model that we've trained. 17 00:00:51,870 --> 00:00:53,070 So let's load this model. 18 00:00:53,070 --> 00:00:58,440 It's quite simple to just using the load model function from Keros, and that's done now. 19 00:00:58,440 --> 00:01:00,090 We can load our amnesty to set. 20 00:01:00,100 --> 00:01:02,390 So to do this, we import the M. 21 00:01:02,400 --> 00:01:10,380 This object here from TensorFlow Dot or data sets we use include data function inside of it to get our 22 00:01:10,380 --> 00:01:14,400 training data or training labels or test data and or test labels here. 23 00:01:14,430 --> 00:01:15,720 So let's run this code. 24 00:01:16,410 --> 00:01:16,870 There we go. 25 00:01:16,890 --> 00:01:18,480 So we have we have a data set now. 26 00:01:19,170 --> 00:01:26,310 So firstly, the first important thing we're going to do is taking a look at the classes or the input 27 00:01:26,310 --> 00:01:29,120 images that our model misclassified. 28 00:01:29,640 --> 00:01:34,140 So to do that, we basically first have to get our predictions out of our model. 29 00:01:34,530 --> 00:01:35,760 So that's simple. 30 00:01:35,790 --> 00:01:37,230 We've done this before in the adolescent. 31 00:01:38,190 --> 00:01:38,460 What? 32 00:01:38,460 --> 00:01:39,900 I'll do it again, just to be sure. 33 00:01:40,260 --> 00:01:43,200 So we take our next test, that's the test data. 34 00:01:43,650 --> 00:01:48,450 We reshape it by adding on this extra dimension one here, 10000 samples. 35 00:01:48,450 --> 00:01:53,990 We know this previously, but you can always examine if you want to make sure by printing out its desktop 36 00:01:54,000 --> 00:01:55,980 shape, which we do here, actually. 37 00:01:56,400 --> 00:01:59,140 But this is that's printed out before and after. 38 00:01:59,160 --> 00:02:03,660 So you can see the difference that how to reshape function does. 39 00:02:04,290 --> 00:02:11,430 And by using Npr.org. Macs to get to the class with the best probability, by using models or predict 40 00:02:11,430 --> 00:02:14,640 X tests as input, we get our predictions right here. 41 00:02:14,640 --> 00:02:20,610 Stored in this variable 10000 values are stored in a variable and you can see the shape difference here. 42 00:02:20,640 --> 00:02:27,150 We have ten thousand twenty at 28 and 28 by 28 images, and we have the same thing ten thousand twenty 43 00:02:27,150 --> 00:02:30,090 eight by 28 images with the appended dimension here. 44 00:02:30,780 --> 00:02:34,600 So we're now going to do this line is actually a bit irrelevant. 45 00:02:34,620 --> 00:02:35,290 Actually, no, it's not. 46 00:02:35,290 --> 00:02:37,280 Sorry, it's predicting the classes for ten thousand images. 47 00:02:37,290 --> 00:02:37,950 That's fine. 48 00:02:38,550 --> 00:02:40,350 So now what do we do here? 49 00:02:40,680 --> 00:02:46,800 Well, we want to find a way to get which which classes of model misclassified. 50 00:02:47,340 --> 00:02:49,380 We can do that by using number here. 51 00:02:49,770 --> 00:02:56,100 So by getting the absolute value here, if we minus the whiteness minus a prediction, remember if it 52 00:02:56,100 --> 00:03:03,390 was white us for a certain class like this in first class whiteness was five and the prediction was 53 00:03:03,390 --> 00:03:04,560 five, it would be a zero. 54 00:03:04,590 --> 00:03:05,520 That's that's fine. 55 00:03:05,910 --> 00:03:10,380 However, when there's a difference like that C, it predicted a six instead of a five. 56 00:03:10,740 --> 00:03:17,400 We'll get a one here and that this is this is no returns, basically a Boolean where we have zeros and 57 00:03:17,940 --> 00:03:21,960 the positive digits because it's an absolute function here. 58 00:03:22,650 --> 00:03:28,410 So we can use that known that stores, we still have the new results and then we use that using NP or 59 00:03:28,410 --> 00:03:29,160 non-zero. 60 00:03:29,610 --> 00:03:32,040 So we just check for all the non-zero values said. 61 00:03:32,040 --> 00:03:36,810 This gives us to bully and I should say, and now we have the misclassified indices here. 62 00:03:37,290 --> 00:03:43,170 So by printing, running this and printing this out, we can see all of these indices here were misclassified. 63 00:03:43,210 --> 00:03:51,060 So the it's technically the ninth index in the test dataset was misclassified the attitude to 60 second, 64 00:03:51,060 --> 00:03:52,530 90 second and so on. 65 00:03:52,860 --> 00:03:55,080 Consider a number of misclassified ones here. 66 00:03:55,390 --> 00:03:58,470 In fact, let's actually print number the length of that. 67 00:03:58,980 --> 00:04:01,280 So we can see sorry. 68 00:04:06,270 --> 00:04:10,830 Run it again and you'll see linked boats one. 69 00:04:11,130 --> 00:04:16,920 And that's because this is actually an array within a very sorry, but so it's 506 values, it got wrong. 70 00:04:17,310 --> 00:04:20,750 So out of 10000 predictions, it got 500 and six wrong. 71 00:04:23,320 --> 00:04:29,080 So now let's take a look at visualizing the actual images that our model in this classified. 72 00:04:29,470 --> 00:04:34,870 This is a very useful technique in diagnosing what your model is doing wrong because also sometimes 73 00:04:34,870 --> 00:04:37,120 they can have mislabeled the actual data. 74 00:04:37,450 --> 00:04:43,450 So it's always a good way to assess what your model is predicting wrong by visually comparing the predictions 75 00:04:43,450 --> 00:04:45,160 to the actual inputs. 76 00:04:45,700 --> 00:04:46,840 So let's take a look. 77 00:04:46,870 --> 00:04:50,650 So this function, his just initial function, it's nothing special. 78 00:04:50,650 --> 00:04:53,440 He would have seen this before in the previous lessons. 79 00:04:53,980 --> 00:05:01,300 And this here's a block of code that plots the input image against a prediction so we can see them side 80 00:05:01,300 --> 00:05:01,840 by side. 81 00:05:02,350 --> 00:05:04,470 So let's run this block of code. 82 00:05:04,480 --> 00:05:12,430 We're going to do 10 random the 10, the 10 random, I think 10 of the first, yes, 10 of the first 83 00:05:12,430 --> 00:05:13,930 two misclassified indices here. 84 00:05:14,320 --> 00:05:19,660 So we're going to predict we're going to get the predictions and the images for the first 10 of these 85 00:05:19,660 --> 00:05:20,110 values. 86 00:05:20,110 --> 00:05:22,360 Here is indices who indexes. 87 00:05:23,320 --> 00:05:27,880 So you can see here this one was a five hour model, predicted a six. 88 00:05:28,330 --> 00:05:32,020 Understandably so, though it does seem to be more four five than the six to me. 89 00:05:32,710 --> 00:05:33,970 This one is definitely wrong. 90 00:05:34,000 --> 00:05:36,280 This one is a four letter, predicted a six. 91 00:05:36,490 --> 00:05:37,510 This one is a nine. 92 00:05:37,630 --> 00:05:38,620 It predicted four. 93 00:05:39,010 --> 00:05:40,940 Again, a nine mistaken for four. 94 00:05:41,650 --> 00:05:43,960 This one looks like a two and predicted the nine. 95 00:05:44,530 --> 00:05:46,140 So you can see the model isn't doing too well. 96 00:05:46,150 --> 00:05:48,610 This one is clearly and it predicted a seven. 97 00:05:49,120 --> 00:05:51,430 This one's clearly a nine and predicted it. 98 00:05:51,640 --> 00:05:54,280 This one is clearly a four and I predicted of two. 99 00:05:54,640 --> 00:06:00,310 So you can see at ninety five percent accuracy on model still isn't doing quite well on these classes. 100 00:06:00,700 --> 00:06:01,900 So we need to improve this. 101 00:06:02,080 --> 00:06:05,470 But that's for a little lesson for no. 102 00:06:05,560 --> 00:06:08,860 Let's take a look at another more elegant way to plot this. 103 00:06:09,250 --> 00:06:17,560 So by using some of my plot lib functions, we can just basically extract the values here. 104 00:06:18,310 --> 00:06:23,590 And there's a mistake in this code only to fix this because the lesson here, I don't fix this mistake. 105 00:06:23,830 --> 00:06:24,640 Sorry about that guy. 106 00:06:25,840 --> 00:06:28,180 So, yes, a fixed a mistake and what we're going to do. 107 00:06:28,720 --> 00:06:34,400 We're going to just print some subplots here, five by five subplots where we just use a title to show 108 00:06:34,450 --> 00:06:37,870 a prediction of the class and the original class and then show the image below. 109 00:06:38,350 --> 00:06:42,310 So let's run this code and take a look at the outputs. 110 00:06:43,570 --> 00:06:44,050 There we go. 111 00:06:44,080 --> 00:06:49,180 So this is a nicer way to actually visualize it as well, and you actually get the actual ground shoot 112 00:06:49,180 --> 00:06:51,000 levels and the predicted levels here as well. 113 00:06:51,010 --> 00:06:54,250 So you don't actually have to infer it or view them side by side. 114 00:06:54,730 --> 00:06:59,410 And as you can see, the model is doing not too good on these losses, is it? 115 00:06:59,440 --> 00:07:03,820 Well, these are obviously will obviously have the impression it's not performing too well because these 116 00:07:03,820 --> 00:07:05,080 are the classes we got wrong. 117 00:07:05,530 --> 00:07:07,290 We're not looking at the classes that got it right. 118 00:07:07,300 --> 00:07:08,710 Otherwise everything would look fine. 119 00:07:09,280 --> 00:07:13,630 But it's good to analyze this so we can see a model definitely does have some weaknesses here. 120 00:07:14,320 --> 00:07:15,370 So we'll stop there. 121 00:07:15,430 --> 00:07:18,820 And next, we'll take a look at creating our confusion matrix. 122 00:07:19,240 --> 00:07:21,100 So I'll see you in the next video lesson. 123 00:07:21,220 --> 00:07:21,670 Thank you.