WEBVTT 00:01.310 --> 00:07.700 This session, we will discuss about different performance metrics that are present in case of classification, 00:08.330 --> 00:16.160 so these are the performance metrics which we will be using for logistic regression and also for any 00:16.160 --> 00:17.720 classification problem. 00:18.230 --> 00:20.980 So let us have a look at those. 00:21.260 --> 00:25.940 So the first one is a confusion matrix. 00:26.120 --> 00:35.150 So a confusion matrix is a combination of the actual values versus the predicted values. 00:36.160 --> 00:47.200 It is all table for predicted and the results that are revealed in nature, so let us consider this 00:47.200 --> 00:50.110 particular confusion matrix. 00:50.350 --> 00:53.740 So here we have a true positive. 00:55.420 --> 01:04.990 BP, which means to positive, then we have Effi, which is false positive, Ifan is false negative 01:05.350 --> 01:08.680 and the end is all negative. 01:08.950 --> 01:11.950 Now, how is it declared how the names are given? 01:12.430 --> 01:20.230 So whatever value that we have predicted is coming at the end of the name. 01:21.340 --> 01:30.640 So if we have predicted something to be negative, then that is it is either false positive or false 01:30.640 --> 01:33.010 negative or negative. 01:33.910 --> 01:43.450 Similarly, if the value has been predicted as positive, then it can be either positive or false positive. 01:44.810 --> 01:47.290 Now, what does this true and false mean? 01:47.740 --> 01:54.780 So let us see that we have predicted something to be positive when the prediction is positive. 01:54.800 --> 01:57.900 It means that that prediction will be positive. 01:57.920 --> 01:59.630 The second word will be positive. 02:00.080 --> 02:03.730 Now, the actual value is also positive. 02:04.040 --> 02:07.460 So the predicted value and the actual value match. 02:07.790 --> 02:10.580 This means that our prediction is true. 02:11.360 --> 02:13.970 That is why it gets the name proof positive. 02:14.420 --> 02:18.490 That is, we have predicted the volume value to be positive. 02:18.950 --> 02:20.780 And this prediction is true. 02:21.830 --> 02:24.390 This means that it is a true positive. 02:25.070 --> 02:33.800 Similarly here we have predicted the value to be positive, but actually the value was negative. 02:33.800 --> 02:37.880 Hence the prediction which we have made is a false prediction. 02:38.700 --> 02:41.410 The prediction which we have made is not correct. 02:41.750 --> 02:44.440 So it is called false positive. 02:44.750 --> 02:48.560 The positive, which we have predicted is a false prediction. 02:50.160 --> 02:56.430 Next, we have false negative, this means that we have made a prediction that the value has to be negative, 02:56.610 --> 02:58.420 but actually it is positive. 02:58.620 --> 03:00.440 So it is a false prediction. 03:00.660 --> 03:03.790 Hence, we have predicted a false negative. 03:03.930 --> 03:06.190 That is it is not a negative. 03:06.210 --> 03:07.680 It is a false prediction. 03:07.980 --> 03:14.850 It is a false negative value, which we have predicted next is true negative, which means that we have 03:15.030 --> 03:16.860 predicted value to be negative. 03:17.310 --> 03:19.020 And it is a prediction. 03:19.020 --> 03:20.350 It is a correct prediction. 03:20.700 --> 03:23.790 That is why it is called a true negative. 03:25.660 --> 03:33.210 Next, we have these different metrics, which we have, so the first metric is accuracy. 03:34.940 --> 03:40.730 Accuracy means the correctly predicted values all out of all the values. 03:41.030 --> 03:42.920 So what are the correct predictions? 03:43.220 --> 03:47.730 Correct predictions will be true, positive and negative. 03:48.200 --> 03:50.980 That is the predictions which we have made correctly. 03:51.290 --> 03:53.590 That will be the proof positive and negative. 03:54.020 --> 03:54.410 Right. 03:54.590 --> 04:02.570 So accuracy is proof positive, plus true negative, divided by the total value, all the values that 04:02.570 --> 04:04.340 this positive plus negative. 04:04.940 --> 04:06.250 So this is accuracy. 04:06.500 --> 04:14.150 Accuracy, which means that the correct prediction out of all the values, what value we have predicted 04:14.150 --> 04:18.530 correctly, that is accuracy makes this precision. 04:20.300 --> 04:29.510 Precision is that out of all the predictions which we have made, these predictions are positive, which 04:30.050 --> 04:32.210 values are actually positive? 04:32.840 --> 04:40.420 That is, out of all the positive which we have predicted, what values are correctly predicted, so 04:40.490 --> 04:42.050 correctly predicted? 04:42.290 --> 04:49.160 Positive will be the positive means that these are predicted positive and these are actually positive. 04:49.940 --> 04:57.470 So this means it is proof positive plus false positive, which means that which values we have predicted 04:57.470 --> 04:58.190 as positive. 04:58.190 --> 05:04.640 So out of all the values which we have predicted positive, which are actually positive. 05:06.770 --> 05:15.260 Then we have sensitivity and the equal, which means that out of all the actual positive values, how 05:15.260 --> 05:21.570 many positive values are predicted that that is out of all the positive values? 05:21.680 --> 05:24.500 So here we have these all the positive values. 05:24.500 --> 05:31.550 So proof positive and false negative are the positive values, are the positive values, the actual 05:31.550 --> 05:35.150 positive cases through positive and false negative. 05:36.120 --> 05:45.540 So these of the actual positive value, the one which we have predicted as to be positive and it is 05:45.540 --> 05:52.650 actually positive, so we divide this correctly predicted positive value divided by the actual positive 05:52.650 --> 05:53.130 values. 05:53.400 --> 06:00.510 This is quite recall how good we are at recalling the positive cases. 06:01.350 --> 06:06.810 Out of all the positive cases, how well can we remember the positive cases? 06:06.990 --> 06:08.240 It's called REPL. 06:10.080 --> 06:12.220 So it is all true positive rate. 06:12.240 --> 06:20.610 It is also known as proof positive rate and precision means that out of all the positive, which we 06:20.610 --> 06:23.580 have predicted positive, which are actually positive. 06:24.000 --> 06:29.760 So these are used in two different ways when which is used that to something which we will discuss in 06:29.760 --> 06:32.490 something, then we have specificity. 06:33.210 --> 06:37.990 So specificity is the opposite of a recall. 06:38.280 --> 06:40.200 So it is correctly predicted. 06:40.200 --> 06:49.700 Negatives out of all the negatives and sensitivity was correctly predicted positive out of all the positives. 06:49.980 --> 06:52.800 So specificity is correctly predicted. 06:52.800 --> 06:54.140 Negatives out of. 06:55.800 --> 07:02.160 Out of all the negatives, so it is too negative, divided by a false positive plus two negatives. 07:04.500 --> 07:08.140 So which metrics should we use where? 07:08.160 --> 07:15.770 So this is how we will decide which metric we should use so we cannot decide upon a single metric. 07:16.140 --> 07:20.130 Hence we try to use a combination of metrics like decision. 07:20.130 --> 07:27.150 And so precision is proof positive, divided by two positive plus false positives. 07:27.150 --> 07:31.220 That is out of the values which we have predicted to be positive. 07:31.560 --> 07:35.890 Which one not actually brought up correctly predicted were positives. 07:36.120 --> 07:44.550 So it is decision and how many positives which we have predicted out of all the actual positive values 07:44.550 --> 07:45.250 is really good. 07:45.750 --> 07:55.530 So let's say we are predicting cancer amongst patients and only if we are absolutely sure means we have 07:55.530 --> 07:56.880 a high value. 07:57.000 --> 08:04.040 So we will have to provide value in case of to someone that they have cancer or not. 08:04.470 --> 08:09.330 So we will let us say we want to tell someone if they have cancer or not. 08:09.630 --> 08:13.230 And for that we want to have a high cost of value. 08:13.680 --> 08:17.820 That is, we want to be 100 percent sure. 08:17.820 --> 08:24.730 We want to have very, very, very sure and to tell someone that they have cancer, otherwise we won't 08:24.730 --> 08:25.150 build them. 08:25.530 --> 08:28.770 So this is the case where. 08:29.890 --> 08:34.790 We are fine on missing out on a few actual positives here. 08:34.810 --> 08:41.950 We want to be assured that we can miss out on an actual positive case, but we don't want to miss out. 08:42.490 --> 08:46.680 We don't want to tell someone who doesn't have cancer that they have cancer. 08:48.110 --> 08:56.030 So instead of telling a negative person, a person who does not have cancer, telling them that you 08:56.030 --> 09:01.580 have cancer, it is better to lose on a few cases who actually have cancer. 09:02.210 --> 09:11.300 So this is the case when we want to have high precision and risk, but that is of whoever we are telling 09:11.300 --> 09:14.950 them they have cancer, they should actually have cancer. 09:14.960 --> 09:20.270 And we should not tell someone who does not have cancer that they have cancer. 09:20.300 --> 09:21.740 So this is the first case. 09:22.490 --> 09:26.360 The second case is when we want to have a look at the value. 09:26.360 --> 09:37.040 That is we are fine to tell someone who does not have a cancer of telling them that they have a cancer 09:37.220 --> 09:43.960 instead of losing out on someone who actually has cancer and not giving them medication. 09:44.510 --> 09:51.110 So we don't want to have a case when all actual cancer patient does not get to know that they have cancer. 09:51.380 --> 09:54.260 So here we want to have a very good recall rate. 09:54.590 --> 09:58.130 So here we want to have a high quality, low precision. 09:59.520 --> 10:09.180 So hence, as we lower the value, the value of the call will increase and precision will decrease. 10:09.330 --> 10:19.020 So here you can see that when we decrease the value, the decline will increase and precision will actually 10:19.050 --> 10:19.620 decrease. 10:20.430 --> 10:29.070 So if we want to miss out on any actual positive cases, if it is fine for missing out on a few actual 10:29.070 --> 10:31.980 positive case, then we will have a high precision. 10:32.130 --> 10:38.790 But if we don't want to miss out on any positive cases, then we will have a high value. 10:40.360 --> 10:46.530 It is very important to understand this, because these are the metrics which we will be using and are 10:46.540 --> 10:47.560 very important. 10:48.550 --> 10:58.420 So let us have a case where we have a five one matter, what is a five, four, five one error or false 10:58.420 --> 11:08.620 positive rate is when we are finding out the number of false positives out of all the. 11:10.100 --> 11:11.300 Negative cases. 11:13.080 --> 11:15.000 So this is also about specificity. 11:16.720 --> 11:24.910 Now, let us see that data is balanced, and in that case, we will be calculating the accuracy rate 11:25.240 --> 11:32.110 and the formula will be true, positive plus negative divided by two positive false negative plus false 11:32.590 --> 11:34.130 positive plus two negative. 11:34.390 --> 11:40.930 So for a balanced dataset, we will be using accuracy rate as the metric. 11:41.530 --> 11:50.470 But for any imbalanced dataset, we will choose from the equal precision one and decide on the basis 11:50.470 --> 11:56.320 of these criteria if we want to have a high precision or we want to have a high equal value. 11:57.570 --> 12:05.910 Now for a false positive, if in case false positive is important, then we will go towards precision 12:06.150 --> 12:11.220 and in case false negative is important, then we will build the world's record. 12:11.850 --> 12:19.630 Now, that is another metric called if we test for this and the score is calculated by one plus the 12:20.250 --> 12:30.120 in two times precision, including one divided by the Beita into two in the recall divided plus precision 12:30.120 --> 12:35.910 value now in gives false, negative and false positive. 12:35.920 --> 12:37.060 Both are important. 12:37.650 --> 12:42.150 That is, we don't want to lose on either false negative or false positive. 12:42.450 --> 12:45.150 Then we will give the better value as one. 12:46.620 --> 12:50.860 But in this, we want to reduce the false positive rate. 12:51.120 --> 12:58.920 We don't want to have high false positive values, that is, we don't want to tell someone that they 12:58.920 --> 12:59.700 have cancer. 12:59.700 --> 13:05.010 When they don't really have cancer, then we will decrease the VW. 13:07.030 --> 13:14.650 And in this false negative is important, that is we don't want to tell someone that they do not have 13:14.650 --> 13:22.330 cancer when they actually have cancer so that they don't really miss out on the treatment and get the 13:22.330 --> 13:24.630 required treatment as soon as possible. 13:25.000 --> 13:27.820 In that case, we will increase the B.W.. 13:27.940 --> 13:34.900 So based on these ideas, you can actually decide if you want to increase the value or do you want to 13:34.900 --> 13:42.010 decrease the value and will you be considering the high precision all of you will be considering, Heidi. 13:42.520 --> 13:51.310 So these are a few points which you might want to keep on Notthoff and try practicing again and again 13:51.490 --> 13:58.270 so that after some period of time you will get a hold of which metrics do you actually need to have 13:58.270 --> 13:58.810 a look at? 14:00.280 --> 14:10.150 So let us have a look at a few more details, so let us say we have this model, one which has a precision 14:10.150 --> 14:16.570 zero point five under the model, has a precision value, zero point nine, and that is a model T because 14:16.570 --> 14:18.040 of precision zero point zero. 14:18.700 --> 14:20.670 Now, here we have model one. 14:20.680 --> 14:25.540 Rules of equal value is zero point for model two has a very low equal value. 14:25.540 --> 14:27.010 That is zero point one. 14:27.280 --> 14:30.820 Model C has a very high legal value. 14:31.150 --> 14:32.020 That is one. 14:32.290 --> 14:35.740 That means that it will never miss out on any. 14:37.040 --> 14:43.390 Positive cases and having a high precision means that it will never miss out on a negative case. 14:43.670 --> 14:49.790 Now, the average of these comes out to be zero point four five zero point five and to point five one. 14:51.040 --> 14:59.640 Why is we have a look at the F1 school, you can see for model one, the F1 score is zero point four 14:59.710 --> 15:05.710 for models to drive one school to zero point one need for more than three, though, if one score is 15:05.710 --> 15:08.290 zero point zero three nine two. 15:09.800 --> 15:19.700 So here you can see that if one school will actually give a better view of the entire precision and 15:19.700 --> 15:27.100 recall metric, because then if one scored when either of the precision and recall is very low. 15:27.260 --> 15:32.900 So in case of Madrid, you can see that the equal value is very low, while in case of model three, 15:32.900 --> 15:34.680 the value of precision is very low. 15:34.970 --> 15:39.740 So for both of these cases, the F1 score comes out to be low value. 15:39.920 --> 15:42.470 These both of these values are low. 15:43.680 --> 15:53.760 While although precision and recall are not very high for model one, but it does not really allow the 15:53.760 --> 15:58.230 run of the value to be very low, hence if one scored is higher for this. 15:58.800 --> 16:05.120 So you can see that F1 score is a better metric when we want to have a balanced precision and vehicle. 16:09.220 --> 16:18.640 So next is a U.S. school, so let us try to understand what a U.S. school is, the U.S. scored indicates 16:18.640 --> 16:25.240 how well the probabilities from the positive classes are separated from the negative classes. 16:26.490 --> 16:27.000 So. 16:28.250 --> 16:37.820 It tells us how much a model is capable of distinguishing between different classes from a U.S. school 16:37.860 --> 16:46.750 perspective from this school, which we have, we want to choose a particular school which has the most 16:46.760 --> 16:49.000 amount of area under this. 16:50.120 --> 16:52.310 And then we have a look at this. 16:52.500 --> 17:00.230 U.S. scored a model which has a threshold value, which is like we have different values plotted for 17:00.230 --> 17:01.520 different social values. 17:02.270 --> 17:06.740 Now, the closer it is to this diagonal line. 17:08.400 --> 17:14.490 The closer it is to this diagonal line, the poorer the performance of, the more the less we want the 17:14.490 --> 17:22.440 model which has the value closer to this top left corner of. 17:23.860 --> 17:32.230 So hence, we can decide upon either this particular threshold value or this threshold value, something 17:32.230 --> 17:34.780 which is actually closer to this particular point. 17:35.560 --> 17:42.040 So the higher the you see, the better the model is at predicting zeros as zeros and ones as ones. 17:42.430 --> 17:44.660 The sensitivity which we have here. 17:45.550 --> 17:47.740 This is also called true positive rate. 17:48.040 --> 17:53.890 And one minus specificity is known as the false positive rate. 17:54.280 --> 18:01.820 And these are the formulas which we have already discussed of of specificity and sensitivity. 18:02.170 --> 18:08.780 So the main task here is to select a particular goal, which is near to this top left corner. 18:09.010 --> 18:16.030 And what we can do is we can plot different charts which are present for different threshold values. 18:16.060 --> 18:19.820 So these are different threshold values for which we are plotting this. 18:20.050 --> 18:26.470 So here we have special value, zero point zero one zero point to zero point three zero point five seven 18:26.470 --> 18:28.480 zero point seven six point nine five. 18:28.690 --> 18:31.210 So all the threshold values are present here. 18:32.410 --> 18:39.830 So out of all of these special values, we want to select a particular threshold value for which this 18:39.830 --> 18:46.170 score is closer to this particular corner so we can decide accordingly. 18:48.790 --> 18:58.030 Now, let us have more information so it is used in the declassification problem, it is suppose we 18:58.030 --> 19:00.580 are implementing a logistic regression. 19:00.580 --> 19:04.030 I suppose the model has predicted some probabilities. 19:04.300 --> 19:08.430 Then we need to decide what is a threshold value and what is a value. 19:08.650 --> 19:15.790 So out of these values, which we have here, out of these values, we want to decide which value should 19:15.790 --> 19:16.340 we select. 19:16.810 --> 19:21.710 So the threshold value or value is decided by the problem statement. 19:22.000 --> 19:29.710 Now, for example, if we need higher false positive ID or law to positively based on that, we can 19:30.070 --> 19:31.540 play with the cutoff value. 19:32.670 --> 19:41.510 Next, we have the sequel, so the Odyssey for calculating the movie required both false positive rate 19:41.550 --> 19:42.530 and positive. 19:42.570 --> 19:49.770 Right now, based on the threshold value, we get the positive and false positive rate that is this 19:49.770 --> 19:50.820 rate and this rate. 19:50.820 --> 19:54.400 So I can select any threshold value legacies, zero point five seven. 19:54.720 --> 19:57.400 So based on this, I will go towards the goal. 19:57.720 --> 20:01.800 So this Goba has a positive rate as zero point six. 20:01.800 --> 20:02.890 I feel positive. 20:03.120 --> 20:08.310 Does zero point for now based on this, I can decide what I actually want to have. 20:09.720 --> 20:14.550 So from this, you can see that here at this particular part of. 20:16.000 --> 20:23.740 I have a higher false positive rate and a higher positive rate, so in that case, I will select this 20:23.740 --> 20:27.490 point to be my cutoff value. 20:27.490 --> 20:32.140 That is zero point seven six can be a good threshold value for this particular. 20:34.640 --> 20:35.090 Now. 20:37.340 --> 20:43.910 It plots a graph joining which point we get the goal of the 80 and the goal is for the U.S. goal. 20:44.180 --> 20:47.270 So 80 under this goal is called the U.S. goal. 20:47.300 --> 20:58.000 So what we can do is we can plot one different U.S. goals and from the different autoclaves and U.S. 20:58.010 --> 20:58.390 goals. 20:58.610 --> 21:03.350 First of all, using different U.S. scores, we can find out which model is better. 21:03.920 --> 21:06.470 So let's say you are comparing different models. 21:07.620 --> 21:10.620 So for each model, you will have a different goal. 21:11.220 --> 21:17.100 So let's say I have one model for which I have a goal of like this and another model for which I have 21:17.100 --> 21:18.090 a goal of like this. 21:18.750 --> 21:25.800 So out of all of these goals, I will select the model which has the highest that this one this one 21:25.800 --> 21:27.930 has the most amount of area under this. 21:28.440 --> 21:28.890 Right. 21:29.310 --> 21:36.960 Next is if I want to select the cutoff, then how I just stated the cutoff has to be the one where the 21:36.960 --> 21:37.830 value of. 21:39.440 --> 21:47.210 False positive rate is less and the true positive rate is high, so we will want to have something like 21:47.840 --> 21:49.250 this particular point. 21:50.400 --> 21:56.520 Or maybe this particular point would be a good decision point where the false positive rate is zero 21:56.520 --> 21:59.030 point four and two positive is zero point six. 21:59.280 --> 22:05.880 So we can have any point selected, which is closer to this particular scenario, and we can then compare 22:05.880 --> 22:12.930 what values we are getting for the bus was for the precision and recall and then compare accordingly 22:12.930 --> 22:15.170 and decide upon the model that you want to select. 22:15.570 --> 22:21.300 But if you want to select upon different models, which we have created, so we will go ahead with the 22:21.420 --> 22:24.670 one which is having the maximum schauder that. 22:26.210 --> 22:34.430 A U.S. school we basically domain how would we are classifying and predicting zeros and zeros and ones 22:34.430 --> 22:37.160 as one, so that is what we are looking out for here. 22:37.490 --> 22:42.290 So this is about the matrix, which we have for classification. 22:42.500 --> 22:51.380 In the next session, we will learn about the implementation of a logistic regression. 22:51.710 --> 22:58.700 And I will have another session in which I will explain about different metrics of a job as intended 22:59.720 --> 23:03.020 so that you can understand more from them.