‫So once we have trained our model I told you that we can quantify the quality of trade of our model
‫using the means squared error term which is given by this formula.
‫It is basically residual Tamal squares divided by n if the predictor responses are very close to the
‫observations.
‫The means could at it will be small
‫but if we find out mean square error on the same data which we used to bring in our model it is called
‫training MSE
‫training it it is not something we are interested in.
‫We are interested in the accuracy of the predictions when we apply our method to previously unseen test
‫data.
‫For example I am predicting house price.
‫I don't really care how well our method predicts house price of previously completed transactions.
‫I get about how well will it predict the whole place of the future transactions similarly if I want
‫to predict the risk of a particular disease in different individuals.
‫I want to do it for future patients and not for the ones I already know the outcome for.
‫So what we're going to do is we are going to split our data into two parts.
‫One will be called training set.
‫This will be used to train the model and the other part will be called The Best said.
‫This will be the unseen data and it will be used to assess the accuracy of our model so mathematically.
‫I have these and pairs of observations X1 by one x2 y2 to extend way in these will be part of my training
‫data and I will use them to train my model.
‫Once I have used them I will have identified the functional form of play that is the ethics.
‫Now I will use previously unseen set of observations x 0 y 0 these observations will come from our best
‫said and I will try to find out the best error which is given by this formula.
‫So test means good.
‫It is average of squared of difference between the predicted value of y to the actual value of y on
‫the given test data so for different types of models.
‫I will compare the value of this test error and then to let the model with least tested it.
‫I hope you understand the idea behind having test entering data basically.
‫We have training data and corresponding training.
‫EDIT body models selected but there is no guarantee that the method with Lewis training error will also
‫have law tested.
‫Roughly speaking many statistical methods specifically estimate we does so that we are able to minimize
‫trainings at data and for such methods all the training so that it will be small but the actual test
‫set it can be quite large.
‫In this graph you can see four different lines this black one is the true function which we want to
‫predict.
‫This orange line is the output of a linear regression model.
‫And these blue and green lines are the result of some other more flexible models.
‫And these small circles that we are seeing are the data points which were used to train the model.
‫You can see as am increasing the flexibility of the model that is I am allowing it to change its shape
‫or its direction.
‫Many times it is touching more points on this graph.
‫So this green card which has highest flexibility is touching the maximum number of points whereas the
‫Orange Girl which has least flexibility is touching very few points you can see after a certain level
‫of flexibility this flexibility is making the curve more wiggly that is it is following the individual
‫data points and not the overall function to the effect of flexibility on training it and tested it can
‫be seen on the graph on the right.
‫You can see that this great plot is of creating error as you keep on increasing the flexibility the
‫training that keeps coming down that is the model will be fitting or will be touching a lot of these
‫sample points but after a certain point the test data which is given by this red go it start increasing
‫with the increasing flexibility.
‫You can see this orange point is the test and train it for this orange glow which is the trade lingo.
‫This blue point is for the blue go which is approximating the true function very closely and this green
‫point is for the green go which is very flexible.
‫It has low training error then the blue go below that is putting the point more closely but it has high
‫test error because it is not approximating the true function so we want to identify this blue point
‫where we get the minimum tested it.
‫Now there are several techniques to split the data into training and test so that we can find this minimum
‫point
‫so we are going to discuss the three most popular techniques.
‫First is called validation set approach.
‫Second is leave one out cross validation and the third one is capable cross validation.
‫The first technique which is validation set approach is the simplest approach in this matter.
‫We will randomly divide the data into two parts a training set and a tested the model will be fitted
‫on the training set and once the model is trained the error for the test it will be calculated to estimate
‫detested we usually do a split of 80 20 that is we use 80 percent of the data for training purposes
‫and 20 percent of the data for testing purposes will be running this approach in our software package.
‫In a separate video there are basically two limitations of this approach.
‫One is that part of the available data will not be used for training and as we know the more data we
‫use during training better will be the performance of the more than So if we keep some data for testing
‫the train model will not be as good.
‫And if you have limited number of observations your training will be severely impacted.
‫Secondly the test error can be highly variable depending on which observation is selected for training
‫and which observation is selected for testing.
‫So to handle these two issues there are these two alternative approaches in the lead one out cross validation
‫so both will and observations will keep the first observation for testing purposes and run the motor
‫on the remaining and minus one of the reasons.
‫Then we will keep the second observation for testing purposes and run the model on the remaining and
‫minus one observations and we will run this model.
‫And times where every time will keep one observation for testing and the other and minus one for training
‫and will take the average of the error on each of these testing observations.
‫So since we need to run this model in times this method can be computationally expensive.
‫An alternative to this leave one out cross validation is the key for cross validation in this.
‫We will divide the data into case it and then we will train the day down K minus one set and use the
‫get set for testing purposes.
‫You can see that leave a note cross validation is a special case of capable validation if you have gays
‫equal to N then give all validation and leave one note.
‫Cross our relation are the same thing so we will not be covering these two techniques in the software
‫package will only be done in the validation set approach in our software package.