1 00:00:04,090 --> 00:00:08,590 We will now take a closer look at the performance and training of the common filter. 2 00:00:09,870 --> 00:00:15,650 How can we tell if the filter is working or tuned correctly and there are two major ways of doing this, 3 00:00:16,050 --> 00:00:21,660 the first way is to compare the comment through the state estimates with the true states or estimates 4 00:00:21,660 --> 00:00:24,570 of what the true state are in the exercise. 5 00:00:24,650 --> 00:00:29,460 So far, we've been using a simulation to attain the truth for comparison and comparing that to the 6 00:00:29,460 --> 00:00:30,630 common field of performance. 7 00:00:31,560 --> 00:00:36,030 But just because we have a simulation that does not necessarily mean that the filter will work in real 8 00:00:36,030 --> 00:00:42,000 life, there might be some differences between the simulation and the actual real life physics and dynamics. 9 00:00:42,830 --> 00:00:46,610 So another way we can do is to use an external reference system. 10 00:00:47,040 --> 00:00:52,140 So on the true system, in real life, we can have another way of estimating what the state is and we 11 00:00:52,140 --> 00:00:55,140 can compare those estimates with the common field of state estimates. 12 00:00:56,240 --> 00:01:00,540 So basically, what we want to do is we want to look at the state estimation error. 13 00:01:00,830 --> 00:01:05,300 So basically, if we know the true state, we can come from the simulation or some external reference 14 00:01:05,300 --> 00:01:08,330 system and we compare it to the common field of state estimates. 15 00:01:08,660 --> 00:01:11,770 We want the state estimation error to be as small as possible. 16 00:01:11,780 --> 00:01:14,390 So we want to minimize this or make this is close to zero. 17 00:01:15,170 --> 00:01:21,530 We also want to make sure that the state estimated covariance matrix for all the time steps is also 18 00:01:21,530 --> 00:01:24,080 representative of how good the estimate is. 19 00:01:24,440 --> 00:01:27,580 So we want to make sure that this relationship is true as well. 20 00:01:29,190 --> 00:01:34,800 Now, in some cases, we might not have a good simulation that we can use to generate the truth, or 21 00:01:34,800 --> 00:01:39,510 we might not have an external reference system that we can put on in real life to generate what the 22 00:01:39,510 --> 00:01:40,530 true signal should be. 23 00:01:41,040 --> 00:01:46,860 So in these cases, we have to use the innovation and we can compare the innovation, statistical properties 24 00:01:46,860 --> 00:01:52,290 to work out how well the filter is performing and the main things that we're going to look at with the 25 00:01:52,290 --> 00:01:57,960 innovation, we want to make sure that it has zero mean and that the innovation sequence variants should 26 00:01:57,960 --> 00:02:02,040 be a similar order of magnitude to the predicted innovation covariance. 27 00:02:03,180 --> 00:02:09,270 So we can see on the slide here that we have an example simulation run and we have the innovation sequences 28 00:02:09,270 --> 00:02:14,760 for this for the X and Y position measurements, when we say we have zero, I mean, we want to make 29 00:02:14,760 --> 00:02:20,490 sure that the police lines here, if we send them all up and take the main we want it to be as they 30 00:02:20,490 --> 00:02:21,090 remain. 31 00:02:21,090 --> 00:02:22,270 And the same thing for the Y. 32 00:02:22,830 --> 00:02:29,190 So if the innovation is centered around zero, it means on average, the main error between the predicted 33 00:02:29,190 --> 00:02:31,440 measurement and the actual measurement is zero. 34 00:02:31,800 --> 00:02:35,460 And this is one of the criteria we assume is true for when we're running the computer. 35 00:02:36,000 --> 00:02:40,950 So if we look at the innovation sequence and we can see that the main is zero centered, then there's 36 00:02:40,950 --> 00:02:42,690 a good chance the field is working correctly. 37 00:02:43,650 --> 00:02:47,670 In some cases, you might see the innovation rise up and become biased. 38 00:02:47,670 --> 00:02:50,830 In these cases, the field is not working correctly or it might diverge. 39 00:02:51,840 --> 00:02:56,460 So we always want to make sure that the innovation sequences for all the different components are zero 40 00:02:56,490 --> 00:02:56,880 main. 41 00:02:58,480 --> 00:03:03,250 The next property that we want to look at is the innovation variance, so we want to make sure that 42 00:03:03,250 --> 00:03:09,190 the variance of the innovation sequence is the same order of magnitude as you predicted, covariance 43 00:03:09,190 --> 00:03:10,390 of the innovation. 44 00:03:11,020 --> 00:03:14,300 So the predictive covariance is the from the matrix. 45 00:03:14,300 --> 00:03:22,090 So the matrix for this case has diagonal elements, our sigma squared X and Sigma squared Y, and these 46 00:03:22,090 --> 00:03:24,520 are actually plotted on the graph here at a time history. 47 00:03:24,530 --> 00:03:32,590 So the green lines here are the plus or minus one sigma of Sigma X and here plus or minus one sigma 48 00:03:32,740 --> 00:03:34,160 of stigma Y values. 49 00:03:34,330 --> 00:03:39,550 So this is how we get these green lines and we want to make sure that the innovation sequence shown 50 00:03:39,550 --> 00:03:45,300 here is consistent with our predicted innovation from the filter so we can calculate the statistical 51 00:03:45,310 --> 00:03:50,760 properties of the innovation sequence from the simulation using a simpler formula as shown here. 52 00:03:50,890 --> 00:03:56,260 And we want to make sure that this sigma value here for the whole simulation is basically consistent 53 00:03:56,260 --> 00:03:56,960 with this sigma. 54 00:03:58,180 --> 00:04:03,580 So from the simulation, we've actually calculated the exposition, measuring innovation and wide precision 55 00:04:03,580 --> 00:04:07,810 measurement, innovation, standard deviations to be 11 and 10 for the X and Y. 56 00:04:08,420 --> 00:04:09,640 So that's these values. 57 00:04:09,640 --> 00:04:12,850 Here are the variances of these blue lines here and here. 58 00:04:13,930 --> 00:04:18,270 So we can see for the X value we have a variant of 11 for the innovation sequence. 59 00:04:18,640 --> 00:04:24,600 We can also see that the one sigma values this green line here is basically at eleven as well. 60 00:04:24,610 --> 00:04:26,530 So it's halfway between zero and 20. 61 00:04:27,220 --> 00:04:29,430 The same thing can be said for the Y innovation. 62 00:04:29,440 --> 00:04:34,070 We have got the innovation level of about 10 and we have a standard deviation of that 10. 63 00:04:34,780 --> 00:04:40,270 So these are basically showing that the innovation sequence is consistent with the actual innovation 64 00:04:40,270 --> 00:04:41,870 predicted from the common filter. 65 00:04:42,700 --> 00:04:47,170 So this is showing that the filter is tuned correctly and it's operating very well. 66 00:04:48,430 --> 00:04:53,260 So now that we've had a look at how to tell if a filter is correctly tuned and working, we want to 67 00:04:53,260 --> 00:04:56,060 have a look at how to actually go about tuning the common filter. 68 00:04:56,890 --> 00:05:01,300 We have already covered one of the important parameters of tuning the filter, which are the initial 69 00:05:01,300 --> 00:05:01,990 conditions. 70 00:05:02,440 --> 00:05:07,630 So we want the initial condition X not to be the best guess or assumption about the initial state of 71 00:05:07,630 --> 00:05:08,170 the system. 72 00:05:08,440 --> 00:05:13,090 Now, this comes through some engineering judgments about what assumptions we can make or what value 73 00:05:13,420 --> 00:05:15,950 the initial step we have when we first start the filter. 74 00:05:16,570 --> 00:05:21,540 And again, we can use the delay, the initialization, to correctly initialize the filter from a known 75 00:05:21,550 --> 00:05:22,010 condition. 76 00:05:22,960 --> 00:05:28,240 We also want to make sure that p not sorry, the covariance, the initial covariance is large enough 77 00:05:28,240 --> 00:05:29,470 to encompass the truth. 78 00:05:29,480 --> 00:05:35,470 So we want to make sure that whatever value we have for Paisner is the variance is large enough that 79 00:05:35,470 --> 00:05:39,000 the truth is contained within that amount around our initial condition. 80 00:05:39,280 --> 00:05:39,580 Not. 81 00:05:41,600 --> 00:05:46,220 Now, the main parameters we're going to use to tune the come and filter are the covariance matrices 82 00:05:46,220 --> 00:05:48,860 for the process model noise and sensor model noise. 83 00:05:49,430 --> 00:05:55,670 So looking at the sensor model noise or covariance, this are matrix here can be generated from information 84 00:05:55,670 --> 00:05:58,300 about the properties of the measurements or the sensor. 85 00:05:58,730 --> 00:06:01,300 So we can look at a data sheet of the sensor. 86 00:06:01,310 --> 00:06:06,650 We could run some tests or experiments on a sensor to come up with some statistical properties of what 87 00:06:06,650 --> 00:06:10,490 the noise characteristics are like in the sensor and in the measurements. 88 00:06:10,730 --> 00:06:13,580 And then we encourage that information into the matrix. 89 00:06:14,630 --> 00:06:20,480 So, for example, let's say we had a sensor and had two measurements, so it said one and said two. 90 00:06:20,780 --> 00:06:28,190 And the measurements are plus or minus a around the truth or plus minus B, the truth for the second 91 00:06:28,190 --> 00:06:28,710 component. 92 00:06:29,330 --> 00:06:31,010 We're using this information here. 93 00:06:31,310 --> 00:06:35,060 We can approximate this inside a Gaussian distribution. 94 00:06:35,570 --> 00:06:43,220 So we could say that our sigma said one or sigma two is just give me three or three B to encourage this 95 00:06:43,220 --> 00:06:46,640 information inside an approximate Gaussian distribution. 96 00:06:47,270 --> 00:06:52,530 Once we have these standard deviations of the Galaxy distribution, we can just throw them into the 97 00:06:52,550 --> 00:06:54,710 diagonal elements of the Matrix. 98 00:06:55,730 --> 00:06:57,770 So we might get this information from a data sheet. 99 00:06:57,770 --> 00:07:02,990 We might make a few approximations and generate our covariance matrix for the sense of noise properties. 100 00:07:03,830 --> 00:07:09,490 The sensory data itself might actually give you the variances of the measurements itself. 101 00:07:09,830 --> 00:07:14,930 It all depends on what information is available and if the information is not available. 102 00:07:14,930 --> 00:07:20,960 You can never run experiments or tests to work out the distribution or we have to make some approximate 103 00:07:20,960 --> 00:07:21,650 assumptions. 104 00:07:24,170 --> 00:07:29,360 And then finally, the last thing we can use to change the filter is the process model covariance and 105 00:07:29,360 --> 00:07:33,410 in general, we want to make the small enough to allow the filter to work in all situations. 106 00:07:35,030 --> 00:07:40,970 If the key metrics is set to small, then we might run into the condition that the it becomes inconsistent. 107 00:07:41,300 --> 00:07:44,710 The variance on the estimate is too small to actually encompass the truth. 108 00:07:44,930 --> 00:07:46,630 So the field becomes inconsistent. 109 00:07:47,330 --> 00:07:52,940 If the relative size of the key matrix to the Matrix in terms of magnitude is too large, it means that 110 00:07:52,940 --> 00:07:55,430 we're injecting a lot of uncertainty into the system. 111 00:07:55,820 --> 00:08:01,160 It means that our uncertainty grows very quickly and it means that our state estimates or the uncertainty 112 00:08:01,160 --> 00:08:06,320 around the state estimates is going to be very large, which is different to what we actually want to 113 00:08:06,320 --> 00:08:10,460 use the common filter for, because we want to use a common filter to obtain better estimates of the 114 00:08:10,460 --> 00:08:10,820 state. 115 00:08:11,330 --> 00:08:14,330 Now we can judge the performance of a common filter. 116 00:08:14,330 --> 00:08:19,850 And if we assume that is correctly tuned and it's consistent, then we can based this performance on 117 00:08:19,850 --> 00:08:25,160 the overall estimation error for a series of runs or the typical performance or the typical estimation 118 00:08:25,160 --> 00:08:26,050 error performance. 119 00:08:26,450 --> 00:08:31,310 So we can say that when we run, this filter is always going to be within plus or minus a certain amount 120 00:08:31,310 --> 00:08:32,480 of the true state. 121 00:08:32,990 --> 00:08:34,970 So this is one way of how to judge the performance. 122 00:08:35,300 --> 00:08:41,320 The next way of judging the performance of the Common Filter is to look at the size of the state variances. 123 00:08:42,350 --> 00:08:47,750 So, for example, if we have a position sensor and the measurements of variance of 10 meters for one 124 00:08:47,750 --> 00:08:54,090 sigma, but the common filter has the position estimates from these measurements to be 20 meters, well, 125 00:08:54,090 --> 00:08:56,710 we can easily see that this filter is pretty useless. 126 00:08:56,720 --> 00:09:01,100 It's just better to use the raw measurements because I have a better accuracy than the actual filtered 127 00:09:01,100 --> 00:09:01,670 estimates. 128 00:09:02,660 --> 00:09:07,830 Now, if we contrast this to the situation where we have a precision sensor giving us our measurements 129 00:09:07,830 --> 00:09:12,800 that have a 10 meter standard deviation, however, the Common Filter estimates have a three made a 130 00:09:12,800 --> 00:09:13,790 standard deviation. 131 00:09:14,000 --> 00:09:18,920 You can see that the opposition estimates from the filter are a lot better than using the raw measurements 132 00:09:18,920 --> 00:09:19,520 themselves. 133 00:09:20,080 --> 00:09:22,470 So this is a good situation to be in. 134 00:09:22,490 --> 00:09:27,440 It means that the current field is working well and it produces better position estimates from the field 135 00:09:27,770 --> 00:09:29,180 than using the raw measurements. 136 00:09:29,330 --> 00:09:32,000 So this is a situation that we typically want to be in. 137 00:09:32,540 --> 00:09:35,810 This is one of the reasons why we want to use the common filter in the first place. 138 00:09:37,640 --> 00:09:42,110 So hopefully now, you know, a bit more information about how to tell if a common filter is chained 139 00:09:42,110 --> 00:09:47,450 and working correctly, how we actually go about tuning the common filter and how we can actually measure 140 00:09:47,450 --> 00:09:48,890 the performance of the common filter. 141 00:09:49,610 --> 00:09:54,440 In the next video, we'll go through the 2D tracker example and actually look at how we can apply this.