1 00:00:00,240 --> 00:00:06,660 So now let's move on to section 25, where we take a look at object tracking with optical flow. 2 00:00:07,260 --> 00:00:10,410 So open that notebook and really have it loaded here. 3 00:00:11,010 --> 00:00:14,310 This notebook has some good blocks that do take a while to run. 4 00:00:14,760 --> 00:00:17,640 So I've run, I've run all of them before. 5 00:00:17,640 --> 00:00:21,840 This lesson started before the recording, so we don't have to wait for them to complete. 6 00:00:22,380 --> 00:00:25,740 But you can run them and wait for them as you go along this lesson. 7 00:00:26,340 --> 00:00:31,170 So firstly, little libraries and images and videos that we're using in this case. 8 00:00:31,350 --> 00:00:35,100 And let's move on to the first part of optical flow. 9 00:00:35,280 --> 00:00:42,600 So optical flow is another tracking algorithm, and it's a much better tracking algorithm than the previous 10 00:00:42,600 --> 00:00:43,770 mean shift and camshaft. 11 00:00:43,770 --> 00:00:45,510 We looked at what it does. 12 00:00:45,510 --> 00:00:52,500 It looks for the apparent flow or movement or direction of an object that's moving in an image and between 13 00:00:52,500 --> 00:00:53,430 consecutive frames. 14 00:00:53,910 --> 00:01:00,300 And then basically tracks that with the 2D vector field, where the feature vector represents the displacement 15 00:01:00,660 --> 00:01:03,150 of the movement of points from frame to frame. 16 00:01:03,660 --> 00:01:08,940 So you can see in this image here, as this Red Point moves here, it generates this vector because 17 00:01:08,940 --> 00:01:11,230 you can see from point one to point to it moved here. 18 00:01:11,250 --> 00:01:18,060 And likewise and likewise so you can see it's projecting this vector that is indicative of the direction 19 00:01:18,090 --> 00:01:19,110 of this object. 20 00:01:19,920 --> 00:01:21,030 So you can see it here. 21 00:01:21,480 --> 00:01:21,810 All right. 22 00:01:22,410 --> 00:01:25,310 So it's quite useful and it's quite a good algorithm. 23 00:01:25,320 --> 00:01:32,100 I remember using this in university during a Masters project while I was helping a friend with a master's 24 00:01:32,100 --> 00:01:37,410 project where she was tracking ants in a frame that was moving. 25 00:01:37,680 --> 00:01:40,350 It was moving story, obviously from a top-down perspective. 26 00:01:40,800 --> 00:01:46,080 And she used what we use optical flow to track the movement of the end because they were trying to figure 27 00:01:46,080 --> 00:01:49,710 out what is the algorithm that the ants use to navigate. 28 00:01:49,720 --> 00:01:50,760 So it was pretty cool. 29 00:01:51,540 --> 00:01:55,950 So let's go on to how we implement optical flow using open TV. 30 00:01:56,560 --> 00:02:02,370 So because we're losing a video, we're going to create a video writing functions here. 31 00:02:03,210 --> 00:02:07,820 And then what we do, we used a she tomasi quantum detection algorithm. 32 00:02:08,280 --> 00:02:09,810 That's what's used in. 33 00:02:10,110 --> 00:02:15,360 Actually, don't mention it here, but that's what's used in optical flow. 34 00:02:15,360 --> 00:02:20,760 At least one of the methods we can use in optical flow to identify the points that we need to track. 35 00:02:21,090 --> 00:02:22,320 So we use that there. 36 00:02:23,010 --> 00:02:25,940 And then this is the parameters that we establish with it. 37 00:02:26,040 --> 00:02:29,540 So you can always measure only these parameters are it's highly configurable. 38 00:02:29,590 --> 00:02:35,790 This algorithm and we declared and Lucas Canada, because that's the person with a group of persons. 39 00:02:35,790 --> 00:02:38,580 I'm not even sure who developed optical flow. 40 00:02:39,660 --> 00:02:45,510 We just create this here with these parameters as well, which is window size max level, which indicates 41 00:02:45,510 --> 00:02:46,980 the amount of pyramids. 42 00:02:47,130 --> 00:02:47,960 So that's we can. 43 00:02:47,970 --> 00:02:53,490 That's a scaling tool that open to the user so we can look at two, we can make it look at different 44 00:02:53,490 --> 00:02:58,320 skills and more robust to smaller or larger objects that we want to track. 45 00:02:59,220 --> 00:03:00,630 And then what we do? 46 00:03:01,050 --> 00:03:02,520 We use this function. 47 00:03:02,520 --> 00:03:07,020 Here's this random function to create some random colors that will create the trails in the image so 48 00:03:07,020 --> 00:03:13,070 that each trail we can distinctly see which each real is and then find the initial coin is here. 49 00:03:13,080 --> 00:03:17,700 So using the CB2 function called Good Features to Track, which is appropriately named. 50 00:03:18,850 --> 00:03:20,370 And this gives us the corners. 51 00:03:20,640 --> 00:03:21,190 Oops. 52 00:03:22,050 --> 00:03:23,250 Let's go back down to the code. 53 00:03:23,250 --> 00:03:23,580 Sorry. 54 00:03:25,320 --> 00:03:30,180 This gives us the coin is that we are going to initial coin as we're going to use to set our tracking 55 00:03:30,180 --> 00:03:33,820 points and then we the mask for the drawing purposes right here. 56 00:03:33,840 --> 00:03:39,210 So you know how it grew previously in open TBE world, we actually do need it for other functions when 57 00:03:39,210 --> 00:03:42,600 we were integrating and demonstrating different functions. 58 00:03:43,140 --> 00:03:45,870 So now let's look at the optical flow code here. 59 00:03:46,440 --> 00:03:51,120 So you can see we just thought this loop would read frame by frame can fit it into a grayscale frame, 60 00:03:51,690 --> 00:03:56,850 and we input the previous grayscale frame, which is the first frame from the image. 61 00:03:57,900 --> 00:04:03,420 With the current frame here, which is now a frame grid with the initial coin is here, and then we 62 00:04:03,420 --> 00:04:07,470 just send this non, which indicates that things are using different parameters for something in the 63 00:04:07,560 --> 00:04:10,080 optical flow function. 64 00:04:10,080 --> 00:04:13,920 And then we send Lucas Canady parameters as well. 65 00:04:14,730 --> 00:04:18,630 So then all we do is select and store all the good points that we want to use here. 66 00:04:18,640 --> 00:04:22,740 So it's Tuttle's equals to and one gives us two points that we want to keep. 67 00:04:23,700 --> 00:04:27,330 And then we just draw tracks using these functions here. 68 00:04:27,330 --> 00:04:32,630 So I'm not going to go into this code in too much detail because it can get a bit complex, especially 69 00:04:32,640 --> 00:04:37,230 for beginners, because you don't know what its function is and enumerate and those things. 70 00:04:37,230 --> 00:04:43,620 But basically what we're doing, we're just iterating through these values here and putting in new and 71 00:04:43,620 --> 00:04:44,400 all in each one. 72 00:04:44,410 --> 00:04:49,170 That's why we use Zip to do it for a list like this or a set like this. 73 00:04:50,010 --> 00:04:57,540 And to this actually, which I should say and we use Laravel to flatten these as well, just give us 74 00:04:57,540 --> 00:04:59,880 two values here and then we just draw the lines and. 75 00:04:59,980 --> 00:05:06,190 Stuff there, and then we just add that frame so we can actually upload it to a video writer. 76 00:05:06,850 --> 00:05:11,890 And then we did the previous values, so we know have a new previous great PBS corner, which is a which 77 00:05:11,890 --> 00:05:13,990 is the current one for a loop. 78 00:05:14,680 --> 00:05:16,480 And then when the video finishes, we break. 79 00:05:16,630 --> 00:05:17,350 So we end that. 80 00:05:17,770 --> 00:05:20,980 So then so this takes about 11 10 seconds to run. 81 00:05:21,550 --> 00:05:24,410 Conversion takes about 18 seconds to run the bubble. 82 00:05:24,440 --> 00:05:28,870 Run all of this already and displaying the video, it takes a little while to run for some reason, 83 00:05:28,900 --> 00:05:33,040 but we can display it here, and you can see this is pretty cool. 84 00:05:33,520 --> 00:05:38,500 You can see as a person walks, you can see the trails behind them and the trails. 85 00:05:38,830 --> 00:05:44,050 And sometimes when it loses that track, the track of that person is always super reliable. 86 00:05:44,590 --> 00:05:49,390 But you can adjust the sensitivity with the parameters, and you can see this is how it works. 87 00:05:49,720 --> 00:05:52,780 You know, it's using the good features, a track which just finds corners. 88 00:05:53,410 --> 00:05:58,000 That's why it is like these interesting areas here, which are probably corner points. 89 00:05:58,060 --> 00:05:58,940 You can see one here. 90 00:05:58,970 --> 00:06:02,110 Lampposts is a bunch over here by this parked vehicle. 91 00:06:03,010 --> 00:06:08,410 So all of those are points that we initialize, but then weren't moving, so they weren't tracked. 92 00:06:08,860 --> 00:06:14,350 And the reason we have different colors is because we used that random color function to just go up 93 00:06:14,350 --> 00:06:24,130 here just to generate random colors between groups between zero and 255 here as a vertical ranges. 94 00:06:24,850 --> 00:06:26,710 So let's go back down. 95 00:06:26,860 --> 00:06:30,850 And now let's take a look at another implementation of optical flow. 96 00:06:31,450 --> 00:06:33,700 This one is called dense optical flow. 97 00:06:34,120 --> 00:06:40,930 So dense optical flow uses a slightly different method, which is illustrated in the gun of funny box 98 00:06:40,930 --> 00:06:41,500 algorithm. 99 00:06:41,500 --> 00:06:44,650 Sorry, but this peroxidation, which is explained here. 100 00:06:45,010 --> 00:06:50,110 Basically, it's a bit more complicated, and I'm not going to go into the actual explanation of how 101 00:06:50,110 --> 00:06:50,590 it works. 102 00:06:51,100 --> 00:06:56,290 However, this is like a good little recap this line that shows you how the original optical flow to 103 00:06:56,290 --> 00:07:02,770 Lucas Canady implementation worked by using the sheet Thomas Tomasi algorithm for getting the corners. 104 00:07:03,460 --> 00:07:08,110 So what we do here and in this video, what we're going to do, which is a little bit different, is 105 00:07:08,110 --> 00:07:14,680 that we're going to basically make the background black and track and only show it hitting the objects 106 00:07:14,680 --> 00:07:18,520 we want to track and direction is going to correspond to Hue. 107 00:07:18,630 --> 00:07:24,670 Remember, Hue is color and magnitude of which is a brightness of it is going to correspond to the speed. 108 00:07:25,180 --> 00:07:25,510 All right. 109 00:07:26,110 --> 00:07:28,780 So you can take a look at a code. 110 00:07:28,840 --> 00:07:31,990 You can see it's it's similar to the code we previously looked at. 111 00:07:32,530 --> 00:07:33,910 We create the video, right? 112 00:07:33,910 --> 00:07:37,090 So he'll get the first frame, establish these things. 113 00:07:37,090 --> 00:07:38,360 We create a blank. 114 00:07:38,360 --> 00:07:43,120 You just feel that we're going to use here, make it all white initially. 115 00:07:43,810 --> 00:07:48,790 And then what we do, we create the optical flow from the back. 116 00:07:49,780 --> 00:07:55,030 So the parameters and a lot of parameters in this algorithm, which personally, I don't know what they 117 00:07:55,030 --> 00:08:01,420 do if it's interesting to you, most of it does affect the sensitivity of the tracker. 118 00:08:02,020 --> 00:08:08,890 So you can take a look at this function in the open, see this documentation and see what it does. 119 00:08:09,970 --> 00:08:11,440 That's something I encourage you to do. 120 00:08:11,770 --> 00:08:15,790 There are tons of open TV functions, and it's impossible for one person to know all. 121 00:08:16,210 --> 00:08:22,360 However, when you're checking your you for your application purposes, you will need to configure some 122 00:08:22,360 --> 00:08:23,280 of these algorithms. 123 00:08:23,290 --> 00:08:27,880 So definitely, if it's something that's interest to you that you're working on, definitely dig up 124 00:08:27,880 --> 00:08:31,270 in documentation and take a look and see what's going on. 125 00:08:31,750 --> 00:08:37,660 So what we do here, we use a flow to calculate the magnitude of which is the speed and angle of the 126 00:08:37,660 --> 00:08:38,050 motion. 127 00:08:39,040 --> 00:08:42,040 And then we use a values to reflect the speed at an angle again. 128 00:08:42,520 --> 00:08:46,270 So that's how we use different colored color combos and just be here. 129 00:08:46,840 --> 00:08:50,800 You can see it's indexed, right like that there, which hue and value. 130 00:08:51,340 --> 00:08:55,840 We ignore saturation in this case, because that doesn't change the visibility to us. 131 00:08:56,200 --> 00:09:02,410 A highly saturated on saturated color will be a bit more difficult to see than the hue and brightness. 132 00:09:03,010 --> 00:09:04,630 So I've run this already. 133 00:09:04,630 --> 00:09:06,700 This one takes a little bit longer to run next. 134 00:09:06,700 --> 00:09:13,090 With just over a minute, the conversion takes about 15 seconds and then we can display it here. 135 00:09:13,090 --> 00:09:17,620 So let's take a look and immediately see, it's a black background, which we created initially. 136 00:09:18,220 --> 00:09:23,830 We remove we created a white background, but then we set those values afterward to zero where it doesn't 137 00:09:23,830 --> 00:09:24,370 have motion. 138 00:09:24,940 --> 00:09:26,350 So let's take a look at the output. 139 00:09:27,010 --> 00:09:32,440 So you can see this looks pretty cool where you can see when someone is moving the left, you can see 140 00:09:32,470 --> 00:09:34,120 dark blue or pinkish. 141 00:09:34,660 --> 00:09:37,450 And when someone is moving right, they're yellow or reddish. 142 00:09:37,990 --> 00:09:43,960 So you can definitely see that the color corresponds to the direction speed, the faster something is 143 00:09:43,960 --> 00:09:44,860 moving to brighter. 144 00:09:44,930 --> 00:09:46,930 Look, you can see it here when it's close to the camera. 145 00:09:46,930 --> 00:09:50,320 The apparent movement does appear to be more as than someone put it away. 146 00:09:50,320 --> 00:09:53,530 So that's why brighter objects because objects will appear brighter. 147 00:09:54,190 --> 00:09:58,120 So that is the implementation of the dense optical flow tracker. 148 00:09:58,570 --> 00:09:59,650 These are very useful. 149 00:09:59,950 --> 00:10:06,520 Is that can be used for many different applications, like tracking vehicles, tracking people, tracking 150 00:10:06,520 --> 00:10:11,080 animals, so feel free to use them for your application purposes. 151 00:10:11,500 --> 00:10:15,760 So that concludes object tracking windows policy with other items. 152 00:10:16,240 --> 00:10:20,560 Next, we're going to make a very simple object tracking by color. 153 00:10:20,920 --> 00:10:22,120 So stay tuned for that.