1 00:00:00,060 --> 00:00:05,640 Hi and welcome back to Chapter 23, where we take a look at background and foreground subtraction, 2 00:00:06,120 --> 00:00:10,440 so open up that notebook and let me explain to you what we're doing here. 3 00:00:10,860 --> 00:00:15,330 So what we're going to do, we're going to try a few different methods of background subtraction. 4 00:00:15,660 --> 00:00:18,480 No background subtraction is what do you expect? 5 00:00:19,200 --> 00:00:21,240 Remember this and all images? 6 00:00:21,570 --> 00:00:27,060 Most images, I should say, is a foreground subject, which is an object that's probably being photographed 7 00:00:27,060 --> 00:00:27,720 or focused on. 8 00:00:28,170 --> 00:00:32,850 And then there's a background, which is the stuff that's behind it, or it doesn't even have to be 9 00:00:32,850 --> 00:00:34,530 behind it to just around it. 10 00:00:35,160 --> 00:00:37,590 So what if we wanted to just remove the background? 11 00:00:37,680 --> 00:00:40,230 So get this black mask over it? 12 00:00:40,230 --> 00:00:41,640 I'm going to keep the white area. 13 00:00:42,210 --> 00:00:46,750 That's what background so production is, and then foreground subtraction is the opposite. 14 00:00:46,770 --> 00:00:51,090 It's where we actually want to remove this boat and you only get the background here. 15 00:00:51,690 --> 00:00:58,380 So open TV has a few background subtraction algorithms implemented, which are quite good. 16 00:00:58,830 --> 00:01:00,240 So we'll take a look at those here. 17 00:01:01,230 --> 00:01:06,930 So let's lodo images and download the little libraries, I should say. 18 00:01:07,560 --> 00:01:09,000 So let's move. 19 00:01:09,240 --> 00:01:13,710 Move ahead, and we're going to use this demo video, which is a walking clip of pedestrians, which 20 00:01:13,710 --> 00:01:16,320 is which is famous in computer vision, which is why I use it. 21 00:01:16,800 --> 00:01:22,080 It's always good to give those previous researchers some credits for when they use these videos. 22 00:01:22,500 --> 00:01:23,880 That's this video here. 23 00:01:24,240 --> 00:01:26,100 You would have recalls seeing it before. 24 00:01:26,850 --> 00:01:30,390 So let's take a look at what backroom subtraction is. 25 00:01:30,390 --> 00:01:34,260 So actually did mention it to you before, but if you want, you can read this in more detail if you 26 00:01:34,260 --> 00:01:34,830 want to get it. 27 00:01:35,370 --> 00:01:37,380 And algorithm, we're going to try first. 28 00:01:37,380 --> 00:01:42,150 It's called a Gaussian mixture based background and foreground segmentation algorithm. 29 00:01:43,080 --> 00:01:46,380 I'm going to explain this here because it's quite complicated. 30 00:01:46,860 --> 00:01:48,300 And here's a link to the research paper. 31 00:01:48,300 --> 00:01:50,130 If you want to see it, see it for yourself. 32 00:01:50,610 --> 00:01:54,630 What I'm going to show you, though, is how we implemented it here in open TV. 33 00:01:55,200 --> 00:01:58,860 So you remember how we actually created this video, which in the video, right? 34 00:01:58,860 --> 00:01:59,880 So which is the output? 35 00:02:00,450 --> 00:02:01,500 So we do that here again. 36 00:02:01,500 --> 00:02:03,720 We load the video and then we create the outputs here. 37 00:02:04,290 --> 00:02:07,080 So this is the part that does a background subtraction. 38 00:02:07,530 --> 00:02:16,530 By subtraction, it's KVUE, Dot, B.G. segment M Dot, Create Background, Subtract MLG, which stands 39 00:02:16,530 --> 00:02:23,520 for that Gaussian mixture got a mixture of Gaussians actually is what sounds sensible. 40 00:02:24,480 --> 00:02:31,340 And then we just go to the loop with which each frame of the video and we get, we extract this this 41 00:02:31,350 --> 00:02:31,590 bit. 42 00:02:31,770 --> 00:02:32,790 Sorry, then extract it. 43 00:02:33,210 --> 00:02:38,820 This gives us the we apply this to the frame to get the foreground mask using the airplane mounted, 44 00:02:39,660 --> 00:02:46,590 and we write that to filter on this, which is the output to the the output video we created here. 45 00:02:47,190 --> 00:02:53,220 So let's run this what we're doing here, we're actually printing out or displaying. 46 00:02:53,220 --> 00:02:57,390 I should see all the images here in the video, so we don't actually have to watch the video in the 47 00:02:57,390 --> 00:02:57,720 end. 48 00:02:58,200 --> 00:03:00,360 We can see it live here frame by frame. 49 00:03:01,320 --> 00:03:07,260 So you can see these are the pedestrians that we're walking here, and you can see that it actually 50 00:03:07,260 --> 00:03:08,340 is working fairly well. 51 00:03:08,580 --> 00:03:11,700 You're getting the pedestrians and everything else does not. 52 00:03:11,700 --> 00:03:15,330 Moving is black, so it is working as expected. 53 00:03:15,840 --> 00:03:21,960 However, it could be better because as you can see, there are a lot more pedestrians here than it's 54 00:03:21,960 --> 00:03:22,470 letting on. 55 00:03:22,830 --> 00:03:30,030 So we can just go back here and now let's try something called the improved adaptive Yorkshire mixed 56 00:03:30,150 --> 00:03:32,190 model for background subtraction. 57 00:03:32,190 --> 00:03:38,520 So we're going to do the same thing again where we display all the outputs and immediately you can see 58 00:03:38,940 --> 00:03:41,490 that it actually is working a lot better. 59 00:03:42,150 --> 00:03:44,910 You can see that the pedestrians are much more defined. 60 00:03:45,330 --> 00:03:47,760 So we have a good foreground mass going here. 61 00:03:48,180 --> 00:03:50,610 This is the background has been eliminated in these images. 62 00:03:50,610 --> 00:03:56,070 So if you wanted to just extract the foreground, you can use a Bitwise operation to do that, which 63 00:03:56,070 --> 00:03:57,630 is a good exercise for you to do. 64 00:03:58,590 --> 00:04:03,420 Actually, a good exercise for you to do is to do the Bitwise operation and on the one, the original 65 00:04:03,420 --> 00:04:09,510 image and write it to an output image and then display that video and call up to previous quarters in 66 00:04:09,510 --> 00:04:10,320 all the notebooks. 67 00:04:10,320 --> 00:04:13,080 You can check it out and everything should be simple. 68 00:04:13,770 --> 00:04:16,770 So let's take a look at foreground subtraction. 69 00:04:16,830 --> 00:04:21,630 Now, moving on to foreground subtraction of foreground subtraction is actually done a bit differently. 70 00:04:21,630 --> 00:04:24,000 It's done a bit more mathematically now. 71 00:04:24,420 --> 00:04:25,650 So what we do? 72 00:04:25,890 --> 00:04:27,810 We load a video and create a video, right? 73 00:04:27,820 --> 00:04:30,450 As well, just just in case you want to see the saved output. 74 00:04:31,230 --> 00:04:35,820 But we use this function called V to accumulate tweeted. 75 00:04:36,420 --> 00:04:43,440 This function allows us to basically, as depicted as a video sequence goes, allows us to store values 76 00:04:43,440 --> 00:04:43,860 of that. 77 00:04:43,860 --> 00:04:49,660 So that accumulates values and then we get the average as well, which is what we specify here. 78 00:04:50,610 --> 00:04:52,140 So we get the average value from the frame. 79 00:04:52,800 --> 00:04:57,030 So it's a way to actually kind of keep track of what the background is. 80 00:04:57,570 --> 00:04:59,640 So then what we do, we use this? 81 00:04:59,950 --> 00:05:03,470 The function here called C-V2X convert skills absolute. 82 00:05:04,010 --> 00:05:09,550 This actually scales and calculates the absolute values and then gives us the result as an eight bit 83 00:05:10,270 --> 00:05:15,070 unit, unsigned end, and that's how we mathematically get the background. 84 00:05:15,190 --> 00:05:19,510 So let's run this and we can see the outputs as they come out. 85 00:05:20,380 --> 00:05:30,290 Let's zoom in a bit and you can see as good as it progresses to the bottom. 86 00:05:30,310 --> 00:05:32,790 Sorry for the haphazard scrolling. 87 00:05:32,800 --> 00:05:35,740 Just because I'm zoomed in, it looks a bit extreme. 88 00:05:36,340 --> 00:05:37,810 This is a disappearing background. 89 00:05:37,990 --> 00:05:41,230 As you can see, it's not that apparent in these images. 90 00:05:41,710 --> 00:05:48,100 However, it accumulates over time, so the longer we leave this, the more it'll accumulate. 91 00:05:48,670 --> 00:05:49,870 So what we can do? 92 00:05:49,870 --> 00:05:57,910 We can show the last image from this video and let's look at this here and you can see. 93 00:05:58,360 --> 00:06:02,710 So it's not the best method, and it should play around with the values to see if we can get it to be 94 00:06:02,710 --> 00:06:04,330 better because I'm sure it can be. 95 00:06:04,870 --> 00:06:08,500 But you can see this guy, they look a bit invisible, these pedestrians. 96 00:06:08,500 --> 00:06:11,920 So it kind of works, but they're better at measurement methods around it. 97 00:06:12,190 --> 00:06:17,650 Let's go back to this, so let's take a look at another background subtraction method called it using 98 00:06:17,650 --> 00:06:18,760 the can a method. 99 00:06:19,390 --> 00:06:22,180 So it's just basically the similarly like we did before. 100 00:06:22,600 --> 00:06:29,680 However, we actually using creating Akinwale from the TV to get structure and element, which I'm actually 101 00:06:29,680 --> 00:06:31,510 not too familiar with what this does. 102 00:06:32,140 --> 00:06:36,940 However, we use the cleaner for this background subtraction, subtract, subtracted key and admitted. 103 00:06:37,900 --> 00:06:40,360 So we can actually apply this now in the frame. 104 00:06:41,440 --> 00:06:43,300 So you go to you create this. 105 00:06:43,300 --> 00:06:44,680 This is the foreground background here. 106 00:06:45,190 --> 00:06:51,190 We used apply method inside of that object applied to the frame and get the foreground must then apply 107 00:06:51,190 --> 00:06:57,190 the morphology x, which is, you know, using function with the kernel that we defined and then we 108 00:06:57,190 --> 00:06:57,940 get the output. 109 00:06:58,060 --> 00:07:00,340 So let's take a look at how this looks. 110 00:07:01,660 --> 00:07:03,610 So you can see this again works pretty well. 111 00:07:04,180 --> 00:07:06,520 You can see the actual people are very well defined. 112 00:07:06,520 --> 00:07:11,560 So this is perhaps the best background removal method we've come across so far. 113 00:07:12,430 --> 00:07:16,210 So that concludes this lesson on background foreground subtraction. 114 00:07:16,720 --> 00:07:20,950 The next lesson is going to be on motion tracking with the mean shifts and camshaft algorithms. 115 00:07:21,400 --> 00:07:22,780 So stay tuned for that. 116 00:07:22,960 --> 00:07:23,530 And thank you.