1 00:00:00,150 --> 00:00:01,140 Hi and welcome back. 2 00:00:01,260 --> 00:00:03,330 So what we're going to do. 3 00:00:03,450 --> 00:00:08,580 We're going to implement neural cell transfer using cameras as well as using the models. 4 00:00:11,100 --> 00:00:12,180 Hi and welcome back. 5 00:00:12,750 --> 00:00:17,850 What we're going to do in this lesson is we're going to use cameras to implement neural cell transfer 6 00:00:18,330 --> 00:00:23,370 as well as use the TensorFlow hub models to implement neural cell transfer, which is quick and easy. 7 00:00:23,850 --> 00:00:24,810 So let's take a look. 8 00:00:24,960 --> 00:00:28,500 So open this notebook, which I've already opened here. 9 00:00:29,220 --> 00:00:34,320 And what we're going to do is we're going to set up and little modules for this and create some hope 10 00:00:34,320 --> 00:00:34,950 of functions. 11 00:00:35,310 --> 00:00:39,100 Then we're going to implement fast style transfer using the TensorFlow hub. 12 00:00:40,080 --> 00:00:44,910 And then we're going to implement, implement our model from scratch, build a model, extract style 13 00:00:44,910 --> 00:00:51,360 and content, then run gradient descent on it, and then also implement one with total variation loss 14 00:00:51,840 --> 00:00:54,390 and then rerun the optimization to generate that output. 15 00:00:54,990 --> 00:00:57,450 So let's take a look at this guide. 16 00:00:57,660 --> 00:01:01,650 So firstly, we just import and set some environment variables here. 17 00:01:02,130 --> 00:01:06,750 So this just tells us we wanted the compressed format of TensorFlow hub models that we're loading. 18 00:01:07,320 --> 00:01:14,370 So TensorFlow hub, by the way, is a repository of models that can be created, TensorFlow models that 19 00:01:14,370 --> 00:01:15,740 can be loaded and reused. 20 00:01:16,140 --> 00:01:18,330 Anyone could for experimentation. 21 00:01:18,510 --> 00:01:19,950 So it's quite it's quite myself. 22 00:01:20,700 --> 00:01:26,860 Next, we're going to just import some libraries, as well as some of the plot lab plotting parameters. 23 00:01:27,090 --> 00:01:28,170 Let's do that. 24 00:01:28,200 --> 00:01:31,410 Also, we're going to use this AI Python display. 25 00:01:31,440 --> 00:01:34,750 This allows us to display images in these notebooks. 26 00:01:34,770 --> 00:01:37,650 However, if you want to display a new image, it overrides the whole image. 27 00:01:37,650 --> 00:01:41,980 So it's quite convenient when you want to update images in real time. 28 00:01:42,000 --> 00:01:42,930 So it's it's nice. 29 00:01:43,950 --> 00:01:49,830 So now we create a helper function here that is converted tensor to image and returns the image and 30 00:01:50,490 --> 00:01:52,500 still image from a format. 31 00:01:53,310 --> 00:01:56,360 So let's take a look and know. 32 00:01:56,400 --> 00:01:57,900 Download some of our test images. 33 00:01:57,900 --> 00:02:03,450 So one of these images to Labrador, which is my Labrador, is his name is Samuel. 34 00:02:03,450 --> 00:02:08,910 With the Chocolate Lab in it, we're going to use him as a content image and then we're going to use 35 00:02:08,910 --> 00:02:14,760 this art painting called The Wave to input to convert style. 36 00:02:15,270 --> 00:02:18,090 And then also, you have few other options here. 37 00:02:18,300 --> 00:02:21,540 You can have music and is available as well. 38 00:02:21,840 --> 00:02:29,070 So let's ludus images create a function to lower the image and then basically convert it into the format 39 00:02:29,070 --> 00:02:33,000 that we want to use within our own networks. 40 00:02:34,260 --> 00:02:36,470 Next, we created a image. 41 00:02:36,600 --> 00:02:38,850 This function is to show the image as well. 42 00:02:39,630 --> 00:02:46,320 And we just load the image from using this little image function that we downloaded, and then we just 43 00:02:46,320 --> 00:02:47,820 display it as well with the image. 44 00:02:48,690 --> 00:02:51,750 So let's take a look at those test images. 45 00:02:52,290 --> 00:02:56,150 You can see my Labrador Samuel with his little reindeer antlers. 46 00:02:56,160 --> 00:03:01,770 This was taken last Christmas 2020, and it was still image since it's called the wave, as you can 47 00:03:01,770 --> 00:03:03,870 see why it's a pretty nice painting. 48 00:03:03,990 --> 00:03:08,700 So we're going to put this style onto this picture might, might be kind of interesting. 49 00:03:09,510 --> 00:03:15,300 So we're going to look at a model first from TensorFlow Hub, and you can check it out here. 50 00:03:15,360 --> 00:03:19,500 TensorFlow Hub, if you want to take a look, haven't been on the site in a while. 51 00:03:19,920 --> 00:03:22,890 This is the model for image stabilization. 52 00:03:23,430 --> 00:03:28,400 So you can go to image style transfer and take a look at many other models they have available. 53 00:03:28,420 --> 00:03:32,430 They've got six right now, including cartoon Gan, which is pretty cool. 54 00:03:33,030 --> 00:03:37,350 So let's go back to our code and let's see how we how we use those models. 55 00:03:37,740 --> 00:03:40,800 So to get your model, you just send us your URL here. 56 00:03:41,310 --> 00:03:42,950 Now, where do you find that URL? 57 00:03:43,200 --> 00:03:43,920 Let's take a look. 58 00:03:44,190 --> 00:03:44,510 So. 59 00:03:46,020 --> 00:03:49,440 So go back to TensorFlow hub and look for the model that you want to use. 60 00:03:49,440 --> 00:03:53,880 That's we're using image stabilization vision one Dash 256. 61 00:03:54,150 --> 00:03:58,150 And what you do, there's some example code and how to use those models. 62 00:03:58,170 --> 00:03:59,940 This is a URL to download a model. 63 00:03:59,940 --> 00:04:05,910 So when you want to bring it in to your collab or your local machine, you can just download it using 64 00:04:05,910 --> 00:04:08,160 hub that it right from the URL here. 65 00:04:08,250 --> 00:04:14,790 So let's head back to our notebook, and all we do is we lower the Model S and model. 66 00:04:14,790 --> 00:04:15,720 There's no hub model. 67 00:04:16,260 --> 00:04:21,090 And then we lowered the content image as a as a TensorFlow constant. 68 00:04:21,690 --> 00:04:23,130 Similar for the style image. 69 00:04:23,130 --> 00:04:28,500 And then we just extract the first element and read this returns is a stylized image. 70 00:04:28,500 --> 00:04:33,780 And then we just use Tensor to image here because this returns a tensor and we have to convert it back 71 00:04:33,780 --> 00:04:34,680 into an image. 72 00:04:34,830 --> 00:04:41,970 So let's run this and it will give us this nice output that will be generated very quickly. 73 00:04:42,220 --> 00:04:46,650 You can see this is quite fast for seconds, and we got a very good implementation of this, to be honest. 74 00:04:46,680 --> 00:04:47,430 This is pretty cool. 75 00:04:47,940 --> 00:04:51,680 It's much better than the all the earlier versions of neural cell transfer. 76 00:04:51,690 --> 00:04:57,570 So it's a good model, and I would encourage you to maybe mess and try the other models on TensorFlow 77 00:04:57,570 --> 00:04:57,810 hub. 78 00:04:58,020 --> 00:04:59,400 There's six of them available here. 79 00:05:00,260 --> 00:05:06,830 So let's take a look back at now implementing this from scratch, know this is going to be a lot more 80 00:05:06,830 --> 00:05:10,340 difficult and the court is a bit confusing, I must say. 81 00:05:10,820 --> 00:05:14,750 However, I'll try to step through it at a high level so you understand what's going on. 82 00:05:15,380 --> 00:05:17,870 So first of all, let's load of a pre-trained network. 83 00:05:17,870 --> 00:05:23,210 We're going to load in 19 and take a look at the layers and the network so we can print them. 84 00:05:23,720 --> 00:05:26,180 And then what we have to do, we have to define. 85 00:05:26,390 --> 00:05:29,960 After that, we have to define content and style representation. 86 00:05:29,970 --> 00:05:31,670 So we'll take a look at how we do that. 87 00:05:32,240 --> 00:05:36,500 So let's do this, which I've already done and print the mean layers here. 88 00:05:36,500 --> 00:05:37,700 These are the 19 layers. 89 00:05:38,150 --> 00:05:41,060 Well, actually, it will be 16 layers in this. 90 00:05:41,450 --> 00:05:46,940 I believe I have the confidence because it doesn't include the top and the top of the Regas Tree fully 91 00:05:46,940 --> 00:05:47,690 connected layers. 92 00:05:48,410 --> 00:05:51,520 So this is where we define the layers we want to use for the content. 93 00:05:51,530 --> 00:05:56,660 So remember, for the content, we need to use layers that are lower down into the network. 94 00:05:57,020 --> 00:05:59,030 So we're using this one to kind of it. 95 00:05:59,060 --> 00:06:05,450 We can use country or conforming Block five, but we choose to use one of two because it should keep 96 00:06:05,450 --> 00:06:07,130 most of the style in this layer here. 97 00:06:07,820 --> 00:06:14,090 And then we have this tile is actually when is it still in and the global localization, not the actual 98 00:06:14,090 --> 00:06:14,480 stuff. 99 00:06:14,510 --> 00:06:15,260 Sorry about that. 100 00:06:15,770 --> 00:06:18,280 So the style is all from different conflicts. 101 00:06:18,280 --> 00:06:23,900 Here you can see we took the first layer from Block one block to block three and form Block five. 102 00:06:24,440 --> 00:06:30,680 So then we just get the lengths of the number of layers here that we will use in our functions. 103 00:06:31,520 --> 00:06:37,400 So now, once we chose those intermediate layers, my style and content representation, we can now 104 00:06:37,430 --> 00:06:38,660 build the model. 105 00:06:39,260 --> 00:06:43,460 So I'm not going to read all of these notes, these notes and more for you guys to run this notebook 106 00:06:43,910 --> 00:06:45,840 and do it on your own. 107 00:06:45,920 --> 00:06:48,590 It's a bit exhausted for me to keep reading off this one. 108 00:06:49,220 --> 00:06:53,780 This is a practical lesson, so it's a little more fun just to go through the code and then you can. 109 00:06:53,780 --> 00:06:58,700 If you want to get a deeper understanding of what's happening and my explanations, my high level explicit 110 00:06:58,910 --> 00:07:00,020 explanations, don't cut it. 111 00:07:00,410 --> 00:07:02,690 Take a look at the notes and comments in the code. 112 00:07:03,650 --> 00:07:07,190 So now we use of do we build a model here? 113 00:07:07,190 --> 00:07:10,420 So this is a simple function here that just takes those layers. 114 00:07:10,420 --> 00:07:16,530 So we know the network here set it to trainable to fall, so it frees up the network weights. 115 00:07:17,060 --> 00:07:20,840 Then we just get those outputs here and we build back the model right here. 116 00:07:21,530 --> 00:07:24,080 So this just returns the Vigilia as a model. 117 00:07:25,250 --> 00:07:27,860 Before that, we were just loading it and experimenting. 118 00:07:27,860 --> 00:07:31,640 It's here to just to visualize it for tutorial purposes. 119 00:07:31,760 --> 00:07:34,100 So now this is the actual function that does it for us. 120 00:07:34,820 --> 00:07:41,300 Next, we have to basically get style layers extractor and then the style outputs from this. 121 00:07:42,020 --> 00:07:46,310 So let's do that now, and let's take a look at the output so you can see. 122 00:07:46,520 --> 00:07:49,320 Let's take a look at this style is and still. 123 00:07:50,360 --> 00:07:56,700 So what we're doing, the name of two layers here block me and the output, which is a outputs here. 124 00:07:56,720 --> 00:07:58,730 This is the output size of that layer. 125 00:07:59,090 --> 00:08:04,790 So we need to capture the output style output size of each of those interim style is just because we 126 00:08:04,790 --> 00:08:06,750 need it for the network. 127 00:08:06,830 --> 00:08:12,650 So let's take a look and see how we calculate style because those things are needed for these these 128 00:08:12,650 --> 00:08:13,760 lost calculations. 129 00:08:15,530 --> 00:08:17,900 So now this brings us to the stylus. 130 00:08:18,440 --> 00:08:25,610 So remember, the stylus basically computes the correlations between filters of the pre-trained CNN 131 00:08:25,820 --> 00:08:27,860 s or G 19 network. 132 00:08:28,370 --> 00:08:33,740 So to do that, remember we talked about the Gram matrix to Gram Matrix is how it how we calculate those 133 00:08:33,740 --> 00:08:34,430 correlations. 134 00:08:34,970 --> 00:08:36,570 This is a function that does it here. 135 00:08:36,590 --> 00:08:41,150 We use this TensorFlow function here called linear alg insome. 136 00:08:41,780 --> 00:08:44,900 And this basically allows us to calculate that. 137 00:08:45,230 --> 00:08:46,630 So let's run this function. 138 00:08:46,640 --> 00:08:51,020 Remember to run each block of code because I've run this entire notebook before to just double check 139 00:08:51,020 --> 00:08:52,040 that everything was working. 140 00:08:52,460 --> 00:08:58,910 So you may need to, I mean, sometimes skip running it accidentally because I know Auth it has been 141 00:08:58,910 --> 00:08:59,740 run still. 142 00:09:00,290 --> 00:09:03,830 But just remember when you're doing it to run all code blocks. 143 00:09:04,850 --> 00:09:10,370 So, no, we extract the style and content from the networks that we loaded. 144 00:09:10,400 --> 00:09:16,430 So those above this here, we would just sort of experimenting to see how we can extract the different 145 00:09:16,430 --> 00:09:21,260 layers and get the size and the mean and max of the pre-trained feature weights as well. 146 00:09:21,710 --> 00:09:25,490 Now we just have a class that basically does all that for us. 147 00:09:25,490 --> 00:09:30,830 So we have a stellar puts the content like the style dictionary and it returns everything here, the 148 00:09:30,830 --> 00:09:34,100 content dictionary, Estelle Dictionary that we can see. 149 00:09:34,100 --> 00:09:36,450 This is what we created up here, basically. 150 00:09:36,470 --> 00:09:37,700 So that's an example of it. 151 00:09:38,240 --> 00:09:42,260 And we just inherit from this class here in this function. 152 00:09:42,800 --> 00:09:50,580 So my style content model and we just created like a DVD model as well, said the way it's defaults. 153 00:09:51,110 --> 00:09:55,910 So this does a lot of the heavy lifting for us here and organizes the code quite neatly. 154 00:09:56,300 --> 00:09:59,390 So run that block of code and we just use that. 155 00:09:59,640 --> 00:10:00,180 Content. 156 00:10:00,240 --> 00:10:01,080 That's our model. 157 00:10:01,410 --> 00:10:01,830 Sorry. 158 00:10:02,130 --> 00:10:08,400 Plus, to get the extractor and then we can run that on the content image itself and cause of the results. 159 00:10:09,060 --> 00:10:14,760 So we have that now, so we have all of those calculated for the input image. 160 00:10:15,390 --> 00:10:20,850 Next, we can run gradient descent here, so we have our stealth targets in our content targets that 161 00:10:20,850 --> 00:10:21,630 we extracted. 162 00:10:22,350 --> 00:10:27,900 Next, we just love the image here and to this from our content image and converted to a TensorFlow 163 00:10:27,900 --> 00:10:28,410 variable. 164 00:10:28,920 --> 00:10:33,990 Then we just clip the outputs from the outputs between zero and one pixel values. 165 00:10:34,800 --> 00:10:39,750 And then we set a optimizer to be Adam with a fairly large loading rate of point zero two. 166 00:10:39,760 --> 00:10:41,400 But you can experiment with these values. 167 00:10:41,970 --> 00:10:44,760 And now we have a content and style weighting loss. 168 00:10:45,130 --> 00:10:52,530 Note this is not the way it's the parameters the alpha and beta parameters that go into the total variation 169 00:10:52,530 --> 00:10:52,890 loss. 170 00:10:52,890 --> 00:10:58,680 This implementation, this basic implementation we're doing here, does not include the total variation 171 00:10:58,680 --> 00:10:59,490 loss loss. 172 00:10:59,880 --> 00:11:04,020 It's just loss and content loss, as you can see here, for a total loss. 173 00:11:04,650 --> 00:11:06,360 So let's run that as well. 174 00:11:06,360 --> 00:11:09,510 And now we can do a function to do a training step. 175 00:11:09,900 --> 00:11:15,330 So we're using the t.f gradient tip to just do two steps similar to how PyTorch does it. 176 00:11:16,830 --> 00:11:21,940 And now we can generate our final output and you can take a look and it looks pretty good. 177 00:11:22,110 --> 00:11:28,230 This is after one epoch of running, so now we can do this, we can see the steps we took to 100 and 178 00:11:28,230 --> 00:11:30,230 trained for 20 bucks of notice. 179 00:11:30,240 --> 00:11:36,930 It doesn't get much better at what it would have, but you can see after 100 steps, it produces this 180 00:11:36,930 --> 00:11:37,260 image. 181 00:11:37,710 --> 00:11:39,060 That's the image we produce above. 182 00:11:39,300 --> 00:11:43,140 But then it's it's not really improving the image, unfortunately. 183 00:11:43,560 --> 00:11:47,580 So on and doing something wrong, this is the best we can get. 184 00:11:48,330 --> 00:11:49,170 But anyhow. 185 00:11:49,290 --> 00:11:52,050 Now let's introduce total variation loss. 186 00:11:52,590 --> 00:11:56,370 So to do that, we have to set a high pass for our. 187 00:12:01,450 --> 00:12:02,680 For artifacts. 188 00:12:02,830 --> 00:12:04,840 So that's what this function does. 189 00:12:05,470 --> 00:12:12,220 Then you can actually just visualize those Haifa high frequency features so you can see what they look 190 00:12:12,220 --> 00:12:13,180 like on the image. 191 00:12:13,330 --> 00:12:20,130 So these are the horizontal filters, the vertical deltas, vertical deltas with style image did horizontal 192 00:12:20,140 --> 00:12:21,650 surface of the style as well. 193 00:12:21,670 --> 00:12:22,540 So you can take a look. 194 00:12:22,960 --> 00:12:28,030 These are basically the artifacts that we're going to try to smooth and reduce. 195 00:12:28,360 --> 00:12:33,070 So now this is using the Sobol filter to basically get the Sorbo edges. 196 00:12:33,400 --> 00:12:35,840 That's an algorithm that looks at the edges here. 197 00:12:35,860 --> 00:12:41,950 So we used it a bit to visualize that the high frequency mapping that you see above is basically an 198 00:12:41,950 --> 00:12:44,260 edge of the detector in case you didn't notice. 199 00:12:44,830 --> 00:12:46,180 That's what's going on here. 200 00:12:46,360 --> 00:12:49,540 So just this observation was noted here. 201 00:12:49,970 --> 00:12:55,340 So now we just create a function here that allows us to get the total variation lost. 202 00:12:55,360 --> 00:12:58,450 That's the high pass X and Y deltas. 203 00:12:58,450 --> 00:13:00,820 And then we use this TensorFlow function. 204 00:13:01,210 --> 00:13:07,390 We use some to get two absolute values of those of those outputs from that pass filter. 205 00:13:08,110 --> 00:13:12,430 And basically, if you run that function on the image, this is what it looks like. 206 00:13:13,600 --> 00:13:13,980 I don't know. 207 00:13:13,990 --> 00:13:16,180 I have this line twice and just delete it. 208 00:13:16,690 --> 00:13:18,970 And it's the same, not death anyhow. 209 00:13:19,060 --> 00:13:24,190 Now we can rerun the optimization, so let's choose a weight for total variation loss. 210 00:13:24,190 --> 00:13:29,220 And we used to go as the default width and we create a new trailing step function. 211 00:13:29,620 --> 00:13:34,390 Because this step brings up function now has a total variation loss in it. 212 00:13:34,930 --> 00:13:40,330 So we can create this here and an image as a TensorFlow variable. 213 00:13:40,960 --> 00:13:42,940 And now we can rerun the optimization. 214 00:13:42,940 --> 00:13:44,470 So let's take a look at it. 215 00:13:45,560 --> 00:13:47,690 You can see after 100 bucks, it looks quite good. 216 00:13:47,720 --> 00:13:52,820 And you can see after each e-book, it's getting better, actually, it's getting more. 217 00:13:53,450 --> 00:13:59,210 It's getting to be you can see it's blending between the images now, so it's smoothing out those artifacts 218 00:13:59,210 --> 00:14:01,600 that we want the total variation loss to effect. 219 00:14:01,610 --> 00:14:08,060 So it's definitely doing that and it's producing a more static image as we run our e-books. 220 00:14:08,930 --> 00:14:15,950 So maybe it'll finish shortly, but you can experiment to the edit finish after 23 seconds, but you 221 00:14:15,950 --> 00:14:21,200 can experiment with this and try different combinations of this. 222 00:14:21,820 --> 00:14:28,260 Wait, here was increased epochs if you want to actually consider wait right here, the variable again 223 00:14:28,440 --> 00:14:28,870 over here. 224 00:14:29,630 --> 00:14:34,880 So if you wanted to save that image, you can just run it, run this block of code and it automatically 225 00:14:34,880 --> 00:14:36,020 downloads that file for you. 226 00:14:36,170 --> 00:14:41,290 I can see I downloaded it before it blocked my download. 227 00:14:41,720 --> 00:14:48,440 When Q&A happens, sometimes they just allow it and you'll be able to download your stylized image output 228 00:14:48,440 --> 00:14:48,740 here. 229 00:14:49,100 --> 00:14:55,220 OK, so that's it for the TensorFlow and Keras implementation of neural cell transfer. 230 00:14:55,700 --> 00:15:00,380 Now we'll take a look at the PyTorch implementation, so I'll see you in the next lesson. 231 00:15:00,500 --> 00:15:00,950 Thank you.