WEBVTT 00:00.720 --> 00:01.860 -: Hello and welcome back to the course 00:01.860 --> 00:03.480 on artificial intelligence. 00:03.480 --> 00:04.320 In today's tutorial, 00:04.320 --> 00:06.210 we're going to talk about an add-on 00:06.210 --> 00:08.820 that we're going to be implementing for our A3C algorithm. 00:08.820 --> 00:11.370 It's called the long short term memory, 00:11.370 --> 00:13.098 or the LSTM for short. 00:13.098 --> 00:15.570 Let's have a look at what we have so far, 00:15.570 --> 00:19.350 and then we'll discuss why we need the LSTM 00:19.350 --> 00:21.660 and what an LSTM is. 00:21.660 --> 00:24.570 So far, we've discussed the A3C algorithm, 00:24.570 --> 00:27.750 we've talked about all three letter A's in the A3C, 00:27.750 --> 00:30.690 and of course, we've seen that it's actually a bit more, 00:30.690 --> 00:33.390 or much more complex than we have on this image, 00:33.390 --> 00:36.330 we actually have three or more multiple agents 00:36.330 --> 00:38.310 going through the environment and they're communicating 00:38.310 --> 00:39.660 between each other and so on, 00:39.660 --> 00:41.700 but for simplicity's sake, for today's tutorial, 00:41.700 --> 00:43.587 we're going to just illustrate everything 00:43.587 --> 00:45.660 with this one agent. 00:45.660 --> 00:47.970 At the end, we have this extra part, the critic part. 00:47.970 --> 00:51.267 Basically, once we have a state, that state, or this image, 00:51.267 --> 00:52.860 goes through a convolutional layer, 00:52.860 --> 00:54.334 then it goes through a pooling layer, 00:54.334 --> 00:55.350 it goes through a flattening layer, 00:55.350 --> 00:57.986 and at this point, we have values or numbers 00:57.986 --> 01:01.710 which are then propagated through the network 01:01.710 --> 01:04.050 and they go into the hidden layers, 01:04.050 --> 01:06.720 and then as the output, we get the policy, 01:06.720 --> 01:10.260 or the actor part, and then get the value of the state, 01:10.260 --> 01:12.840 or we get the critic part. 01:12.840 --> 01:14.970 And what we're going to do today 01:14.970 --> 01:16.725 is we're going to talk about this hidden part. 01:16.725 --> 01:18.270 In the hidden layers, 01:18.270 --> 01:20.460 we can actually take it to the next level 01:20.460 --> 01:21.720 and we can add a modification. 01:21.720 --> 01:24.720 And we've already seen that multiple modifications exist 01:24.720 --> 01:25.650 for the A3C algorithm. 01:25.650 --> 01:27.180 We've seen one of them, we've seen that, 01:27.180 --> 01:30.780 in some cases, you can have this main part of the network, 01:30.780 --> 01:32.910 which is individual to every agent, 01:32.910 --> 01:35.040 or you can have this main part of the network, 01:35.040 --> 01:38.070 which is shared, and that's what we saw that 01:38.070 --> 01:39.960 in our previous Intuition tutorials, 01:39.960 --> 01:42.180 we had a shared part of the network, 01:42.180 --> 01:44.190 this network was shared between the agents, 01:44.190 --> 01:47.790 and as Atlan will tell you more in the practical tutorials, 01:47.790 --> 01:50.790 that really helps with the challenge of Breakout. 01:50.790 --> 01:53.490 And there are lots and lots of lots of other ways 01:53.490 --> 01:55.385 you can modify the A3C algorithm, 01:55.385 --> 01:59.910 lots of other additions that can be implemented. 01:59.910 --> 02:01.440 And one of them we're going to discuss, 02:01.440 --> 02:02.550 because we are actually going to have it 02:02.550 --> 02:04.410 in the practical side of tutorials, 02:04.410 --> 02:06.000 here, before your hidden layers, 02:06.000 --> 02:08.640 what you can add is an LSTM layer, 02:08.640 --> 02:13.640 a neural network layer, which allows your A3C algorithm, 02:13.757 --> 02:15.990 which allows your algorithm to have memory, 02:15.990 --> 02:18.870 which allows the algorithm to remember what happened before. 02:18.870 --> 02:21.150 And we'll talk about LSTM just now in more detail. 02:21.150 --> 02:22.830 But basically, you can add an extra layer here, 02:22.830 --> 02:25.800 which is the LSTM layer, and enhance your algorithm 02:25.800 --> 02:28.020 with some additional memory of another feature. 02:28.020 --> 02:30.000 And what you'll actually see in the practical tutorials 02:30.000 --> 02:32.670 is that we didn't even need any hidden layers 02:32.670 --> 02:33.810 after the LSTM layer. 02:33.810 --> 02:36.210 You'll see that in Atlan's implementation, 02:36.210 --> 02:38.850 he's got the flattening layer right away, 02:38.850 --> 02:40.440 after that, he's got the LSTM layer, 02:40.440 --> 02:43.620 so basically, this box represents the LSTM layer, 02:43.620 --> 02:46.200 and then after that, right away you've got the output. 02:46.200 --> 02:49.500 You didn't even need any other hidden layers after that, 02:49.500 --> 02:52.110 simply because that's how much power the LSTM layer 02:52.110 --> 02:53.340 adds to the algorithm. 02:53.340 --> 02:55.242 And again, the algorithm, 02:55.242 --> 02:57.810 or the architecture of your neural network, 02:57.810 --> 03:00.120 it's a very individual thing, it's a personal preference, 03:00.120 --> 03:01.410 it's a very creative thing, 03:01.410 --> 03:03.750 so you might want to have two LSTM layers, 03:03.750 --> 03:05.337 you might have one LSTM layer 03:05.337 --> 03:08.970 and then several, like five hidden layers after the LSTM 03:08.970 --> 03:10.020 That's totally up to you 03:10.020 --> 03:11.940 and for you to experiment and explore. 03:11.940 --> 03:16.584 But this is what we came up with in the practical tutorials. 03:16.584 --> 03:19.320 You'll see that we have a flattening layer, 03:19.320 --> 03:22.710 and after that, we've got an LSTM layer, 03:22.710 --> 03:24.581 and then the output. 03:24.581 --> 03:27.150 Now that we've talked about the LSTM layer so much, 03:27.150 --> 03:28.830 what is this LSTM layer? 03:28.830 --> 03:30.900 Well, the LSTM layer adds memory, 03:30.900 --> 03:33.420 gives a feature, like allows the neural network 03:33.420 --> 03:35.520 to have memory about what happened 03:35.520 --> 03:37.020 in the previous situations, 03:37.020 --> 03:42.020 and it is often symbolized or shown with a symbol 03:42.180 --> 03:43.260 which looks like this. 03:43.260 --> 03:45.270 This is us just getting started into it, 03:45.270 --> 03:47.100 and I'm just putting here, I know it looks very crooked, 03:47.100 --> 03:49.320 but I'm putting it here so you can see 03:49.320 --> 03:51.780 when we further discuss this image, 03:51.780 --> 03:52.992 you can see what's going on. 03:52.992 --> 03:56.850 The output of this layer goes into here, 03:56.850 --> 04:01.590 and that is our, so this is a whole layer going in here, 04:01.590 --> 04:04.050 so it's a vector of values, X is a vector of values, 04:04.050 --> 04:06.030 goes into our LSTM, which we'll just discuss this now, 04:06.030 --> 04:07.860 and then as an output, you get another vector, 04:07.860 --> 04:10.110 which is the concatenation of this store, 04:10.110 --> 04:12.810 or somehow it ties in with network, in our case, 04:12.810 --> 04:15.660 as an output, you get this and you get this. 04:15.660 --> 04:17.632 Let's have a look at this in more detail. 04:17.632 --> 04:19.040 We're just gonna focus on this part. 04:19.040 --> 04:20.400 In fact, we're going to, 04:20.400 --> 04:22.620 as you probably noticed by the letters being on this side, 04:22.620 --> 04:25.974 we're gonna turn it to the side, so like that, 04:25.974 --> 04:28.530 and this whole like jumble around 04:28.530 --> 04:30.510 was just to reiterate the fact 04:30.510 --> 04:32.250 that even though it looks like this, 04:32.250 --> 04:35.370 what is actually happening is a layer of values, 04:35.370 --> 04:37.620 a whole vector of values, is going in here, 04:37.620 --> 04:39.570 something is happening, which we'll discuss just now, 04:39.570 --> 04:41.460 and then a whole vector of values is going on here. 04:41.460 --> 04:45.461 This is the layer, it's not just one element of it, 04:45.461 --> 04:47.550 this is the layer itself. 04:47.550 --> 04:50.970 Let's go back, again, just to reiterate. 04:50.970 --> 04:54.570 Layer goes into this layer, something happens, 04:54.570 --> 04:56.143 layer comes out. 04:56.143 --> 04:59.040 That's the LSTM, it's just on its side. 04:59.040 --> 05:00.690 It's just easier to discuss this way, 05:00.690 --> 05:02.280 and that's the common representation. 05:02.280 --> 05:06.030 All right, now that we agree why this image was on its side 05:06.030 --> 05:07.950 and how we're going to proceed with this, 05:07.950 --> 05:11.880 let's start digging into this LSTM situation a bit more. 05:11.880 --> 05:14.280 What happens inside the LSTM layer? 05:14.280 --> 05:15.570 This is what it looks like, 05:15.570 --> 05:18.000 and of course, this looks very complex, 05:18.000 --> 05:19.800 and we're definitely not gonna go 05:19.800 --> 05:21.581 through all of this right now 05:21.581 --> 05:23.910 simply because there's quite a lot to discuss. 05:23.910 --> 05:25.080 These are point-wise operations, 05:25.080 --> 05:26.880 these are layer-wise operations, 05:26.880 --> 05:29.790 and there's just a lot going on, 05:29.790 --> 05:32.723 or a lot of intricate details, 05:32.723 --> 05:35.340 which we're not going to go into 05:35.340 --> 05:37.767 because otherwise, it would blow out this course. 05:37.767 --> 05:40.270 And the purpose is not to talk about LSTMs here, 05:40.270 --> 05:42.450 we're just gonna utilize these LSTMs. 05:42.450 --> 05:45.660 And if you'd like to learn more about LSTMs, 05:45.660 --> 05:50.122 you can either go to over here, Christopher Olah's blog, 05:50.122 --> 05:52.170 he's got a good description of LSTMs, 05:52.170 --> 05:57.170 or we also talk about LSTMs in our deep learning HSN course. 05:57.360 --> 05:58.193 You can check it out there. 05:58.193 --> 06:00.240 We've got a whole section on recurrent neural networks 06:00.240 --> 06:02.040 and LSTMs, as well. 06:02.040 --> 06:06.510 Basically, this is the internal part of the LSTM. 06:06.510 --> 06:09.240 And what happens is like the layer goes in, 06:09.240 --> 06:12.180 so we're gonna talk about this on an intuitive level, 06:12.180 --> 06:13.500 on a very basic intuitive level, 06:13.500 --> 06:16.110 just what is gonna be enough for us to understand 06:16.110 --> 06:18.390 what happens, why there's memory, 06:18.390 --> 06:20.730 and so that you can also better understand 06:20.730 --> 06:24.198 what Atlan is talking about when he's implementing this. 06:24.198 --> 06:26.067 A layer goes in, all of this, 06:26.067 --> 06:29.700 something basically goes on here, a layer goes out. 06:29.700 --> 06:34.470 What we need to actually see is that these parts, 06:34.470 --> 06:37.620 there's actually additional inputs into this layer. 06:37.620 --> 06:40.336 Remember this, usually you have like an input 06:40.336 --> 06:42.630 from a previous layer, then this layer, 06:42.630 --> 06:43.560 and then you have an output, 06:43.560 --> 06:45.780 if you think of that image we had previously, 06:45.780 --> 06:49.620 the normal network, which is not on its side, 06:49.620 --> 06:52.830 which is like from left to right, not from bottom to top. 06:52.830 --> 06:55.470 But with an LSTM, you actually have more inputs. 06:55.470 --> 06:57.450 I know it's getting even more complex. 06:57.450 --> 07:00.240 But these things, at least we can understand them. 07:00.240 --> 07:04.380 This is your memory cell, this is the key, 07:04.380 --> 07:07.260 and this is what you'll hear Atlan talk about. 07:07.260 --> 07:12.090 The memory cell is something that is saved in the LSTM. 07:12.090 --> 07:15.450 These inputs and these outputs, they're actually, 07:15.450 --> 07:18.468 here what you're looking at is the time access. 07:18.468 --> 07:20.440 This is unraveled in time. 07:20.440 --> 07:24.930 In one specific iteration, this happens, 07:24.930 --> 07:27.060 but then this value is taken from the past 07:27.060 --> 07:28.890 and this value is passed into these values, 07:28.890 --> 07:30.600 these values are taken from the past, 07:30.600 --> 07:32.670 and these values are passed onto the future. 07:32.670 --> 07:35.580 And how they pass, well it's through the way the LSTM works. 07:35.580 --> 07:38.580 Without worrying about too much what's going on here, 07:38.580 --> 07:42.060 all we need to understand is that the layer goes in, 07:42.060 --> 07:45.840 and here we already have a value which came from the past, 07:45.840 --> 07:49.140 which is stored inside the LSTM, 07:49.140 --> 07:51.767 inside the long short term memory, 07:51.767 --> 07:54.780 we've got this memory cell, 07:54.780 --> 07:59.670 and whatever value was here before, it just stays here. 07:59.670 --> 08:00.870 As you can see, it just goes through, 08:00.870 --> 08:02.340 it flows through freely, 08:02.340 --> 08:04.530 except for these point-wise operations 08:04.530 --> 08:05.850 where it can be either closed off 08:05.850 --> 08:07.320 or something can be added to it. 08:07.320 --> 08:08.760 But regardless of that, 08:08.760 --> 08:11.610 it's just some value that flows through freely. 08:11.610 --> 08:14.400 It's basically, it's passed on to the next point in time, 08:14.400 --> 08:15.233 the next point in time, 08:15.233 --> 08:17.490 so you can just think of it as like some memory 08:17.490 --> 08:20.070 that's like a flash drive or something like that 08:20.070 --> 08:21.420 that this cell has. 08:21.420 --> 08:23.550 And so, it just remembers the previous value 08:23.550 --> 08:24.450 of this was in here, 08:24.450 --> 08:27.750 and then it can use that to either add to it 08:27.750 --> 08:30.360 or read from that value and so on. 08:30.360 --> 08:33.240 And this value is the hidden state. 08:33.240 --> 08:34.920 It's the, so hence the h, 08:34.920 --> 08:38.580 and the hidden state is basically another value 08:38.580 --> 08:42.000 that comes from the past and then is used inside the LSTM. 08:42.000 --> 08:45.360 And as you can see, at the end, after all of this happens, 08:45.360 --> 08:48.240 what you get is you get a layer that comes out 08:48.240 --> 08:50.970 and it is, so you get this value that comes out, 08:50.970 --> 08:53.430 and it is the same value that's passed forward. 08:53.430 --> 08:55.800 Basically, the LSTM remembers two things. 08:55.800 --> 09:00.150 There's a constant value that is just like stays in the LSTM 09:00.150 --> 09:02.520 that can be changed, like there's a flash drive 09:02.520 --> 09:06.030 for like a constant value, so the memory cell. 09:06.030 --> 09:09.420 And so, you have the luxury of storing something 09:09.420 --> 09:11.640 in that space, in that memory, 09:11.640 --> 09:13.590 and it'll be passed onto the future. 09:13.590 --> 09:15.270 Whenever, in the next iteration, 09:15.270 --> 09:18.810 so like the algorithm was in an environment, 09:18.810 --> 09:20.580 it saw something, it did something, and so on, 09:20.580 --> 09:23.250 and then in the LSTM, you can store a certain value, 09:23.250 --> 09:25.170 then it will remember this value 09:25.170 --> 09:27.090 even when it's in the next state. 09:27.090 --> 09:29.370 And also, the other value that it'll remember, 09:29.370 --> 09:31.410 it'll remember its previous output, 09:31.410 --> 09:33.750 it automatically will remember its previous output, 09:33.750 --> 09:35.790 so the output goes here and goes here. 09:35.790 --> 09:40.140 That's basically the very, very very high level 09:40.140 --> 09:42.150 of what happens in an LSTM. 09:42.150 --> 09:43.740 Once again, if you'd like more details, 09:43.740 --> 09:46.140 there's lots of resources where you can find, 09:46.140 --> 09:47.220 and at this stage, 09:47.220 --> 09:49.830 we just don't need to go into that much detail 09:49.830 --> 09:51.090 on all of these things. 09:51.090 --> 09:53.700 We just need to understand what a memory cell is, 09:53.700 --> 09:57.201 what a hidden state is, 09:57.201 --> 10:02.201 and how they facilitate memory for the LSTM. 10:02.340 --> 10:03.630 And the question is, 10:03.630 --> 10:06.240 so now that we kind of have a general overview 10:06.240 --> 10:11.240 of all of this, to reinforce or to solidify this knowledge, 10:12.240 --> 10:16.230 or kind of like give a reason for this knowledge, 10:16.230 --> 10:19.200 let's ask the question, why do we need memory? 10:19.200 --> 10:23.160 Why do we need memory in our A3C or other algorithms? 10:23.160 --> 10:24.990 Well, let's look at our example, 10:24.990 --> 10:27.090 the challenge that we're taking on in the section. 10:27.090 --> 10:29.970 The challenge is Breakout, and what happens in Breakout? 10:29.970 --> 10:32.430 Well, in Breakout, you've got this environment, 10:32.430 --> 10:34.830 these little blocks that you need to destroy 10:34.830 --> 10:36.780 with this little ball, and you need to make sure 10:36.780 --> 10:40.140 that this is your kind of like racket or platform 10:40.140 --> 10:42.300 that's moving around, and it must, 10:42.300 --> 10:44.250 wherever the ball is flying, it must catch the ball 10:44.250 --> 10:46.020 and it'll bounce off the platform 10:46.020 --> 10:46.890 and it'll go back and hit, 10:46.890 --> 10:48.120 it'll bounce off the walls as well, 10:48.120 --> 10:50.040 and go back, hit a block, and come back. 10:50.040 --> 10:54.300 And so, that's the essence of what you need to accomplish. 10:54.300 --> 10:55.749 But now, let's look at this ball. 10:55.749 --> 11:00.749 Imagine you are an A3C algorithm, 11:00.750 --> 11:04.140 or an agent inside, one of those agents inside A3C. 11:04.140 --> 11:05.490 You see this picture. 11:05.490 --> 11:07.680 What do you extract from here? 11:07.680 --> 11:10.050 What would your actions be here from here? 11:10.050 --> 11:11.850 You can see the ball is flying, right? 11:11.850 --> 11:14.340 Well, it's flying, right, so it's going somewhere. 11:14.340 --> 11:16.252 And maybe it's flying towards you, right? 11:16.252 --> 11:17.820 Could you make this conclusion? 11:17.820 --> 11:20.220 Could you like anticipate that it's coming towards you? 11:20.220 --> 11:22.890 You probably could and maybe you're in the right spot 11:22.890 --> 11:24.870 to catch the ball, but what if the ball 11:24.870 --> 11:26.670 is actually not flying that way, 11:26.670 --> 11:28.890 what if it's flying that way, what if it's flying that way? 11:28.890 --> 11:32.040 The thing is, you cannot tell from this one image 11:32.040 --> 11:33.990 which way it's flying because you don't know 11:33.990 --> 11:36.780 where it was in the previous moment of time. 11:36.780 --> 11:39.402 If it was here, then it's flying this way. 11:39.402 --> 11:41.760 If you knew the previous moment in time, 11:41.760 --> 11:43.440 if you knew that it was here, 11:43.440 --> 11:45.480 then now you know here is, as a human, 11:45.480 --> 11:47.280 you just draw a line from these two and you'd be like, 11:47.280 --> 11:48.930 oh cool, so it's going this way. 11:48.930 --> 11:50.231 But if you knew it was here, 11:50.231 --> 11:52.470 you draw a line for these and know it's going this way. 11:52.470 --> 11:54.330 Moreover, look at this, 11:54.330 --> 11:57.120 it could have actually been somewhere like over here. 11:57.120 --> 11:59.790 Maybe it's going up, maybe it's actually going that way. 11:59.790 --> 12:01.620 Maybe it was here and now it's going upward. 12:01.620 --> 12:04.650 Just from the one image, it's very hard, 12:04.650 --> 12:05.940 it's actually impossible, 12:05.940 --> 12:08.370 it's just like geometrically impossible 12:08.370 --> 12:10.530 to tell which way the ball is flying. 12:10.530 --> 12:14.460 And so, that's why the LSTM, the memory, 12:14.460 --> 12:17.130 actually really, really helps the algorithm. 12:17.130 --> 12:19.860 Without the memory, you can still do a good job. 12:19.860 --> 12:21.060 You could probably like guess 12:21.060 --> 12:24.630 or find other ways of understanding where to go. 12:24.630 --> 12:27.720 But with the LSTM, with just even that one memory, 12:27.720 --> 12:30.852 so if we go back, even with that one value, 12:30.852 --> 12:34.530 that was kind of like the output of the previous value, 12:34.530 --> 12:38.670 or maybe you can store it here, or based on this value, 12:38.670 --> 12:40.830 like based on the information that it gets 12:40.830 --> 12:43.260 from the previous point in time, 12:43.260 --> 12:45.900 so let's say from what happened here, 12:45.900 --> 12:48.060 so that's where your ball was before, 12:48.060 --> 12:50.760 so you can parse out information about the environment 12:50.760 --> 12:53.250 from the previous point in time through here, 12:53.250 --> 12:54.180 then now you have it. 12:54.180 --> 12:57.510 Now, not only have your information from the image, 12:57.510 --> 12:59.940 unless if we go back even further, 12:59.940 --> 13:02.850 you'll remember that information from the image, 13:02.850 --> 13:03.780 well this is Doom, 13:03.780 --> 13:05.340 but we're actually working with Breakout, 13:05.340 --> 13:07.740 information from the image came here, here, here, 13:07.740 --> 13:10.170 turned into these flattened values. 13:10.170 --> 13:11.910 And so, that's information from the image 13:11.910 --> 13:13.350 coming into the LSTM. 13:13.350 --> 13:14.772 And now, all of a sudden, 13:14.772 --> 13:18.480 as you remember, coming not from somewhere, 13:18.480 --> 13:21.300 but from the previous point in time, 13:21.300 --> 13:22.800 so that's why you kind of actually demonstrate 13:22.800 --> 13:24.570 it's coming from the top or from the bottom, left, right, 13:24.570 --> 13:28.470 it's actually, it just stays in the LSTM layer, 13:28.470 --> 13:29.820 you have that information, 13:29.820 --> 13:31.650 just through the architecture of the LSTM, 13:31.650 --> 13:34.110 you have the information about what happened previously. 13:34.110 --> 13:36.180 And so, we go back, 13:36.180 --> 13:38.820 that information here helps you now 13:38.820 --> 13:40.920 make a decision on what to do, 13:40.920 --> 13:42.990 helps the algorithm make a decision. 13:42.990 --> 13:45.120 And now all of a sudden, it knows that, 13:45.120 --> 13:48.360 oh okay, so the ball is actually flying in either area, 13:48.360 --> 13:50.340 let's say it's flying in this direction 13:50.340 --> 13:52.140 or in this direction, so I'm in the right place, 13:52.140 --> 13:53.220 so I should stick around here, 13:53.220 --> 13:54.810 the ball is coming in my direction. 13:54.810 --> 13:56.940 Or if it realizes that the ball is flying there, 13:56.940 --> 13:58.290 it should start moving to the left, 13:58.290 --> 13:59.880 because if it waits a bit longer, 13:59.880 --> 14:02.063 it'll be too late and it'll miss the ball. 14:02.063 --> 14:06.780 Basically, that's how the LSTM layer really helps 14:06.780 --> 14:09.273 in this algorithm, and that's exactly what we will see 14:09.273 --> 14:12.662 when you're doing the practical tutorials with Atlan. 14:12.662 --> 14:14.460 So there we go, that's how LSTMs work. 14:14.460 --> 14:18.240 And just one additional note, as we mentioned at the start, 14:18.240 --> 14:20.730 LSTMs are not a hundred percent necessary. 14:20.730 --> 14:24.600 They're not a complete, they're not completely attached 14:24.600 --> 14:25.560 to the A3C algorithm. 14:25.560 --> 14:27.990 You might want to have them in the A3C algorithm, 14:27.990 --> 14:29.880 you might not want to have them, depending on the situation, 14:29.880 --> 14:31.230 depending on the architecture you choose. 14:31.230 --> 14:32.280 There are lots of additions, 14:32.280 --> 14:36.600 and we've already discussed the addition or the modification 14:36.600 --> 14:38.910 where the neural network is shared between actors, 14:38.910 --> 14:43.170 or is not shared between agents or not now the less LSTM. 14:43.170 --> 14:44.910 There's another one that you'll see 14:44.910 --> 14:46.260 in the practical tutorials 14:46.260 --> 14:47.640 where we add entropy, 14:47.640 --> 14:49.500 which is calculated through our policy loss, 14:49.500 --> 14:51.570 and Atlan will walk you through that. 14:51.570 --> 14:54.630 Basically, there's lots of different modifications 14:54.630 --> 14:57.330 that can happen in an A3C algorithm. 14:57.330 --> 15:00.540 Just remember that it depends on what you want to achieve 15:00.540 --> 15:03.930 and it's also something that we'd encourage you to explore 15:03.930 --> 15:06.210 if you're going to be implementing lots of these 15:06.210 --> 15:08.193 and trying out different algorithms. 15:09.150 --> 15:10.320 We've already discussed a couple, 15:10.320 --> 15:13.950 and maybe you can find some additional modifications 15:13.950 --> 15:15.540 that might be of interest to you. 15:15.540 --> 15:17.130 Or maybe when you're watching these tutorials, 15:17.130 --> 15:19.830 maybe by then more modifications have come out 15:19.830 --> 15:22.050 which are very interesting. 15:22.050 --> 15:24.630 Definitely that's something that you could look into 15:24.630 --> 15:27.420 and that could further enhance your knowledge 15:27.420 --> 15:29.130 of artificial intelligence 15:29.130 --> 15:30.750 and how to create these algorithms. 15:30.750 --> 15:32.760 And on that note, I hope you enjoyed today's tutorial 15:32.760 --> 15:34.200 and I'll look forward to seeing you next time. 15:34.200 --> 15:35.733 Until then, enjoy AI.