WEBVTT 00:00.750 --> 00:03.510 -: Hi guys. Welcome to the final tutorial for Breakout. 00:03.510 --> 00:05.580 -: Hey guys, super excited to have you here. 00:05.580 --> 00:08.701 -: And super happy to be all together again after Doom. 00:08.701 --> 00:09.534 -: Yeah. 00:09.534 --> 00:11.070 -: So this time we're gonna watch the result for Breakout 00:11.070 --> 00:14.370 which I remind was much more challenging than doom. 00:14.370 --> 00:15.960 So let's hope we get some good results. 00:15.960 --> 00:16.793 -: Okay. 00:16.793 --> 00:17.626 -: Yeah. -: So what are we doing? 00:17.626 --> 00:19.335 -: So here is the main code. 00:19.335 --> 00:20.168 -: Yeah. 00:20.168 --> 00:21.270 -: So, you know, we have several codes. 00:21.270 --> 00:22.325 The, all the codes are here, 00:22.325 --> 00:23.158 -: Mhmm 00:23.158 --> 00:23.991 and here is the main code. 00:23.991 --> 00:24.824 -: Yeah 00:24.824 --> 00:26.880 -: The code that will execute all the other codes. 00:26.880 --> 00:30.030 So build the new network and train the, the agents. 00:30.030 --> 00:30.863 -: Yeah. 00:30.863 --> 00:31.696 -: And the critic., okay? 00:31.696 --> 00:34.800 So basically what we're gonna do is just execute 00:34.800 --> 00:36.450 just to see how it's going, 00:36.450 --> 00:37.979 but we will not wait for the results 00:37.979 --> 00:40.020 because actually it takes several hours. 00:40.020 --> 00:40.853 -: Yeah. 00:40.853 --> 00:43.410 -: So we, we prepared the results in this test folder. 00:43.410 --> 00:44.790 We already have the video 00:44.790 --> 00:46.992 and we will show you right after that. 00:46.992 --> 00:47.825 -: Okay. 00:47.825 --> 00:48.658 -: Okay? 00:48.658 --> 00:52.200 So to execute, basically we just select the whole code here 00:52.200 --> 00:53.703 and here we go. 00:56.970 --> 00:57.990 Yeah. -: All right. 00:57.990 --> 01:00.780 -: And as you can see, the videos are populated. 01:00.780 --> 01:01.613 -: Okay. 01:01.613 --> 01:02.550 -: All right. 01:02.550 --> 01:04.770 So right now we already have two videos 01:04.770 --> 01:06.810 and then the agents are training 01:06.810 --> 01:08.550 but it's gonna take a long time. 01:08.550 --> 01:10.757 So now what we're gonna do is watch the results. 01:10.757 --> 01:11.590 -: Okay. 01:11.590 --> 01:12.423 -: Okay. 01:12.423 --> 01:13.256 -: Which I haven't seen yet. 01:13.256 --> 01:14.089 -: You haven't seen it yet, yeah. 01:14.089 --> 01:14.922 -: All right. 01:14.922 --> 01:15.755 -: Right. 01:15.755 --> 01:18.690 So, in the template folder that you will find 01:18.690 --> 01:21.141 on the SDS website, you will find two folders. 01:21.141 --> 01:21.974 -: Mhm. 01:21.974 --> 01:22.920 -: Code no comments, 01:22.920 --> 01:24.150 and code with comments. 01:24.150 --> 01:26.460 So that's, even if you missed any detail 01:26.460 --> 01:27.628 you can still have a look at this code. 01:27.628 --> 01:28.650 (mouse clicking) 01:28.650 --> 01:30.630 You have all the comments well explained. 01:30.630 --> 01:34.260 So, each line is well commented so that you can understand. 01:34.260 --> 01:37.260 And then in this folder, we'll find this test folder 01:37.260 --> 01:39.840 and that's the folder that will be populated 01:39.840 --> 01:40.673 with the videos. 01:40.673 --> 01:41.506 -: Okay. 01:41.506 --> 01:42.690 -: So that's where the videos are. 01:42.690 --> 01:43.560 So this- 01:43.560 --> 01:46.440 these are the videos of our previous training. 01:46.440 --> 01:48.480 You know, we trained the model for a couple of hours 01:48.480 --> 01:50.700 and we obtained these videos. 01:50.700 --> 01:52.170 So we're gonna watch them one by one 01:52.170 --> 01:53.183 starting with the first one. 01:53.183 --> 01:54.016 -: Mhm. 01:54.016 --> 01:54.930 -: And we're gonna see if, 01:54.930 --> 01:56.457 you know, -: Yeah 01:56.457 --> 01:57.870 -: we see some artificial intelligence playing breakout 01:57.870 --> 02:00.190 and if we manage to solve breakout. 02:00.190 --> 02:01.080 Okay? -: Okay. 02:01.080 --> 02:01.913 Let's have a look. 02:01.913 --> 02:03.180 -: So let's have a look at the first video. 02:03.180 --> 02:05.280 So this first video is, you know... 02:05.280 --> 02:06.330 Actually what we're gonna do. 02:06.330 --> 02:07.163 Yeah. 02:07.163 --> 02:08.190 What we're gonna do is stop the training 02:08.190 --> 02:11.138 because it's slowing down my computer right now, so, okay. 02:11.138 --> 02:11.971 -: Mhm. 02:11.971 --> 02:12.804 -: So interrupted, 02:12.804 --> 02:13.980 but all good. 02:13.980 --> 02:14.813 -: Yeah. 02:14.813 --> 02:16.020 -: So let's watch the first video. 02:16.890 --> 02:20.087 It's gonna be really bad because it's really the first one. 02:20.087 --> 02:21.720 The, the pedal might not even move. 02:21.720 --> 02:22.553 There we go. 02:22.553 --> 02:23.386 Yeah, it doesn't move. 02:23.386 --> 02:24.219 -: Why doesn't it move at all? 02:24.219 --> 02:26.100 -: Because you know, it's the very beginning. 02:26.100 --> 02:27.750 The AI doesn't know what to do. 02:27.750 --> 02:28.583 -: Yeah. 02:28.583 --> 02:30.995 -: And it's still pre-processing the images, so yeah. 02:30.995 --> 02:32.010 Nothing. -: Gotcha. 02:32.010 --> 02:34.920 -: And it might even be the case for the second video. 02:34.920 --> 02:36.420 If we look into the second one 02:36.420 --> 02:38.490 the paddle might not, 02:38.490 --> 02:39.540 might still not move. 02:41.910 --> 02:42.743 -: Okay. 02:42.743 --> 02:44.070 -: So how, what do you think of this 02:44.070 --> 02:45.362 artificial intelligence so far -: A3C sucks(chuckles). 02:45.362 --> 02:47.220 (laughing) 02:47.220 --> 02:49.590 -: So that was worth spending so much energy 02:49.590 --> 02:50.423 on the A3C, right? 02:50.423 --> 02:51.689 -: Yeah (chuckles). 02:51.689 --> 02:52.522 -: Okay. 02:52.522 --> 02:54.360 So next video, we might have something. 02:54.360 --> 02:55.193 -: All right, let's have a look. 02:55.193 --> 02:56.220 -: Maybe a little something, 02:56.220 --> 02:57.720 a little bit of artificial intelligence. 02:57.720 --> 03:00.047 -: Okay, you're giving away too much. 03:00.047 --> 03:02.380 (chuckling) 03:05.160 --> 03:06.621 -: Not even. 03:06.621 --> 03:07.454 -: Not even, no. 03:07.454 --> 03:08.287 -: Not even. 03:08.287 --> 03:09.120 -: Useless. 03:09.120 --> 03:10.020 -: Are we gonna break breakout or... 03:10.020 --> 03:10.853 -: No. 03:10.853 --> 03:11.780 -: No? 03:11.780 --> 03:12.843 -: All right. Come on. Let's do something, 03:12.843 --> 03:13.676 -: Okay 03:13.676 --> 03:16.713 -: Let's, let's hope that we get something for the next one. 03:17.850 --> 03:19.470 Ah, yeah, I think we get something 03:19.470 --> 03:22.222 because now that's 11 seconds. 03:22.222 --> 03:23.055 -: Okay. 03:23.055 --> 03:24.570 -: There we go. 03:24.570 --> 03:25.403 -: Up. 03:25.403 --> 03:26.236 -: Better, okay. 03:26.236 --> 03:28.890 So now the AI has eyes, it can see. 03:28.890 --> 03:29.887 -: Yeah. 03:29.887 --> 03:30.720 -: It can sense what it's doing. 03:30.720 --> 03:31.553 -: Yeah. 03:32.400 --> 03:33.690 -: But it's not very quick yet. 03:33.690 --> 03:34.523 -: Yeah. 03:34.523 --> 03:35.356 -: It's not a smart AI. 03:35.356 --> 03:36.269 -: Okay gotcha. 03:36.269 --> 03:37.102 -: But better. 03:37.102 --> 03:37.935 -: Yep. 03:37.935 --> 03:38.768 -: Okay, so four. 03:38.768 --> 03:39.601 -: Four. 03:39.601 --> 03:40.434 -: Okay. 03:40.434 --> 03:41.384 So next video now. 03:41.384 --> 03:42.900 -: Mhm. 03:42.900 --> 03:44.400 -: Ooh, it looks much better now. 03:44.400 --> 03:45.570 -: Yeah. 03:45.570 --> 03:46.403 More time. 03:46.403 --> 03:47.236 -: Yeah. 03:47.236 --> 03:48.069 Are you ready? 03:48.069 --> 03:48.902 -: Yep. Let's have a look. 03:48.902 --> 03:49.924 -: Yeah. 03:49.924 --> 03:50.757 -: Okay. 03:50.757 --> 03:51.590 Got it. 03:51.590 --> 03:52.830 I got another one. 03:52.830 --> 03:54.120 Ah, no, you lost it. 03:54.120 --> 03:54.953 -: Yeah. 03:54.953 --> 03:59.160 Still not perfect, but definitely it's like a human playing. 03:59.160 --> 03:59.993 -: Yeah. 03:59.993 --> 04:01.373 -: Not very good at breakout, but still it's like a- 04:02.731 --> 04:03.564 -: Like a 04:03.564 --> 04:04.397 -: human playing. 04:04.397 --> 04:06.810 -: like a five year old, two year old human playing. 04:06.810 --> 04:08.250 -: Yeah yeah. 04:08.250 --> 04:09.083 -: Okay. 04:09.083 --> 04:10.380 Wow, It's actually getting them. 04:10.380 --> 04:12.000 -: Yeah, a good two year old now. 04:12.000 --> 04:12.833 -: Yeah. 04:14.610 --> 04:17.550 -: But it's like, it's doing, you know 04:17.550 --> 04:21.150 trying to catch the ball with some weird tactic. 04:21.150 --> 04:22.440 -: Yeah. 04:22.440 --> 04:24.150 -: It's not like a perfect 04:24.150 --> 04:25.770 prediction of where the ball is going to be. 04:25.770 --> 04:26.959 -: Gotcha. Okay. 04:26.959 --> 04:27.792 -: Okay. 04:27.792 --> 04:28.625 -: Let's hope it gets better. 04:28.625 --> 04:29.458 -: So much better at 27. 04:29.458 --> 04:30.291 -: Uh-huh. 04:30.291 --> 04:31.124 -: Okay. 04:31.124 --> 04:31.957 Next one, 04:33.510 --> 04:34.421 37 again. 04:34.421 --> 04:35.254 -: Oh, okay. 04:35.254 --> 04:36.087 -: Okay. 04:36.087 --> 04:37.528 -: Yeah, that was 27, this is 37. 04:37.528 --> 04:40.170 -: Okay. No, I mean the time. 04:40.170 --> 04:41.190 -: Okay. 04:41.190 --> 04:42.023 Oh okay. 04:42.023 --> 04:43.140 -: Yeah. 04:43.140 --> 04:44.644 Now I would say it's a five year old. 04:44.644 --> 04:46.977 (chuckling) 04:50.550 --> 04:51.870 -: Okay. 04:51.870 --> 04:52.830 -: Hmm. Pretty good, huh? 04:52.830 --> 04:54.060 -: Yeah. 04:54.060 --> 04:56.763 -: And now the goal, the goal is going faster. 04:56.763 --> 04:57.596 -: Mhm. 04:57.596 --> 04:59.550 -: So it's of course much more difficult. 04:59.550 --> 05:00.950 But still, it's doing great. 05:02.910 --> 05:06.300 So let's see if we can beat 27 this time. 05:06.300 --> 05:08.707 23, 24, 05:08.707 --> 05:10.290 28. Okay, here we go. 05:10.290 --> 05:11.401 See it's getting better and better. 05:11.401 --> 05:12.234 -: Yeah. Yeah. 05:12.234 --> 05:13.067 -: That's a good sign. 05:13.067 --> 05:14.010 -: Mhm. 05:14.010 --> 05:14.843 -: Okay. 05:14.843 --> 05:15.853 -: Cool. 05:15.853 --> 05:16.903 -: Lasted three lives. 05:18.270 --> 05:19.206 Next one. 05:19.206 --> 05:20.250 (mouse clicking) 05:20.250 --> 05:21.990 Whoa. 05:21.990 --> 05:23.135 One minute, 10 seconds. 05:23.135 --> 05:24.180 -: Oh okay. 05:24.180 --> 05:25.013 -: Let's see. 05:28.530 --> 05:29.850 -: Straight there. 05:29.850 --> 05:30.683 That's really cool. 05:30.683 --> 05:32.100 -: Yeah, that's definitely a pro now, 05:32.100 --> 05:32.933 isn't it? 05:32.933 --> 05:34.350 -: Yeah. And you can tell it's not a human playing 05:34.350 --> 05:37.860 because it's like it's moving sporadically like super fast. 05:37.860 --> 05:38.820 -: Yes. 05:38.820 --> 05:40.410 -: And it's doing a lot of, 05:40.410 --> 05:41.790 I would say even unnecessary movements, 05:41.790 --> 05:42.930 but it's still getting there. 05:42.930 --> 05:43.763 -: Yeah. 05:43.763 --> 05:46.013 Let's see how it does when the ball is going really fast. 05:46.920 --> 05:49.050 -: Where does the ball speed depend on? 05:49.050 --> 05:50.223 -: Um. 05:51.420 --> 05:53.010 I don't know. I think it's a.... 05:53.010 --> 05:55.620 -: Does it depend on the color of the block it destroys? 05:55.620 --> 05:56.700 -: Maybe. 05:56.700 --> 05:58.140 -: Or is it just like random? 05:58.140 --> 06:00.600 -: Mm, it looks like random but 06:00.600 --> 06:02.520 we should check out the breakout rules. 06:02.520 --> 06:03.540 -: Yeah. 06:03.540 --> 06:05.323 -: But it's, it's really doing great now. 06:05.323 --> 06:06.156 -: Yeah. 06:06.156 --> 06:07.560 -: 34. 34 score. 06:07.560 --> 06:08.430 -: Nice. 06:08.430 --> 06:11.370 -: Wow. It's really, really doing great. 06:11.370 --> 06:12.480 So yeah we definitely have 06:12.480 --> 06:15.390 some A3C playing and breakout now. 06:15.390 --> 06:16.223 -: Nice. 06:18.720 --> 06:20.130 With an LSTM, yeah? 06:20.130 --> 06:21.090 -: Yeah, this was an LSTM. 06:21.090 --> 06:22.740 So it's like we have a brain with a memory. 06:22.740 --> 06:23.573 -: Yeah. 06:23.573 --> 06:24.477 -: Not only with eyes. 06:24.477 --> 06:25.310 -: Mhm. 06:25.310 --> 06:26.143 -: It has good eyes, you know? 06:26.143 --> 06:26.976 It's a quick, pretty quick 06:26.976 --> 06:27.809 -: Yeah. 06:27.809 --> 06:28.642 -: to catch the ball. 06:28.642 --> 06:30.040 But also it's got some memory, but 06:32.430 --> 06:33.263 I don't know. 06:33.263 --> 06:34.920 Would you use your memory to play breakout yourself? 06:34.920 --> 06:36.210 -: Of course. 06:36.210 --> 06:37.043 -: Yeah. 06:37.043 --> 06:37.876 -: Okay. 06:37.876 --> 06:39.030 So, well done. 06:39.030 --> 06:41.387 Well done to the A3C, the AI. -: Okay. What's the, 06:41.387 --> 06:42.870 what's the last one? 06:42.870 --> 06:43.923 -: The last one? 06:46.020 --> 06:46.920 Oh. 06:46.920 --> 06:47.760 55 seconds. 06:47.760 --> 06:48.660 So a little less. 06:48.660 --> 06:49.493 -: Yeah. 06:49.493 --> 06:50.610 -: You remember what the score was? 06:50.610 --> 06:51.443 -: No. 06:51.443 --> 06:52.276 -: On the previous one? 06:52.276 --> 06:54.540 Let's check it out to see if we get some improvement. 06:54.540 --> 06:57.011 We might not get some improvement every video. 06:57.011 --> 06:58.271 -: Yeah. Yeah. Exploration. 06:58.271 --> 06:59.452 -: Okay, so that was 78. 06:59.452 --> 07:00.660 -: Okay. 07:00.660 --> 07:02.910 -: Yeah, that's due to exploration as you say. 07:06.450 --> 07:07.713 Okay, there we go. 07:10.410 --> 07:11.687 -: Hmm hmm hmm 07:16.800 --> 07:17.633 Okay. 07:17.633 --> 07:19.980 Think we should have like some music playing while was- 07:19.980 --> 07:21.510 -: Yes (chuckling). 07:21.510 --> 07:22.533 Like hypnotizing. 07:27.600 --> 07:28.533 You can do it. 07:39.468 --> 07:40.548 It's like on steroids. 07:40.548 --> 07:42.873 Clicking left to right all the time. 07:45.750 --> 07:46.830 -: But it's doing good. 07:46.830 --> 07:47.663 -: Yeah. 07:47.663 --> 07:52.200 -: But we might not solve the whole game of breakout but- 07:52.200 --> 07:53.033 -: Yeah 07:53.033 --> 07:54.990 -: we just need to have it trained more. 07:54.990 --> 07:58.080 -: Yeah, that was already really good. 07:58.080 --> 07:59.330 -: It's really good, yeah. 08:01.200 --> 08:02.760 Oh boy -: That speed. 08:02.760 --> 08:04.440 -: Oh yeah, that was a little bit too fast. 08:04.440 --> 08:05.273 -: Sixties. 08:05.273 --> 08:06.750 So it beat the score again. 08:06.750 --> 08:08.370 -: That was 78, wasn't it? -: I know. 08:08.370 --> 08:09.240 -: Yeah. 08:09.240 --> 08:11.473 Okay, so we did it, didn't we? 08:11.473 --> 08:12.306 -: Good stuff. 08:12.306 --> 08:13.320 -: Definitely some AI playing breakout. 08:13.320 --> 08:14.153 -: Yeah. Nice. 08:14.153 --> 08:15.120 Very nice, very nice. 08:15.120 --> 08:15.953 Awesome. 08:15.953 --> 08:17.010 -: So I hope you liked it. 08:17.010 --> 08:21.360 And mostly, I hope you liked this powerful implementation 08:21.360 --> 08:23.130 of the A3C 08:23.130 --> 08:24.000 and um- 08:24.000 --> 08:27.270 -: Yeah, you can see what how, how it can do really well 08:27.270 --> 08:28.920 with the LSTM. 08:28.920 --> 08:29.753 Yeah? 08:29.753 --> 08:30.686 -: That's right. 08:30.686 --> 08:33.600 Like at- at the end of the A3C paper, 08:33.600 --> 08:35.850 they have like the comparison of different algorithms 08:35.850 --> 08:37.590 and this one is the best one by far. 08:37.590 --> 08:39.090 -: Not only they have the best, 08:39.090 --> 08:40.620 the comparison of these models 08:40.620 --> 08:43.500 but also I experienced several of them. 08:43.500 --> 08:47.250 And so I experienced one without and LSTM and one with 08:47.250 --> 08:49.950 and definitely the LSTM improved the algorithm. 08:49.950 --> 08:52.140 -: How much better is it within those? 08:52.140 --> 08:52.973 -: Much better. 08:52.973 --> 08:56.760 Like I, I barely some artificial intelligence 08:56.760 --> 08:58.230 before playing breakout. 08:58.230 --> 09:00.030 It last like 15 seconds. 09:00.030 --> 09:00.863 -: Okay. 09:00.863 --> 09:01.696 -: Or 20 seconds. 09:01.696 --> 09:04.473 But definitely not more than one minute as we saw here. 09:04.473 --> 09:05.306 -: Yeah yeah. 09:05.306 --> 09:06.510 That's good, that's good. 09:06.510 --> 09:07.380 Okay, cool. 09:07.380 --> 09:08.213 -: Cool. 09:08.213 --> 09:09.240 Thanks a lot guys. 09:09.240 --> 09:10.073 -: Thanks a lot. 09:10.073 --> 09:11.850 That was the end of the third module 09:11.850 --> 09:12.960 and therefore the end of the course. 09:12.960 --> 09:16.320 So, congratulations for having competed the three modules 09:16.320 --> 09:19.498 and mostly the A3C, which was a pretty big (chuckles). 09:19.498 --> 09:20.331 Pretty big. 09:20.331 --> 09:21.164 -: Yeah yeah. 09:21.164 --> 09:23.580 And for sure explore the different modification that, 09:23.580 --> 09:25.980 modifications that exist for the A3C. 09:25.980 --> 09:27.083 -: Yes. 09:27.083 --> 09:28.230 -: And you know, maybe at the time you're watching this 09:28.230 --> 09:31.050 there's some really cool new advancements 09:31.050 --> 09:32.323 in A3C and you can have some- 09:32.323 --> 09:33.480 -: Probably probably, 09:33.480 --> 09:36.210 or even a new model even better than the A3C. 09:36.210 --> 09:37.043 -: Yeah 09:37.043 --> 09:37.876 For sure. 09:37.876 --> 09:39.210 -: Let's see what 2018 says. 09:39.210 --> 09:40.104 -: Okay. 09:40.104 --> 09:40.937 All right. -: All right 09:40.937 --> 09:41.770 Thanks all you guys. 09:41.770 --> 09:42.603 -: Thanks. 09:42.603 --> 09:43.436 Bye. 09:43.436 --> 09:45.936 (piano music)