1 00:00:00,590 --> 00:00:08,480 Lesson seven Project St. Augustine recognition with microphones specifically for model training and 2 00:00:08,480 --> 00:00:13,850 development, convolutional layers are the main parts of convolutional neural networks. 3 00:00:13,880 --> 00:00:15,920 They are used to build the networks. 4 00:00:16,550 --> 00:00:22,670 A convolution is the simple act of applying a filter to an input that results in activation, which 5 00:00:22,670 --> 00:00:28,850 is called convolutional repeated use of the same filter results in a map activation called a feature 6 00:00:28,850 --> 00:00:35,360 map, which shows the location and strength of a detected feature in an input such as sequence of data 7 00:00:35,360 --> 00:00:40,940 or the one d an image, the two D or Point Cloud the 3D. 8 00:00:41,480 --> 00:00:43,730 This map is called a feature map. 9 00:00:44,360 --> 00:00:44,540 Three. 10 00:00:44,630 --> 00:00:51,050 The people who use convolutional neural networks see that they can automatically learn a lot of filters 11 00:00:51,050 --> 00:00:57,380 in a short time that are specific to weight training data and a specific predictive model problem, 12 00:00:57,830 --> 00:01:00,920 like predicting which image belong to which people. 13 00:01:01,550 --> 00:01:08,420 The result is a set of very specific features that can be found anywhere in the input data. 14 00:01:08,810 --> 00:01:15,080 Now, using a C and can help you find simple patterns in your data, which can then be used to make 15 00:01:15,080 --> 00:01:17,960 more complex patterns in the higher layers of your data. 16 00:01:18,680 --> 00:01:24,230 A one designed and is very useful when you want to find interesting features in these short term fixed 17 00:01:24,230 --> 00:01:24,530 length. 18 00:01:24,530 --> 00:01:32,330 Part of the overall dataset and the location of the feature in the segment isn't very important now 19 00:01:32,330 --> 00:01:35,080 for preparation for expansion task. 20 00:01:35,330 --> 00:01:40,790 Make sure that we are terminal facilities, wife aware and Wi-Fi libraries. 21 00:01:41,090 --> 00:01:47,180 You can check it by uploading the following code and then open serial monitoring, and you should see 22 00:01:47,180 --> 00:01:49,910 a female version displayed as output on the screen. 23 00:01:50,420 --> 00:01:54,200 If there is no output installed, the latest Wi-Fi library. 24 00:01:54,680 --> 00:02:04,700 Step one Open Arduino ID and then click Sketch Locate include library and then look at manage libraries. 25 00:02:05,150 --> 00:02:12,200 Step two The name of the library that we need and select the latest version from the dropdown menu if 26 00:02:12,200 --> 00:02:12,770 available. 27 00:02:13,190 --> 00:02:15,830 Then lastly, you can now click Install. 28 00:02:16,520 --> 00:02:19,550 Also, you need to make sure you have installed the following libraries. 29 00:02:19,970 --> 00:02:25,910 You can search for these libraries by typing the library name in the search box of Arduino Library Manager. 30 00:02:27,020 --> 00:02:30,500 We have seen Arduino RBC Wi-Fi. 31 00:02:31,970 --> 00:02:34,580 We have seen Arduino, RBC Unified. 32 00:02:34,850 --> 00:02:35,800 We have seen it. 33 00:02:35,930 --> 00:02:38,470 Do we know anybody else? 34 00:02:38,900 --> 00:02:40,400 We have seen Arduino FC. 35 00:02:40,850 --> 00:02:45,980 We also have seen the Arduino SFE with the now among the processing blocks. 36 00:02:45,980 --> 00:02:53,810 We see three familiar options, namely raw spectral analysis, which is essentially for your transfer 37 00:02:53,810 --> 00:03:01,970 of the signal spectrogram and MFP, or known as the Mel Frequency Energy Backs, which correspond to 38 00:03:01,970 --> 00:03:03,920 four stages of audio processing. 39 00:03:04,310 --> 00:03:09,590 We described earlier for this part try all of them if you like trying new things. 40 00:03:09,920 --> 00:03:15,800 The only one that might be too much for our small neural network is wrong, which has a lot of data 41 00:03:15,800 --> 00:03:16,730 for us to handle. 42 00:03:17,060 --> 00:03:24,380 For this lesson, we'll just use the NFC or the Mel Frequency Energy bands, which aren't the best choice 43 00:03:24,380 --> 00:03:25,220 for this task. 44 00:03:25,880 --> 00:03:31,490 It is time to go to the Classifier tab and choose a modal architecture that fits your needs. 45 00:03:32,000 --> 00:03:35,900 We can use the one deep cons or the two con. 46 00:03:36,680 --> 00:03:41,390 It doesn't matter which one you choose, because both will work if possible. 47 00:03:41,630 --> 00:03:46,490 We should always go with the smaller model because we'll be using it on embedded device. 48 00:03:47,150 --> 00:03:52,130 It took us a long time to write the materials for discourse, so we did four different experiments. 49 00:03:52,760 --> 00:04:01,280 They were one second or the 2D gone with NFC or MST features, and the results are shown in this table. 50 00:04:01,850 --> 00:04:08,840 And for us, the best model was one they called network with MFI processing block by tweaking MFP parameters, 51 00:04:09,230 --> 00:04:14,060 namely the increasing straight to zero point zero two and decreasing low frequency to zero. 52 00:04:14,330 --> 00:04:19,580 We have achieved accuracy of eighty nine point four percent on validation data set. 53 00:04:20,090 --> 00:04:24,830 You can find a trained model in the course materials and try it out for yourself to see how well it 54 00:04:24,830 --> 00:04:25,250 works. 55 00:04:25,850 --> 00:04:31,790 It is good at separating crying sounds from the background, but its ability to detect totting sounds 56 00:04:31,820 --> 00:04:32,840 isn't very good. 57 00:04:33,260 --> 00:04:38,300 It might need to get more samples, actually, as soon as we've made a model that's accurate enough 58 00:04:38,300 --> 00:04:38,960 to train on. 59 00:04:39,350 --> 00:04:41,330 We can test it in a new data. 60 00:04:41,780 --> 00:04:48,980 The labs classification and then send it to the real terminal now in the adrenal libraries will download 61 00:04:48,980 --> 00:04:54,600 it as an adrenal library, put it into adrenal library folder and open examples. 62 00:04:55,130 --> 00:05:01,910 The name of your project and then locate the Nano three dialysis microphone continuous to see how it 63 00:05:01,910 --> 00:05:02,330 works. 64 00:05:02,840 --> 00:05:11,060 Now, based on the adrenal Nano 33 B Elite and the EDM library for Real Terminal, we will use the DMA 65 00:05:11,060 --> 00:05:18,260 or direct memory access to get samples from the ADC or the analog to digital converter and then send 66 00:05:18,260 --> 00:05:21,470 them to the inference buffer with the help of the MCU. 67 00:05:22,070 --> 00:05:28,460 In that way, we can get the sound samples and figure out what they mean at the same time when we want 68 00:05:28,460 --> 00:05:31,730 to move sound data from the PD and library to DME. 69 00:05:32,150 --> 00:05:33,710 We need to make a lot of changes. 70 00:05:34,190 --> 00:05:39,170 If you get a little lost during the expedition, check out the full sample code, which you can find 71 00:05:39,290 --> 00:05:40,550 in the course materials. 72 00:05:41,600 --> 00:05:44,020 Don't delete the PDM library from sketch. 73 00:05:44,870 --> 00:05:50,180 Add the main descriptor structure and other settings consent right after losing to the statement. 74 00:05:50,840 --> 00:05:57,080 Then, right before a set of function, create variables for buffer arrays, volatile variables for 75 00:05:57,080 --> 00:06:03,200 passing the values between ESR callback and the main code, and also high pass Butterworth's filter, 76 00:06:03,560 --> 00:06:09,110 which we will apply to signal to eliminate most of the DC component in microphone signal. 77 00:06:09,740 --> 00:06:16,220 Now, make sure to add three blocks of code after that first one is called callback function called 78 00:06:16,220 --> 00:06:19,940 by ESR or the interrupt service routine. 79 00:06:20,510 --> 00:06:26,840 Every time one of the two buffers field inside the function, we read the elements from recording buffer. 80 00:06:27,410 --> 00:06:33,290 The one that was field just now processed them and put it into an inference buffer. 81 00:06:33,800 --> 00:06:36,200 Next block contains the ESR itself. 82 00:06:36,500 --> 00:06:40,250 It is executed by a timer at a certain period of time. 83 00:06:40,550 --> 00:06:48,080 Inside of that function, we check if the DMA see Channel one has been suspended, if it has been suspended. 84 00:06:48,110 --> 00:06:53,420 It means that one of the buffers for microphone data has failed, and we need to copy the data from 85 00:06:53,420 --> 00:06:53,660 it. 86 00:06:54,380 --> 00:06:57,730 Switch to another buffer and restart the DMA. 87 00:06:57,740 --> 00:06:58,610 See ADC. 88 00:06:59,330 --> 00:07:06,350 The next block contains configuration data for ADC, Demasi and timer controlling EOS R or interrupt 89 00:07:06,350 --> 00:07:07,280 service routine. 90 00:07:07,760 --> 00:07:14,180 So basically on this slide, you will see the data set for the ADC, the MACIE, as well as the timer 91 00:07:14,180 --> 00:07:16,070 for the Izaak. 92 00:07:16,850 --> 00:07:24,980 Now add the debug condition on top of the setup function, just like this now in the setup function. 93 00:07:25,580 --> 00:07:31,160 After the run on the classifier underscore in it with open and closed parenthesis. 94 00:07:31,760 --> 00:07:38,810 Add the following code that creates inference buffers, configures DMA and start the recording by setting 95 00:07:38,810 --> 00:07:45,320 volatile global variable recording to one, then right after that, delete the PDMP dot and opening 96 00:07:45,320 --> 00:07:53,690 close parenthesis and free sample buffer microphone underscore, inference, underscore and void opening 97 00:07:53,690 --> 00:07:58,470 doors, parentheses function and also microphone inference start. 98 00:07:59,330 --> 00:08:07,010 You are entitled to underscore D and underscore samples be the same underscore data underscore underscore 99 00:08:07,010 --> 00:08:12,770 inference on this group called that void functions since we are not using them. 100 00:08:13,520 --> 00:08:18,950 After compiling and uploading the code open to serial monitor and you will see probabilities for every 101 00:08:18,950 --> 00:08:20,030 class is printed out. 102 00:08:20,720 --> 00:08:24,560 Play some sounds or scoff at real terminal to check the accuracy. 103 00:08:24,920 --> 00:08:29,240 If you're having difficulties on listening to my voice instructions. 104 00:08:29,600 --> 00:08:33,020 You may refer the illustration on the side of this line. 105 00:08:33,920 --> 00:08:40,220 So since we are terminal cannot connect to the internet, we can take these samples demo and make it 106 00:08:40,220 --> 00:08:48,170 into a real iottie application with the Blink, Blink is a platform that lets you quickly build interfaces 107 00:08:48,170 --> 00:08:53,420 for controlling and monitoring your hardware projects from your iOS and Android devices. 108 00:08:53,810 --> 00:08:55,460 You can do this quickly with Blink. 109 00:08:56,000 --> 00:09:02,330 In this case, we will use Blink to push notifications to our smartphone if we are terminal detects 110 00:09:02,330 --> 00:09:03,980 any sounds we should worry about. 111 00:09:04,730 --> 00:09:09,740 Now get the app set up an account and start a new project to get started with Blink. 112 00:09:10,310 --> 00:09:12,590 When you're finished, click the play button. 113 00:09:13,100 --> 00:09:17,060 Then that's your setup by compiling and uploading simple and push button. 114 00:09:17,510 --> 00:09:24,980 For example, make sure you change Wi-Fi Assist, ID password and your Blink API token, which you can 115 00:09:24,980 --> 00:09:25,760 get it in the app. 116 00:09:26,750 --> 00:09:33,680 And if the code compiles the test is successful, pressing the left button on your terminal causes a 117 00:09:33,680 --> 00:09:35,690 push notification to appear on your phone. 118 00:09:36,230 --> 00:09:38,130 Then you can move to the next stage. 119 00:09:39,890 --> 00:09:45,290 On this part, we are going to move all the neural network inference code in a separate function and 120 00:09:45,290 --> 00:09:52,280 call it in the loop open and just parenthesis function right after doubling that run open and close 121 00:09:52,280 --> 00:09:52,970 parenthesis. 122 00:09:53,660 --> 00:09:58,930 Similar to what we did before, we checked the neural network prediction probabilities. 123 00:09:58,970 --> 00:10:06,740 And if they are higher than the threshold for a certain class, we call link that notify open and close 124 00:10:06,740 --> 00:10:12,440 parenthesis function, which, as you might have guessed, pushes the notification to your mobile device.