1 00:00:03,040 --> 00:00:12,850 Okay, so let's start talking about the prototyping phase and the main challenges land chain has identified 2 00:00:12,850 --> 00:00:18,520 in this phase A with the LM app development teams. 3 00:00:18,520 --> 00:00:21,670 So first let's talk about the prototyping phase. 4 00:00:21,670 --> 00:00:31,480 It involves quick experimentation between prompts, model types, retrieval strategy and other parameters, 5 00:00:32,170 --> 00:00:40,960 the ability to rapidly understand how the model is performing and debugging where it is failing is incredibly 6 00:00:40,960 --> 00:00:43,540 important for this phase. 7 00:00:43,540 --> 00:00:51,640 Okay, so we have four main challenges in the prototyping phase. 8 00:00:52,540 --> 00:00:55,540 First when things go wrong. 9 00:00:55,720 --> 00:00:58,690 How to identify what is failing. 10 00:01:00,230 --> 00:01:07,100 Second, how do we iterate an experiment in a fast and easy way? 11 00:01:08,340 --> 00:01:17,520 Third, how to compare the performance of alternative prompts, retrieval strategies and model choices. 12 00:01:18,480 --> 00:01:24,090 And fourth, how do we test the performance of the prototype? 13 00:01:25,810 --> 00:01:32,830 We are going to see how we solve these challenges with Lang Smith in the next lesson. 14 00:01:32,860 --> 00:01:40,180 Next, let's talk about the challenges we face in the beta testing phase. 15 00:01:40,210 --> 00:01:42,580 Let's talk about the beta testing phase. 16 00:01:42,970 --> 00:01:52,570 So the beta testing phase allows developers to collect more data on how their LM applications are performing 17 00:01:52,570 --> 00:01:54,820 in real world scenarios. 18 00:01:55,390 --> 00:02:03,100 In this phase, it is important to develop an understanding for the types of inputs the app is performing 19 00:02:03,100 --> 00:02:09,759 well or poorly on, and how exactly it's breaking down in those cases. 20 00:02:10,630 --> 00:02:17,020 Both feedback collection and run annotation are critical for this workflow. 21 00:02:17,050 --> 00:02:24,730 This will help in curation of test cases that can help track regressions, improvements, and development 22 00:02:24,730 --> 00:02:26,740 of automatic evaluation. 23 00:02:26,890 --> 00:02:32,500 This is how long chain is describing as the beta testing. 24 00:02:32,500 --> 00:02:37,420 Okay, so we know what beta testing is about. 25 00:02:37,420 --> 00:02:46,090 And what they have identified is that the top challenge that development teams are finding in this stage 26 00:02:46,090 --> 00:02:56,440 is to be able to answer this question, how to process and analyze the feedback of the initial users. 27 00:02:56,680 --> 00:02:57,910 And remember. 28 00:02:58,530 --> 00:03:05,190 These beta testing phase is in many cases just. 29 00:03:09,740 --> 00:03:18,770 In many cases, many LM app development teams, they simply don't go through the beta testing phase. 30 00:03:18,830 --> 00:03:23,810 They just go with their gut feeling and language. 31 00:03:23,810 --> 00:03:32,540 Myth is a great tool to use a much more scientific approach in the development process. 32 00:03:32,540 --> 00:03:37,640 And this is how classic app development is doing. 33 00:03:37,640 --> 00:03:47,060 So long chain, as we will see next, is providing us a very good way of doing beta test beta testing. 34 00:03:47,450 --> 00:03:51,080 Finally, let's talk about the production phase. 35 00:03:51,620 --> 00:04:02,570 So in the production phase, closely inspecting key data points, growing benchmarking data sets, annotating 36 00:04:02,600 --> 00:04:06,410 traces, and drilling down into important data. 37 00:04:06,410 --> 00:04:15,350 In Trace View are workflows you will also want to do once your application hits production. 38 00:04:16,720 --> 00:04:19,269 This is from the Long Chain article. 39 00:04:19,269 --> 00:04:27,550 However, especially at the production stage, it is crucial to get a high level overview of application 40 00:04:27,550 --> 00:04:33,520 performance with respect to latency, cost and feedback scores. 41 00:04:34,060 --> 00:04:44,020 This ensures that it is delivering desirable results at scale, so production is very important and 42 00:04:44,020 --> 00:04:47,290 is not just about monitoring. 43 00:04:47,560 --> 00:04:53,170 It's also about a key processing and analyzing user feedback. 44 00:04:53,170 --> 00:04:53,860 Right. 45 00:04:53,860 --> 00:05:02,170 So this is where long chain is telling us that this is the it is the first a there is the first big 46 00:05:02,170 --> 00:05:11,140 challenge for LM app development teams, how to keep processing and analyzing user feedback in this 47 00:05:11,140 --> 00:05:12,460 production phase. 48 00:05:13,050 --> 00:05:18,240 Second challenge how to measure the performance of the application. 49 00:05:18,860 --> 00:05:22,400 And third, how to keep improving the application. 50 00:05:22,400 --> 00:05:24,320 So in the next lesson. 51 00:05:24,620 --> 00:05:34,580 In the next lessons, we are going to see how Lang Smith is going to help us solve all these challenges 52 00:05:34,580 --> 00:05:35,360 in the. 53 00:05:36,020 --> 00:05:46,310 Prototyping, beta testing and production phases in order to a develop a professional LM application. 54 00:05:46,310 --> 00:05:55,610 And once we finish with that explanation, we will see long chain doing it at work with real applications 55 00:05:55,610 --> 00:06:00,650 first with a very basic application, then with a professional application.