1
00:00:03,000 --> 00:00:11,610
Okay, so in this detailed video, let's talk about how Lang Smith helped us to solve the main challenges

2
00:00:11,610 --> 00:00:14,280
we face in the production phase.

3
00:00:15,420 --> 00:00:17,430
Let's talk about the production phase.

4
00:00:17,430 --> 00:00:23,190
So remember what the Land Change team tells us about this phase.

5
00:00:23,190 --> 00:00:33,120
Closely inspecting key data points, growing benchmarking data sets, annotating traces, and drilling

6
00:00:33,120 --> 00:00:42,510
down into important data in trace view arc workflows that you will also want to do once your application

7
00:00:42,510 --> 00:00:43,890
hits production.

8
00:00:43,890 --> 00:00:52,410
Okay, so everything we did in the beta testing phase, we also want to do that in the production phase.

9
00:00:53,010 --> 00:01:01,860
However, especially at the production stage, it is crucial to get a high level overview of application

10
00:01:01,860 --> 00:01:07,620
performance with respect to latency, cost and feedback scores.

11
00:01:07,620 --> 00:01:12,030
We are talking about monitoring key metrics.

12
00:01:12,810 --> 00:01:17,580
This ensures that it is delivering desirable results at scale.

13
00:01:18,730 --> 00:01:25,630
So let's see the first challenge how to keep processing and analyzing user.

14
00:01:26,450 --> 00:01:29,690
Feedback in order to solve this challenge.

15
00:01:29,690 --> 00:01:35,990
As we said, we need to keep using lamb-smith as in the beta testing phase, so no need to repeat what

16
00:01:35,990 --> 00:01:38,150
we said for the beta testing phase.

17
00:01:38,510 --> 00:01:43,520
About the second challenge how to measure the performance of the application.

18
00:01:43,760 --> 00:01:47,780
We will use Lamb-smith to monitor key metrics.

19
00:01:47,780 --> 00:01:50,690
Let's see more details about that.

20
00:01:51,560 --> 00:01:58,220
Lang Smith provides monitoring charts that allow you to track key metrics over time.

21
00:01:58,880 --> 00:02:04,790
Currently, these monitoring charts are, I would say in the early version.

22
00:02:04,790 --> 00:02:10,009
I think this is going to improve a lot, but right now what we have is good enough.

23
00:02:10,669 --> 00:02:18,650
You can expand to view metrics for a given period and drill down into a specific data point to get a

24
00:02:18,650 --> 00:02:21,470
trace table for that time period.

25
00:02:21,470 --> 00:02:25,820
This is especially handy for debugging production issues.

26
00:02:26,570 --> 00:02:34,070
The platform also allows for tag and metadata grouping, which allows users to mark different versions

27
00:02:34,070 --> 00:02:41,090
of their applications with different identifiers, and view how they are performing side by side within

28
00:02:41,090 --> 00:02:42,200
each chart.

29
00:02:42,890 --> 00:02:45,830
This is helpful for a B testing.

30
00:02:45,860 --> 00:02:50,120
Changes in prompt model or retrieval strategy.

31
00:02:50,120 --> 00:02:52,820
So these are the three areas where we.

32
00:02:53,450 --> 00:02:56,960
We'll want to compare different versions.

33
00:02:56,960 --> 00:02:57,440
Okay.

34
00:02:57,440 --> 00:03:07,700
We have our, you know, a final version of the application, but still we need to check, you know,

35
00:03:07,700 --> 00:03:12,020
different prompts, different models and different retrieval strategies.

36
00:03:12,020 --> 00:03:12,620
Why?

37
00:03:12,650 --> 00:03:15,860
Because things are always going to change.

38
00:03:15,860 --> 00:03:22,940
We are going to have new models or better models, better versions of the current models.

39
00:03:23,330 --> 00:03:25,790
We are going to have new prompting strategies.

40
00:03:25,790 --> 00:03:27,980
We are going to have new retrieval strategies.

41
00:03:27,980 --> 00:03:35,750
So even when we are in the production phase, even when we have an application which is live, you know,

42
00:03:35,750 --> 00:03:43,550
being used by, by by real users and customers, we still want to test, you know, different, uh,

43
00:03:43,550 --> 00:03:44,990
approaches and solutions.

44
00:03:44,990 --> 00:03:50,330
And for this long chain is going to be very interesting uh, in this stage.

45
00:03:50,660 --> 00:03:54,950
So in order to keep improving the app.

46
00:03:55,590 --> 00:04:04,440
We will use Lang Smith to mark different versions for a B testing of prompts, models or retrieval strategies.

47
00:04:04,470 --> 00:04:12,450
Okay, what we said and in order to measure the performance of the app, we will use Lang Smith to monitor

48
00:04:12,450 --> 00:04:13,650
key metrics.

49
00:04:13,680 --> 00:04:26,400
Okay, so this is how Lang Smith is going to help us solve the main challenges in the production phase.