1
00:00:03,150 --> 00:00:12,660
Okay, let's see how Lang Smith helps us solve the main challenge we have in the beta testing phase.

2
00:00:13,960 --> 00:00:20,410
So you remember our main challenge now is going to be how to process and analyze the feedback of the

3
00:00:20,410 --> 00:00:21,250
initial user.

4
00:00:21,250 --> 00:00:32,140
Remember, in the beta testing, we are going to test our prototype with a small initial group of users

5
00:00:32,140 --> 00:00:34,330
that we call the beta users.

6
00:00:34,940 --> 00:00:42,350
So one of the problems that the long chain team has observed during this year, working with different

7
00:00:42,350 --> 00:00:49,640
LM app development teams, is that they were, uh, they were having problems in order to understand

8
00:00:49,640 --> 00:00:53,600
how to process and analyze the feedback of the initial users.

9
00:00:53,930 --> 00:01:02,030
So this is how this is how Lamb-smith can help us, uh, solving this problem.

10
00:01:02,030 --> 00:01:09,200
So here in in the notebook, you have a detailed, uh, explanations about that.

11
00:01:09,200 --> 00:01:12,140
And we are going to take a look also at the platform.

12
00:01:12,620 --> 00:01:21,800
So how to use Lang Smith to filter traces with negative human feedback to understand the problems behind

13
00:01:21,800 --> 00:01:22,250
them.

14
00:01:23,030 --> 00:01:28,070
Remember that this is the thumbs down button right.

15
00:01:28,070 --> 00:01:31,820
So in our application we are going to have a thumbs down button.

16
00:01:31,820 --> 00:01:36,440
And what we want is to understand and what is happening.

17
00:01:36,560 --> 00:01:39,560
Uh, when the users click on that button.

18
00:01:39,560 --> 00:01:40,220
Right.

19
00:01:40,220 --> 00:01:46,970
So in the project traces project Traces view.

20
00:01:48,050 --> 00:01:52,610
Click on any of the tags displayed on the right sidebar.

21
00:01:52,610 --> 00:02:01,310
So if you see here we have some tags in the right sidebar.

22
00:02:01,490 --> 00:02:08,810
We click on this, uh, tag correctness with ease, which is one of the tags we have included in this

23
00:02:08,810 --> 00:02:09,860
particular project.

24
00:02:09,860 --> 00:02:12,680
You see that we are filtering the traces.

25
00:02:12,680 --> 00:02:19,730
So in this project we don't have many traces, but in a regular project in the beta testing phase you

26
00:02:19,730 --> 00:02:22,070
may have thousands of traces.

27
00:02:22,070 --> 00:02:30,290
So it would be very interesting for you to have this filter feature in order to filter the traces with

28
00:02:30,290 --> 00:02:32,210
negative human feedback.

29
00:02:32,570 --> 00:02:33,200
Okay.

30
00:02:33,200 --> 00:02:39,620
So in the Project Traces view, click on any of the tags displayed on the right sidebar.

31
00:02:39,620 --> 00:02:47,630
Those tags were created by you when you coded, for example, the thumbs up and the thumbs down buttons

32
00:02:47,630 --> 00:02:49,700
of your LM application.

33
00:02:49,700 --> 00:02:55,550
You will see this in the production in the in the professional project later.

34
00:02:56,000 --> 00:03:04,160
We can use this if we have something like thumbs up or down in the LM application user interface.

35
00:03:04,160 --> 00:03:11,600
Then we can filter with the tag that thumbs up or down add button.

36
00:03:11,600 --> 00:03:13,520
Okay, this is this.

37
00:03:14,690 --> 00:03:23,420
Then we can filter with the tag that thumbs up or down let's say button.

38
00:03:27,430 --> 00:03:31,690
Okay, so I think the concept is clear.

39
00:03:31,690 --> 00:03:43,120
So long as myth allows us to use filtering in order to select only the traces that have a negative human

40
00:03:43,120 --> 00:03:43,750
feedback.

41
00:03:43,750 --> 00:03:51,550
This is going to be very helpful for us during the beta testing phase, in order to understand better

42
00:03:51,550 --> 00:03:52,720
what is happening.

43
00:03:53,850 --> 00:04:03,120
Second, how to use lamb-smith to inspect interesting traces and how to add annotations to one trace.

44
00:04:03,120 --> 00:04:08,880
So annotations is the feedback that we include in one trace.

45
00:04:08,880 --> 00:04:17,519
So the the user of the application can add feedback using this thumbs up or thumbs down button.

46
00:04:17,519 --> 00:04:24,720
But we in Lang Smith can add manual annotations to our traces.

47
00:04:24,720 --> 00:04:33,270
And this is interesting not just for us as engineers, but also for the product managers or people from

48
00:04:33,270 --> 00:04:36,840
the marketing department, subject matter experts, etc..

49
00:04:36,840 --> 00:04:40,350
Okay, so how to do that?

50
00:04:40,350 --> 00:04:48,600
Click on each trace to see the trace details and after clicking on a particular trace, click on Send

51
00:04:48,600 --> 00:04:50,040
to Annotation queue.

52
00:04:50,070 --> 00:04:54,510
You can do this in many different locations in the platform.

53
00:04:54,510 --> 00:05:00,660
So if you go to projects then you select one project.

54
00:05:00,750 --> 00:05:09,390
You will see that in this initial table you have this option to include a trace in the annotation queue.

55
00:05:09,420 --> 00:05:17,700
You also have the option if you open one trace and you have here the button annotate okay.

56
00:05:18,270 --> 00:05:21,840
So a so here you have the explanation.

57
00:05:21,840 --> 00:05:26,040
After clicking on a particular trace click on Send to Annotation queue.

58
00:05:26,040 --> 00:05:35,130
There you can add human feedback to the trace like for example you can include default language, myth

59
00:05:35,130 --> 00:05:42,540
tags, correctness, faithfulness, conciseness, context, relevancy, etc. or you can also add custom

60
00:05:42,540 --> 00:05:44,190
tags and notes.

61
00:05:44,190 --> 00:05:47,040
Okay, so a.

62
00:05:49,440 --> 00:05:53,400
You are going to see this in the example, but it's very easy.

63
00:05:53,430 --> 00:05:53,880
Okay.

64
00:05:53,880 --> 00:05:58,170
So here you have some of the the the tags we have included.

65
00:05:58,170 --> 00:06:03,510
You can include a manual uh notes and you can add tags.

66
00:06:03,510 --> 00:06:07,770
You see here the default tags that come with a lamb-smith.

67
00:06:07,770 --> 00:06:08,610
Okay.

68
00:06:08,610 --> 00:06:09,660
So.

69
00:06:10,310 --> 00:06:15,830
You can also annotate one individual run inside a trace.

70
00:06:15,860 --> 00:06:19,670
Okay, so apart from the trace, you can also come here.

71
00:06:19,670 --> 00:06:23,120
And in one run you can annotate the run okay.

72
00:06:23,150 --> 00:06:23,630
Annotate.

73
00:06:23,630 --> 00:06:24,110
Run.

74
00:06:26,900 --> 00:06:34,700
For example, if you find out that the problem with a bad trace is in the retrieval step, it could

75
00:06:34,700 --> 00:06:41,000
be interesting to annotate the retrieval run with feedback that can help you to fix it later.

76
00:06:41,000 --> 00:06:46,550
Okay, for example, this one, we can see that the problem is here when we check, you know, all the

77
00:06:46,550 --> 00:06:47,300
operations.

78
00:06:47,300 --> 00:06:54,950
So we say okay, check, check this step because I think here is the problem okay.

79
00:06:57,210 --> 00:07:04,710
Annotation is especially useful in a collaboration environment when the LM app developer and the subject

80
00:07:04,710 --> 00:07:09,960
matter expert can both comment on the highlighted trace traces.

81
00:07:09,990 --> 00:07:15,720
Okay, so the subject matter expert is going to be the product manager, or people from the marketing

82
00:07:15,720 --> 00:07:22,890
department, or a people with special knowledge of the field we are working in.

83
00:07:22,920 --> 00:07:23,460
Okay.

84
00:07:24,410 --> 00:07:31,310
Next thing a Lang Lang Smith does for us in this beta testing beta testing phase.

85
00:07:31,310 --> 00:07:38,840
How to use Lang Smith to expand the test data set by adding runs as examples.

86
00:07:39,050 --> 00:07:46,640
You can add traces as examples to the test database, and runs are the different steps that compose

87
00:07:46,640 --> 00:07:49,190
a trace so you know how to add.

88
00:07:50,260 --> 00:07:52,660
A trace in the data set.

89
00:07:52,840 --> 00:07:53,440
Okay.

90
00:07:53,650 --> 00:07:59,770
And you know how to add a run in the data set as well.

91
00:07:59,830 --> 00:08:00,700
Okay.

92
00:08:00,790 --> 00:08:02,050
We show you before.

93
00:08:02,050 --> 00:08:11,140
But just a reminder, in the next lesson we are going to see how Lang Smith helps us solve the main

94
00:08:11,140 --> 00:08:13,960
challenges we find in the production phase.

95
00:08:13,960 --> 00:08:24,760
Remember, we are having a first round, a first view of Lang Smith at work, but in the next, uh,

96
00:08:24,760 --> 00:08:30,820
blocks you are going to see Lang Smith working with real projects.

97
00:08:30,820 --> 00:08:36,370
You will see a professional project and you will see a simple project as well.

98
00:08:36,370 --> 00:08:44,169
So in the next lesson, we are going to see how Lang Smith helps us solve the main challenges in the

99
00:08:44,169 --> 00:08:45,610
production phase.