Hello everyone and welcome.
In today's session we will cover model as a judge for evaluations.
For this I will go ahead and hit on create button.
In there I'll select model as a judge option.
So here I'll give the name as L.l.m. as a judge evaluation.
The description is model.
As a judge evaluation I'll go ahead and select the model anthropic and haiku.
So this is what is needed for the automatic evaluation that generates evaluation matrix.
Then here we have another model that we want to evaluate.
So in this case here I'll go select the same model haiku from anthropic Apply just like our previous
video.
I'll select two different metrics here helpfulness and correctness.
I'll go with responsible A metrics as Harmfulness.
Here I'll have to provide the prompt data set and the evaluation results very similar to how we did
it in our previous video.
So I'll go ahead and share with you how the evaluation dataset looks like.
So here there is a prompt right 15 words.
Summary of text.
Fargate is a technology.
And all of this you can go through it.
I'll upload it along the section.
And then there's a response.
Fargate allows AWS Fargate allows running containers without managing servers or clusters, simplifying
container deployment and scaling.
So there are two results that or two line items I am providing for the LM to evaluate.
And this is how the the content or the prompt is.
And then this is what the response is from the LM dot.
All right.
Let me upload this to the S3 bucket.
Right.
So the upload is complete in my bucket I uploaded the The file.
Large language model as a judge evaluation data set.
So now I'll go to the job that we are creating and then browse provide the evaluation dataset as large
lambda model as a judge.
And then I'll have the results here.
You can see results.
Or I can actually create another folder here by the name as judge evaluation results.
Then create a folder.
Go back up browse results.
So this is the part.
This is not clear.
But what I'll do is go ahead and add this path here.
Results as a judge.
Results.
I'll use an existing role bedrock provisioning role.
And then I'm not giving a new KMS key.
Go ahead and create.
It looks like there's a problem here so I missed there's a space in here.
Let's hit the create button.
Use an existing role bedrock provisioning role.
Create.
There you go.
The job is in progress.
I will go ahead and pause.
It takes a little while for this to run.
So the evaluation job that we ran for as a judge is completed.
So now I'll go ahead and click on this and we'll go check out the summary.
This was the job that we ran.
Description model and all the details that we we had provided.
And then we had three different scores that we wanted.
One of them was helpfulness correctness and harmfulness.
So these three were the metrics.
And if you notice here the breakdown is provided where we have the score as 0.83 for helpfulness.
And then here are the prompt details.
Expand this.
So if you notice here we have two different line items.
One said by 15 words for this text.
And then the Fargate is the technology and all the details that we provided.
This is the output.
This is the ground truth that we wanted.
So this is how it compare the input and the output with actual ground truth that is available or that
we passed.
Here's the score that it provided.
And it gives you the details about why it scored it as 0.83.
So if you go through it, you'll understand why this is 0.83.
Say okay.
Same goes for correctness score here is one out of one.
And you go to the prompt details and then expand this.
And you'll see the very similar way of evaluating prompt input generation output and ground truth values.
So one thing that I want to note wanted to highlight is all scores have been normalized to value zero
and one.
So this is going to be always between 0 and one.
There is no harmful content.
The value is zero.
Here, let's check this as a prompt.
Details here.
And then the score is zero because there are no harmful contents.
Great.
So we got all the details.
Now if you want to check out the then the evaluation of the entire data set for Rag and large language
model.
Then you can go to S3 go to buckets.
In here we have the bedrock evaluation as the bucket and results.
So we have two results.
This was for the rag and this is for large language model as a judge.
And then you go here go dive deeper.
And then general data sets.
Yeah.
There you go.
So this is where you find all the relevant details.
You can download this file and go through the entire data set instead of just two.
Same goes for the results of Rag where you go in here Inference configs.
Zero data sets.
Reg data sets.
And this is the JSON line item.
This is where what you can download.
I'll quickly show you how this looks like.
So here's how it looks like.
There is conversation turns input record.
Was this the output details are here where this there's a model that we selected.
Here is the text chunk and the retrieved passage content from the text and the details that was provided,
the location of the file or the data source.
All of this you can find here for the entire PDF that we selected.
Well, actually this is not the entire PDF dot.
This is like two line items with five chunks each.
So that is what it is because the chunk size is five.
It would go ahead and check it for five different chunk items.
Thank you so much and I'll see you in another video.