Hello everyone and welcome.
In today's session, we will cover evaluation capabilities in Amazon bedrock evaluations help you streamline
testing and improve generative AI applications.
As of now, bedrock offers a couple of ways to evaluate your results.
One of them is models based evaluation.
Here we have automatic and human evaluations.
In automatic, you can evaluate performances using just the model and metrics that you select.
Bedrock now offers model.
As a judge.
It's a pre-trained model that evaluates your model's performances using metrics that you selected under
humans option.
You can work with AWS Managed Teams, where you can evaluate responses from up to two models.
You can define evaluation metrics specific to your use case.
You can also do the same with your work team if you have one other way to do this is knowledge basis.
You can now run an automatic knowledge based evaluations to assess and optimize Rag applications using
Amazon Bedrock knowledge basis.
So now let's go ahead and see how this works in practice.
To evaluate knowledge bases we'll have to first create one.
In this case here I created a knowledge base by the name evaluation KB.
And the data source that I have here is from a PDF that is publicly available called Lambda PDF.
So this is the bucket first link Udemy Bedrock evaluation.
And in here I have lambda hyphen dg pdf.
That is my source.
So I'll go here select this option and add document from S3.
Browse DG choose add.
And then I'll add this as a source.
It checks the permissions.
So now the knowledge base is successfully synced with Lambda.
So let me go over the source that I have right now, so that you will be able to understand and relate
to some of the post-processing that I would do for evaluations.
So here is my source.
In this case here I have developer guide AWS Lambda.
And you can find this on their website.
I'll also attach it to the section where you can download it, but it basically covers different aspects
of of Lambda serverless and how you can use the Lambda service on AWS console.
So I'll go ahead and test whether my data is ingested or not on the serverless OpenSearch serverless.
So in this case here I will open a new tab.
Go to open search.
In here I'll go to the serverless dashboards.
And then it did create a new collection I'll go in here go to the dashboard URL and then developer tools
run this query.
So now I see that the source URL is first link Udemy bedrock evaluation Lambda.
So it did sync up the source.
And my vector database is ready with Lambda as a source.
Now I'll go ahead and close all of this.
And now I would go ahead and create a knowledge base evaluation.
I'll give the name here first link Udemy Lambda Evaluation.
I'll provide a description which is optional Lambda knowledgebase evaluation.
Then you have to select a model that would do the rag on the existing data source.
So in this case here I'll go ahead and select cloud 3.5 Sonet.
I'm now going to add tags as of now.
And then I'll select the knowledge base KB that we just created.
In here there are two different options available for knowledge base evaluation.
One is retrieve only and the other one is retrieve and response generation.
I will select, retrieve and generate a response generation option available here.
It not only retrieves the content, but also generates the content and then will evaluate the entire
pipeline.
I'll have to select the model for that.
I'll go ahead and select a cloud model.
Haiku apply.
So this is a very important part here where you have to select the matrix.
There are about five matrices that you can measure.
Quality Helpfulness, correctness, logical coherence, faithfulness, and completeness.
I'll go over each one of them in detail right now before we create the pipeline or evaluation.
Let's quickly go over the matrices that are designed for evaluation retrieval with response generation.
The score is an average score for responses across all prompts in your data set.
The scores are normalized from 0 to 1 for ease of interpretability.
The first one that we saw is correctness.
Correctness means accurately answering the question.
It checks how accurately the responses generated by the language model were in context to the data given
to the large language model.
The next one is completeness.
Completeness means answering and resolving all aspects of the question.
It checks how closely the answering went for the context given to the language model.
Helpfulness.
Helpfulness means holistically useful responses to the questions.
This logical coherence, where the logical coherence means responses are free from logical gaps, inconsistencies,
or contradictions.
The last one is faithfulness.
Faithfulness means avoiding hallucination with respect to the retrieved text chunks.
The higher the score, the more faithful the generated responses are.
Now let's go back to the console.
So now we selected helpfulness as one matrix.
Let's select one more matrix as correctness.
Scroll down a little bit and then we'll see is responsible AI as an harmfulness.
It measures how harmful the responses are and using hate in certain violence.
I'll select that there are a couple of others responsible AI options available as refusal.
It measures how evasive the responses are in refusing to answer questions.
Stereotyping measures generalized statement about individuals, a group of people, and responses.
So now I'll go ahead and select the data set for evaluation.
Go ahead and select the same lambda source that we have as the evaluation.
Where do I want the S3 location to store the results?
For that I'm going to create one more directory here or a folder in S3 bucket.
Amazon S3 buckets under first link Udemy bedrock evaluation.
I'll go ahead and create results and create a folder that would store all of our results.
Go back.
Browse results.
So this is the thing that is little confusing here.
It doesn't choose.
Give the option choose here.
What I'll have to do is copy the S3 location evaluation and results.
I'll go ahead and create a new service role.
You can also use an existing service role.
Let's go ahead and give Amazon Bedrock service role and hit the create button.
So we landed up in a problem.
So what happened here is data.
I think I did a mistake.
So I had to do data set for evaluation.
This is not the same as data set as for source.
I'll have to give a data source here.
So for that for the data set options I'll have to provide a JSON that says for each conversation turns
refer the responses.
The content here is text.
A trigger is a source or configuration that invokes Lambda functions such as AWS service, and the question
or the prompt that goes in there is what is AWS Lambda trigger?
Same goes with the other line item.
What is AWS Lambda event?
And the answer should be along these lines.
So I'll go ahead and upload this file to the data to the S3 bucket.
So now I uploaded the JSON file evaluation data set I'll go ahead and select that file.
Browse evaluation data set.
This will be the evaluation criteria.
Now if you notice here you do not have the required course setting which is also one of the other problem
in this bucket.
Here I'll have to go to permissions and allow a course setting which is the Cross-region resource sharing
option.
Evaluations do need this.
So I'll go ahead and paste the solution.
So we say aloud method is get put post delete allow from all different origins and allow control allow
origin.
So I'll go ahead save the changes.
Go back to my evaluation job here.
Add a new one.
I'll go ahead and actually create a new one for now I know what.
Let me select.
The S3 bucket.
So I'll come down here and datasets and evaluation result S3 location.
So if you notice here the very first thing that it needs is the data set for evaluation.
What does this mean.
This means that we'll have to provide prompts and the desired result, how it should look like to the
large language model for it to assess the responses, let me share with you how this this JSON file
looks like.
So if you notice here there are two line items.
And it's very important to understand how this works.
This section would determine the matrix that you want to generate from the responses.
So here we have conversation turns as a root element.
A reference response.
So how does a response look like to us.
It's like a text that says a trigger is a source or configuration that invokes a lambda function, such
as an AWS service.
Now this is the context when the question asked is what is an AWS Lambda trigger?
Then the response should look something like this.
That is the expectation that we are setting up for the matrix.
Similarly, for another question or prompt, what is an AWS Lambda event?
The content should look like an event as a JSON document defined by the AWS service or.
The application invoking a Lambda function that is provided as an input to lambda function.
So now with this, let me upload this to the S3 bucket and give the location for the data set.
So I uploaded the data set here.
Now let me go back to the evaluation data set.
Browse and provide the JSON file that would use to evaluate the Rag result from retrieve and generate
responses.
The second option I have here is results for evaluation.
So where do we want to store the results for evaluation?
For that I have created directory results.
I'll select that.
And then down here I'm going to use an existing role.
Let's give the role name as provision role.
And then I'll go ahead and click the create button.
Looks like we have some errors.
So it does not have the required course setting.
So the bucket that we use for evaluation.
Evaluation jobs they do need the permission for cross-origin resources.
So I go here to edit.
All right.
So in here I already had the course setting available which has allowed headers.
It allows all the four method for the the other jobs to access the bucket allow origins.
It allows origins from different regions expose headers as access control allow origin.
So now I'll go ahead and save the changes.
Go back here and create.
Let's go up.
No errors.
So there was an error.
It does not have access to the cloud 3.5 provision role.
So I'll have to go ahead and make changes and retry it.
So I might go ahead and select a different model.
Maybe that works I'll select Haiku Create.
It worked.
So I gave permission for haiku and not sonnet.
So you can change the the IAM permissions for your account and make it work for now.
The entire evaluation pipeline is up and running.
This does take a while, so I'll pause the video and grab some coffee and come back.
So after a while the job is now complete.
Let me go to the evaluation job and let's understand some of the evaluations that it came up with.
So now let's evaluate the results in the matrix summary.
Here we give three different matrix correctness helpfulness and harmfulness.
The correctness is one.
That means it satisfied the Responses that we got.
So let's scroll down a little bit and let's understand how the chart looks like.
So in this case here helpfulness is 0.83.
And let's say you want to understand why the helpfulness was not one.
So in this case here the conversation that we had was what is an AWS Lambda trigger.
This is the output we got from the law.
There were five sources that were chunked and they were referenced for summary.
And then this is the ground truth.
This is what we gave in the JSON file for evaluation.
And it compared this ground truth versus the generation output content.
And the score is not one.
It's close to 0.8.
Sometimes llms are a little stricter in conversation.
So you do get responses on the on little lower at times, but if you want to understand why you got
the score is 0.83, you can go ahead and understand the explanation for that particular conversation
that was used.
So this is how you can take a deep dive into the evaluations of each matrix that you select.
Then there is correctness correctness.
Give us one out of one for the two conversations that we give.
Harmfulness is zero.
There was no harmful content.
You can go here and check the same data set.
What is an AWS Lambda trigger?
There was the output retrieved count retrieved chunks ground truth that we provided and the score is
zero.
There is no insult, harm or violent content generated.
So these are the evaluations that were created from the data that we had.
Thank you.
I'll see you in the next video.