So what is a SaaS evaluator?
Also known as semantic answer similarity can use SaaS evaluator component to evaluate answers predicted
by a haystack pipeline, such as a Rag pipeline against ground truth labels.
You can provide a BI encoder or cross encoder model to initialize a SaaS evaluator.
By default, it uses sentence transformer paraphrase multilingual MP.
Net base v2 model.
Note that only one predicted answer is compared to the ground truth answer at a time.
The component does not support multiple ground truth answers for the same question or multiple answer
predicted for the same question.
Let's take a example on SaaS evaluator.
So here I have imported SaaS evaluator.
We provide a variable and we'll do a warm up to import the model.
Now let's run this sense evaluator and provide the ground truth answers.
This case here, let's say we do Berlin and Paris, and let's do the predicted answers here that we
got from the generator as Berlin and Lyon.
Now, just when you're on this, spend the results as individual scores.
Or we can just do score.
Let's save this sass evaluator and then run this.
It might take a minute or so to download the model.
Sometimes it's quicker.
There you go.
We actually got the answer pretty quickly.
It's 0.7.
So what it did is it compared Berlin with Berlin and Paris with line and gave us the answer.
Let's check this one more time as a result.
Individual scores.
Now let's run this again.
we'll get individual answers, like how bonus similarity between Berlin and Berlin pairs with line.
So if you notice, the similarity was one for Berlin because it matched, whereas with Paris it was
low.
So the average of these two is 0.7587.
This is the similarity answer evaluator which evaluates against ground truth answers.
Let's move on to the third evaluator, which is context relevance evaluator.
You can use context relevance evaluator component to evaluate documents retrieved by his tech pipeline.
Just rag pipeline without ground truth labels.
Unlike the Sass evaluator, we do not need the ground truth labels here.
The component breaks up the context into multiple statements and checks whether each statement is relevant
for answering a question.
The finance score for the context relevance is a number from 0 to 1, and represents the proportion
of statements that are relevant to the provided question.
Let's take a real world example and understand how context relevance evaluator works.
So now let's understand this.
Here I have imported the evaluator context relevance evaluator.
In here I'll ask a very similar question that I asked before who created the Python programming language.
And then we give the same context here that we provided in our previous example period of the Python
when it was created, and what the Python language is all about.
Now, instead of using faithfulness evaluator, in this case we'll use the evaluator as context Text
relevance evaluator, which would evaluate the context.
We don't need the predicted answer here, which is unlike the faithfulness evaluator where we have predicted
answers as well.
This is just to evaluate whether, given this question can weaken this context.
Answer the question or not.
Let's go to.
The result is evaluator run.
Provide a list of questions and contexts as context.
Print the score as result.
Score.
Print individual scores and then also print the results.
Let's go ahead and execute this.
So if you notice here when this executes we get all 111.
And the result is here as well.
So we got one.
Because clearly the question here is in context.
Actually it's not the way the context here is relevant to the question.