Dive into faithfulness.
Evaluator faithfulness evaluator component evaluates document retrieved by haystack pipeline such as
Rag pipeline without ground truth labels.
The components place the generated answer into statements and checks each of them against the provided
context within NLM.
A higher faithfulness score is better, and it indicates that a large number of statements in the generated
answer can be inferred from the context.
A faithfulness score can be used to better understand how often and when the generator in a rag pipeline
hallucinates.
Let's work with a real world example with a faithfulness evaluator.
So here's the psalm editor.
Let me import a haystack components called evaluator and a specific evaluator called Faithfulness evaluator.
I'll ask the question here.
who created Python programming language.
Let's provide the context here.
Which is creator of the Python.
Python was created in late 1998, in a programming language.
And then it has two sentences here, the creator of it and what Python as a language entails.
So these are the context.
Now let's go to the predicted answer for this.
So let's say the generator provided an answer like this.
Here.
This is the answer that came back from a generator and which is Python high level internal purpose programming
language that was created by George Lucas.
Now let's run.
Let's initialize Sarri the evaluator as faithful evaluator and result evaluator.
Run!
Run the pipeline here with questions, contexts, and predicted answers.
Now, once we're in the pipeline.
Let's print result of individual scores.
That's one.
Now it says print the results.
Print result scores.
So in this case we have the entire program ready.
Let's execute this.
I'll go ahead and execute this I got an error.
So I believe it's not scores it's results.
I'll execute this one more time.
So now perfect.
So if you see here the score is 0.5.
Individual score.
And the reason we got the score as not one and 0.5 is because there are two statements here.
One of them has an answer to the question and the other one does not.
And that is why it's one and zero.
And it's not one.
It's a it's an average of the two statements.
That's why we got 0.5 as the individual score.
So now if you notice the creator is also different here.
So here is Gurudom von Rosen and here is George Lucas.
That's the difference here.
So what I can do here is let's take the creator of the context here and put it here.
And let's see how the question and context and the answers are evaluated.
And we get the answer.
Cut the relevancy here is the creator here was different than the predicted answer here.
That's what's causing the accuracy to go down.
And we receive.
And let's understand how this works here.
Faithfulness evaluator.
Now if you scroll down a little bit on this, on this class, you would notice this is a prompt that
has been used for faithfulness evaluator to evaluate the predicted answers.
So you see here your task is to judge the faithfulness or groundedness of statement based on the context
information.
So context information here is the source of truth.
Extract the statements from the provided predicted answers to questions.
Calculate the faithfulness score for each statements made in the predicted answer, and we are determining
that the score is one.
If the statement can be inferred from the provided context, or zero if it cannot be inferred.
So that is the prompt that is passed along with the context questions and predicted answer to the output
to the OpenAI model for evaluation.
That is about faithfulness evaluator and how it works.
We will use faithfulness evaluator in an end to end pipeline in a more advanced topic later on, but
this course and tutorial or video is more about how faithfulness works.
Let's move on to another evaluator called SAS evaluator.