Evaluators that we learned in our previous videos and use it for evaluating a Rag pipeline.
The goal here is how to evaluate your Rag pipeline with model based metrics available in haystack offering.
The basic understanding is that you have known how to execute pipelines, and you're pretty aware of
the pipelines in haystack.
So first things let's create a rag pipeline.
In this case here I'll quickly run through the process here where I have to load the datasets, import
those documents, use sentence transformers, and write that in an in-memory document store.
So for that, let me first import the documents.
In this case here I'm going to use a pre-existing document from the hugging face PubMed QA instructions.
It will load datasets.
It's a huge data set.
So we will use only top 25 data.
And then in document I will use the context of the document and get all the documents.
So once we have all the documents, let's understand how the document looks like.
So what I'm going to do is I'm going to just quickly print the document so you understand how the documents
look like.
So now I'll just quickly execute this so we understand how the documents look like.
There you go.
So there's a document here.
And it has content.
And each document has a content and document ID.
Now let's move on to get two different things here.
Also the instructions from the document that we have.
So we wrote the documents here very similarly.
We get the context.
We also get the instructions and the response.
So these are all the instructions of the questions and the all ground truth answers as a response.
So we get three different values from the documents.
So now we have documents questions and ground truth answers.
Let's initialize in memory document store and document writer and sentence transformers.
So in this case here I'm initializing the document store as in memory document store there is sentence
transformers document embedder.
We're using mini lm v6.
And then we initialize the document writer with the document store.
And if there is a duplicate then we skip.
Now let's create a pipeline.
So I created a pipeline here index indexing and the component as document Embedder which is a document
Embedder is a document writer.
And then you connect document Embedder documents to document writer documents.
And then we run the indexing pipeline with all the documents that we had.
So that's step one where we will do all the indexing pipeline would run.
Now let's move on to the other phase where we want the Rag application.
So in this case here we need the prompt builder, the answer builder sentence transformers text Embedder
generator and the retriever.
This will help us create the Dirac pipeline.
And now what I'm going to do is provide a prompt.
So this is the prompt from all the document.
We say answer the following question based on the given context for documents in the document.
This is the question and what's the answer.
And then moving on we connect.
We'll initialize the Rag pipeline.
This is a pipeline.
So in this case here in the Rag pipeline we are adding components like sentence transformer text Embedder
Retriever, which is in memory document store.
We add the document store here.
Prompt builder a generator and answer builder.
Now we added all the components to the req pipeline.
And then we will connect different components to each other.
So in this case I'll begin with query Embedder retriever or query embedding.
Then the retriever is a sender.
Here it will send the retrieved documents to the Prompt builder documents.
Prompt builder sends it to the generator and generator replies has sent it to the answer builder to
build the answers.
There's one more Rag pipeline component that needs to be connected, which is the retriever that retrieves
the document here, sends it to the answer builder documents.
So now everything is connected to each other.
And then we will ask the question, do high levels of procalcitonin in the early phase after pediatric
liver transplant indicate poor postoperative outcome?
That's the question here.
And then I will go ahead and run the pipeline.
So in this case here I have the code that we can use to run the pipeline.
So here we have the question that we asked.
Prompt builder is the same question.
And the answer builder has the same question.
Everything's great.
And then let's run this.
It's a big data set.
So what I'm going to do is even though we are getting just top 25, I'll pause the video and we'll come
back.
It was quicker than I thought.
So in this case, we got the generated response from the generator about the question that was asked
here.
So I hope you know about all of this and you're aware of how to execute the pipeline.
This is where we start.
the evaluators whether the results that we got like how good were they?
So to evaluate that we'll need to extract this questions the ground truth answers and the ground truth
document.
So this is the original set of questions that we have stored.
So we extract that top 25 and store it.
And so these are the first set of questions that are stored as a zip file.
And we will use it to evaluate the Rag pipeline going forward.
Let's initialize the Rag answer and then retrieve docs.
So now instead of executing this once what we want to do is we want to execute this and the list of
25.
So we get a list of 25 questions, ground truth answers and ground truth docs.
And then we capture the responses from the generated LLM as rag answers and the retrieved documents.
So we execute this in a loop.
So if you see here we have the list of questions that we got here.
So we execute the entire pipeline that we had here.
We execute in a loop.
And each response that we get we get we save the answer.
And we also store the retrieved documents as the document part of the answer.
So we'll have 25 sets of responses that we receive.
Now we will go ahead and evaluate this.
So what does this mean is create an evaluation pipeline.
And then eval pipeline I'll add components.
And this time the component that we want to add is faithfulness and evaluator.
And then let's import that.
We have seen how this works independently.
But now it's time where we see how this works within a pipeline.
There was a typo there.
And then going to quickly use sass evaluator and then provide the model which is mini lm V to evaluate.
So now we have things in place.
We add the component and now let's run the pipeline.
To run the pipeline we'll have to provide three different things for faithfulness evaluation run pipeline.
Here we have to provide the questions that we had.
List of questions.
Then the ground truth documents and the rag answers.
So this is what were the answers that we got from the generator or the LM.
And this is the ground truth doc.
That is the relevant answer.
And these are the questions.
So faithfulness would help us understand how close the rag answer was compared to the ground truth docs
and similarly sass evaluator.
We have got all the predicted answers and the ground truth answers right here from the list of the ground
truth answers that we originally captured.
So this is actually ground truth documents.
And here is the ground truth answers.
Sass evaluator does need the the ground truth answers to evaluate against the generated answers.
So now we get everything that we want.
And then let's go ahead and run this and print the results.
Run this.
I will pause this until the pipeline gets executed and see you back soon.
So now the pipeline got executed along with the evaluators running on it.
So now let's go ahead and copy that and paste it in a Visual Studio editor.
And let's analyze this.
So if you notice here we got the root of the JSON as one of the evaluator is faithfulness.
And scroll down a little bit here.
The other one here is sass evaluator.
So we got two distinct evaluators and their evaluation performed in case of SaaS faithfulness.
The result has statements.
So in this case here there were statements.
One statement is two, three, four and five.
And then each statement had a score.
And it was evaluated as 1111.
And this is the overall score.
Same goes for other 24 statements.
And you can check it out how the responses were when you execute this at your own time.
But in a nutshell, this is how it works.
Whereas the SaaS evaluator had had a score and then which is the average of all scores, and the individual
scores are here.
These are 25 SaaS evaluator scores for the 25 items that we we looped against.
So this is how the evaluators work in haystack.