Hello everyone!
In today's video we will cover detecting hallucination using model based approach.
The model that we are going to learn and have hands on today is going to be vector based model which
is hosted on hugging face.
The model name is Hallucination Evaluation model.
It's an open source model available on hugging face.
Let's understand what this model has to offer.
By hallucination we mean a text is not supported by other text.
You always need two pieces of text to determine whether a text is hallucinated or not.
When applied a rag, the LLM is provided with several pieces of text retrieved from some datasets,
and a hallucination would indicate the summary or the hypothesis is not supported by those facts called
evidence.
A common type of hallucination in Iraq is factual but hallucinated.
For example, given the premise the capital of France is Berlin.
If the hypothesis is the capital of France is Paris is a hallucination.
Although it is true in the real world scenario, this happens when do not generate based on the textual
data provided to them as part of the Rag retrieval process, but rather generate content based on their
pre-training knowledge.
Let's run through a real world example in our Colab notebook.
Okay, so here is my colab notebook.
I have done a couple of predefined steps which take some time.
I have installed the torch.
I went ahead and logged into my huggingface account to download the model, and then I have also installed
the FFT library.
P is parameter efficient fine tuning of billion scale models on low resource hardware.
This library would help us run model efficiently and optimize it on a low resource hardware.
So that's the steps that I have already taken.
Now let me go ahead and download the model.
For that I am going to use auto model.
Sequence classification is a transformers utility.
And from there I'm going to download the vector hallucination model.
Let me run this.
It's going to take a while for it to download the model.
Let me pause the video.
So it downloaded the model.
Now let me import the pipeline and tokenizer.
All right.
So I have imported the tokenizers.
Now what I'm going to do is I'm going to provide two things.
One is the test data.
So that let's start with that.
So this is the test data.
So the capital of France is Berlin.
The capital of France is Paris.
So there are two parts to this.
One is the premise, which is a fact that we are giving to the model.
And now we know that the capital of France is Paris, which is right.
But this is the, the factual data that we are giving to the to the model that consider the out the
capital of France as Berlin.
But this is what the response we got from the from the model.
It's very important to understand these two things.
Here.
This is the fact that we are providing to the to the model.
But this is what it it responds back with.
So this is what it learned from the training.
While this is what is being told as the context that we give to the to the model.
Now, once we do that, let's provide the prompt here.
So determine if the hypothesis is true given the premise.
And the premise is a text one which is this.
This is the premise and this is the hypothesis that we got from the the model response.
Now let's run through the the loop of the list.
So we will format the prompt pair zero is this value.
It goes here in the premise whereas x2 is hypothesis.
So now let's go ahead and create the pipeline.
I'm going to do here is this classifier as the model.
Go ahead and do a pipeline which is a text classification.
We go to model as we have the model name which is vector based.
Okay.
And then the last part that we have here is tokenizer.
We want to use auto tokenizer from pre-trained.
I'm going to use the Google T5 base tokenizer.
And that should be good enough.
And then let's get the full scores from the classifier.
Input is top k equals none.
And then let's print this course.
So what we're doing is we are executing the pipeline classifiers on it, and then we are running the
input pairs.
Top case none and printing the full scores.
Let's go ahead and run this.
All right.
It's executing.
There is a problem.
Let's see what the what the problem is.
There is a pipeline error.
I had to add the part that says trust remote code as true.
Now let's run this one more time.
We are still having a problem.
Unknown task classification.
I'm just type the name.
Now let's execute this.
So we got an answer here, which is how much the model hallucinated.
So if you notice here, for each premise we saw that this is the response we got.
The the capital of France is Berlin.
That's the response we got from the model.
And the capital of France is Paris is actually the the right answer.
So if you see here this it categorized this as hallucination whereas this was consistent whereas this
was hallucination.
All right.
Thank you I will see you in the next video.