Hello everyone!
In today's video we will cover the hallucination model that is Phi three Hallucination judge from grounded
AI.
It's an open source model hosted on the hugging face.
The model would help us with the detection of hallucination.
We'll do a hands on example and understand how this works.
For that, I have opened a colab notebook.
And if you notice here I have a pro account that helps me get a little more GPUs in memory.
I went ahead in the runtime and I have 100 GPU as my desired compute that I want to run on, and I'm
running this on Python three run type.
So that's the setting that I have.
And I went ahead and did pip install torch that.
So it installed the torch on the notebook which takes a little longer time.
So I did that already.
The next thing that I'll have to do is I'll log in to hugging Face and provide the tokens.
So in this case here I will add the hugging face notebook and then let's run this.
In this case here, it asks me for the token.
I'll have to copy the token from the hugging face account.
So give me just a second.
So I have copied my account and token and make sure that it's not a GitHub credential.
You add that tokens invalid?
Maybe on this one more time.
Maybe I copied the wrong one.
Let me go back and do this one more time.
So now I copied my tokens and it logged in successfully.
We need the hugging face token so that we can install or download the model onto the notebook.
Great.
So now this is done.
Now the important part here is let's install theft.
Now what is theft on a high level while this installs.
Interesting, looks like it got installed quickly, but tier is a parameter efficient fine tuning of
billion scale models on low resource hardware.
It optimizes model parameters to work with low resource hardwares.
So that's why we need Pemf.
So I installed Pemf.
And now let's go ahead and write some code here.
So from Pemf model and pemf config.
And then from transformers import for casual lm.
So these are the two libraries that that we need.
Pemf and Transformers.
And then what I'm going to do is I'm going to download the model for that from Pemf from pre-trained
model.
I need this grounded I then from the auto model for casual M.
We are doing this as a base model phi three mini 4k instruct and then path model for pre-trained.
So okay it looks like I haven't defined them properly, so let me copy them.
Okay.
Auto model for.
All right let me execute this.
It looks like there's a problem.
Wrong path.
Import path.
Okay.
All right, all good.
Let's run this.
Okay.
Perfect.
So if you notice, it is going to download the model.
It's a big model relatively big.
So it might take a little while.
I'll pause the video and I'll come back.
All right.
So it downloaded the model right.
So in this case here see it was a slightly bigger model five six, seven and seven and close to 7.5GB,
which is which is relatively big model to host.
Let's write some code here that we can invoke on the model.
So for that we have to first use a tokenizer.
So what's a tokenizer?
Tokenizers will convert your input to input all our natural language query to a language that the model
understands.
So for that we have to use Tokenizers create.
So we have this already.
I'm going to use a tokenizer from the model we just download which is 53 minute 4k instruct and then
from transformers import tokenizer phi three mini 4k instruct.
So we got the tokenizer which is ready.
So in this case here.
This is the text that I got from the hugging face examples that they have.
So let me go here and share with you in this case here.
And you see scroll downwards.
You'll find this example here.
It's not properly done in many ways.
This example had to be tweaked in terms of how it's written.
I have the code here that works well.
So if you notice here this is this is the prompt.
Your job is to evaluate whether a machine learning model has hallucination or not.
And this is the prompt.
This is the knowledge base.
This is the context that we provide to the model.
Walrus are the largest animal.
The user input is a question what is the biggest mammal?
And the model response is walrus.
So if you notice here, there are four distinct elements that are getting injected to the model.
One is a prompt.
Overall, this is a prompt.
There's an info that which has a model which has a knowledge base, user input and model response.
And we are instructing the model that provide the response or provide model output of hallucination.
Respond with only yes or no and return the prompt right.
So this is the entire prompt.
But technically this is divided into four distinct aspects of the input that goes to the model.
So now we have that.
What I'm going to do is okay we'll get the text.
So we got the text right.
The next thing that we want to do is pass create a message.
So how does the message get created.
So the message that gets created is like this.
This is the message.
The role is a user.
And this is the context we got from the from the the prompt here.
Great.
Let's execute this.
Oh, let's run this.
Great.
Everything worked well.
The next phase is that we run the pipeline.
So in that case here, we will have to generate the pipeline.
So what I'm doing here is a pipeline of text generation.
We have base model and a tokenizer.
Let me bring this up from here and let's put it up here.
Okay.
So now if you notice we have a base model.
We have a pipeline and tokenizer.
So let me change this a little bit.
We need a tokenizer.
That's one.
Then I have to import pipeline.
So now we have the pipeline imported.
We have the base model that we imported all the way up here which is the base model here.
Right.
And then we have the tokenizer.
Everything is ready.
Now we have to give a list of arguments.
The generation arguments that would be required like how many tokens temperature and sample Temperature
is basically how much variation do you want in your model output response?
You can change the temperature and set that up to however much you want for for the model that works
best for your inputs.
And then the next last steps is to.
The last step is to invoke the the pipeline.
So in this case here we work in the pipeline here with the model here.
Pass the messages and generation arguments that we just passed.
And let's check the output.
So the messages here is the list of messages that we have created.
And then the generation arguments.
Let's run this.
Let's see how the response turns out to be.
So it looks like we have some problem.
GPU is available in the environment but more than the environment is passed.
Let me try this one more time.
So after playing around with different problems and solutions, I figured that I had to restart the
the session and done everything again.
This is what I did.
Once I did, that colab was falling in place and I actually was able to run this.
Perfect.
And I do get the text here which says no, the hallucination did not happen.
So that's what it is all about.
Let's change this a little bit.
Let's see what the user inputs.
Smallest mammal right.
In this case let's see.
It did hallucinate here because clearly the largest mammal is walrus.
The user input is about mammals.
Smallest mammals.
Sorry.
And response was walrus.
So definitely that deviated from the actual context that was provided.
We were able to detect hallucination using a model based approach.
And we tested this ran through the entire exercise of installing, doing hands on activity and understanding
this one model.
Thank you.