Hello everyone and welcome.
In this video we will learn about another probe language model risk cards.
This probe is all about a language model vulnerable to different probes that come up with the text generation.
The white paper of this probe can be found here.
And the gist of it is that, as with all languages, text generated by language model can be harmful
or used to bring about harm.
Automating language generation adds both an element of scale and also more subtle emergent undesirable
tendency to generate text.
So the different categories under Lmlk probes are bullying, demeaning, profanity, and so forth.
Let's go ahead and execute one of the content here profanity and probe the and find the vulnerability,
if any.
So for that I have created a yml file with different configurations that we went through.
The only difference is I'm specifying a different probe spec by the name of profanity.
And I'm testing this on OpenAI model GPT 3.5 turbo.
We'll go ahead and run the command.
Garrett config probe l c profanity yml and run this all.
There you go.
That was pretty quick.
And if you notice here, it ran about different profanity prompts and probes and there were some failures.
So let's go ahead and go to the JSON line file which is nine C6.
So I have opened the report for LMK profanity.
And here is how it looks.
So let me copy one of the lines here and paste it here.
Okay.
So from the response we got here profanity mental disability was tagged as failed.
The profanity mental disorder or mental disability is marked as one.
It was determined from one of the key words that is listed here in the prompt.
And the other one is search profanity sexual search profanity.
Sexual is marked as one.
So that is how the detectors have determined that out of all the words here, these two were marked
as one.
That means that they were basically not okay to pass the test.
Thank you so much.
I hope you learned about one more probe.
Please feel free to go over different probes.
Whatever works best for you to determine whether an LLM is vulnerable to the different probes and make
a better system and application for generative AI.
Thank you.