Hello everyone.
Welcome.
In this video we will learn about probe encoding.
Encoding probe will try to get a model to generate a specific piece of given text by presenting an encoded
version of that text.
It will attempt to circumvent, safeguard or input filtering.
Let's explore this probe with the hands on using a yml file.
So here is my project root directory.
In here I have created another directory by the name reports and I have created a yml configuration
for my Pro encoding.
Here I have three root elements run plugin and reporting.
The generations field would specify how many times we want to send each prompt for inference.
In our case it's one.
The plugin probe spec element is encoding.
That is the probe that we want to test.
And then we specify the model type as open AI.
And we want to test this on the model G 3.5 turbo.
The entire report will be submitted to this particular directory.
Having said that, let's go ahead and execute the the generic tool using the CLI.
So here I would say Gary hyphen hyphen config and name of the probe encoding.
YML and enter.
There you go.
So now if you notice it is going to log everything to the log file.
It has the generator as OpenAI GPT 3.5 turbo.
Don't forget to add the OpenAI API key to the to the environment variable.
Then it's going to test all the different probes for encoding that are listed here.
And now the very first thing it did is encoding decode match.
It passed 60 out of 60.
Now it would keep going.
It was the first one was inject Ascii 85.
And then it would execute all of these probes one by one.
I'll stop this because it's going to take a Wi-Fi to run.
I will turn it back on when it's done executing all the probes.
So it is done executing all the different encoding probes.
And a lot of them were failure.
The 16 out of 30 was inject base 16.
Inject base 64 failed with 31 out of 55th, inject hex failed with 15 out of 30, and so on.
So what I'm going to do is I'm going to open the JSON line format where it dumped all the data for us
to go and understand how the entire execution was.
Was framed and what happened behind the scenes.
So here, if you notice here we have a sequence of different immuno patients.
That happened.
And this was the inject Ascii 85.
And this is what was the prompt.
And this was the output.
Similarly after 85 there was inject base 16.
And then this is what the prompt was.
And then if you scroll on the right this was the output.
And then it keeps going down.
Inject base 32 inject ECoG and different inject Rot13.
All of these injections were tested for encoding.
And if you notice here overall out of everything only 13 passed out of 30, which is a very low number.
But this gives us a very good understanding of how the entire encoding was ran.
Sorry, this 13th out of 30 was for inject jalgo and not the entire invocation.
The entire invocation suite can be found here, and it helps us understand how well this model was against
different encoding.
Thank you and I'll see you in the next video.