-: Okay, let me show you how to use a tool called promptfoo.
Okay, so promptfoo is an evaluation framework,
which runs locally, it's quite useful,
and you can add pretty simply a bunch of tests
in order to test your prompts.
So, this is the interface you get when you run it.
It works in the command line as well,
and it just gives you kind of like a pass/fail, pass/fail
for various different metrics.
It's really interesting.
Now, the way that you can get it working
is you just copy this command, npx promptfoolatest init,
then you run that in the terminal,
and this is going to give you a prompts.txt
and a promptfooconfig.yaml file.
You don't actually have to run this init necessarily,
you can just set these up yourself.
But essentially, the way it works is,
if I just maybe open up my config,
that you would set in here
everything you need to run your evaluation.
That you would put your prompts,
and these can be in line like this
or you could put something like,
if I show you in the guide
actually it's a little bit easier.
You can put like a list of files in here as well,
prompt1, prompt2, etcetera.
And then, you have,
you can just do it in line like this,
which is nice if you're doing a simple test.
Here I've got one prompt,
which is just tell me a funny joke about a topic.
And then another prompt,
just tell me a funny joke about topic
in the style of Dave Chappelle.
Then we have the providers that we want to test against,
so you can just keep adding as many as you want here.
And then you have the tests.
So you can have test cases with specific tests.
So in this case, going to tell
a joke about the topic of bananas
and we want to make sure that the output contains
the word bananas.
And the way you can do that, which is very simple,
is you just have an assert and then type icontains
and then the value bananas.
And that's just going to check that has that in there.
So here we're going to have the word avocado,
here we're going to have the word New York
for these different tests that we're going to run.
So what this is going to do,
it's going to run every prompt,
every combination of every prompt,
every provider, and then every topic here
because we have the different topics
that are inserted into this prompt.
The other cool thing you can do
is you can set default tests for everything.
So this runs across all of the different outputs.
And you can do basic things like JavaScript functions,
which is pretty cool.
If you want to calculate shorter outputs,
you can do output.length, which is really helpful.
But then, what I mostly use this for
is using an LLM to test whether the prompt was correct.
It's called LLM rubric,
and then you literally just write your prompt in here,
"Ensure the joke will be funny enough
for stand up comedians to tell on stage."
And what it will do is, for every version of this,
it's going to call the LLM again,
it's going to check, you know, this is the question,
and then it's going to give you a response and a reason.
And you can just keep adding as many of these as you want.
I want to make sure there's no disclaimer or text
other than the joke,
that it doesn't describe itself as an AI,
and the joke should not be a dad joke
or too politically correct
because it's like a problem that I've had in general.
All right. And once you have this,
you can just keep adding prompts and stuff to it,
which is really helpful.
All right, so, we'll just save this.
And I'll show you how it looks when you run it.
So if we're going into the terminal here,
so you can just run npx prompt through latest eval.
And then it's going to run everything.
Here we go.
So I think it caches everything here.
We don't have to run it again
if it's already run, which is nice.
But you can see that we have a lot of fails
and we can actually see that in an interface here.
So if we go use the view command,
it's going to ask this if we want to open a URL.
And here we go, we have our full test.
Here are the different outputs,
you can see that, you know, the topic bananas,
and all of those failed.
It didn't do a good job.
Avocado toast, it actually got six passed here.
So, this one had one fail, five pass, one fail, five pass.
You can click into this and see what failed and passed.
And this is the actual prompt compiled afterwards.
And then this is the output,
and you can see that it was successful and everything.
It contained the word avocado.
It wasn't a dad joke.
I give the reason the joke is not a dad joke.
It does not appear to be too politically correct,
blah, blah, blah.
So that's really cool.
And then you can see where it fails.
You can click on this and say,
"Okay, it's not funny enough.
It's not fresh enough."
Okay, so that's pretty interesting.
And if we look at what are the different things that failed,
it's quite often this, "It's just not funny enough," right?
What's this one again?
Okay, no disclaimer or text other than the joke, yeah,
so here we go.
It's because we have this at the beginning.
All right, here we go. (chuckles)
And that's not what we want,
we want the just the joke back.
Cool. So, really quick way to do this.
You can also, you know, add more test cases if you want.
And then you can go back and see your different prompts,
and click into it and see the score of the different things,
like, well, it's a pass rate, success rate, etcetera.
You can see the datasets that you've added,
and then you can see the progress,
so like how is it improved over time
as you've tested more stuff.
Now, you can also, in theory, add another evaluation
or set up an evaluation here without using the YAML
if that's easier for you as well.
So you just add the providers, in this case,
yeah, I want gpt-3.5-turbo.
And then I want to add a prompt.
Yeah, hilarious joke.
And then, you can duplicate that if you want to edit it,
which is nice.
Kind of funny joke on topic.
Okay, so now we've got our two prompts,
and then we can add the test case.
You can just say, "I want a joke about chocolate."
All right. So, here we go.
It gives us our YAML.
And we can run that test.
So if we run the evaluation.
Here we go, 100% passing.
There's no tests.
I want to set up an evaluation here.
Add an assert.
We can just soon say,
here we go.
And just be,
mind numbingly funny.
Run the evaluation.
Yeah. So, it's just a simple pun
is not mind numbingly funny. (snickers)
Cool. So you can see how simple this is to run.
And yeah, I find it quite quick and easy
to mess around with.
So, feel free to use this for your prompt optimization,
connect it to different things you're going to be doing.
But, yeah, I could quite like this.
And I think you can export it as well,
I haven't done that, but I think you can.
Yeah, you can download the table,
JSON as well, which is quite useful.
Cool. All right, enjoy. Happy prompting.