In-context learning.
Don't worry, it's not that complicated as it sounds.
It's just adding examples into the prompt.
This is the killer application of Llms.
If you get examples from the prompt, it is much better at following this example.
They should do a good job of the task that you're asking it to do that this is one of the most famous
studies that introduced the GPT three MLM, but it's still one of the best, I think, in terms of showing
you the value of prompt engineering and adding examples as you add more examples.
The prompt for bigger models, not only do they get better at performance, they get they actually keep
getting better over time, which is really nice.
And obviously just adding one example here.
So zero shot is no examples.
Going from 0 to 1 takes you from 10% to almost 50% accuracy on Twitter.
And I'd say that really matches my experience as well.
This is if you add ten examples and after ten examples, that's when it starts to level off a little
bit.
But yeah, anything if you can do anything just to get one example on the prompt, that would make a
big difference.
Great.
So if you want to read more about that there's a link to the paper there.
Let's give you an example of using examples on your front.
Here we just have a zero shot prompt.
There are no examples in this.
We're just asking it to do something with style of Steve Jobs.
And then we're telling you the format.
And it's kind of an awkward pretend there's magic soulmate futzing.
And then we're just giving it examples.
So we have an array of examples.
That is how I usually add things to my prompt because it's cleaner.
And then you can swap out the examples much easier rather than writing them all out into print.
So what I'm doing here is the shoe shop prompt is just a zero shot prompt.
And then I've added an example section and four example examples.
And then I'm just iterating through and adding these with the line break.
So when you run this what you get is the full prompt with the examples.
So we've added this section here.
Now if you've made a change to these examples and we wanted to add something new in, maybe we wanted
to add an additional one here.
Then we could do that.
And it's uh, just the case of changing the array.
And then we have this extra example which is really helpful.
Also I really like this tip before.
And you can also then just make a change.
So if you wanted this to be in terms of you want it to be labeled differently or whatever, you could
do that here in the code rather than having to rewrite your whole prompt.
All right.
That's our performance.
And let's see how well we do.
Now you can notice something about these names is that they all start with I.
And I didn't tell it to say make every name start with I.
But it just learned that from the examples.
So here I put in I bar fridge I fridge beer I drink beer fridge I bought I space, I time every single
example product name I put in starts with an I.
It's just like Steve Jobs iPod or iMac or iPhone.
And I did that specifically, not something that I actually really want from a product name generator,
but it did that specifically to show you just how much it followed the examples.
With just three shots, we've managed to get it using the word letter in front of every single name.
Now what do we want an evaluation metric for this.
And maybe there are different things to evaluate.
But this is just an example of a programmatic evaluation because we can just tell whether it starts
with the word I or not.
In encode we have something to run quite quickly, and then we just want to get the percentage there
actually returned it with it, starting with an I.
So split the product names into an array that's split by comma, and then we'll see if it starts with
an I.
We'll run that.
We can see that 66% of the names started with an I.
So this one didn't start with an I.
If we did have I in front then it would say 100%.
And then if we then we can see that the error metric is working.
So now that's working.
We could do some a B testing which is really fun.
So I want to see how many examples are needed in order to get a result that it starts with I every time.
We're including three examples traditionally here, and we want to know if including that third example,
that second example is worth it.
Maybe it just learns from one example and it would hit this.
And just now we've got the different prompts.
We've got the three shot prompts, two shot prompt, one shot prompt.
And all I've done is kind of built them up by just adding the next prompt in here, say the next example.
So we have this terror shot prompt already had.
And then we just add the example, the first example to get the word shot prompt.
And then this one has two examples.
And then this one had three samples.
All right.
Those are the different prompts.
And we're going to test this asynchronously if there's a common pattern because we just run it once.
And we can see that they all begin to die or appear.
But that might not be reliable enough for your for your application.
Say, what is quite common is you'll create a function which would help you run and evaluate, and then
you would then have a number of around.
In this case, I'm testing it 30 times now.
The reason why we're doing it asynchronous is if you think of a supermarket where you have multiple
cashiers on shift, they can all handle a customer each at a time, and therefore the line can go down
quite quickly.
Whereas if you only have one cashier on, then everyone's going to have to wait their turn and she can
only process one at a time.
Asynchronous is just a way of getting more cashiers on the desk, right where where I'm going to call
all 30 prompt at the same time, and we should get them back basically as fast as if we'd only called
one.
So that's the benefit of async.
Otherwise it would just take 20 minutes or something.
For this to one at a time.
Call.
Call the prompt 30 times and run this and add the the parts that make it async.
By the way, we have an async API client, and then we have to await that function.
And we have to create the function as async def into just def that async def.
And then we're spending up the tasks and then we're gathering those tasks.
And what that means is now it can take all 30 tasks at the same time and then just gather them.
So when they all arrive back then we're gonna get the correct count.
Like what?
What percentage of these tasks are have the right result and we could see him.
There are actually still had some about a third of the names, 27% still began with I, which is pretty
good for Steve Jobs did begin a lot of product with I'd.
We've found that even with just one example hit, 100% of the time, it starts to produce a thigh that's
reliable enough for it.
And this is a pretty made up evaluation metric of studying the name with I.
But if you had any other evaluation metric, you could just drop that in and then you see the results
here.
Well, what do you take away is that it's really important to know how many examples you need to provide,
because the more examples you provide, the more expensive the prompt is, the more longer it takes
to run.
And that's going to be something you have to trade off while you're building a real production AI system.