-: Hey, let's start practicing what,
let's start exploring how prompt caching works
in a Jupyter Notebook.
So the first thing we're gonna do
is just run this pip install openai and anthropic command.
That's gonna make sure
that you've got both the OpenAI package
and the Anthropic package installed inside of Python.
After doing that,
we've also got a bunch of different imports
that we're gonna need.
And then you need to set your OpenAI API key,
and we're using the getpass module and the getpass function,
which will basically allow you to do this in a hidden way.
Now I've already set my API keys,
and we're also loading the Anthropic client
and also modifying the OpenAI client
to have our general API key.
Now, the first thing you're gonna need to do
is create a new function called check_openai_caching,
which will take both a system prompt and a user prompt,
and it will have a doc string.
And then what we're then gonna do is have a response,
and that response is gonna basically do
an openai.chatcompletions.create.
We're gonna use the GPT-4o model,
and we'll pass in both the messages
for the system prompt and the user prompt.
Now we're gonna get the usage out of this
by doing usage = response.usage,
and then we're gonna just print some metadata,
so the total tokens, the prompt tokens,
the completion tokens,
and then we're gonna build some logic here
that just allows us
to easily extract the prompt tokens details
and getting the cache tokens from those,
whether they're a dictionary or if they're not a dictionary,
and we have some fallbacks for those here.
And now what we're gonna do is also return the response.
What you should see is the first one
doesn't have any cached tokens.
However, the second time we make that call,
you'll end up with cached tokens.
And that basically means you're getting reduced costs
on your input tokens.
All we did was use that long system message prompt
and then rerunning the response
within that five minute time window
is gonna automatically enable
that bit of the system prompt to be cached.
And you'll get a 50% cost savings using OpenAI with that.
Now, it works slightly different when it comes to Anthropic,
so we're gonna have a different function here
called create_anthropic request.
Now we'll call cached_message,
and I'm gonna take in both a system prompt
and a user prompt, and we're gonna have a doc string.
And then what we need to do is we need to have a response,
which is going to use the Anthropic client.
And then we will have
the .beta.prompt_caching.messages.create.
And inside this you're gonna put a model,
and I'm gonna change this to latest.
Then we're gonna put the max tokens at 1,024.
Then we're gonna put our system,
which we will have a type of text,
and the text will be here from the system prompt.
Now this is the bit that's interesting
is where you actually add a cache control key
inside of here.
And inside of that cache control key,
you put a type of ephemeral.
Now we're also gonna have the messages,
which just have a type of text and a user prompt.
Now, after that, we now need to get usage.
So what we're then gonna do is we're gonna go below here
and we're gonna do usage is equal to response.usage,
and then we're gonna print the cache creation tokens,
which is actually usage.cache_creation_input_tokens.
We also have the cache read tokens,
which is, that is correct, cache_read_input_tokens,
and we also have the regular input tokens.
And that will be the usage.input_tokens.
We have the regular output tokens,
which is just the .output_tokens,
then we're gonna return the response.
Cool, so let's have a look and we'll use the same context
that we had earlier with the long system prompt.
And we're gonna call our new function.
So that will be create_anthropic_cached_message,
and then we're gonna call this twice.
And just one more thing
that we need to do is rather than have a type of text here,
we need a role of user and then the content.
So this, we do actually need to change this to a role.
So I've just updated that function
and then we're gonna run these two again.
And you can see we have created some input tokens.
We don't have any cached read tokens.
And then the second time that we actually do that,
we end up with not creating any cache creation tokens.
We do read from the cache.
We have some regular input and regular output tokens.
So again, you can see
that you can get very granular with Anthropic
by deciding what should be cached, okay?
So you can specifically say
in this specific type of message,
I want to cache this system prompt
and you also have the amount of cache creation tokens
and the cache read.
So you have to specify with Anthropic
using their beta client at the moment
with the prompt caching.
And obviously this is a much different approach to OpenAI.
They do offer additional savings when it comes to Anthropic.
So if you're doing a large amount of work
and a lot of LLM requests,
then you might want to look at Anthropics caching
because it provides 90% cost reductions
after you've created the cache.
However, if you are just,
you know, going about your day-to-day things,
and you're not particularly interested
in latency and cost optimization,
then I would recommend OpenAI.
So it really does depend on the use case
that you're currently working on.
The final function that we're gonna work on is a function
that we're gonna use
and we're gonna calculate the cost savings.
I'm gonna paste in the cached tokens,
and then we're gonna get the model.
And in this case, I'm gonna put claude, the latest.
And then what we're gonna have here
is a way of calculating the cost per million tokens.
So I'm gonna call this prices,
and we're gonna have two models.
We'll have the claude-3.5-sonnet,
and inside there, we will have the base input,
which at this time of writing is 0.003.
So that's essentially $3 million,
$3 per million tokens.
We also have the cache read,
which is 0.0003,
and we also have GPT-4o,
which has a base input of 0.0025.
And we also have a cache read of 0.00125,
but basically half price.
And then what we then need to do is we then need to go
and get the model prices based on the model that we put in.
And then we need to work out the base cost.
And the base cost is basically the number of cached tokens
times by the base input,
the cached cost, which is the cached tokens
times by the cache read.
And then we return these figures
and we'll also print these out as well.
So let's go and print those out.
And we'll also then return the savings, which is here.
So let's also have another variable called savings.
Okay, great, cool.
So now we've got the ability
to do the calculate the cost savings using this.
And then what we could do is, for example,
we could do a couple of different edge cases here.
So we could say OpenAI and we could put in some tokens.
So let's say we're putting in 10,000 tokens.
And we will do the same here.
And you can see here the base cost of that is $25.
The cached cost is $12.50, and the savings $12.50.
So you get 50% cost reduction with OpenAI.
If you look at Anthropic, we have our base cost of $30,
the cached cost, $3, so our savings are $27.
So a saving of 90%.
So hopefully this gives you a good indication
as to how prompt caching works,
how you can use this for both OpenAI and Anthropic,
and also how you can calculate cost saving metrics
based on any input tokens
that prompt caching currently provides
for different types of models.