Hi, and welcome to this AI and C-Sharp video on the Microsoft Agent Framework.
Today, we're going to look into OpenAI and its caching tokens and how they work.
So, let's jump into it.
So, here I have some sample code and a small prompt.
In that, we just make an Azure OpenAI client here.
We could use OpenAI client, they work exactly the same.
We are going to give our model a little text with some aromatic equations,
and we tell it this is the only knowledge it has.
Then we are starting a session, and we are going in
and writing out the answer, of course, but also the token count.
And we're using GPT-5 Mini here, and the price is in million tokens,
and it would be 0.25, and then we can see caching is one-tenth of that.
Then we have output, and reasoning is part of output.
We're not really going into reasoning today,
but I've just put it out so we have all four of these.
So, if we ask our AI here,
what is this text about?
What we will see is we get our answer back.
And because this is our first question, we don't get any cache or anything.
It's just the input and the output, and much of it was reasoning.
Probably wouldn't need reasoning for this specific question,
but just so we have all of it.
And we can do a follow-up question.
Are the answers correct?
And we will get back that, yes, they are correct,
but what you will notice is we didn't hit the cache.
So, the cache didn't work.
Does that mean we need to set some settings over here? Why is that?
Well, we don't need to set any settings.
Caching is on by default, but caching is only happening
when we hit a thousand or more tokens in terms of OpenAI.
So, because we were below that,
they, being OpenAI or Azure in this case,
didn't go in and use the cache because probably it's too much of a hassle
for very, very low token counts and would hurt our performance.
So, they have chosen 1,000 to be when you begin,
and you can't really change that.
So, in order for us to see caching,
we need to go in and use a longer text.
So, in my case, I have a book here.
It's a prefix of Pride and Prejudice,
and then the first chapter.
It's roughly 500 lines of text.
And the reason for a longer text is, of course,
to hit over that 1,000 token limit.
So, if we go in and ask again, what is this text about?
We will now see that once it's come back,
and it will take a little longer now because we have a longer text,
it comes back and gives us a summary of what that text is about.
And we can see our input here is almost 10,000 tokens.
So, we are above when caching should go into effect.
So, right now, it has actually cached some of our tokens
and are on their servers.
Their servers being Azure, in this case, but also OpenAI.
So, if we do a follow-up question to this,
we can see that we have 1,000 tokens.
If we do a follow-up question to this,
what characters are in Pride and Prejudice?
We will get that back.
And we don't really care about what it says here.
We are interested in seeing that of the 9,800 tokens in total,
we added a little more because we added more questions.
We saw that 9,600 of them were cached,
meaning we paid only one-tenth of the price for those.
And that's because whenever our question and answer is exactly the same,
and in this case, this text is exactly the same as we sent it the first time,
it doesn't need to predict the next token for each one of those 9,600 tokens.
But it needed to predict the rest.
So, if we asked,
who is Darcy?
We will again see that our cache is quite high.
So, again, we only really paid a 0.25 in the difference between these two.
So, this is really good, we don't need to do anything,
but of course, for the small cases, we don't get any caching value,
but that doesn't matter too much.
So, this is good.
One interesting thing is that if I restart this,
we are in a new session, and if I say, what is this text about?
What we will see is that it's actually still caching.
So, the cache is in per API key we use.
So, even between sessions, if we ask exactly the same thing,
we will get the same.
But if we go into the book and put in something different here,
we lose our cache.
So, if we do this, what is this text about?
And because I changed the first character, it's not the same text anymore,
the cache is missed, and we see that we end up using full input.
So, what if I go down here and in the middle, just type something in there?
Let's see how that affects our cache.
Here we can see we can cache up to this area, so this was the 1100.
So, it's only when something changes,
but if we have exactly the same text, then we get the cache.
If you go see OpenAI's documentation,
they talk about caching, caching, caching, caching, caching, caching, caching,
They talk about caching being there for like 5-10 minutes until inactivity,
and no matter what, the cache will go away after an hour.
So, of course, they need to keep that up in memory, up in their servers,
so cache is fleeting,
but it will stay around long enough for a follow-up question like this.
And everything is transparent, you don't need to write any specific code
in order to hit the cache, so they're just giving you long text cheaper automatically.
And it's roughly this tenth of a price every time we do caching,
but, of course, we pay more because we spend more.
So, that is actually everything how caching works.
You don't need to be worried that you need to do anything,
it just works out of the box, which is cool.
So, that's everything, see you on the next one.