-: Hey, welcome back.
So far, you've learned about how to manage messages
in both a stateful environment,
i.e. OpenAI saves the messages,
and also the stateless,
where the store=False with the Responses API.
The next thing we're gonna have a look at
is how we can do custom token counting
if we want to restrict the token size
when we're using the Responses API.
Now, you can either use the notebook,
or you can follow along and get that muscle memory building.
The notebook you would want to open
is the managing_token_messages_manually_responses_api
inside of the openai_features_and_functionality" folder.
All right, so the first thing that we're gonna do
is we're gonna install the tiktoken package
and the openai package.
After that, what we're then gonna do
is we're gonna import the OpenAI package,
import the model that we're gonna be using,
is GPT-4.1-mini.
You're going to need to update the client
and add in your API key here.
Now if you scroll down,
we've got this managing tokens manually
using tiktoken and a custom function.
So we've copied across that function
from the other notebook,
which is basically,
we have this import on tiktoken
and we can easily find out how many messages we have
for all of these different types of models
as a custom helper function.
And we're gonna have a look at a more complicated example
of how to manage tokens for the input tokens.
We have a bunch of article headings here
and we're gonna set up the system prompts.
So we'll say system_prompt
= You are a helpful assistant
for a financial news website.
And then we are also going to say
system_prompt += All of the subheadings.
So at this point here,
we're gonna do for heading in article_headings.
Then we're gonna do system_prompt +=
and then we're gonna add in the f string here with heading,
\n for the new line.
After that we're gonna do
messages.append,
and then we're gonna paste in the role of system,
and the content of that will be equal to our system_prompt,
and we need to put a comma here.
We're also gonna set the maximum token size,
so MAX_TOKEN_SIZE,
we're gonna set this to 2048.
Now we have some pseudo code here,
where we're gonna loop over
every article_heading in the headings.
And whilst the chat history object
is more than 2,048 tokens,
we're gonna remove the oldest non-system/developer message.
Basically how we'll do that
is find the index of the first message.
It's not a system or developer message,
and if there isn't a non-system message,
we're going to remove it.
So the first thing that we need to do for this
is we need to add on a messages here
telling them that we want to write
a very large paragraph about that heading.
The next thing we're gonna do
is tell ChatGPT to generate a response,
and notice how we are using the store=False
so that we can control the message history.
We're putting in the messages that already exist.
And then we are gonna then update the conversation history
with what comes out of that.
So we'll just create a new role with a role of assistant
putting in the content from the response.output_text.
And then our while loop,
we're gonna have some custom logic here
where we iterate over all of these.
And what you'll see is we basically end up doing this
where we look to see in our messages
if the message role is not in the system or developer,
then we found the first index of that specific message.
Then what we do is if it's not equal to None,
then we pop that message out
and we also then print out the current number of tokens.
So if you run this,
what you'll see is it will actually grow in the token size
as this runs again and again.
So we'll just wait for this to come back.
The first token count was 1,163 tokens.
The second token count will be even larger
because we're just continually adding in
all of the message history.
So this is why I wanted to show this to you
and demonstrate this to you
because as you add on these messages,
the input length is gonna get increasingly large,
which will add onto your costs.
So what we are doing here is having a custom function
that allows us to basically reduce the token count.
And then as we see here,
the current token count is roughly hovering just above 2,000
because we decided to remove some messages from this.
So hopefully this gives you a good way
of you can combine the store=False
when you're using the responses.create method
as well as the custom token utility function that we have
to basically generate any type of long output
whilst truncating earlier messages
that aren't your system or developer message.
In the next video, we're gonna have a look
at the differences in the API reference
and how you use the Chat Completions endpoint
versus the Responses API endpoint.
Both endpoints are currently fresh.
And the Chat Completions endpoint,
you will see that in some of our content.
It is being continued indefinitely,
so it's good for you to be aware
of how to use both the Responses API
and the Chat Completions API.
Cool, I'll see you in the next one.