-: Hey, in this video, we'll have a look at what tokens are,
and also we'll look at something called token limits
and how that relates to large language models.
Firstly, tokenization is the process of taking sentences
or words and making those into smaller units called tokens.
On the right here, you can see we've taken
a bunch of sentences and we slowly break that down
by a process called tokenization into tokens.
Now, a token doesn't always have to be a word.
You can have smaller amounts
such as a character or two letters,
but the idea is that ChatGPT takes a series of words
and converts that into tokens.
Each model has a different number of input tokens
and output tokens that can be used.
For example, GPT-5.2 has a $1.75 for input tokens
and $14 for output tokens.
If we have a look at a different model,
like for example, GPT-5 mini,
you can see that the input tokens is $0.25
and the output is $2.
Now, both of these have different context windows,
and obviously as the models change,
these context windows will grow over time
and the prices will also get cheaper
for input and output tokens.
For example, in GPT-5 mini,
you can see that this has 400,000 context windows
and it can output up to 128,000 max output tokens.
If we have a look and go back to see GPT-5.2,
you can see that this has the same context window.
So different models have different prices,
and sometimes they have the same.
Different types of models
can have different lengths of context windows,
and also they could potentially generate
different numbers of output tokens.
They've also got different types of latency.
So you can see GPT-5 nano is very fast,
but it has average reasoning, so use that on smaller tasks
that you don't need a heavier model to use.
Whilst, if we go back and have a look at GPT-5.2,
you can see it's got the highest reasoning.
Its speed is medium,
and obviously it's their flagship model.
It costs quite a lot of money to run that at scale.
There's a really good page
that I'll put a link to in this video
where you can easily find lots of different types of models
that are on OpenAI and all of their varying trade-offs
in terms of the input tokens versus output tokens.
You'll find that if the document is too large
in comparison to how many input tokens you can use,
then that basically will cause the model
to reject that response
and you won't be able to see anything.
So I've just produced a load of text
just to show you what this is like,
and you'll see that actually
I can't even submit this message
because the message is too long.
So every large language model has a restriction
on how many tokens it can go in
and how many tokens it will produce out as the response.
Okay, so the next thing I want you to do
is just a quick exercise.
I want you to go to your browser like Google Chrome.
I want you to type in OpenAI pricing.
I want you to click on the first result,
and we're just gonna have a look at the different types
of pricing for different types of models.
So I'm just gonna go to browser,
to a browser, go OpenAI pricing in Google,
and then you're gonna click this one here,
the openai.com one.
And what you'll see is you've got a range
of different pricing for different types of models.
So go and have a read of this, have a look.
There's some other different types of prices
that you can see here,
but the main one I wanted you to look at
is, for example, o1 versus o3-mini,
and it will also show you the context length
of these models too.
Have a look at the input price,
and the output price are the ones
that you'll want to be concerned with if you're a developer.
Now, let's say you've got some texts
and you're looking to find out
how many tokens are inside that text.
Well, OpenAI have built a really easy to use tool
called a Tokenizer, which you can access
by the platform.openai.com/tokenizer.
If you go and click on this link,
basically it will take you to a webpage,
and you can see different types of models.
You can put in some text here,
and it will show you how many tokens that text has,
and you can see if you look at the color,
the separate tokens, you can also click on a show example,
and then you can see how many characters
and tokens those have.
They also have IDs,
so each token gets translated to a numeric value,
which is allowing the large language model
to understand what that token is in real time.
It's also possible to automatically get the token input size
from text by using a Python package called Tiktoken.
This allows you to take any model that OpenAI offers,
put the model name in,
and figure out how much the token count is
for a specific message.
There are lots of different techniques that you can use
to overcome the token limit.
The first one is if you are struggling with the input size
for what you're working with,
use a bigger model or more powerful models.
You could also shorten your prompt
or the context that you're including into these models.
You can also chunk, which basically means
rather than putting all of the context in one go,
you could put that in several messages.
You can also do something called windowed chunks,
which is you have a sliding window of chunks,
and we'll talk about that later in the course.
And also summarizing the chunks.
So when the message streams gets too long
or the context is too long,
asking for smaller summaries of the information
so that you can put those summaries in instead.
Cool, I'll see you in the next one.