Hi, and welcome to this AI and TCR video and the second part of the Reasoning Deep Dive mini-series.
In this video, we're going to look at how OpenAI do its reasoning.
If you haven't seen part one, I heartily encourage you to do so before seeing this video.
So here we are inside the Visual Studio, and over in our sample repo, we have Thinking
and Reasoning settings, and we are running the OpenAI settings here.
So OpenAI, as mentioned in the first video, was the first one to actually have reasoning models.
And the first one was the O1 model.
You can still hit it in certain areas, but it's kind of a legacy model now.
Then they made O3, there was never an O2 because of a legal trade name with the O2 company in the UK.
So the next one was called O3, and the next one was called O4, there's various mini-models
and pro-models of them.
But what you really need in models right now is the ChatGPT5.
So ChatGPT5 was the first model, so for example, the famous 4.0 is not a reasoning model.
We should expect that all future models are reasoning models from OpenAI, because this
is the future, but again, these videos are important because we need to control how much
thinking is going on.
So in all these samples, I'm going to ask the question, what is the capital of France
and how many people live there, with an extra thing for the AI to understand is to answer
back in max three words.
I also do that so it's easier for you to see the output over here.
But what we can do is, if we just take a baseline, and the baseline here, if we go
down and have a look, we can see we're using a ChatGPT5 mini, and we're just asking the
question, nothing changed other than just telling, hey, ChatGPT5 mini, answer this question.
So this is what I call a baseline, and what many people don't know is that reasoning is
on by default on these models, and it's by default medium.
That might sound okay, but for a simple question like, what's the capital of France and how
many people live there, it is probably a bit too much even for that.
So if I just press F10 and let it run the code down here, which is just a normal run
and then give some output, we will see that it takes quite a while.
So it's still thinking, and it came back.
And what we see is it wrote back that Paris is 2.1 million, not the important part in
this video series, but more that we had an input token that's also not too much because
it will always be the same input, but we used 528 output tokens, and of those, 512 of them
were used for reasoning.
So if we do the same, but now we're going to do reasoning equals minimal.
And to do equal minimal, we need to do quite a lot of extra here in the raw format because
of this not being a standard yet on how to do reasoning.
So we need to, inside our agent, give the chat client options of going into the chat
options and inside this doing something called raw representation factory.
And this is an object directly from OpenAI that we need to set and set the reasoning
effort to a number.
And over time, they have made various different versions of this reasoning effort.
There's actually two more now, one called none, and one called extra high.
Some models support them, but for example, it's only the chativity 5.2 and higher that
understand the very high, for example.
But if we do all this extra step, and to me, this is very cumbersome, and I will also show
how I, in real life, use something on top of agent framework in order to make this easier.
But if we do this and let it run, so if I come up here again, write out the chat and
let the code we just saw run, you'll see it comes back much, much faster because now
we had minimal reasoning, gave back the same answer because it's more or less facts that it have.
It doesn't need to reason on it.
So imagine you're the user being up here versus the user being down here when it comes to
such a simple question like this.
You would much rather be the user down here because you got a fast answer, and at the
same time, you much want to be a developer down here because it costs much less tokens.
So controlling reasoning for minimal like this is a good thing, but if it had been a
very, very advanced question, of course, we might end up having problems here because
it couldn't follow the rules or didn't think enough about it.
If we go on the other hand and go much higher because what we have also seen so far has
only just been the chat client, and when we saw down here, we can only really set one
thing about this when it comes to reasoning.
There's nothing more that is about reasoning in the chat client.
But if we go up here and begin to use the responses API, which does it in a slightly
different manner as well.
Here it's a create response, while here it's a chat completion options, making it difficult
for you to understand and remember in day-to-day.
Whenever I need to set these things, I always need to go back to a previous sample, hence
the reason why I've made something better that I will show in one second.
But this is the raw way of doing it, and we can now set something called a reasoning options.
And in that, we can set again an effort level from high, extra high, and so on, but we can
also set a reasoning summon verbosity, because whenever something reasons, we might want
to see what did it reason about?
Why did it come to that conclusion?
And in the chat client part here, that is not an option, while if we use the responses
API, it is an option.
In general, OpenAI is a big mess of different settings with different options, and when you begin to...
When we see in video two and three on Google and Entropic, we will see that this is kind
of a big patchwork of different things as they came along.
They will probably streamline down the road, but here we need to go in and actually go
between auto, concise, and detailed in how much we want to see back.
And we can see it's a summary, so it will actually think more inside, but we only get
the summaries of what it thought about.
If we go into this example, in this case, I've set the reasoning to high to see how
much it thinks, because, again, if you set reasoning to minimal, there will not be a
lot of reasoning summary back, while if we set it to high, there will be a lot of reasoning summary back.
So if I let this run, again, it will think a lot, and I'm even using the ChatGP 5 Mini.
If we use the ChatGP 5, the time we would now sit and wait would be even longer.
And again, high is higher than the default, which was medium, so this will be the longest
time we wait here.
So behind the scenes right now, it's doing the internal monologue, and we can see now
that it come back, that this is determining the response form.
User wants to have a response of maximum three words, and a lot of things it now thinking about.
So we are getting this data back.
I didn't show it, but down here, when we ask the question, we can, inside the messages,
find the contents that are subtype text reasoning content, and then we can write this out, and
this is what we're seeing right now.
So this is not what the user will see, but you could optionally show this or log it or
whatever you wish to do with it in order to understand why did it end up saying Paris 2.1 million.
And we can see, because of reasoning, we now ended up with 1,472 of the tokens were used for reasoning.
So it took a long time to answer this, and that's because this is a long reasoning effort.
You ask the question, and had it been a human, that human would have sat for perhaps 30 seconds,
a minute, before they even answered your question.
But again, if it's more complex, this is really the way for the LLM to actually get the answer
right, at the cost of more tokens and more time.
Let's move on, because I really, really honestly hate code like this.
It's difficult to remember.
It's annoying to look at.
And in real life, when I sit and do these things, I think, yeah, I should use the ChattyPT5
for its intelligence, but I don't need it to reason.
So I want to make it quickly go down to minimal, but that's not easy, because I need to sit
and copy-paste this along, and I make helper methods to do it, and so on.
So I have created what is called the Agent Framework Toolkit.
So what we're about to see here is exactly the same as we saw with the raw here, going minimal.
And the Agent Framework Toolkit does it in a much more concise way, in that we just create
an agent factory, and then we get an agent.
And among the features, just like models and tools and such, is just the reasoning effort directly here.
So this does exactly the same, just in a much more concise manner.
So if we do this, we can see it's fast, not because it's Agent Framework, but because
we chose minimal.
And we get back that it's 1.2 million and a low token count.
And in the same manner, if we want to use Agent Framework Toolkit for responses part,
the only thing we need to tell is it's a client-type of responses and the two settings.
So again, these few lines of code versus all these extra things you need to set in
the raw framework.
So if we do that, again, it will be slow to do it, because we are now doing the reasoning,
and we should roughly end up with the same as we saw in the raw responses API.
What you will also see down here is that when we have reasoning, I have a handy method
just directly on the response to say, get reasoning content.
So you don't need to write all this part as well.
This is just tucked away in an extension method.
So if we get back, it will just give us the reasoning, and we can print it out to the
user, save it, and so on.
So Agent Framework Toolkit does not make reasoning better or faster or anything, but it just
makes your part of the code much more simple to look at.
So if we look at this part of the code versus the raw version with responses API, we can
see it can't even fit on half the screen here.
So it's just a more convenient way to do it.
But that's not the important part here.
If you do it one way or the other, the most important thing is go in for the love of God
and help your users not get slow answers just because you've gone to the new model.
And if you know up front you don't need reasoning, you can go to a chat.gtkform1 model, which
is not a reasoning model, but when it's easy, I tend to stay with five on everything and
then controlling the reasoning on the fly.
So that's everything for OpenAI.
In the next one, we're going to look at Google.