Congratulations on getting through three of the four
major concepts of LLM.
So we have gone through chat, tools, and
structured output.
We're still missing RAG, but before we go
there, I want to do a little intermission
of the life of an LLM call, meaning
what actually goes on when you call the
LLM, what requests and what response are going
on.
The reason why I've waited for this moment
is that we need to know about tools
and structured output in order to understand this
lecture.
And we now have that, so now is
the time to do this.
I will do this through slides here, but
the code is also in the sample repo.
So what we see here is that we
have a normal system and there should be
nothing new except this little line that enables
us to get the raw calls back and
forth.
There will be a bonus video just after
this explaining a bit more what's going on,
because there's a bit more before this and
after this in order to do this.
But for us, it's just a normal client
and a normal agent.
We are giving it one tool called getWeather,
which is just down here.
We tell it to speak like a pirate
as an instruction.
We make a message called, what is the
weather like in Paris?
And we want it back as a weather
response, meaning structured output.
So nothing new that we haven't learned over
the last couple of sections.
And this is what happens when we call
this specific line of code.
So we call this URL, and in my
case, I had my resource called sensum365ai,
but yours will be something else here.
But beyond that, we're just calling OpenAI's deployments
of chat-gpg5 in the chat system on
completions with a version number.
And then we sent the first thing we
sent.
In general, you should look like these three
sections are one long JSON file.
I just put it up in three columns
here, so it's easier to look at.
But the message, we sent two messages and
not just one.
And that's because, of course, we have our
instructions, which is the system message.
So this is turned into one message.
This is turned into another message.
And that's why we have two messages here,
one in the role of system and one
in the role of user.
We then send the gpt5 as the model,
and then we send this block of code.
This block of code is the JSON schema
for our weather response, meaning this object over
here.
So we know that it's a weather response.
We know that we have a city of
type string, a condition of type string, a
degrees of type integer, and a degrees Celsius
of type integer.
We know that these four of them are
required, which is what we see over here.
And then a bit more like additional properties,
which we have none and so on.
So this is also being sent along with
the messages.
And then the final thing we send is
our tool, which is down here, and turned
into a schema.
That is a function.
The function has no description, because it didn't
provide one.
The name of getWeather.
It had one parameter, which was called city,
and that city was of type string.
And that is what's being sent.
And then you might think, oh, then it
will tell us back it's sunny and 90
degrees.
But that's not what happens.
What happens is it gives us back this
response.
And in this response, we are getting back
a finishing reason of tool calls.
And that means it says, hey, I want
you to make a tool call.
And in our case, it says, I want
you to call getWeather, and I want you
to give the parameter city the value of
Paris.
Beyond that, there's a few small things here
that are mostly empty in our case.
There's some creation date.
There's some ID.
There's what model and what its date is.
It gives us back some information about hate
speech, jailbreak, and so on, if any of
this is violating that, which it's not.
And then it's giving us back some information
about how many tokens we used in order
to do this.
So we can see that it used 190
tokens for input, meaning all we see here
cost 190 tokens.
We saw that it totally used an output
of 536, and 212 of those were for
reasoning, because it was a reasoning model.
So the last token, the number between these
two is this part, because this is the
only thing it generated.
The rest is just logging around our system.
And this means that Agent Framework will receive
this call and say, oh, that means the
user will like to call this tool.
So they invoke this tool on our behalf,
send in Paris as the city, and you
can see I set a breakpoint here.
And now we could expect that the LM
sits and waits for us, and there could
be some time out or something, and that's
absolutely not true.
The LM doesn't care at all what we
do now.
It doesn't wait for us.
It doesn't think it's missing to do something.
It, in its mind, it's complete, because it
told you, hey, I want to call this
tool.
And now it's off serving other people.
There's no connections.
There's no persistent connection.
There's nothing between us and the system right
now, because what happens is when we call
again, we send the entire message once again.
And technically, it's an entire new thing for
the LM, and it doesn't know that we
called it before.
So we sent the same information, speak like
a pirate, user, and stuff like that, but
we're sending this extra part in that, hey,
you told us to call this tool, and
here's the result of that tool.
It's sunny and 90 degrees.
Beyond that, we sent the same two starting
messages.
We sent the model again.
We sent the response format, which we not
used in the first one.
So why did we send it?
Well, it could have been that the question
would just be hello, and then we would
need to use it, because there was no
tool call.
We get the same prompt thing about self
-aid and self-harm and so on, and
when we sent the second request, we sent
the response format exactly in the same structure
as before.
We also sent a tool again, because it
could be that it needed to call the
tool again.
Had the message been, what is the weather
like in Paris and Berlin, it would technically
send one request, call the tool, send one
request, call the tool, and first there, it
would have all this data.
So there's a lot of back and forward
that we really don't see when it's abstracted
away, and when we send this again, we
pay for all these tokens again.
So that is also the reason why the
second we put on a tool or structured
output, it will cost us more.
It's, of course, incredibly powerful, so we shouldn't
think that it's wrong to cost extra, but
that's just the way it works.
So this would call, had we known this
up front, we could have made this call
instead, and it would know, because everything is
here that it needs to know in order
to answer the question now, and the second
response comes back, and in this case, it's
a finishing reason of stop, meaning it's done.
It has sent us what it thinks is
our final result, meaning the city equals Paris,
the condition equals sunny, the degrees Fahrenheit equals
66, and degrees Celsius equals 19.
It gives again back the IDs and stuff,
the self-harm part and stuff, and the
final tokens being used.
So the completion token is small, because it's
only what we get back here, and the
prompt token is bigger than the first time.
It was 190, now it's 227, meaning this
costs the difference between 227 and 190.
And then we're done.
We can take this JSON and turn it
into our object, and that's everything that is
to it.
It is completely request-response, request-response with
a magic happening up with the next token
prediction, but beyond that, it is just a
stateless machine of running.
It's not like it's sitting there waiting or
having a hook into your machine, being able
to call your tools or magically know how
to run C-sharp up there.
All the tools are run on your machine
with your access and so on, and it's
quite a brilliant system in order, in the
way it works, and this is really helping
a lot of people understand what is actually
going on, why it is so incredible, magical,
and at the same time so damn simple.
So now we know the life of an
Olicon, and we are ready to go on
and tackle RAC.