In this lecture, we will talk about the
code interpreter.
Code interpreter is a tool a little similar
to web search, which is a hosted tool,
so you don't need to make it yourself.
But what it can do is it can
run code.
And since it's AI, it can run Python
code.
But not that we need to write the
code ourselves.
We get the AI to write this Python
code and have it execute.
The main reason for having such a tool
is that, for example, if you give your
AI a very complex mathematical formula, it might
not be good at using next prediction tokens
in order to figure out what the number
is.
But it can certainly write a formula in
Python and then execute the Python code and
just get the answer back instead, which is
more efficient and more secure to be correct.
But it can also use the code interpreter
tool to generate things, for example, images and
stuff like that.
And this is what we are actually going
to do here.
So I'm going to show it first, and
then we'll go into the code, because this
is probably the most advanced example we have
seen so far.
But what I'm going to ask it is,
create me a pie chart of top five
most populous countries in the world.
Because before this, let me just start it,
because it will take a little while.
Before this, we have only gotten text back.
But we can actually get things like images
back.
Not that we're doing image generation, but that
we're using a code interpreter tool to actually
get the image by generating it using Python
code.
So right now, what's happening is, it is
spinning up a small container up in the
cloud and generating this image.
It is okay at it, but not good.
You can see top five here is not
put in.
We would be able to tell it a
bit more that the title should be this,
and so on and so forth.
But they have given us the right information,
and I'm now running a PNG right now
in my temp folder.
So let's see how this is done.
We also see it gets the information back,
but we are most interested in getting the
image right now.
So first up, this only works with OpenAI.
Azure OpenAI have code interpreter, and they can
use it for mathematical formulas and run things
like that, but they cannot generate images back.
It's a limitation, bug, call it what you
want.
So as you can see here, instead of
the normal Azure, I'm using OpenAI for this.
And then I'm making a client.
In this case, it's a GPT 5 Nano,
and I'm giving it one tool, meaning this
hosted code interpreter tool.
And as we saw with web search, these
host tools only work with the responses client,
which was the one that were in evaluation.
So for that reason, we have it up
here, because there's more of this that needs
evaluation.
And then we do our chat loop, but
instead of just writing out the answer, we're
also expecting the response back, because the response
have a lot of extra information.
And let me set a break point and
do it again, so we can better understand
this code.
So I'm just going to ask the same
question again.
And once it comes back, we will begin
to loop to this, because the response underneath
have one or more messages, meaning whenever we
ask just hello and the AI say hello
back, we get two messages, one from the
user and one from the assistant.
But inside these messages, there can also be
content.
All content are by default AI content, but
we can ask if it's various different types
of content.
So for example, annotation content, which is how
we get a file ID that is generated
by the tool.
So now it has done it, and let's
see this response object with the debugger.
So what we can see is we have
inside it one message.
This message is from the assistant.
Let it come back.
And inside that, we'll find various content.
So we have content about how it reasons.
We have content about it calling the tool,
where we see some input in that it
makes some Python code.
It used some matplotlib, stuff like that.
And what we are most interested in is,
you can see it come back with further
things here, where we actually get a container
ID, meaning what container up in the cloud
was running this.
We need that later on.
We also see, let me see if I
can find it here.
Let's just run this code, so we can
see that when we go to the messages,
we can go to the content, and check
if that content has any annotations.
So the first content didn't have.
The second didn't have.
The third didn't have.
We have eight of them in total.
Fourth didn't have.
Fifth didn't have.
Sixth.
Seventh was an annotation.
So if we look at this content, we
see we get one annotation, and that annotation
being, what is the file ID of what
we did, and what is the container ID,
which are the two things we need.
So if we go in here, we can
check that this raw representation is a container
file citation message annotation.
And if it is not, which it is,
we now have what we need, because we
can now use the raw client to get
a container client, meaning now we are not
working with chat messages and anything, but we
are actually going to get files that are
associated with my account up in OpenAI.
So we get a client in order to
download our file.
So we take our citation and take the
container ID and the file ID.
So what happens is it actually downloads it
all.
It gives me back the file in a
binary format, so I can't really see anything
other than it's linked here.
And then I check the file name of
the system, which is this type, pi thing.
And then I write all the data, in
my case, down into a temp folder.
And once I do that, I just use
normal C-sharp to run that part, meaning
it will open up in my file or
in my image viewer.
So it will open up.
Now it looks a little better here in
this case.
But again, we could use prompt engineering to
tell exactly how we want it to look.
We then go to the final content and
messages.
We're done.
And we have our image that we could
then use in our system.
So while it's very easy to just give
the tool, and if we had not used
it to get images back but just used
it for a very advanced formula, we wouldn't
need to do anything down here.
But if we get files back, we need
to do stuff like this, going through the
content, figuring out where the annotations are, and
figuring out where the container file citations are,
so we can actually get our files and
show them to the user.
So more advanced down here.
And in general, you will now see that
this response, which we have so far just
used the text order to string, has so
much more information into it when we need
to get things back in a more advanced
stuff in multimodal.