Hi, and welcome to this ANC-Sharp video on the Microsoft Agent Framework.
Today we're going to look into Gemini and its code execution options.
Code execution is something OpenAI already have,
but Gemini have it as well.
So let's go into it.
So let me bring up a sample here,
and that sample being, under Google,
Google Gemini Specific Features, Google Code Execution.
So code execution is a way to run Python
in a sandbox environment up at Google,
and then either use it to do some advanced calculations,
like prime numbers and something like that,
or use it to generate, for example, charts
that you would like to get back as images in the tool.
So I have both here, first being the chart,
and I'm asking make a chart, use the code execution tool
to get the top five countries by the amount of cars they produce per year.
So as we have seen the other tools, special tools from Gemini,
of course we can use normal tool calling,
but if we need these special tools,
we need to go in and do a little of the breaking glass options here,
where we go into the chat client options,
the chat options, and set the raw representation factory
to a generate content config.
If we do that, we can get a tool called code execution.
There's nothing more to it, there's no options or anything,
it's just that we get a sandbox environment that can run Python,
and it can only run Python.
That's very much specified in the documentation.
But it doesn't really matter for us, we can just write it,
and we can just leverage the benefit of it.
In this case, if we run down here to letting it run,
it will now go in, use its world knowledge, find the top five countries,
make some Python, and then generate an image.
And we can get the data back here, so we see the top five countries
according to 2023,
and we can see that we spent quite a lot of tokens.
Let's come back to those at the end.
If we want to know a little more about how this was taken
from the world knowledge, but what Python was executed,
and how we can get the image, we need to go in to take the response
and get the raw representation factory as a chat response,
and after that into a generate content response.
This is the Microsoft Extensions AI version,
and this is the raw Google version that we need.
So once we have that, we can go in and get the executable code,
which in this case is this.
So if you, for some reason, wanted to run the code yourself
and generate the image, that would be possible.
But in line in this, there's also something called parts.
And in here, there is multiple parts of what it has done.
So the first one being the code that you generated,
then that was okay, the result of that code.
Then it gives us back the image in a blob,
so we get the raw data inside here.
It seems to also have a bug that it gives that back twice.
I have seen sometimes that it gives it back once,
and sometimes it gives it back twice.
In our case, we get it back twice,
so you need to take into account if there's multiple images
and you didn't expect that.
And the final one is actually the text being back.
So that is what we get,
and if we go in and find the parts that have inline data,
so the first one didn't, the second one didn't either,
and now we get to the data.
We can then make a path, in my case in my temp folder,
and write all the raw bytes into that and run the image.
So the image looks like this,
and that was generated up in the sandbox environment using Python,
and sent back to us as raw bytes.
If we go further,
we will see that we will end up with one more image.
Again, in my opinion, this is a bug
because it's exactly the same two images,
so we need to take care of that in our code,
unfortunately, until Google fixes this.
But at the end, we just have our image and our text,
and if we want to, our executed code.
Another way we can use the code tool is, for example,
give us the first 50 prime numbers,
generate them and give some calculations on them. If we do this,
we can again see that we get a response back here,
and that it used a fairly amount of tokens.
And again, we can go and get the raw contents,
and we can get the executable code,
so this is how it calculated the first 50 prime numbers,
and the raw result, if we want that for some reason,
which is what this code generated.
So we have full access to the raw data,
but also just letting the AI respond back if we want to.
The reason why I've talked a little about the tokens,
in input and output tokens here and up here,
is that they actually just bill you the tokens you use for code execution,
meaning we pay for this part of the code and so on,
as it was input tokens and output tokens.
And that's the reason why these are fairly high,
but of course still being good that it can do this.
And this deviates from OpenAI,
where you actually need to pay per session you run.
Google have decided to say, okay, we pay tokens,
but we don't pay for you to run a sandbox environment.
They take the charts on that and instead bill you in tokens,
which I think will be much cheaper than actually paying
for the container apps running up in the cloud.
In my book, that is a better solution and a cheaper solution for us,
but of course if you make some very, very advanced stuff,
you pay a lot of tokens in that code execution.
But that's actually everything there is to it.
We still need to do breaking glass like this,
but beyond that, it's very simple to just use,
and then a little cumbersome to get out again from the system.
But that's everything. See you on the next one.