Hi, and welcome to this ARN CSTR video on the Microsoft Agent Framework.
In this video, we're going to look into the Kodak and Hyperlite integration.
So, you might ask, what is Kodak? What is Hyperlite?
Well, it has also to do with the tool calling,
and tool calling in a different way.
Because when you have traditional tool calling in a LLM,
we, of course, have our prompt and our tool definitions,
and then we ask the LLM to do some kind of task in that prompt.
And it might need to use three different tools in order to do that.
And the way tool calling is working is, of course, that the LLM gets the request
and tells back how to call the tool,
then get back, call the tool, get back, call the tool.
So, we're not only doing one request and response,
but for every single tool call, we're doing multiple requests.
And that means that we are wasting tokens and time
because there is this back and forward all the time.
Kodak is yet another way of trying to do something similar,
but instead of sending all these different tool calls
and giving it all the definitions, which can be token costly,
Kodak wants to do one tool.
They want to do one tool that can execute code.
So, instead of all our tools in the definition,
we just give an execute code tool.
The LLM gets that and builds up a piece of code
to do the same task as we do up here,
but it only has this tool,
and it knows that that tool has various underlying tools to do this.
So, we still do normal tool code in C Sharp and so on,
but it just happens to be now that it's a piece of code
that executes those tools instead of it's the LLM coming back and saying,
I need to call this, I need to call this.
So, it's a more orchestrated one call to the LLM,
get back how would I do this in these three steps of calling these tools.
The execution tool then executes the code,
gives that back to the LLM, and then get the result.
So, we save some tool calls,
and we have a more orchestrated way of doing this.
But, of course, since we are executing code now,
it is a bit insecure because we need to have that code run in a sandbox.
So, we have seen code interpreter videos in the past
where we let the LLM use Python code up in the cloud
if you use the responses API.
But, of course, that is costly because someone needs to run that code.
This code can run locally, but again, since it's insecure,
we need some way to sandbox that code.
And that is where Hyperlite part comes in.
So, Hyperlite is a set of NuGet packages
that allows for code execution in sandbox,
not using C Sharp, that's not part of the spec,
but it can use JavaScript or Python.
So, this is the setup.
Let's see a few examples.
So, in my sample repo, I have something called Code Act here,
and in that I put in an introduction.
We might make more videos going forward.
But let's just set a breakpoint, and let's run the code here.
So, it's just getting started.
And let me move this a bit.
Getting started.
And let me move the console in here as well so we can see.
There we go.
So, we are just nearing up a new client, like always.
In this case, we are just using JTP 5.4 Mini.
And then let's try to use Code Act with JavaScript,
meaning Hyperlite JavaScript,
because Code Act is just the way of executing the code,
and the JavaScript is the sandboxed environment.
So, we're going to make an agent like normal,
but what we're going to do is we're getting one AI contact provider that is new,
and this package, you need to go in and have
the new Microsoft Agents AI Hyperlite NuGet package
in order to be able to react to this.
So, we're simply just saying,
always use execute code for calculations,
write the code in JavaScript, show the result,
and in my case, I've also said, show the code you used to do it.
And then we have our question here.
Let's get rid of this.
We want to calculate the 20th Fibonacci number,
divide it by 42, and subtract by 30,
just so it has a little extra thing to do than just Fibonacci.
And in some cases, this can actually be done by an LLM without any tools,
but that's not the point here.
But we make our agent like normal, and we call the agent.
And what we'll see once it's done
is that it ran,
and it did a function call where it said execute code,
and then it sent in what code it wanted to execute,
and it got a result back, and we can see the standard out here
is the number that we need over here.
Let's pack it into some JSON.
So what we can see over here is that it figured out
what the 20th Fibonacci number was,
made the calculation, and it gave us back the right result.
And this was not just generated by the LLM.
It actually executed this piece of JavaScript in order to do it,
where it made some JavaScript code
and did divide by 42 and minus 30 in order to log out what that was,
and that was what we saw in the output that it then used.
So this is just like code interpreter up in the cloud,
but you can do it locally now.
Cool. But we talked about tools,
so let's see a little more with tools here.
So we do the same as the code act here,
but because we want to add tools, we are just making some options here.
So we are making the options, and we want to use JavaScript,
and we then give it a tool.
And this tool is just like we have done so many times before.
It's sunny and 90 degrees, so I can even set a breakpoint here.
And of course, we would have multiple tools.
Just having one tool over the other might not be the smart thing,
but in this case, we are just getting it that way.
Then we again make an agent,
and we give our fiberlight code act provider with our options this time,
so it actually knows that it has a tool inside it as well.
So if we do it like that, we can, for example,
ask, what is the weather like in Paris?
And in normal scenarios, we would just have called that tool.
Now you can see it's calling the tool,
but let me show you what called the tool in one second.
And...
Oh, I'm running too far here, but let's just see the script here.
So what it did was it came back, of course, with the value,
but as you can see, it was a JavaScript call that made a constant city,
and then it made a call tool, which is part of the code act specification,
where it called the getWeatherForCity,
which was the one that we hit down here.
So our JavaScript actually called back,
told the C Sharp that it needed to call that tool,
and got the results back for us.
And you can imagine if I had three tools,
then it would have been within that call this, get the result,
do this about it, get the result, do this about it,
and finally produce one output instead of having the multiple tool calls.
So that is the JavaScript part.
Then it ran already this one,
where you can see that if we want to not just do JavaScript,
which is out of the box and default because it's very easy
to make a sandbox for that without us needing to do anything special.
If we want to use Python instead,
we need to have some kind of Python sandbox,
in this case an AOT, but it can also be a Wasm sandbox.
And I didn't know how to make this, I'm not smart with Python,
but I just had my codecs AI figured out,
and it gave me this file.
In my case, there's this extra file, it's 40 megabytes,
so it's kind of big, and I can't guarantee that this works
in every machine you have, so if it doesn't work
when you run this, go in and check out.
Just ask an AI, make this file for me.
But it's essentially Python packet up in a sandbox,
where we can run code.
And then we do exactly the same as before,
the only difference is that we're now using Python,
and telling it to use Python to write the code.
But it also did exactly the same, and we can see that over here,
where it shows the Python it used instead of some JavaScript it used.
And then we have, if we don't want to have these AI context providers,
we can actually just get the raw tool and the raw code.
So we do that by just making the same sandbox here,
but then we can say Hyperlite execute code function with those options.
And what that produces is actually a tool,
so a tool like normal, where we have a name for it,
and a description, and inside it will be able to do
the call to the Hyperlite.
And in order for it to understand what that is,
you can also, from that tool, say, build instructions.
So that tool can get back and say,
and this is actually what happens inside the AI context provider.
So let's quickly have a look at what the instructions it gives on the fly is.
You can execute code in a secure sandbox
by calling the execute tool.
Any tools listed in the tools description
are only accessible within the toolbox via the call tool,
as we saw up here, the call tool here.
They cannot be revoked directly,
so it's not by mistake begin to call some tools that are not there.
So this is the way we can do that,
and again, the tool is up here,
and I think it's here in the description.
If there had been any tools extra we have given here,
it would have been in this, that inside this tool,
you can call this, this, and this.
So that way, we can just put our instructions in manually,
and we can put in the tools manually
instead of using the AI context provider.
Probably easier to do this and let it inject on the fly,
but if you, for some reason, know up front
that you need the tool, don't need the tool, and so on,
it is possible to get them by tool calling.
And then it will run like normal.
So pretty nice feature
that is able to be sandboxed
and save you a lot of tool calling
if you have a lot of chaining of tools.
So that is actually everything.
See you on the next one.