Hi, and welcome to this Microsoft Foundry video,
which is part three, where we talk about
agents.
But before we do that, a small disclaimer,
because Microsoft Foundry and the agent experience is
still in preview [December 2025].
And honestly, it's quite buggy and half complete
in various places, almost to the point where
I didn't make this video.
So I will at the end of the
video, after I explained how it should work
and what works, I will explain a lot
of the things that don't work and then
talk a little about what we should do
from here.
But let's first dive in and see what
the claim is that we can do when
this goes out of preview.
So in order to understand this best, I
have put in a normal scenario, which is
working with agents in memory, the old AI
Foundry, classic agents, as they're called now, and
this new Microsoft Foundry, and see them side
by side in order for us to better
understand what is new in Foundry, at least
on paper.
So when you make a new agent, you,
of course, just take the raw client, Azure
OpenAI or OpenAI or similar.
You do something similar in AI Foundry, but
you do it using the azure.ai.agents
.persistent NuGet package instead.
And now in Microsoft Foundry, they have made
yet another package.
In this case, it's Microsoft Agents AI Azure
AI.
So, so far now, nothing special.
AI Foundry and Microsoft Foundry needs to do
it using role-based access control.
So you don't have an API key while
you can choose up in normal systems.
But here it begins to differ.
In a normal scenario, you just create your
agent in agent framework.
You give it model, instruction, tools, and it
all happens in memory.
In AI Foundry, you create an agent as
a persistent agent, meaning something where you create
it, it's there, and then you can reference
it later so you don't need to create
it again and again.
So the first time you create it, and
then you use it after the fact by
just referencing it.
So you only need to go and change
the agent if you need to update something.
This is also the same for Microsoft Foundry,
but can just do a little more.
Because up here, we have model, tools, instructions,
tools.
Down here, we have knowledge up front, which
is vector stores that it can automatically search
into and stuff like that.
That's more a manual process up here.
In the new Foundry, you also have a
new memory component where you can let it
remember certain things about the user if you
give it the user scope, like what is
their name, location, and such things.
A little like the AI provider context in
agent framework.
And then again, we reference them by the
existing name.
Up here, it's by ID.
Down here, it's by name.
And it can also now work with versions,
which I will get into in a bit.
So on paper here, an even greater system
than the AI Foundry.
And the final part is, of course, consuming
it up in a normal one.
You just consume the agent you just made
in code.
In AI Foundry, you either consume the agent
you just made or got the reference to
the agent, and then you used it.
And then on the fly, you could add
more knowledge and more tools to the file.
In Microsoft Foundry, this is completely changed in
various different ways.
First up, all agents have a version.
So you can now make a version 3,
but still reference version 2, for example.
So whenever you start here, you either need
to...
If you don't specify, you just get the
latest, but you can specify an earlier version.
I will show that in a live sample
in just one second.
And then you consume it, but the new
system is that you should actually not do
that.
They're kind of vague on what you should
do and what you shouldn't do.
You can do it, but you can also
publish an agent, meaning that it gets its
own endpoint.
And the reason for that is that then
your users should not know about your Foundry,
and if you make a new version and
you publish that, you can choose when it
should be published to the user.
So you can set and build, while here
you can choose when to publish.
And these public versions can then be consumed
by Agent Framework using the Responses API, or
it can be consumed in Microsoft Teams and
Microsoft 365.
This is what the documentation and everything tells
you what you can do and what's new.
And this is really exciting because it's a
really good system and having these things.
But a few things break down.
I will show a little of them now
in a live demo, and then we will
talk a little about the rest of them
after the fact.
But let's jump in here.
And I have Microsoft Foundry here in my
sample repo, which is linked in the description
of this video.
And I have a lot of old classic
agents from the AI Foundry before.
And then I have this new one, Agents
Basic.
And I made a basic one if we
make more advanced stuff with tooling and stuff.
And if we look at what we need
in order to do this, we just need
Azure Identity, and we need Azure AI package
from the Agent Framework.
And up in the portal here, I don't
have any agents right now.
I could go in and make an agent
and then just use it in code, so
I don't need to build the agent in
code if I don't want to.
I could simply say, just create an agent,
choose what model I want to use, write
some instructions, add some tools, and then I
could test it from here, or I could
go in and use it in code just
using the name test, because all agents in
the same project needs to have a unique
name.
I could then publish the agent.
And when published, I will get some URLs.
It takes a little extra time to publish
today.
And I could get an activity protocol that's
for Microsoft Teams and the bot service and
response API where I get that this URL
is specific to this agent, meaning the name
is in the URL.
So everything can be done up from here.
I could go in and say I want
the code interpreter tool, and then it would
know the code interpreter tool and so on.
But I can also make them in code.
And you can see I actually saved it
twice, one when I published and the other
when I added the code interpreter tool.
So now I have two versions, by the
way.
And I can go back to version 1
where I didn't have the tool.
So pretty neat that you can do it
up here.
Of course, it's all about normal resources.
This infrastructure as code probably would be best
if we made the agents in code, and
they will auto-create themselves if someone goes
and, by mistake, delete the agent up here.
But that's all a matter of opinion.
Let's see it in code at least.
So if we run the code, we will
just take it line by line.
So first we make something called an AI
project client.
Let me just move it down here.
And this comes from this Azure AI project.
And then there's another subfolder for OpenAI, which
is kind of silly now that Claude can
be used as tools.
So we are using technically a namespace called
projects.openai to work with Claude as well.
But we have done that for ages with
the Azure OpenAI tool that can run Mistral
and all the other models anyway.
In any case, we have the AI project
client where we just give an endpoint and
our Azure CLI credentials.
In my case, I set up Azure CLI.
Let me find it up here.
There's the AZ tool where I've logged in.
And that will give me my credentials.
Up in Azure, of course, they are given
by the tool and the managed identities.
And here you could use default Azure credentials
in order to get them from the Visual
Studio and stuff instead.
But I just use this because it's the
fastest one, despite it being slow.
Then if we want to make an agent,
we, of course, need the model.
And this client can actually also work with
the deployments.
It can list the deployments, but unfortunately, it
can't create the deployments.
So we couldn't go from just having a
project to getting the model deployed and up
and running.
But at least we can check if our
model exists.
I'm doing it with a try-catch method
here, but I could also just list all
deployments and then loop through, is there a
deployment called dbt-41-mini.
In my case, it exists, so I don't
get this error, but else I would get
the error.
And in my case, I would just write
out that it wasn't found.
I do this similar with the agents, where
I just ask for the agent by name.
And if that fails, which it will, because
we don't have it, we get a 404.
And we will just say, okay, the agent
doesn't exist.
Let's create it.
And in my case, I'm sending in the
instructions here, because I later on want to
make a version 2 with other instructions.
And in this case, it was urnici.
And then we use the client.agents.createAgentVersion.
So we always make versions instead of agents,
because the first time it will just be
version 1.
We don't specify the version or anything.
That just happens automatically.
So in order to create an agent, we
need a name, and we need to give
it some options in terms of agent version
creation options.
And there's various different agent versions, some being
hosted, some being file-based, image-based, and
so on.
But we want a prompt agent definition, meaning
a normal agent.
I haven't seen what the others can be
used for at the moment, so it might
be for some future projects or something I
haven't noticed in the system.
But if we make this, let me just
make it so we can see the code.
We can give tools in here.
So we give tools, for example.
And if we want to make a code
interpreter tool, like OpenAI have it, we can
do that.
We need to tell what container that should
be in, and we can make it create
an automatic container so we don't need to
do anything.
This is a little more cumbersome than the
old Azure AI Foundry, where we just told
it code interpreter tools, but it's because it
can also hook up to existing containers so
it doesn't need to spin them up.
In our case, we just do that.
We can also give it web search, which
is fine.
And you can technically give MCP tools, but
this is one of the buggy things that
are right now, in that we can make
a tool, meaning a remote MCP tool, but
if we do that, it will just call
it but never complete it because built into
the portal is that it needs approval for
MCP tools, and Agent Framework can't handle that
approval process right now.
So that's the reason I've taken it away,
but else it's just a name and a
URL, which also is a bit buggy in
my mind because if you had a remote
MCP tool that needed an API key in
its headers, you wouldn't be able to give
that through this agent NuGet package right now.
Moving on, we can do instructions.
We have talked about that.
And there's also options for reasoning options, but
this is also buggy at the moment, so
if you set them, the portal will completely
ignore it, set every model to high for
some reason, even though the default is medium,
and if you try to set it like
this, it will not recognise it, and it
will ask every time you do anything to
an agent, hey, it's unsaved because it looks
at the behind-the-scenes data and that's
different from what it sees itself.
So right now, effectively, we can't set the
reasoning, which is a huge bug in my
opinion.
The old portal also has it, unfortunately.
There's also file search tools and such things,
but these were just a simple one to
set up.
So now we have created our agents, and
now we can get it as Microsoft Agent
Framework.
And for that, we just simply say client
.getAIAgent, and we just give it a name,
and behind the scenes, it will go up
and find the right one because everyone has
a unique name.
So now we have a normal client with
instructions and our normal steps with the tools
we have made.
And now we can ask it, of course.
Let's say hi there to it, and it
will come back.
Hello, how can I assist you?
So nothing special there.
Then I will tell it what is this
last number times this line now divided by
this number and ask it to use tools
to calculate, showing that it can use the
code interpreter tool.
This will take a little while because it
needs to spin up a container up in
Azure, do the Python code it will generate
to answer this, execute that Python, get it
back, and then we are on run.
But we can see that it calculated what
it was, and behind the scenes in the
messages, we can see some content about that
it was called with code interpreter and what
code it was behind the scenes and stuff.
So it was this down here.
Nothing special.
We get the result back.
Now we try to say what is the
biggest story for today, and today is the
9th of December, 2025, when this is recorded.
And if I know my American politics, it's
probably something about Trump.
But let's see.
So this is behind the scenes using Bing
now, but in the past, you needed to
make a Bing account and stuff.
Now they just say you don't need to
do that, but you're still built for the
calls, just like you are for the code
interpreter calls.
So it will say that it's something about
the Supreme Court and President Trump.
Don't want to go into politics, so we
just have it.
So now we have asked it and used
its tools and stuff, and so far, sort
of okay, except for a few mistakes or
limitations.
Now we're going to try and call the
create agent again.
By the way, if we just look up
in the portal and refresh, we can see
our agent up here, and we can see
that it has what it was.
It has the code interpreter tool, and here's
one more mistake the system has.
It can't show the web search tool, despite
it being here, because the real web search
tool is in preview, but it still reacts
to web search, so a bit messy again.
Up here, we can also see the conversations
we had with it.
I've used this name before, so it has
multiple conversations, and we can see that it
used the web search tool to get this
text and generated the response back and so
on.
But what we can also see is that
this is in version 1, and if I
go in and give it a new instruction,
meaning I just call the same code as
I did when I created it, and the
only difference is that it has to speak
like a pirate now.
So if I do that, and we go
up in system here, we can now see
that it's version 2, and it has speak
like a pirate and version 1 had UL
and ICI.
What I can do is I could also
just drag it up here and say, oh,
I want a version 3.
So if we do that and call the
same code again and go to the portal,
we will see we don't get a version
3 because it would like to check, is
there any differences in the agent definition, and
if not, then let's not make a new
version.
So it's perfectly safe to call create version
if you are okay with the extra time
it takes in order to be sure that
your agent is just like as you have
defined it in the code.
In any case, we have now said it
needs to speak like a pirate, so we
just get the agent again.
So this is version 2 of the agent,
and we'll just say hi there, and it
will come back with ohoi matey.
So we now hit version 2, but if
we want to hit version 1, we can,
in that we can go to client-agents
.get-agent-version-async, and we can say
give me my agent in version number 1,
and if we do that, we get a
version back, something called an Asian version from
the portal, and we can now get our
agent by version.
So this one might be a good idea
if you use it like this and you
want to build new versions, but have the
users still run on a version and then
in your code bump up, now I want
to run version 3, now I want to
run version 4, and so on.
So agent by version we get, and we
can now ask agent 1.
It doesn't know it's agent 1, but it
was fun to write, and when it comes
back, it doesn't say hello matey because this
is the version that had it was just
a nice AI, but if we asked version
2, it would speak like a pirate.
And that is all the code, but let's
run it again and see what happens up
here.
Now that we have our agent, and it
shouldn't surprise that, of course, we have our
model.
We have that before, and if we ask
for our agent, it has the agent, so
it didn't show an exception, so we can
just use the agent.
So now we're not calling create agent anymore,
and we're just using the one that was
up there.
And the answers will be the same, but
remember that now that we're just calling the
agent, it will be version 2, which is
the one that speaks like a pirate.
So for that reason, we will see that
it will come back, but else it will
be exactly the same.
Once it wakes up.
But I think you get the point.
We could then make a version 3 that
didn't speak like a pirate, so when we
hit this, it will essentially not do anything.
It will make version 2 again.
So now we come back with that, and
I don't want to spend your time on
this, but we need to talk a little
about the limitations, because it has been a
real struggle making this video.
It's taken me all day to find these
simple things, because when you just look at
the surface, 80 to 90% works, but
the last percentages, which are very important, at
least in my mind, makes this a very
tough sell, meaning I would never, ever go
live with what we have right now, to
be honest.
But let's talk a little about the eight
things I have found that are broken and
buggy right now, and there might be more.
First up, one of the big problems is
that we can't set the reasoning effort.
We could go up in the GUI and
do it, but then we couldn't use code
to make our agents.
So if you want to live with you
needing to set the reasoning effort up in
code instead, or in the GUI, you might
could live with this problem.
To me, reasoning effort is so important, because
it's simply too expensive to run the Chatterjee
V5 models with reasoning on for every single
little question.
For the good stuff and hard and complex
stuff, it's really good, but for the other
ones I want to go down to minimal
or none, and we can't do that.
Second, we can't mix hosted tools and ad
hoc tools.
You could in the old AI foundry, so
it might come back.
But right now, for example, we couldn't go
and have the code interpreter tool and then
a local ad hoc tool, like get the
weather or whatever we want.
That is simply not possible.
We can get ad hoc tools to work,
but then it's only ad hoc tools, or
we can get hosted tools to work, and
then it's only ad hoc tools.
But we can't mix the two in a
good manner at the moment.
A few things works, but not all, and
this is a really deal breaker for me
as well.
Hosted MCP tools, yes, we can define them,
but the agent framework will just get back,
hey, I want to do this tool call,
but it doesn't know about the tools, so
it can't understand what's going on and ask
us to approve it.
And on the foundry side, we don't have
the setting to say if something doesn't need
to ask for approval, and we can't set
headers in any way or authentication.
So that's also a big deal breaker compared
to running them locally.
The web server tool works, and again, it's
easier to work, but the GUI doesn't display
it.
That is not a big deal.
We can go around that, of course, but
it's annoying that it isn't there and being
shown.
Then I didn't show it, but up here
in the agents, there's something called a memory
component, and if you turn it on, it
will make a memory store, and then you
should be able to go in and say,
hey, my name is Rasmus, remember that, go
to a new thread, a new conversation, and
ask what is my name.
That works 50% of the time, it
seems, and we can't define this in code
in any way.
It's simply not in the NuGet package that
we can do.
So I wouldn't rely on this.
I would use AI context provider instead myself,
but it simply doesn't work at the moment.
Then there's the knowledge system, which is kind
of a big mess.
You can press knowledge here and say ads.
I have never really gotten it to work
really well, and even when I get it
to work, it can call it sometimes, but
it can't really do anything with it, and
I've tried various ways to try and get
this to work together.
The entire knowledge section here is rough too,
to be honest, and it's mostly built on
AI search, which is a really expensive resource
to run.
At the same time, so I would probably
not use it in real life anyway.
I haven't looked endlessly into it, but when
I even can't use this GUI to set
it up, then there is something wrong, and
you can't set it up in the code
either.
Then there is that these publish agents are
not properly supported.
You get this URL, but the agent framework
or OpenAI's own NuGet packets can't support that
URL.
There is some breaking glass, meaning go in
and do some hacks to get it working
in the system, but I talked with the
team, and they said that there's no documentation
and no information about it, and you need
to go into the raw call and on
the fly get rid of stuff to get
it working, and I would not be very
happy to do that.
And then we end up, why have they
made publish agents in the first place when
we can already hit them by a version
and stuff?
It must be because they want us to
actually go live in production scenarios with these
versions instead of hooking up to Microsoft Foundry,
but given that it doesn't work, it's yet
another issue.
I could probably live with this one because
why not just use the Foundry, but bigger
companies might have a problem with this.
Then publish to Microsoft Teams, and Microsoft 365
sounds so cool on paper that you can
more or less without any code, if you
just use the portal, get an agent in
there with your own instructions and everything.
I have not made it to work.
It makes some agent services, and it says
prepare agent, press okay, and nothing happens, and
then you can try again, and then you
can't publish the agent again.
I might be doing something wrong.
It could be that I might have not
enough permissions in the setups I have, but
if it works, it's at least not easy
to do.
So this and then the portal being slow
at times and stuff, but these are the
big things I have found that really makes
me hesitant on what to do from now
because what do we do with this series?
I can soldier on.
The next video would be about workflows, which
probably are also a lot of buggy.
So I'm a bit unsure if it's better
to wait a couple of months to continue
this series, make a version three, take two,
where all the stuff works, and then continue
there, or if you want to see everything
that is in there right now with all
the buggy stuff as well, where there are
certain things I can't show.
So I'm generally interested in figuring out if
you want more in this series, or this
was the first step in the series, and
we will take a break and come back
when things are actually working because you can
definitely not go live on this, in my
opinion, or I wouldn't do it.
The needs might be less than what I
have, but especially reasoning effort and mix of
tools is a major deal breaker for me,
in my opinion.
But sound off in the comments, and let's
see if there will be a part four
or not in this series.
Up to you guys.
But in any case, see you in whatever
next video is there.