Hi, and welcome to this AI and C-Sharp video on the Microsoft 18 Framework.
Today we're going to look into Foundry Local,
which just came out as 1.0, meaning general availability.
So before we jump into code, let's talk a little about what it is and its history.
Because this is Microsoft's own definition,
Foundry Local is an end-to-end local AI solution, blah, blah, blah.
It's about being able to run LLMs locally.
And using some SDKs, we are, of course, using the C-Sharp version.
The alternative to Foundry Local is OLama,
which can run the OLama models, but also a lot of other models.
But I think, as far as this goes, Microsoft put out Foundry Local
in order to satisfy enterprise businesses
that can't really just be dependent on an open-source model like OLama,
which is not owned by Meta.
So let's talk a little history about this,
because I have actually covered Foundry Local before,
but it has been a very bumpy ride on what this is.
It was introduced back in May 2025 at the Build Conference from Microsoft,
and in June, the first version came out, and that was version 0.1,
because we were in preview mode at that time.
So quickly, we got over June, August, September,
we got version 0.1, 0.2, and 0.3.
It required to be downloaded using Winget before we could work,
but that was pretty okay to work with,
but it was a little odd that you actually needed to download some
to your machine before it worked.
That might be good for a local machine,
but if you want to run it up in a cloud or something,
that should be very annoying that you can't make the code do it all.
So in my samples back then, I actually automated Winget
in order to get it to work.
So I was okay with it at that point.
I'm not really a big fan of offline models,
but it worked, and it could connect at that time to the semantic kernel,
which was the predecessor to the ADIN framework.
Then in November 2025, something very, very odd happened.
Something very, very odd happened.
They released, suddenly went from 0.3 to a version 0.8.0.1, and 0.8.0.2.
This was a complete rewrite of what we have seen in the early days.
Winget was gone, and it was made with some open-source NuGet package
that was not the official OpenAI standard,
but something it felt totally randomly picked off,
and it made it completely incompatible with everything else Microsoft did.
So when I saw this and tried it out and tried to make it work again,
I simply gave up and went back to 0.3 because I knew that worked.
And since then, they must have come to their senses
because now in April 2026, we have version 1.0.
And again, it's not dependent on Winget,
and it has compatibility with ADIN framework,
as we'll see in some code in one second.
But let's also evaluate a bit if it's worth doing
because there is some oddnesses in it still,
but that's perhaps to be expected from local.
But let's have a look at the code.
So here I have the code.
It's in 0 to 1st agent under OpenAI-based
because it's still OpenAI-based,
and 0 to 1st agent local foundry.
So let me get rid of this.
So the model I'm going to use is a very, very small model,
GWEN3, 0.6 billion parameters.
And that's because my graphics card is very, very small.
And for that reason, I can also only use the CPU models,
not the GPU or NPU models in here.
So I'm just going to use Direct4CPU,
but you can change here and also go up to higher models.
What models are available, we will see in one second.
One thing to note is that in order to use this,
you either need to use a foundry-local WinNL
or a foundry-local on other types of machines.
So you need conditional NuGet package in here to do this,
and also you need conditional target frameworking
that you need to target the windows before they work,
not just the normal .NET 10.
So already here, it's a bit funky, but it is doable.
So if we do this, we make a folder
where we can store all these things.
In my case, I would put it in my user folder here,
and you can see it's empty at the moment.
So we're going to make what is called a foundry-local manager,
and we're going to make it with a name and a path
and where we want to host the models in terms of a local web server.
So we're going to do that, and once it's created,
we can take its instance.
Again, a little funky way of doing things, but it is possible.
Then we're going to get a catalog of models,
and then we can list these models.
And let me run it down here so we can see.
These are the models that are supported,
and among them there is this Quintree 1.6 billion.
So it's not everything in the world that can be used.
The biggest and most interesting one is GPT-OSS 20 billion parameters,
which is the OpenAI ChatGPT open source model that was released
that are roughly equivalent to, I think,
between ChatGPT 4.0 and ChatGPT 5.0.
It's a little stronger than 4.0 and not as smart as 5.0.
So it's a fairly okay model, but there's also all the FHIR models,
the Quint models, and the Whisper models, if need be.
So once we have these, we can get the specific model,
and we give this alias.
And in there we need to find among…
In this case there's only one variant that is CPU-bound,
but there could have been a GPU-bound version of some of them,
and then you would be able to choose the model of…
if it should use GPU, CPU, or NPU.
In our case we do that, and then we check if it's already in the cache,
meaning have it already been downloaded.
And by now we don't have any models downloaded,
so there won't be any, and we'll go in and download.
This will take a little while, not too much, it's not a big model.
This one has a progress download, which is broken at the moment,
so don't use it.
It will download, but you can't really show the process, it has a bug.
So it's a rough 1.0 in my opinion.
But we now have our model downloaded, so we can see in here
that we got a 511 megabyte big model here.
So behind the scenes it's our next models.
So let's load the model into memory.
And again, on the bigger models this will take longer,
but here it was only 500 megabytes we needed to load in,
and it's CPU bound, so it's taking up my memory and my CPU.
And then we're going to start a small web server that we set up here,
so we have a local URL to do this.
So now we have the Outry part up and running,
it now runs a mini web server that we can connect to these models,
similar to when you do OLAMA and just start the OLAMA service.
And you can, of course, wrap this into any kind of service you want
in order to run these models.
In this case we are just running it in a console app
where we first both run the server and consumes it,
but in real life you will, of course, have two different processes.
Then we go in and make, based on our service URL, which we have here,
so we can see it gave us on the fly a port.
Up here we didn't specify it, but it will on the fly serve a port.
And then we're going to make a normal OpenAI client,
but instead of an API key, because it's local, we don't need one,
so we can just write anything here. In my case I wrote no API key.
And then we need to pass the endpoint in.
And once we do this, we have everything like normal.
We can do a chat client and we can get an AI agent like normal.
And then we can begin to ask our questions.
So first question is, what's the capital of Sweden?
And it will come back using normal agent framework.
And come back and hopefully say something here.
Oh, so it's also showing it's thinking.
Apparently that's the way that model works.
Strange, but different models, different way of doing things.
But it come back at least and say the capital of Sweden is Stockholm. That's okay.
And then we can do the same in streaming.
I will just set a breakpoint down here and let it stream on how to make soup.
First it will give us back the thinking part.
Again, it's not all models that just does this.
So we don't need to take it away in real life.
So in this case it was how to make tomato soup and it will show us that.
And then we can stop the web service
and we can unload the model from memory so we can free up our memory.
And then we're done.
So this is completely different from what you have seen previously from my videos
because, again, it's complete rewrite.
It works.
So if you're into offline models, this is probably a good way to go
or go with Olama.
They work more or less the same way
in that they make a OpenAI client with a special URL.
But that's everything there is to it.
It's up to you if you feel like this is a good way of doing things.
If you want to go Olama or, like me, want to use the paid models up in the cloud.
But see you in the next one.