Hi, and welcome to another video in AI
in C#, where we look into the
Microsoft Agent Framework.
In this video, we're going to look at
how we can work with input data other
than text.
So here we are in Visual Studio, and
we have seen a lot of examples so
far where we have just given some chat
message with some text, or just given the
text directly here, same output.
But what if we want to use other
things?
These LLMs, or most of them, are multimodal,
where they talk about they are able to
work with images and PDFs and stuff.
In the example, I have put in some
scenarios for text, image, and PDF, because that
is more or less what they can work
with.
Because if you try to upload a .txt
file or some kind of file that you
have, they just come back and say they
are not able to work with them, they
are not able to process them.
And of course, if it's a file or
some kind of content that it doesn't understand,
you can, of course, use normal RAG or
Retrival Augmented Generation, and put it, extract data
out of a file, let's say you had
some kind of technical specification in a format
that is not common.
You could, of course, take that data out
and then just give it to the LLM
as text.
But there's this few scenarios, images and PDFs,
that are supported.
In one of the cases, PDF is only
partly supported, but let's see how that works.
And in order to run this sample, you
need both Azure AI and OpenAI, because OpenAI
actually have some features that they support that
Azure OpenAI, for some reason, don't support.
But let's have some breakpoints and let's start
with the normal text, which is nothing special
in the scenario.
And we just used Azure OpenAI here, but
both Azure and OpenAI, of course, can understand
text.
So what is the capital of France and
a response back, that just says the capital
of France is Paris.
So nothing special there.
Images are also supported.
So let's run and see that whenever we
use an image, we can either use an
image that are online, meaning a URL to
an image.
So you can simply go in and say
the chat message, instead of just writing the
text, you can make a list of different
things you want to send in.
In our case here, we want to send
in a text content and a URI content,
because if we just send the image, it
might not know what to do about it.
But in this case, we want to ask
what is in the image and the image
is a picture of the settlers of Catan.
I can show it here, so we know
what's going on.
Just bring it up on another screen and
show that it's this image.
So we're asking, what is in this image?
And if we do that, take a little
longer, of course, and let me jump over
here.
So it can look at the image and
tell what's in it.
And we could have asked what colours are
used in the image, what numbers are used,
and so on.
So that's just image recognition and working with
them.
And it doesn't require any special model other
than a multi-model.
And we can begin to see our input
tokens are becoming bigger, because it needs to
get the entire image and stuff.
If we have local files, I have the
same image, just as an image.jpg. And
I put in that it's only called image
.jpg. So it's not like it's looking at
the URL and figuring out it's settlers of
Catan.
It's actually looking at the image.
And images can be processed in two ways,
either as Base64 with a data UI.
And then we will just ask what is
the content of the image.
So we will see exactly the same response
back, more or less, and taking the same
amount of input tokens.
And in the same manner, we can use
the read-only memory, which is the three
options we can send data to a system.
So in these cases, it's data content instead
of UI content.
And we can see we get exactly the
same message back.
So nothing special there.
And these work both for Azure AI and
Azure OpenAI.
But then we come to PDFs.
And let's run again.
And in this case, I have taken a
PDF that is the rules of settlers of
Catan, just to keep the same theme here.
And what we will see is that this
is using the OpenAI agent, because if I
had used the Azure OpenAI agent, it would
have come back and said, hey, I don't
support files.
So Azure OpenAI, for some reason, can only
do this with images and not PDFs.
And it's not something that is anything about
the agent framework.
It is the pure LLM behind the scenes
that are the problem here, not the framework,
because that doesn't care.
But if we have the rules here, because
we can only work with local data and
local inquisition, because if you have some UI,
you, of course, can download it, have it
in memory and give it to the system.
But we can't just point to a PDF
out there in the world and say, what
is in this PDF?
But if we get it as a base
64, just like before, we can ask, what
is the winning conditions in Attach PDF?
And this takes quite a while and takes
like 15,000 tokens, because it's a long
rule book with images and stuff.
But had it been a simpler PDF with
just text, it would probably go faster.
So I get a response back, and it
will tell that we need to reach 10
victory points, and it will even point out
that is specified on page 5 and page
14 and tell a little about the rules.
So that is fairly cool, but we can
see other token costs, of course.
And data in memory works exactly the same
way.
And again, it's only possible using OpenAI agent.
So it is thinking for the next one,
and then we are done.
But we'll get the same answer back, of
course.
But actually making a small failure here that
says it's on page 7, it's about it
and not page 14.
So that's AI for you.
So again, other files, I have tried to
put in .txt and so on, and it
doesn't seem to really be able to understand
that.
It just says, I can't process external files
like that.
And that is the case where you need
to use your C-sharp skills to extract
data from the various files and give it
as text for the system or convert something
to PDFs and give it that way.
So if we look at this, you can
see we can give this list of contents.
And you can see there's a bunch of
other contents here, but these are both input
and output content.
And we are not really going to send
at any point an error content.
That is for the LLM to send back
the MCP tool call results and things like
that.
Hosted files, if it's up in Azure AI
Foundry, it will give it back as a
hosted file.
So don't give the file back, but it
gives a reference to the file that you
can then download by yourself to save tokens.
So that is actually everything there is to
this.
Of course, if you work with images and
PDFs, it's very simple, but the rest is
extraction and giving as text instead.
So see you in the next one.