Hi, and welcome to this ANC SHARP video on the Microsoft 18 Framework.
In this video, I'm going to show you a neat trick about filtering data
without passing the data to an LLM.
And the reason why you might not want to pass the data
is perhaps it's a big amount of data, so it will cost a lot of time and tokens,
or it could be sensitive data, so you don't want it to be up in the cloud.
So there's ways around this, but first, let's see what the problem is
in terms of speed and token count.
So in here, I have a sample, and I have, if we look into the samples,
it's the one on other topics, AI as data filter.
So what I have is a bunch of books, namely 100 of them,
with just being some JSON, and we can see that each book
have a title, a year of release, an author, a genre, and a synopsis.
So I turn these books into data code here,
and I put in a string on these so I get it out as some XML
because AI are better at XML than the JSON.
Then I make a client, and in this case,
I'm using chat-tbt-41-mini as the first model.
And I'm just telling it's a librarian,
and then I'm giving all the data in the prompt,
the prompt meaning very, very long string of text.
And the reason I do this is what we're about to do
is not really something RAC can do
because we are going to ask some questions,
like, for example, how many of these books were released in 1980?
How many have an author that ends with the letter S
in their name, and so on and so forth.
So things a user could ask,
but we can't really use anything in terms of RAC
because that would be a similarity search
that has nothing to do with these kinds of filters
or what you can call it.
So I'm going to ask three questions.
So how many of them were released in 1980?
Then give how many and list the books
where the authors have an S as the last letter in their name.
And finally, a bit more advanced,
where have the last letter of S in their name.
This genre is not fantasy,
and was released between 1800 and 1900.
So progressively more advanced questions to learn.
So let's first see how ChattyBT 4.1 Mini handles question 1.
So we ask the question by giving the book data and the question.
And we can see it came back fairly quickly, 1.6 seconds,
and it says that these two books are the right ones.
It used 4,185 input tokens
because we needed to give all the books,
and output tokens were very low
because it just needed to write this simple text out.
And the real answer is actually 2.
But if we try again.
In this case, it's still clear.
I've seen it give back 3
because it thought there was three books.
That's not the most important thing here.
So let's go to some more advanced things
where the end of the author's name should be S.
This will begin to take a little longer.
But it says it has 27 books, the author ending.
And we can already see that's wrong
because Lewis Carroll is not ending in S.
Mark Twain is not ending in S.
It does not even have an S in it.
So it's definitely wrong with this,
and the real answer is actually 15
because I'm just using a little link here to check.
So it definitely got it wrong,
and despite that, it still used a lot of our tokens.
And it goes even further wrong
if we give it a much more advanced question here,
like the author's name should end in S,
it should not be a genre of fantasy,
and it's released between 1800 and 1900.
In this case, it says it's 14 books.
Again, it goes wrong.
David Copperfield is...
Let's re-find it.
Dostoevsky is not ending in S, for example.
And the real answer is actually 7.
So sometimes a model like this gets it right.
Sometimes it doesn't.
So let's move this all the way back up here
and try with another model.
So now we are going in and hot reloading,
so we use Chatterjeeb T5 Mini instead
in order to do the same.
I'm going to get our three questions,
and we get the first answer back.
Now it takes longer
because now we are having a reasoning model,
and it will actually reason over what this is.
And it comes back again with the same two books, which is good,
where the other one, the ones you saw here, it could handle it,
but in some cases it can't.
A model like Chatterjeeb T5 Mini is pretty good at this, so it's okay,
and if we went to one of the even bigger models,
it would be even better.
But we are still using a lot of tokens,
and now we are even using more in output due to reasoning.
If we go to question two,
it now becomes much, much harder for it
because now it needs to really reason over if this is right.
So I've seen these take up to half a minute to run back,
despite it being a fairly simple link query,
if you knew what you were doing, of course.
But it will think about it quite a while,
and hopefully come back in one second.
Yeah, it's a difficult question among 100 books,
but now it comes back,
and it comes back with the answer of 15,
which is actually the right answer.
So reasoning does help with things like this,
and it's quite good at it.
But we again use a lot of tokens, and now you can see
we're really using a lot of reasoning tokens.
But the real answer is 15. We got it right.
But now let's go and do the advanced question
with both the letter S, not fantasy,
and release between 1800 and 1900.
I will just pause the video here when it's back
because this will take a little long time.
And we are back.
You see it fought for 32 seconds in order to do this.
It got the answer right, which is 7, which we'll see in one second.
But it used a lot of tokens again in order to do this.
But at least it gets it right.
But what I want to show you now is that there is a better way.
And we can even go up here.
Switch back to the simple model
because even the simple model can do it
in the other way I want to show you.
So we're going to load up the model,
and we're going to load up the three questions.
But then I'm going to jump down to some new code down here.
And what we're going to do here is
instead of actually giving the model all the books,
meaning the 100 books in XML format,
we're instead going to ask it to make a filter for those books.
And the book filter is just we want to have the fields,
the comparison operation, and the value back.
So we have among title, year of release,
author, genre, and synopsis.
We have equals, not equals, start with, end with,
contains greater than, greater than equal,
less than, less than equal, and regex.
And then a value, just represent a string.
But we could have value as integer, value as string.
But it's fairly OK to do it here.
So see what happens now when I just say, hey, LLM,
make a filter for the following query and then the question.
It is almost instance that it can make a filter.
And we get back year of release equals 1980.
Because it's an easy question now.
It doesn't have all the token count.
And this response, if we look at it,
it only used 154 tokens.
So very, very simple to do.
And then, of course, it's our job to filter the books now,
which I do in this, which is two.
And this is not fun code to write.
But I think it's worth getting rid of 30 seconds of wait time
and 7,000 tokens by sitting down and writing this code.
It can be more advanced if you have more advanced samples, of course.
But in this case, I'm just going through the filters
and slowly applying the filter based on what it gave back.
I'll let it run to here.
We can ask the second question.
And again, it can easily make that filter
because it's really, really good at understanding
what we are talking about.
Because it doesn't have all the context to work with.
So order ends with s. Simple to do.
And again, we get the books back.
And we have the 15.
And the final one, which is a much more advanced question,
it actually goes fairly quick as well
and turn it into a filter of four different things.
Order ends with s.
Genre not equal fantasy.
Year for lease greater than or equal to 18.
And year for lease less than or equal 1900.
And again, we get the books.
And it has the seven books.
So this is a much more advanced way.
It saves a lot of time.
And it gives the right books back.
It can tend to end up being a little more wooden answer
because now you just have the books
and you need to show the answer.
So it doesn't show the normal.
I looked at it and it's this and this number of books and so on.
But often you want to have it in a structured format anyway.
So giving it the books like we did up here
is just waste of time, in my opinion, in certain scenarios.
And letting it build filters like these down here,
which is fairly simple to do.
Then we, of course, have the annoying part of taking the filter
and turning it and applying it.
But it's fairly simple code.
It took me like five, ten minutes to write this code.
So I definitely think it's worth doing filters
instead of the system.
And again, if the data had been very sensitive,
we could still have used an LLM to filter it,
which is really, really cool.
So that is how we can do it.
And thank you for seeing this video.
See you in the next one.