Welcome to the RAG section of this course.
So, what is RAG and why do we
need it?
RAG is kind of difficult to understand, so
I'm going to spend some quieter time here
in PowerPoint to explain to you what it
is, before we go into the components and
actually some code.
So, what's RAG?
Well, RAG stands for Retrieval Augmented Generation, but
that doesn't really make us any wiser, so
let's see an example.
So, if we ask an LLM what is
the capital of France, it can quite easily
answer back that the capital of France is
Paris.
So, this is an easy question, and we
can give it very, very advanced questions like
what is the Einstein-DeHaas effect?
I just randomly found that, and it's apparently
something about spinning objects.
Not important, but this feels like, oh, I'm
really talking to an intelligent being here.
But then I can ask it, what is
the guest Wi-Fi at the office?
And it doesn't know.
And of course it doesn't know, because in
the world of knowledge there's different sections of
that.
There's the Internet, of course, and LLMs know
the Internet.
It's distilled, meaning that all side malware, the
dark web and such things has been taken
away, but else it knows kind of everything
on the Internet.
All depending on if you use a big
model, it knows the entire thing in details.
If you use smaller models, it's only the
highlights.
But there's a bunch of other knowledge in
the world.
The Internet is big, but all your personal
data, your emails, your bank data, your documents
and so on, despite them might being in
a OneDrive or a Google Cloud or in
Gmail or something, it's still not data that
are publicly available.
So an AI, of course, don't know.
And you probably can understand that you didn't
want to know that already, because that could
be medical information, that could be sensitive information
in there.
So, of course, an LLM out of the
box don't know.
In a similar manner, it doesn't know your
company data, your documents, your contract with the
clients and so on.
Of course it shouldn't know.
It also, of course, don't know other people's
data.
You even don't know and can't get access
to other companies, other people's private data.
So how should an LLM know?
And then, of course, there's a bunch of
offline data.
Despite me wanting to know something, it might
be that it's only in a book in
a different country that have that specific knowledge
somewhere, because it hasn't been scanned yet, digitised
in any way.
So there's a lot of data in the
world, and an LLM has lots of it,
but doesn't have all of it, of course.
So RAG is the process of giving some
of this extra data to the LLM so
that it doesn't know on the fly.
So let's say that this box represented everything
in the world I knew.
I knew that my middle name was Rasmus,
I knew my favourite colour was blue, I
knew my parents had a dog called Mr.
Buttons, I knew the password to the guest
Wi-Fi at the office is 42, I
know I live in Denmark, I know John
is in charge of support, and I know
my birthday is February 25th.
If I knew all that, I could, in
a prompt, or in what we know now
as instructions, give everything I know, and then
ask what is the guest Wi-Fi password
at the office.
And then the AI would know, because it
could shift to all of it and find
that this one specific one is the part
we need.
But this box, of course, is not everything
I know.
Everything I know is a lot of things,
if we begin to count it up.
So how could I give it everything I
know?
Well, some of it would be fairly easy
to give.
It's just some raw text I have in
various online documents and stuff.
But there's also a lot of messy stuff
in binary formats.
For example, if I have a thousand images
and they show different things on the places
I have visited, I wouldn't be able to
ask my AI which countries have I visited,
if it's only in images.
So I would need to give something to
an AI and actually extract all that data
from the images.
So some of them are difficult to get
to.
And there's also things that I only have
in my mind that are never written down.
I don't think I have any single document
here that shows what my favourite colour is.
That's only in my mind.
And that is not something the AI would
be able to be given.
And there's a bunch more.
And if we begin to tally up everything
I know in terms of personal data, my
company data, and so on, it would be
so much data that an AI would be
information overloaded.
Because what we're talking about here, if I
gave it up front, we saw it before,
this was fairly easy to give up front.
That would be like a hundred tokens or
something like that.
If we go here, there is actually a
max to how much we can give.
Just like if you place some text into
a field, there's often a max limit on
how big that field can be.
When it comes to the question we can
give, the context window, as it's called, so
this is the words that are being used.
So if you hear the word context window,
it's actually how much can you put into
a prompt.
And it varies from 128,000 for some
of the big models to the really, really
capable models is up to a million tokens.
And turn that into pages of text, it's
roughly between 500 and 2,000 pages of
text.
So Google is the one that are highest
up right now, while ChatGPT is a bit
further down, for example.
But giving all this data comes as a
cost.
It comes as three different costs.
First off, giving a million tokens, we have
seen the prices.
So if we need a big model and
that, it costs, just to make that one
prompt, one or two US dollars.
It also comes at the cost of speed.
So if we give them all these million
tokens and send this prompt, it will take
quite a while to get back to us.
And if we give it everything among all
these, let's say there was 10,000, a
million different entries in this context, one of
them would say what the guest Wi-Fi
password is.
That's suddenly a needle in a haystack.
So it might even not find it despite
it being given because it has so much
other data.
But again, even if I filled up all
this, that would not be nearly what I
know in my mind.
And some of it will still be difficult
to get because it's not even possible.
It can't read faults.
So we need some kind of cert index
for all this, everything I know.
And of course, I need to curate what
do I want to put into an AI.
I might not want to put my medical
information, but I might want to put my
financial information or the reverse.
But we need some kind of cert index
for all this, everything I know.
And I need to put it into something
a little like Google.
So when you go to Google and you
search, it kind of knows the entire Internet
and can find the right pages, but it's
just the index of it.
It's not like it has all the information
about every site, but it has enough information
about the site in order to give it
back to you as a search result.
So we need to take our data and
put it into a cert index.
And then we need to ask the question
again, but instead of giving everything I know,
meaning all this data, I need to make
a search result on the fly.
So I need to search for Wi-Fi
password office, for example, or just Wi-Fi
password or however I find it.
It's a little like going to Google and
putting in the right words to give the
best search result.
And if I do that, that search result
will give back let's say among all this
over here, three things back, that the Wi
-Fi guest password is 42, that my home
password is Maverick123, and my last Wi-Fi
bill, whatever that was in terms of money.
Because just searching for Wi-Fi password office
could give the right things back, but also
other things back.
It depends on how many search results I
want to have back.
In this case, if this was the most
relevant one, and I said only one, it
would have gone okay, but if I asked
for the home Wi-Fi and said only
one, I would only have gotten this, and
it wouldn't have gotten number two and number
three here.
So in this case, I'm asking for three
things, and these are the three most relevant
just like when you make a Google search
for the most relevant.
And then the AI would be able to
answer my question.
It would feel intelligent again, because despite it
not having everything, it on the fly got
the most relevant things, and among that, we
were lucky enough that the answer was actually
there and was able to ask, give it
back to me.
This is actually Rack, and what we are
doing here is what is called embedding and
ingestion of data into a vector store, which
is where we save Rack data, and whenever
we make this request to the search index
or the vector store, that's a Rack search
where we at the top of our search
query, augment the data with Rack data so
the AI can answer.
So this is actually Rack, what we have
just seen.
Now let's go into what the different components
are so we can go in and see
some code soon.