Now that we have our understanding of vectors
and what we are going to put into
a vector store, let's build the vector store
and ingest some data.
And in order to do that, I've made
up a few dummy samples of some kind
of internal handbook or knowledge base at a
company.
And one of the questions is our previous
question about what is the office password.
But there's other things like is Christmas Eve
a holiday, how to register for vacation and
such a thing.
Doesn't really matter.
And this is just 10.
That would be several hundred in a normal
company probably.
But we have this data, and we're going
to make a normal connection as before.
And we're going to make the vector generator.
And then we're going to choose our vector
store.
So there's various options, but I have chosen
in this case a SQLite because it is
easy to work with, although a bit limited.
There is certain that have just taken on
the job and say, hey, I also want
to be a vector store.
SQLite was officially just a SQL.
While others have been dedicated from day one
to being a vector store.
So there's different options.
And up here, we can see Azure AI
Search, Cosmos DB, Elasticsearch in
memory, and all kinds of different vector stores.
I will show a few examples down here
of the ones I have tried.
But we need a new NuGet package in
order to do this.
So I have put in one more NuGet
package.
That NuGet package is Microsoft Semantic Kernel Connectors
SQLite Vector Store.
So this is part of something called Microsoft
Extensions.VectorData. It just has a little funny
name because this is actually the old version
of Agent Framework, but it's only by name.
It has nothing to do with the old
stuff.
But this is our implementation, and they have
other connectors to the other vector stores.
So what we're going to do is we're
going to make a connection string to a
SQLite.
So that's just writing data source and then
a path to some place.
In my case, I'm putting it down in
my temp folder.
And then I'm going to new up a
vector store.
In this case, it's Microsoft Semantic Kernel Connectors
Vector Store SQLite.
And I'm putting in the connection string, and
then I'm making a vector store options object,
which I give the generator.
So we're hooking this up to this because
then it can generate on its own.
We could technically sit down and take any
kind of thing that can support vectors and
begin to generate the vectors ourselves and put
them in using the SQL language or the
language of the vector store.
But this is just a convenient way to
not do that.
So we have this vector store now.
We could have made others.
So, for example, if we wanted to have
Azure AI Search, which is a powerful but
quite expensive version of a vector store.
And in that, we would just need to
give our endpoint and our key in order
to do this.
If we wanted to use SQL Server, it
works from SQL Server 2025 and forward.
We would just need a connection string.
And if we wanted to use Cosmos DB,
who can also do it, we would need
a connection string, the database name, and tell
a little about serialisation.
So all of them is just about some
connection string to get something up and running,
but they all turn out to be a
vector store.
So we are having a general system.
And then we are going to make what
is called a collection.
So we're getting a collection.
We are telling that the primary key is
GWT.
And our data is equal to this object
we have here.
I didn't mention this one up here was
just a question and an answer.
But this down here is a bit more
in that we are going to tell that
we have an object that should represent a
record in our vector store.
In a SQL, it's a table.
But in Azure AI Search, it's an index.
So it's called collection just to be more
generic.
But in a SQL Server, it's just a
table.
And then we're going to tell we have
an ID, which is the key of this.
And then we can put in one or
more pieces of data, because we haven't really
talked about this yet.
But whenever we do a vector similarity search,
once we have found the right search, we
actually need the real data to give back
to the LLM.
It doesn't really want to have the vector.
It just want to have the data that
was similar to it in its original form.
So we're saving the question and the answer.
And we are putting in vector store data
here.
And if you have a lot of data,
you can put in to make it indexes
and give it new names if need be.
The final one we need to have is
a vector store vector.
So the vector we have, how many dimensions
it is, and what should represent the vector.
In our case, we want to have that
the question and the answer is the vector.
So we get it in Q, question dash
A, answer.
That would be what we want to embed.
So when we do that, we get this
back, and we can give it a name.
So in our case, since it's a SQL,
it will be the table name.
In an Azure AI search, it would be
the index name.
So once we have one of these, we
can call a method called ensure collection exists,
meaning a creation of it.
So in, again, a SQL, it will be
create table.
And in an Azure AI search, it will
be create index.
So now we have it, our table, and
we can ask if we want to import
data.
So I just put it in so we
don't need to import every single time.
But if we run it the first time
here, we can see it comes and asks
us, do we want to import data?
And just so it's easy here, normally you
would do a little more data manipulation.
I will talk a little about that in
one second.
But in this case, just to ensure that
we get rid of all the old data,
I just say we actually delete the table
again and recreate it.
Then we are sure that it's empty.
We can actually go in and work with
the data, delete certain things.
But my main problem, if I just added
new data all the time, is that right
now my ID is a new grid every
single time.
Had it been something that were fixed for
my records, I could just run this again
and again and absurd it, meaning if it
already exists, it would update it instead.
So I'm going to delete and create.
And then we are going to go to
our 10 knowledge bases.
And we are just going to say absurd
async with this new record that we want.
So we just give it the ID, the
question, and the answer.
And behind the scenes, absurd will go in
and do our embedding, like we saw in
the previous lecture.
We can actually send the embeddings in to
absurd if we want to.
But this is just an easier way.
So it will take a little time, because
every time we run this, we go up
into Azure and do our embedding.
But if we run it, we now have
our 10 embeddings in.
And what I do down here, let's try
and rerun it so we can see what
this is about.
Let's just say no.
We can also list our data.
So there is a get async.
It's not like we can write SQL against
it, because again, this is a generic setup
against indexes, SQLs, all kinds of ways of
implementing a vector store.
So we need to do a get async
where we do a small func record.
And in our case, I'm just saying give
me all records that doesn't have the empty
GWT, meaning all of them.
And here I'm setting how many records do
I want back.
I want as many as possible.
And doing that, I will get my first
record back.
So the first record here is our question
about Wi-Fi.
Print it out.
Next.
So we're just doing a for each on
all the data.
Behind the scenes, this implementation will use a
SQL in order to get this information, while
an Azure AI search will use their API
to get the data instead.
So we just get our 10 questions out
here just for us to see that they
are in there.
We could also go to the SQLite database,
but it has a slight odd way of
storing the vectors, which again is abstracted away
from us.
In SQL, it's just a column, but in
SQLite, it's some more extra tables that have
the data for some reason that they have
chosen.
It doesn't really matter to us.
We now have our vector store.
So with our data put in, so now
we should be able to actually finish this
and begin to search against it as well.