Eden: Hey there, Eden here,
and in this part, we are going to implement LangChain code
to ingest our medium blog into our vector store.
It's going to include taking our data
and loading it into a LangChain document object,
split this object
with a LangChain text splitter into smaller chunks,
going to embed those chunks and turn them into vectors,
and then we're going to store those vectors
in the Pinecone vector store.
And now this may sound a lot,
but actually in LangChain it would only take us
a couple of lines of code.
And that's what's cool about LangChain.
It saves us a lot of boilerplate code.
This video will include the imports
and the LangChain objects that we'll be working with.
And the following video will include
the ingestion implementation.
And let's start by importing from LangChain,
the text loader.
So when we talk to LLMs, we send them data
and that data is text.
So the text can be something that we write to the prompt,
but we can attach to it some other things like
what if we want to process something
on our WhatsApp messages
or if we want to download some PDF files
and process them,
or if we want to do something with a Notion notebook.
So all of those elements I described earlier
are simply text, but they have different formats
and different semantic meaning.
So the document loaders are actually classes implementations
of how to load and process data
in order to make it digestible by the large language model.
So to be honest, it's pretty easy to understand.
So I don't mind getting my hands dirty
and going inside this repo and reviewing the source code.
So this is the LangChain Python implementation.
It's a GitHub repository, which is public,
so you can easily find it online.
I also list it in the course's resource.
So if I'll head up and I'll go to the LangChain directory.
And right over there there is this directory
which is called document_loaders.
So inside it is the entire implementation
of all the document loaders.
So for example, here is the WhatsApp document loader
and if we'll go up there is the text document loader
under the text.py file.
So you can see this is the entire implementation
of the text document loader, which we're going to use.
So all we're doing right now is simply taking a file path,
opening it up like we would usually open a file in Python,
attaching some metadata that is going to hold the source,
which is going to be equal to the file path of the file.
And that's pretty much it.
We wrap it inside a list and return it.
So nothing novel here, nothing special.
Simply an easy wrapper that helps us abstract things
and you'll see it later because we will have the same
interface for all documents.
So let's check out what's happening
with the WhatsApp document loader.
Let's see its implementation.
So I'm going to go back
and I'll go head up to the downside of this document.
And yeah, this is the WhatsApp chat loader.
Now here we're also loading a file, which is the text file
that represent our WhatsApp chat.
And all we are doing right now is simply opening up the file
like we did with the regular text loader,
then using some regular expressions to extract the names
of the sender, of the receiver, the text,
and the date that it was sent.
And simply concatenate one row
after another who sent which row.
So it's simply taking the file, loading it,
doing its formatting and returning it as is.
So basically we're making the data
more ingestible by the LLM.
So it's super, super easy to use
and there are tons of implementations of different file
and document loaders that we can use to our disposal.
So that's what makes LangChain so great
that it has all this variety of options that we can choose
to handle our documents.
We can handle documents of any kind, of WhatsApp,
of Google Drive, of Notion, of PDF, whatever we want,
LangChain can help the LLM digest it
in the best possible way.
Cool, so let's go back to the code
and now let's import the character text splitter.
Now text splitters are here to help us
with long pieces of text.
Those texts have tons of tokens inside them
and if we'll send them directly to the LLM,
then our request will probably fail
because it surpassed the token limitation
the model enforces.
So for example, in the GPT-3.5,
we have 4K tokens limitation.
So to conclude, the text splitter
allows us to take text which is large
and to split it into chunk.
Now to be honest, text splitters
have a lot of logic in there
because there are a lot of splitting strategies
and there are a lot of smart ways to do it
with calculating the appropriate chunk size.
Now the chunk size is not trivial
because it may change according to
what we want to accomplish.
Now this is different to the LLM we're choosing
and to the different embedding system,
but we'll cover it later in this course.
In the meantime, let's take a look at the documentation
of the character text splitter.
Now notice that we give it the separator
that we want to separate and we supply the chunk size.
Now here it's the size of 1000 tokens
and the chunk overlap.
The chunk overlap parameter specifies the amount of overlap
between the chunks when we split the text
into smaller parts.
This overlap can be super helpful
to ensure that the text isn't split up in a way
that disturbs the context or the meaning.
The length function is usually len
and it helps LangChain to determine the chunk size.
Now sometimes you want to do it with tokens
and there are special functions we can write
that help us find out how many tokens our chunk having.
Alright, let's now import our embeddings object
and we'll use OpenAI embeddings.
Now I remind you that an embedding model or encoder
or whatever you want to call it, is simply a black box,
which takes in inputs as text
and outputs vectors in an embedding's vector space.
So I have a text that I want to embed.
How do I actually do it?
Well, a lot of embedding providers simply exposed to us
an API slash embed that we can use
where we send the text and get back the vector.
There are a lot of models that aspire to create
a good and smart embedding, the one of OpenAI,
the last of them text-embedding-ada-002,
is a very good one
because in embedding
there is a great significance for the price.
Sometimes you embed an entire database
and if a certain model is cheaper
then it's much better to use.
So de facto, the text-embedding-ada-002
is cheaper by 98% than their last one.
So if we return back to the import of OpenAI embedding,
then the whole idea
of text embedding models in LangChain
is to create a uniform interface for us
to access different embeddings
from different embeddings providers.
So it doesn't matter if we're using Cohere, Hugging Face,
OpenAI, it doesn't really matter who are we using.
We have the same interface.
So switching between a one to another
is pretty straightforward with just changing a parameter.
So that's the entire abstraction of the embeddings.
And here we are going to use the OpenAI embeddings.
Okay, we're getting closer to the end of the video.
That's good because it's a long video.
So now we're importing Pinecone.
Now we talked about embeddings that we create from the text,
the vectors.
We need to store those vectors somewhere.
We need persistent storage.
We want the ability to search in the vector space
for closest vectors of the current one.
We also want to be able to add new vectors
to the vector space.
So all of this is being handled by the vector databases
that does all the work for us.
Now, an excellent vector database
that has recently gone viral is called Pinecone
and we'll be exploring and using Pinecone in these videos.
It has a free tier,
so don't worry about spending a lot of money.
And that's all for our imports.
And in the next video,
we will be implementing the ingestion.