WEBVTT

00:00.080 --> 00:01.560
Hey there, Eden here.

00:01.560 --> 00:08.760
And I think now is an excellent time to pause and to reiterate what we did so far and what we have left

00:08.760 --> 00:12.880
to do in the context of our rag ingestion pipeline.

00:13.640 --> 00:16.680
And luckily for us, we are done with the hard part.

00:16.720 --> 00:20.680
And by the way, this is very typical for production grade Rag applications.

00:20.720 --> 00:26.400
A lot of times the hard part is just to get the data to our system.

00:28.040 --> 00:28.440
All right.

00:28.440 --> 00:30.120
So just to elaborate on that.

00:30.400 --> 00:38.440
So far we have taken the source which is the documentation, and we have loaded it into long chain documents.

00:38.640 --> 00:41.320
So this entire process uses external APIs.

00:41.320 --> 00:43.320
It uses to really map to really extract.

00:43.360 --> 00:44.960
We ran everything concurrently.

00:44.960 --> 00:49.120
And there is a lot of depth and a lot of things to optimize over here.

00:49.640 --> 00:55.640
So right now we want to go and prepare everything to be indexed into our vector store.

00:55.880 --> 01:01.280
So this will include two chunk ify and to transform our original documents.

01:01.440 --> 01:04.920
And we're going to split them into smaller chunks.

01:05.160 --> 01:06.560
And we'll soon discuss why.