WEBVTT 1 00:00:00.180 --> 00:00:01.320 Hey there, Eden here, 2 00:00:01.320 --> 00:00:05.430 and in this video we are going to do some boilerplate setup 3 00:00:05.430 --> 00:00:08.640 and we're going to set up the environment for this project 4 00:00:08.640 --> 00:00:12.090 where we are going to be showing an entire RAG pipeline 5 00:00:12.090 --> 00:00:14.790 from ingestion to retrieval. 6 00:00:14.790 --> 00:00:18.090 So let's go and let's set up our environment. 7 00:00:18.090 --> 00:00:21.090 So here I'm in the LangChain repository 8 00:00:21.090 --> 00:00:23.070 and the branch we're going to be working on, 9 00:00:23.070 --> 00:00:26.940 on this section is called project/rag-gist. 10 00:00:26.940 --> 00:00:29.010 This is here the project 11 00:00:29.010 --> 00:00:31.230 and hopefully I'm going to be adding 12 00:00:31.230 --> 00:00:34.350 a much nicer readme file that is going to explain 13 00:00:34.350 --> 00:00:35.580 what's happening here. 14 00:00:35.580 --> 00:00:37.680 Anyways, here we can see this entire code, 15 00:00:37.680 --> 00:00:39.600 which we're going to be implementing. 16 00:00:39.600 --> 00:00:42.990 And what's interesting is to go here to this commit list. 17 00:00:42.990 --> 00:00:44.820 And here we're going to see every commit 18 00:00:44.820 --> 00:00:47.160 for every lesson that we're going to be having now. 19 00:00:47.160 --> 00:00:50.670 Now I want to go now to the first commit, 20 00:00:50.670 --> 00:00:52.230 which you can see right over here 21 00:00:52.230 --> 00:00:54.720 where I added a bunch of files 22 00:00:54.720 --> 00:00:57.300 and this is going to be our starting point. 23 00:00:57.300 --> 00:01:00.120 So we want to go and clone the repository 24 00:01:00.120 --> 00:01:03.540 and switch to this branch of RAG gist 25 00:01:03.540 --> 00:01:05.940 and specifically to go to this commit. 26 00:01:05.940 --> 00:01:09.180 So let's go and let me open up a terminal 27 00:01:09.180 --> 00:01:12.150 and let me go and cd into desktop. 28 00:01:12.150 --> 00:01:15.390 So now let's go to the repository and let's go 29 00:01:15.390 --> 00:01:18.420 and copy here the URL of the repository 30 00:01:18.420 --> 00:01:21.210 and let's go and write git clone 31 00:01:21.210 --> 00:01:23.460 and then paste here, this repository. 32 00:01:23.460 --> 00:01:25.950 So now we can see, we cloned the repository. 33 00:01:25.950 --> 00:01:30.360 Let's cd into langchain- course. This is the repo. 34 00:01:30.360 --> 00:01:33.330 And here now I want, let's clear it out. 35 00:01:33.330 --> 00:01:36.840 So now let's go and switch to the branch that we want 36 00:01:36.840 --> 00:01:40.170 in the exact commit of the initialization. 37 00:01:40.170 --> 00:01:43.710 So let me write git checkout -b. 38 00:01:43.710 --> 00:01:48.450 That branch is going to be project/rag-gist. 39 00:01:48.450 --> 00:01:50.280 Now here we want to paste in 40 00:01:50.280 --> 00:01:52.890 the hash of the commit we want to go to. 41 00:01:52.890 --> 00:01:57.780 So let's go now to the branch rag-gist to the commit list 42 00:01:57.780 --> 00:02:00.540 and let's go and we want this initial commit. 43 00:02:00.540 --> 00:02:05.040 So let me go and copy this hash and let me paste it 44 00:02:05.040 --> 00:02:09.240 and we switch to a new branch project/rag-gist. 45 00:02:09.240 --> 00:02:11.790 So now let me fire up cursor 46 00:02:11.790 --> 00:02:16.790 and we can see cursor now is with the default configuration. 47 00:02:16.980 --> 00:02:18.990 So let me open the sidebar 48 00:02:18.990 --> 00:02:22.020 and here we can see we have all the files here. 49 00:02:22.020 --> 00:02:26.430 And if I'm going to go and click here this icon 50 00:02:26.430 --> 00:02:29.280 and we now have all the GIT information. 51 00:02:29.280 --> 00:02:32.640 And we can see now we are in the commit of initial commit. 52 00:02:32.640 --> 00:02:34.890 So this is exactly where we want to be here. 53 00:02:34.890 --> 00:02:36.870 So let's examine now the files 54 00:02:36.870 --> 00:02:38.700 and let's see what's happening here. 55 00:02:38.700 --> 00:02:40.350 So we have the git ignore file 56 00:02:40.350 --> 00:02:42.180 for files we do not want to commit 57 00:02:42.180 --> 00:02:44.760 and to track in our GitHub repository. 58 00:02:44.760 --> 00:02:48.480 Let's make sure that we have here .env. This is good. 59 00:02:48.480 --> 00:02:52.830 And here we can see we have the Python version I'm using. 60 00:02:52.830 --> 00:02:57.240 Here we have a boilerplate for the ingestion.py file. 61 00:02:57.240 --> 00:02:59.730 And notice here we have here an error. 62 00:02:59.730 --> 00:03:03.690 And this is because we haven't yet configured the cursor IDE 63 00:03:03.690 --> 00:03:05.220 to the virtual environment 64 00:03:05.220 --> 00:03:07.410 that is going to have all the packages installed. 65 00:03:07.410 --> 00:03:09.420 We're going to do it very, very soon. 66 00:03:09.420 --> 00:03:14.420 Here we have a medium blog that I put here as a TXT file 67 00:03:14.700 --> 00:03:17.520 and I simply went to Google, 68 00:03:17.520 --> 00:03:22.520 simply search for what is a vector db medium 69 00:03:22.770 --> 00:03:26.460 and went to this Medium blog here. 70 00:03:26.460 --> 00:03:27.780 I'm going to link everything 71 00:03:27.780 --> 00:03:29.400 in the videos resource of course, 72 00:03:29.400 --> 00:03:32.730 and I simply took all of the text here and pasted it. 73 00:03:32.730 --> 00:03:34.020 So this is what I did here. 74 00:03:34.020 --> 00:03:37.920 So this is an article regarding vector databases 75 00:03:37.920 --> 00:03:40.230 and now this is its text format. 76 00:03:40.230 --> 00:03:45.230 So let's go now and check out the pyproject.toml file. 77 00:03:45.690 --> 00:03:48.657 And here we have a list of all the dependencies 78 00:03:48.657 --> 00:03:50.010 for this project. 79 00:03:50.010 --> 00:03:52.020 So let's review them for a second. 80 00:03:52.020 --> 00:03:55.500 We have black, we have isort for formatting 81 00:03:55.500 --> 00:03:57.930 and we're going to install langchain. 82 00:03:57.930 --> 00:04:00.210 And don't worry about these versions. 83 00:04:00.210 --> 00:04:01.710 We are going to be installing 84 00:04:01.710 --> 00:04:03.960 a much later version of langchain, 85 00:04:03.960 --> 00:04:07.380 which currently is version 1.2 86 00:04:07.380 --> 00:04:09.690 at the point of filming the video. 87 00:04:09.690 --> 00:04:12.210 However, when you are going to be installing it, 88 00:04:12.210 --> 00:04:14.040 you're probably going to have a new version 89 00:04:14.040 --> 00:04:15.480 and I want to promise you 90 00:04:15.480 --> 00:04:17.670 we're going to be having up-to-date code 91 00:04:17.670 --> 00:04:20.610 and I'm going to refill the videos if needed. 92 00:04:20.610 --> 00:04:22.290 So I do this continuously. 93 00:04:22.290 --> 00:04:24.450 Actually this video is a refill. 94 00:04:24.450 --> 00:04:27.000 Anyways, we are going to need langchain community 95 00:04:27.000 --> 00:04:29.460 for community contributed code. 96 00:04:29.460 --> 00:04:31.050 And for here we're going to be using it 97 00:04:31.050 --> 00:04:34.170 for a certain document loader that we're going to be using. 98 00:04:34.170 --> 00:04:36.330 We want to use langchain-OpenAI. 99 00:04:36.330 --> 00:04:40.170 This is the vendor I'm using now for the embeddings model 100 00:04:40.170 --> 00:04:42.690 and for the LLM we're going to be writing. 101 00:04:42.690 --> 00:04:45.000 Of course you can choose any LLM you want. 102 00:04:45.000 --> 00:04:47.160 Here we are installing langchain-pinecone 103 00:04:47.160 --> 00:04:48.690 and this is the integration 104 00:04:48.690 --> 00:04:50.760 of the pinecone managed vector store, 105 00:04:50.760 --> 00:04:52.860 which we are going to be using in this section. 106 00:04:52.860 --> 00:04:55.530 And we have here langchain hub. 107 00:04:55.530 --> 00:04:59.280 Actually we don't really need it, so don't worry about it. 108 00:04:59.280 --> 00:05:01.620 And we have here python-dotenv. 109 00:05:01.620 --> 00:05:04.260 So this is going to help us load the environment variables. 110 00:05:04.260 --> 00:05:06.180 And here we have the uv.lock file 111 00:05:06.180 --> 00:05:08.583 with all of the exact langchain versions 112 00:05:08.583 --> 00:05:11.490 that I installed when recording it. 113 00:05:11.490 --> 00:05:15.390 Alright, so now let me go and actually delete this file 114 00:05:15.390 --> 00:05:17.670 because I want now to create it again 115 00:05:17.670 --> 00:05:21.390 to have the latest versions at least at the moment. 116 00:05:21.390 --> 00:05:24.390 And here it's December 15th, 2025. 117 00:05:24.390 --> 00:05:26.520 You're probably going to have new versions. 118 00:05:26.520 --> 00:05:31.170 So let me open the terminal and here let me write uv lock. 119 00:05:31.170 --> 00:05:33.090 And this is going to take the TOML file 120 00:05:33.090 --> 00:05:35.430 and it's going to create a new lock file 121 00:05:35.430 --> 00:05:39.120 with all the new packages and dependencies that we need. 122 00:05:39.120 --> 00:05:41.370 Let's go and look for langchain. 123 00:05:41.370 --> 00:05:44.030 And here we can see the langchain version is 1.2.0. 124 00:05:45.240 --> 00:05:47.550 And this is good and this is currently the latest versions. 125 00:05:47.550 --> 00:05:49.650 All right, so the uv.lock file 126 00:05:49.650 --> 00:05:52.230 has only the information about the package. 127 00:05:52.230 --> 00:05:55.110 We want to install them also in our virtual environment. 128 00:05:55.110 --> 00:05:57.930 So let me write uv sync. 129 00:05:57.930 --> 00:06:00.570 And this is going to be installing all the dependencies 130 00:06:00.570 --> 00:06:02.070 from the uv.lock file. 131 00:06:02.070 --> 00:06:03.180 And you can see right over here, 132 00:06:03.180 --> 00:06:07.980 it created a .venv directory, which is untracked by Git 133 00:06:07.980 --> 00:06:10.050 because of our gitignore file. 134 00:06:10.050 --> 00:06:12.090 And now we installed all the dependencies 135 00:06:12.090 --> 00:06:13.560 in a new virtual environment. 136 00:06:13.560 --> 00:06:16.170 So if I'm going to open a new instance 137 00:06:16.170 --> 00:06:17.670 of the terminal in cursor, 138 00:06:17.670 --> 00:06:19.650 it's going to automatically load 139 00:06:19.650 --> 00:06:21.570 this new virtual environment. 140 00:06:21.570 --> 00:06:23.070 So this is great. 141 00:06:23.070 --> 00:06:26.190 Let's go now back to the ingestion.py file. 142 00:06:26.190 --> 00:06:28.830 However, here we can see we still have an error 143 00:06:28.830 --> 00:06:31.830 and this is because we haven't yet configured the IDE. 144 00:06:31.830 --> 00:06:33.930 So let's go and configure the IDE 145 00:06:33.930 --> 00:06:35.730 to the virtual environment. 146 00:06:35.730 --> 00:06:37.980 So, inside the virtual environment 147 00:06:37.980 --> 00:06:40.710 I'm going to write which Python3. 148 00:06:40.710 --> 00:06:42.720 And this is going to output the path 149 00:06:42.720 --> 00:06:45.810 to the interpreter with all the dependencies installed. 150 00:06:45.810 --> 00:06:47.220 So I'm going to copy it. 151 00:06:47.220 --> 00:06:49.200 Let's me go command+shift+P, 152 00:06:49.200 --> 00:06:52.440 let me go write select interpreter 153 00:06:52.440 --> 00:06:55.740 and here I want to enter an interpreter path. 154 00:06:55.740 --> 00:06:58.200 And here I want to select the recommended interpreter 155 00:06:58.200 --> 00:07:00.330 and we can see it fixes the issue. 156 00:07:00.330 --> 00:07:01.530 If it cause you problem, 157 00:07:01.530 --> 00:07:04.020 you should go and write Command+Shift+P 158 00:07:04.020 --> 00:07:05.760 and in the select interpreter, 159 00:07:05.760 --> 00:07:10.020 you can enter the path to the interpreter we copied. 160 00:07:10.020 --> 00:07:12.480 Let's now run everything as a sanity check 161 00:07:12.480 --> 00:07:14.790 and we can see it's printing this ingestion. 162 00:07:14.790 --> 00:07:17.370 Let's go now and create a .env file. 163 00:07:17.370 --> 00:07:18.720 And here we're going to be storing 164 00:07:18.720 --> 00:07:21.660 all the environment variables and all of our API keys. 165 00:07:21.660 --> 00:07:24.510 So let me paste in my OpenAI API key 166 00:07:24.510 --> 00:07:26.340 like we did in previous sessions 167 00:07:26.340 --> 00:07:28.110 and do not worry about my API key. 168 00:07:28.110 --> 00:07:31.140 Of course, I'm going to revoke every API key you see here 169 00:07:31.140 --> 00:07:33.090 after I finish filming this video. 170 00:07:33.090 --> 00:07:34.050 And let me now paste 171 00:07:34.050 --> 00:07:36.780 all the LANGSMITH related environment variables. 172 00:07:36.780 --> 00:07:39.900 So we have the LANGSMITH_API_key for the tracing. 173 00:07:39.900 --> 00:07:42.870 We have LANGSMITH_PROJECT RAG GIST 174 00:07:42.870 --> 00:07:45.270 so we can see it in the LANGSMITH UI. 175 00:07:45.270 --> 00:07:48.180 And we have LANGSMITH_TRACING=true. 176 00:07:48.180 --> 00:07:51.780 So now we want to go and configure our vector store. 177 00:07:51.780 --> 00:07:53.970 So we are going to be using pinecone. 178 00:07:53.970 --> 00:07:56.640 And Pine Cone is a managed vector store, 179 00:07:56.640 --> 00:07:57.900 which is cloud-based. 180 00:07:57.900 --> 00:08:00.690 And in the future videos we're also going to see 181 00:08:00.690 --> 00:08:04.080 open source alternatives and pinecone have a free tier, 182 00:08:04.080 --> 00:08:06.870 so it's more than enough for this course. 183 00:08:06.870 --> 00:08:09.600 So let me now open up my browser 184 00:08:09.600 --> 00:08:13.200 and let me go to pinecone.io. 185 00:08:13.200 --> 00:08:15.270 And here let me go and log in. 186 00:08:15.270 --> 00:08:18.750 Alright, so here we are in a place where we can see 187 00:08:18.750 --> 00:08:20.370 all of my indexes. 188 00:08:20.370 --> 00:08:24.480 Of course you should have an empty indexes directory. 189 00:08:24.480 --> 00:08:27.540 So I want to go now and create a new index. 190 00:08:27.540 --> 00:08:32.010 Let's call this index medium-bogs-embeddings-index. 191 00:08:32.010 --> 00:08:34.410 And you can call it in any name you want. 192 00:08:34.410 --> 00:08:37.080 We are going to configure it in the environment file. 193 00:08:37.080 --> 00:08:40.650 So let's go now and in the configuration 194 00:08:40.650 --> 00:08:42.090 we have custom settings 195 00:08:42.090 --> 00:08:44.700 and here we can enter manually the dimension 196 00:08:44.700 --> 00:08:46.260 of the vector store. 197 00:08:46.260 --> 00:08:48.120 So this is the lengths of the vectors 198 00:08:48.120 --> 00:08:49.440 that are going to be stored. 199 00:08:49.440 --> 00:08:52.800 And we can see we have a bunch of vector types, 200 00:08:52.800 --> 00:08:56.160 we have Dense, we have Sparse, we're going to go with Dense. 201 00:08:56.160 --> 00:08:57.930 And we have here the metrics. 202 00:08:57.930 --> 00:09:00.240 So those are the metrics to calculate 203 00:09:00.240 --> 00:09:02.340 the similarity between vectors. 204 00:09:02.340 --> 00:09:04.440 We have COSINE which is the default. 205 00:09:04.440 --> 00:09:07.080 We have Euclidean, we have dotproduct 206 00:09:07.080 --> 00:09:10.470 and we will be elaborating on that later in this course. 207 00:09:10.470 --> 00:09:12.780 And this is if you want to go with custom settings. 208 00:09:12.780 --> 00:09:15.990 What I like to do is I want to go and find 209 00:09:15.990 --> 00:09:18.150 the embeddings model that I'm using. 210 00:09:18.150 --> 00:09:21.300 And here we are going to be using at least me in the video 211 00:09:21.300 --> 00:09:23.850 text-embedding-3-small by OpenAI. 212 00:09:23.850 --> 00:09:27.120 And here we can see this is the default configuration 213 00:09:27.120 --> 00:09:30.450 and here we can see the dimension is 512 214 00:09:30.450 --> 00:09:35.100 and I'm going to select it to be 1536 215 00:09:35.100 --> 00:09:38.160 because I want it to hold more information. 216 00:09:38.160 --> 00:09:41.730 So the longer the vector, the more information, 217 00:09:41.730 --> 00:09:44.790 semantic information and semantic meaning it can hold. 218 00:09:44.790 --> 00:09:46.830 So this is the general rule of thumb. 219 00:09:46.830 --> 00:09:48.870 And here in the capacity mode 220 00:09:48.870 --> 00:09:50.820 I'm going to go with serverless. 221 00:09:50.820 --> 00:09:53.790 We have also an option to have dedicated read notes 222 00:09:53.790 --> 00:09:56.490 and maybe I'll dive into vector store configurations 223 00:09:56.490 --> 00:09:57.630 in later sections. 224 00:09:57.630 --> 00:09:59.400 Here we have the cloud provider. 225 00:09:59.400 --> 00:10:01.350 Now I do not care which cloud provider 226 00:10:01.350 --> 00:10:02.790 I am running for this tutorial. 227 00:10:02.790 --> 00:10:06.390 However, for enterprises who has compliance 228 00:10:06.390 --> 00:10:08.550 and privacy restrictions, 229 00:10:08.550 --> 00:10:11.460 they sometimes want to run on a certain cloud provider 230 00:10:11.460 --> 00:10:14.220 and pinecone supports the major three providers. 231 00:10:14.220 --> 00:10:15.600 We can also choose the region 232 00:10:15.600 --> 00:10:17.700 which our vector store is going to be deployed. 233 00:10:17.700 --> 00:10:19.950 I'm going to go with the default one. 234 00:10:19.950 --> 00:10:23.460 Here, we can see we have also in Ireland, in Oregon, 235 00:10:23.460 --> 00:10:24.510 and this is important 236 00:10:24.510 --> 00:10:26.820 when we deploy an application to production. 237 00:10:26.820 --> 00:10:29.970 We want our vector store to usually be in the same region 238 00:10:29.970 --> 00:10:31.830 as our RAG application 239 00:10:31.830 --> 00:10:34.920 because our RAG application is going to be making requests 240 00:10:34.920 --> 00:10:36.540 to the vector store. 241 00:10:36.540 --> 00:10:38.220 And if it's not in the same region 242 00:10:38.220 --> 00:10:40.740 then we're going to have egress cost. 243 00:10:40.740 --> 00:10:43.290 Anyways, I do not want to complicate things right now. 244 00:10:43.290 --> 00:10:45.810 We're going to be discussing those kinds of trade-offs 245 00:10:45.810 --> 00:10:47.700 in the production section of this course. 246 00:10:47.700 --> 00:10:51.180 And I'm going to go and create this index. 247 00:10:51.180 --> 00:10:55.290 So right now we can see that the index is created. 248 00:10:55.290 --> 00:10:58.380 So let me go and copy here this name 249 00:10:58.380 --> 00:11:02.820 and let me write here the environment variable of index name 250 00:11:02.820 --> 00:11:04.350 and let me paste it in. 251 00:11:04.350 --> 00:11:08.640 And I also want to go and create an API key and use it. 252 00:11:08.640 --> 00:11:11.520 So here I have my already created API keys. 253 00:11:11.520 --> 00:11:13.680 Let me go and create a new API key. 254 00:11:13.680 --> 00:11:18.680 Let me call it rag-gist-video and let me copy it 255 00:11:18.900 --> 00:11:20.520 and let me go and paste it 256 00:11:20.520 --> 00:11:25.143 under the name of PINECONE_API_KEY. 257 00:11:26.610 --> 00:11:29.850 Now the name PINECONE_API_KEY here is important 258 00:11:29.850 --> 00:11:33.660 because this is what langchain is going to be looking for 259 00:11:33.660 --> 00:11:35.040 when it's going to be using 260 00:11:35.040 --> 00:11:37.230 the langchain pinecone integration. 261 00:11:37.230 --> 00:11:39.750 So it's going to be looking for the environment viable 262 00:11:39.750 --> 00:11:41.460 with this exact name. 263 00:11:41.460 --> 00:11:45.870 Cool, so let's go now to the ingestion file. 264 00:11:45.870 --> 00:11:50.100 And now let me go and write print 265 00:11:50.100 --> 00:11:52.770 and let's see that we managed to successfully load 266 00:11:52.770 --> 00:11:54.510 all the environment variables. 267 00:11:54.510 --> 00:11:55.950 So for example, let me go 268 00:11:55.950 --> 00:11:59.103 and access OS ENVIRON 'PINECONE_API_KEY' 269 00:12:00.592 --> 00:12:05.592 and let me run it and we can see we get here an error. 270 00:12:06.600 --> 00:12:09.360 This is because I did not save the .env file. 271 00:12:09.360 --> 00:12:12.780 Let me save it and now let me run it again. 272 00:12:12.780 --> 00:12:15.690 Boom. We can see it's now printing it. 273 00:12:15.690 --> 00:12:18.810 Cool. So we finished initializing our environment. 274 00:12:18.810 --> 00:12:20.970 This was the boring part of the course 275 00:12:20.970 --> 00:12:25.950 and in the next video we are going to ingest our blog here. 276 00:12:25.950 --> 00:12:28.770 So we are going to run the ingestion pipeline, 277 00:12:28.770 --> 00:12:30.570 which is going to take our blog, 278 00:12:30.570 --> 00:12:34.410 it's going to chunk it up into smaller pieces of text, 279 00:12:34.410 --> 00:12:37.860 it's then going to embed every piece of text 280 00:12:37.860 --> 00:12:39.570 and turn it into a vector. 281 00:12:39.570 --> 00:12:42.120 And then we are going to take those vectors 282 00:12:42.120 --> 00:12:44.340 and we are going to store all of them 283 00:12:44.340 --> 00:12:46.143 in the PINECONE Vector store.