WEBVTT

00:00.000 --> 00:03.600
Instructor: Llms.txt is a standard file,

00:03.600 --> 00:07.530
which is designed to help LLMs and AI agents

00:07.530 --> 00:11.670
to better understand and process website content.

00:11.670 --> 00:14.280
So this file typically is placed

00:14.280 --> 00:17.010
in the root directory of a given website

00:17.010 --> 00:19.410
and provides a concise summary

00:19.410 --> 00:22.800
of the site's most important content and structure

00:22.800 --> 00:26.310
in a machine-readable Markdown format.

00:26.310 --> 00:28.230
So the purpose of this is to give

00:28.230 --> 00:33.150
those AI systems, like ChatGPT, LLMs, like Google Gemini,

00:33.150 --> 00:36.060
or your application-based AI agents

00:36.060 --> 00:38.070
to process the web content

00:38.070 --> 00:41.220
more accurately and more efficiently.

00:41.220 --> 00:44.280
So the format is usually a Markdown file,

00:44.280 --> 00:46.680
and the content will include the URLs

00:46.680 --> 00:48.720
to the important pages of the website

00:48.720 --> 00:51.840
with a brief description of each page content

00:51.840 --> 00:53.790
and the purpose of that page.

00:53.790 --> 00:57.150
And it may have an optional additional information.

00:57.150 --> 01:00.470
Now, the benefits of using llms.txt,

01:00.470 --> 01:02.670
it will improve the AI accuracy

01:02.670 --> 01:05.250
when extracting the information from the website,

01:05.250 --> 01:08.640
so to downstream that information makes it very, very easy.

01:08.640 --> 01:12.060
It also enhances the content discoverability

01:12.060 --> 01:14.670
of LLMs with the website content,

01:14.670 --> 01:17.430
so the website owners have an incentive

01:17.430 --> 01:19.830
to really create this file.

01:19.830 --> 01:22.650
It also provides obviously better context

01:22.650 --> 01:25.050
for understanding the website structure

01:25.050 --> 01:26.880
and can even improve the SEO

01:26.880 --> 01:28.980
by making the content more accessible

01:28.980 --> 01:31.020
for AI-driven search engines.

01:31.020 --> 01:33.360
And while it's not an official standard,

01:33.360 --> 01:36.180
llms.txt is gaining massive popularity

01:36.180 --> 01:38.520
in the GenAI community.

01:38.520 --> 01:39.900
The page we are seeing right now

01:39.900 --> 01:41.880
is in the LangGraph official documentation.

01:41.880 --> 01:46.110
You can simply head up there and go to llms.txt,

01:46.110 --> 01:50.580
and the LangChain team has even taken this one step ahead.

01:50.580 --> 01:54.120
So they have two types of llms.txts.

01:54.120 --> 01:57.150
One is a regular one, like we saw earlier in the example,

01:57.150 --> 02:00.240
and the other is llms-full.txt.

02:00.240 --> 02:03.990
Now, they have both for the LangGraph documentation

02:03.990 --> 02:06.870
in Python and also in JavaScript.

02:06.870 --> 02:08.280
So what's the difference?

02:08.280 --> 02:11.190
The difference is that the llms.txt

02:11.190 --> 02:14.640
is only going to include the URLs and a short description

02:14.640 --> 02:19.320
while the llms-full.txt is going to be all the information

02:19.320 --> 02:21.570
and all the texts of those pages.

02:21.570 --> 02:23.880
So, it's going to be a huge file.

02:23.880 --> 02:25.650
Now let's talk about the use cases

02:25.650 --> 02:27.690
and when to use the llms.txt

02:27.690 --> 02:30.360
and when to use the llms-full.txt.

02:30.360 --> 02:34.890
So I'll use the llms.txt, the brief one,

02:34.890 --> 02:39.060
when, for example, I have an AI agent or an MCP server,

02:39.060 --> 02:44.060
which has the web scraping tool, like Firecrawl,

02:44.580 --> 02:47.640
which enables us to download the content of a website.

02:47.640 --> 02:49.080
So once we have it,

02:49.080 --> 02:52.290
we have in the context all the mapping of the website,

02:52.290 --> 02:54.930
and we know exactly which one to download.

02:54.930 --> 02:56.460
So in case, for example,

02:56.460 --> 03:00.870
we need to download the content about LangChain memory,

03:00.870 --> 03:04.110
then we can simply go and give it in the context,

03:04.110 --> 03:06.540
and the LLM would be able to choose the tool

03:06.540 --> 03:09.540
with the correct URL to download only the memory.

03:09.540 --> 03:11.190
So we'll download,

03:11.190 --> 03:13.980
and we'll retrieve only the information that we'll need,

03:13.980 --> 03:16.800
kind of similar like Rack.

03:16.800 --> 03:18.180
And this is very cool,

03:18.180 --> 03:21.630
and this really enhances the quality of agents

03:21.630 --> 03:24.000
which can get information in real time,

03:24.000 --> 03:27.963
if those websites have llms.txt implementation in them.

03:29.580 --> 03:31.830
And don't worry if it sounds too abstract.

03:31.830 --> 03:32.700
In the next video,

03:32.700 --> 03:34.860
I'm going to show you hands-on

03:34.860 --> 03:39.210
how to go and how to use llms.txt and how to use MCP

03:39.210 --> 03:42.060
in order to get really good information

03:42.060 --> 03:44.913
and really enhance your application's answers.

03:46.200 --> 03:49.260
And it's going to fetch that information in real time,

03:49.260 --> 03:51.330
so this is the key point here.

03:51.330 --> 03:53.970
Now, the llms-full.txt,

03:53.970 --> 03:55.740
it's going to have much more information.

03:55.740 --> 03:59.610
It's going to be a huge file with all the information in it.

03:59.610 --> 04:02.220
Now, when and how to use that,

04:02.220 --> 04:03.630
we can simply download it,

04:03.630 --> 04:05.700
and if we'll chunk it ourself

04:05.700 --> 04:08.130
and index it into our vector store,

04:08.130 --> 04:10.950
we can get very similar functionality.

04:10.950 --> 04:13.170
If we're using LLMs with large context,

04:13.170 --> 04:15.330
we can even send the entire thing.

04:15.330 --> 04:18.090
Or if those LLMs have context cache,

04:18.090 --> 04:20.790
we can cache this information.

04:20.790 --> 04:23.670
Now, there are advantages and disadvantages

04:23.670 --> 04:28.670
for using llms.txt and using llms-full.txt.

04:29.280 --> 04:33.330
So if we want to use the abbreviated version, the llms.txt,

04:33.330 --> 04:35.610
with only the brief information and the links,

04:35.610 --> 04:37.980
then the usage pattern of this is usually

04:37.980 --> 04:42.060
by using an agent with a scraping tool and a search tool.

04:42.060 --> 04:43.440
So once we do that,

04:43.440 --> 04:46.020
we are going to get real time information,

04:46.020 --> 04:47.520
so that's one advantage.

04:47.520 --> 04:50.310
However, it's going to take us a bit more time

04:50.310 --> 04:52.410
because we first need to download this page

04:52.410 --> 04:56.460
and to fetch it with the URL scraper.

04:56.460 --> 04:59.490
We then need to process it with an LLM,

04:59.490 --> 05:01.950
then the LLM will tell us what we need to download.

05:01.950 --> 05:03.810
So then we need to make another fetch

05:03.810 --> 05:05.880
and to download another file

05:05.880 --> 05:09.480
and another web page content and to process that,

05:09.480 --> 05:11.340
and only then we'll get a result.

05:11.340 --> 05:13.800
So while we're getting realtime information,

05:13.800 --> 05:16.650
because it's going to fetch the information dynamically,

05:16.650 --> 05:19.470
the latency is going to be a bit higher.

05:19.470 --> 05:20.940
And by the way, don't freak out

05:20.940 --> 05:23.040
about the demo I'm showing you right now,

05:23.040 --> 05:26.460
we're going to implement this hands-on in the next video.

05:26.460 --> 05:27.510
So it's really cool,

05:27.510 --> 05:30.510
and it's a very cool pattern you should be familiar with,

05:30.510 --> 05:32.463
and it's going to use MCP.