1
00:00:04,570 --> 00:00:09,370
In this lesson we will talk about what are tokens?

2
00:00:16,010 --> 00:00:26,450
In my opinion, the best way to define tokens is saying that they are the atoms of the LMS.

3
00:00:26,540 --> 00:00:32,090
So it's like the minimum unit to measure.

4
00:00:32,090 --> 00:00:34,940
What do we do with LMS?

5
00:00:35,930 --> 00:00:40,040
A token is typically a small part of a word.

6
00:00:40,040 --> 00:00:44,840
In some cases, one complete word is a token.

7
00:00:44,870 --> 00:00:52,370
A token, but in many cases a token is typically a small part of a word.

8
00:00:52,370 --> 00:01:01,430
So if you remember, we said that ChatGPT has been trained with the text equivalent to 10 million books.

9
00:01:02,470 --> 00:01:04,569
So if we.

10
00:01:05,620 --> 00:01:09,790
Take a look at the number of words in 10 million books.

11
00:01:09,790 --> 00:01:16,930
We are going to have around 800,000 million words.

12
00:01:17,530 --> 00:01:29,380
So the equivalent in tokens is a little bit more than that is actually 1,100,000,000 tokens.

13
00:01:29,410 --> 00:01:29,920
Okay.

14
00:01:29,920 --> 00:01:31,030
So.

15
00:01:32,060 --> 00:01:33,890
Yeah, we would say that.

16
00:01:36,050 --> 00:01:43,220
1.5 tokens equals to one word, something like that, more or less.

17
00:01:43,550 --> 00:01:51,080
So you have also other kinds of a names.

18
00:01:51,080 --> 00:01:58,760
In LMS, multiple tokens form a sequence and many sequence form a vocabulary.

19
00:01:58,760 --> 00:02:05,540
But in our case we are going to stay with tokens.

20
00:02:05,540 --> 00:02:10,580
We don't want to, uh, worry ourselves with sequence or vocabularies.

21
00:02:10,580 --> 00:02:18,140
Tokens is going to be enough for us a in building LMS applications.

22
00:02:21,330 --> 00:02:26,220
In the next lesson we are going to talk about prompts.

23
00:02:26,250 --> 00:02:28,410
This is an important topic.