WEBVTT 00:00.040 --> 00:04.720 Large language models do not process text the way humans do. 00:05.320 --> 00:13.160 While we naturally read words and sentences, llms transform all input into a very different internal 00:13.160 --> 00:19.600 representation made up of tokens, vectors, and fixed length context windows. 00:20.160 --> 00:23.600 This transformation is not a minor technical detail. 00:24.000 --> 00:29.400 It explains nearly every strength and limitation of modern AI systems. 00:29.960 --> 00:37.800 Understanding these three concepts answers some of the most common questions people have about LMS. 00:38.200 --> 00:40.760 Why do prompts suddenly stop working? 00:41.160 --> 00:44.720 Why does the model forget earlier parts of a long conversation? 00:45.120 --> 00:48.360 Why do API costs scale with usage? 00:48.640 --> 00:51.120 And why do token limits even exist? 00:51.600 --> 00:56.840 All of these behaviors trace back to how text is represented internally. 00:56.960 --> 01:04.760 Once you understand tokens, Embeddings and context windows LLM behavior stops feeling mysterious. 01:05.120 --> 01:08.680 You begin to see patterns and constraints clearly. 01:09.320 --> 01:16.280 This knowledge shifts you from being a user of AI to being an engineer who can design systems around 01:16.280 --> 01:17.400 these constraints. 01:17.880 --> 01:25.040 In practice, this understanding is what allows you to build reliable, scalable, and cost efficient 01:25.080 --> 01:26.480 AI applications. 01:26.720 --> 01:33.240 Rather than fighting against model limitations, tokens are the atomic units that large language models 01:33.280 --> 01:34.600 actually process. 01:34.880 --> 01:38.480 Unlike humans who perceive complete words and phrases. 01:39.600 --> 01:42.920 Break text into smaller pieces called tokens. 01:43.320 --> 01:50.320 These tokens can represent entire words, subwords or even individual characters, depending on the 01:50.320 --> 01:52.000 tokenization strategy. 01:52.040 --> 01:58.080 For example, the word unbelievable might be split into tokens, un and believable. 01:58.560 --> 02:00.240 This splitting is intentional. 02:00.640 --> 02:08.160 It allows models to understand word components, reuse common patterns, and handle rare or unfamiliar 02:08.160 --> 02:13.160 words more effectively by sharing tokens across related words. 02:13.280 --> 02:17.600 Models generalize better and require smaller vocabularies. 02:17.960 --> 02:23.720 One of the most important points to understand is that token count is not the same as word count. 02:24.200 --> 02:30.840 A single word may map to multiple tokens, while several short words may collapse into fewer tokens. 02:31.240 --> 02:33.880 This difference has real world consequences. 02:34.200 --> 02:39.440 Token counts affect pricing, context limits, and how you design prompts. 02:39.920 --> 02:47.000 Once you think in tokens instead of words, many practical LLM behaviors become easier to predict and 02:47.000 --> 02:47.800 control. 02:47.840 --> 02:54.280 Tokenization strategies define how text is broken into tokens, and each strategy represents a trade 02:54.320 --> 02:58.070 off between simplicity, flexibility, and efficiency. 02:58.630 --> 03:04.630 The most intuitive approach is word based tokenization, where each word is treated as a single token. 03:05.190 --> 03:11.550 While easy to understand, this method struggles with rare words, typos, and multilingual text, and 03:11.550 --> 03:14.270 it leads to extremely large vocabularies. 03:14.310 --> 03:21.430 Modern models instead rely on subword tokenization techniques such as byte pair encoding, wordpiece, 03:21.430 --> 03:23.390 and unigram language modeling. 03:23.790 --> 03:29.910 These approaches break words into meaningful chunks, balancing vocabulary size with flexibility. 03:30.550 --> 03:35.030 They allow models to handle unseen words by combining familiar subwords. 03:35.470 --> 03:40.350 This is why models like GPT and Bert can generalize so effectively. 03:40.630 --> 03:46.110 Another approach is byte level tokenization, which operates directly on raw bytes. 03:46.510 --> 03:53.950 This method is fully language agnostic and can handle any input including emojis, code, and special 03:53.950 --> 03:54.790 characters. 03:55.430 --> 04:01.590 The downside is that it often produces longer token sequences and less interpretable tokens. 04:02.110 --> 04:04.990 The core trade off is clear, more flexible. 04:04.990 --> 04:11.630 Tokenization handles edge cases better, while coarser tokenization produces cleaner, more interpretable 04:11.630 --> 04:12.950 representations. 04:12.990 --> 04:19.110 Modern models carefully optimize this balance once text has been converted into tokens. 04:19.350 --> 04:21.470 The next transformation occurs. 04:21.870 --> 04:26.430 Each token is mapped to a high dimensional vector known as an embedding. 04:26.870 --> 04:29.030 These embeddings are not random. 04:29.310 --> 04:34.270 They are learned during training by exposing the model to billions of text examples. 04:34.710 --> 04:41.470 You can think of embeddings as coordinates in a large semantic space with hundreds or thousands of dimensions. 04:42.030 --> 04:46.710 Each dimension captures some aspect of meaning, usage, or context. 04:47.150 --> 04:53.910 For example, the embedding for king will be positioned near queen and monarch, but far from unrelated 04:54.030 --> 04:55.630 words like bicycle. 04:56.070 --> 04:58.390 This transformation is fundamental. 04:58.870 --> 05:02.230 Llms do not understand words in a human sense. 05:02.550 --> 05:07.390 They operate entirely on mathematical relationships between vectors. 05:07.910 --> 05:13.390 All reasoning, generation, and pattern matching happens in this abstract space. 05:13.630 --> 05:15.710 The pipeline is straightforward. 05:15.710 --> 05:17.750 Raw text becomes tokens. 05:17.790 --> 05:22.150 Tokens become embeddings, and embeddings flow through transformer layers. 05:22.710 --> 05:30.390 Once you internalize this process, it becomes clear why llms excel at tasks like semantic similarity, 05:30.550 --> 05:32.870 reasoning, and generalization. 05:33.310 --> 05:35.390 Everything starts with embeddings. 05:35.710 --> 05:42.430 The real power of embeddings comes from how they are organized in space in an embedding space. 05:42.670 --> 05:48.510 Similar concepts cluster together while dissimilar concepts lie far apart. 05:48.710 --> 05:55.910 This geometric structure enables many advanced capabilities that go beyond simple text generation. 05:56.390 --> 05:59.950 Semantic search is a direct result of this structure. 06:00.350 --> 06:05.750 Instead of matching keywords, systems can retrieve information based on meaning. 06:06.270 --> 06:14.110 A query like reduce costs can automatically match documents discussing budget optimization or expense 06:14.110 --> 06:17.430 reduction, even if the exact words differ. 06:17.870 --> 06:24.710 Embeddings also enable clustering, allowing large collections of text such as customer feedback or 06:24.710 --> 06:30.190 research papers, to organize themselves by topic without manual labeling. 06:30.550 --> 06:31.350 Retrieval. 06:31.390 --> 06:37.990 Augmented generation relies heavily on embeddings to find relevant information from large knowledge 06:37.990 --> 06:40.670 bases and inject it into prompts. 06:40.950 --> 06:42.950 Context matters as well. 06:43.470 --> 06:50.630 The word Python may cluster neo, Java, and JavaScript in a programming context, or near Cobra and 06:50.630 --> 06:53.270 reptile in a biological context. 06:53.790 --> 07:00.910 Modern models dynamically adjust embeddings based on surrounding tokens, allowing meaning to shift 07:00.910 --> 07:02.070 with context. 07:02.110 --> 07:08.590 Every large language model has a context window, which is the maximum number of tokens it can process 07:08.590 --> 07:09.550 at one time. 07:10.190 --> 07:12.470 This limit is not arbitrary. 07:12.790 --> 07:17.110 It is a fundamental architectural constraint that shapes how models work. 07:17.590 --> 07:25.150 The context window includes everything the model sees system instructions, conversation history, retrieved 07:25.150 --> 07:32.190 documents in a rag, setup, the user's current input, and even the space needed for the model's response 07:32.190 --> 07:35.110 when the total number of tokens exceeds the window. 07:35.350 --> 07:38.590 Older tokens are truncated and effectively forgotten. 07:39.070 --> 07:41.790 This explains many real world behaviors. 07:42.110 --> 07:45.910 Long conversations gradually lose early context. 07:46.190 --> 07:49.510 Large documents must be split into smaller chunks. 07:49.780 --> 07:53.340 Multi-step debugging sessions eventually hit a wall. 07:53.780 --> 07:57.940 Context is not infinite memory, it is a fixed resource. 07:57.980 --> 08:03.060 Each token in the context window consumes memory, compute, and cost. 08:03.420 --> 08:07.540 For engineers, this means context must be managed intentionally. 08:07.980 --> 08:13.540 Understanding what fits in the window and how information flows through it is essential for building 08:13.540 --> 08:15.340 reliable AI systems. 08:15.980 --> 08:19.620 Context design is just as important as prompt design. 08:19.620 --> 08:26.340 Context window size represents one of the most important engineering trade offs in systems. 08:26.700 --> 08:33.700 Larger context windows allow models to process longer documents, maintain conversation history, and 08:33.700 --> 08:36.100 reason over complex inputs. 08:36.300 --> 08:43.660 However, they also require significantly more memory, increased API costs, and slow down inference. 08:43.900 --> 08:50.940 Smaller context windows are faster, cheaper, and more efficient, but they limit how much information 08:50.980 --> 08:53.060 the model can consider at once. 08:54.700 --> 09:00.500 Long conversations lose context quickly, and large documents must be carefully managed. 09:00.700 --> 09:04.140 This trade off means that bigger is not always better. 09:04.580 --> 09:11.100 Effective LLM applications are designed around context constraints rather than ignoring them. 09:11.460 --> 09:16.180 Critical information should be placed where the model pays the most attention. 09:16.420 --> 09:19.220 Unnecessary verbosity should be removed. 09:19.460 --> 09:21.100 Every token counts. 09:21.580 --> 09:25.980 Techniques like intelligent chunking, sliding windows with overlap. 09:26.140 --> 09:28.420 Summarization and retrieval. 09:28.420 --> 09:32.460 Augmented generation helps stretch limited context further. 09:33.100 --> 09:35.660 The key takeaway is simple but powerful. 09:36.100 --> 09:39.380 Context is a scarce, expensive resource. 09:39.820 --> 09:42.620 Treat it like memory in traditional programming. 09:43.060 --> 09:44.740 Allocate it intentionally. 09:45.060 --> 09:48.700 Manage it carefully, and optimize relentlessly.