WEBVTT 00:00.160 --> 00:04.360 This opening slide establishes the central theme of this section. 00:04.680 --> 00:10.760 While large language models have dramatically expanded what software can do, they do not form complete 00:10.760 --> 00:12.280 applications on their own. 00:12.920 --> 00:19.200 As highlighted on page one, the real challenge lies in integrating Llms into production systems that 00:19.200 --> 00:22.160 are reliable, scalable, and maintainable. 00:22.920 --> 00:29.160 This section focuses on back end architecture because that is where most LLM applications succeed or 00:29.160 --> 00:29.720 fail. 00:30.280 --> 00:37.640 A powerful model cannot compensate for poor system design without proper APIs request handling, error 00:37.640 --> 00:39.600 recovery, and observability. 00:39.840 --> 00:42.480 Even the best models produce fragile systems. 00:43.040 --> 00:50.040 The visual of server infrastructure reinforces an important idea llms live inside distributed systems. 00:50.480 --> 00:55.760 They must coexist with databases, caches, APIs, and external services. 00:56.280 --> 01:01.520 Throughout this section, we will focus on the engineering patterns and architectural discipline that 01:01.520 --> 01:05.580 separate robust applications from experimental prototypes. 01:05.580 --> 01:13.100 By the end of this section, you should think of lrms not as magic components, but as dependencies 01:13.100 --> 01:17.500 that must be carefully orchestrated within a well-designed back end. 01:18.580 --> 01:25.260 This slide explains why back end architecture is a first order concern for LM applications. 01:25.820 --> 01:33.100 As stated on page two, LMS are powerful inference engines, but they are not applications by themselves. 01:33.580 --> 01:37.700 Production systems require the same rigor as any distributed system. 01:37.940 --> 01:45.020 Authentication, authorization, scaling, observability, and fault tolerance for back end design leads 01:45.020 --> 01:46.500 to cascading failures. 01:46.900 --> 01:49.580 High latency degrades user experience. 01:50.060 --> 01:53.580 Uncontrolled API usage causes runaway costs. 01:54.100 --> 01:56.820 Inconsistent behavior erodes trust. 01:57.340 --> 02:01.700 Importantly, these are not model problems, they are architectural problems. 02:02.140 --> 02:07.670 The slide emphasizes a critical insight LMS are dependencies in your system. 02:07.990 --> 02:14.510 Just like databases or external APIs, how you integrate them determines whether your application succeeds 02:14.510 --> 02:15.350 or fails. 02:15.750 --> 02:23.270 The listed production requirements and common failure modes reinforce that back end concerns, not prompt 02:23.270 --> 02:23.950 quality. 02:24.150 --> 02:26.590 Dominate real world reliability. 02:27.110 --> 02:33.750 This mindset shift is essential for engineers transitioning from experimentation to production grade 02:33.750 --> 02:34.510 systems. 02:34.950 --> 02:42.590 This slide presents a reference architecture for production LM applications as shown on page three. 02:42.830 --> 02:48.990 A typical stack includes several interconnected components working together at the edge. 02:49.150 --> 02:54.150 A client application, web, mobile, or CLI initiates requests. 02:54.670 --> 03:01.430 The API layer, commonly built with fast API handles validation, routing, and authentication. 03:02.030 --> 03:06.030 LM services perform inference and orchestrate tool calls. 03:06.510 --> 03:09.990 Supporting layers include vector databases for retrieval. 03:09.990 --> 03:16.770 Augmented generation reads for caching and PostgreSQL for application state and user data. 03:17.250 --> 03:23.730 The diagram clearly illustrates a layered design and the key architectural goal is emphasized loose 03:23.730 --> 03:25.730 coupling with strong boundaries. 03:26.210 --> 03:31.130 Each component should be independently deployable, testable, and replaceable. 03:31.610 --> 03:38.410 This modularity is what allows teams to swap Lem providers, upgrade vector databases, or refactor 03:38.410 --> 03:41.450 business logic without rewriting the entire system. 03:41.890 --> 03:45.290 Good architecture creates flexibility, not lock in. 03:45.450 --> 03:54.210 This slide explains why fast API has become the framework of choice for Lem powered backends. 03:54.610 --> 04:03.330 As described on page for fast API combines Python's ecosystem advantages with performance comparable 04:03.330 --> 04:05.330 to Node.js and Go. 04:05.770 --> 04:08.290 Native async support is critical. 04:08.890 --> 04:16.780 Lem API calls can take seconds, and blocking the event loop would cripple throughput fast APIs. 04:16.820 --> 04:22.100 Async first design allows systems to remain responsive even under heavy load. 04:22.500 --> 04:24.980 Type safety through pedantic catches. 04:24.980 --> 04:30.460 Errors early preventing malformed requests from reaching downstream services. 04:31.140 --> 04:36.780 Auto generated open API documentation keeps front end and back end teams aligned. 04:37.220 --> 04:43.420 Streaming support enables server sent events, allowing users to see tokens as they are generated. 04:43.540 --> 04:47.180 The rule at the bottom of the slide is decisive. 04:47.740 --> 04:52.340 Async first architecture is mandatory for LM applications. 04:52.900 --> 04:57.980 Synchronous code may work in prototypes, but it becomes a bottleneck in production. 04:58.660 --> 05:06.620 This slide reinforces that framework choice is not cosmetic, it directly impacts scalability and reliability. 05:06.620 --> 05:13.460 This slide focuses on API design as the contract between clients and your LM infrastructure. 05:13.860 --> 05:21.560 Page five outlines common endpoint patterns such as chat for conversational interfaces, slash completion 05:21.560 --> 05:28.880 for stateless generation, slash embed for vector creation, and slash tools for function execution. 05:29.360 --> 05:33.320 Each endpoint should have a single, well-defined responsibility. 05:33.960 --> 05:38.000 This clarity simplifies testing, debugging, and scaling. 05:38.400 --> 05:45.160 The slide highlights three core design principles fin controllers, logic, isolation, and explicit 05:45.160 --> 05:45.840 schemas. 05:46.080 --> 05:52.680 Route handlers should delegate business logic to service layers rather than embedding it in HTTP code. 05:53.240 --> 05:57.000 LM orchestration should remain separate from transport concerns. 05:57.440 --> 06:02.960 Pedantic schemas should define all requests and response payloads to enforce structure. 06:03.400 --> 06:05.320 The best practice callout is key. 06:05.800 --> 06:09.360 Treat lm API calls like any external service. 06:09.760 --> 06:16.360 Wrap them in abstraction layers so they can be mocked, retried, monitored, and replaced without disrupting 06:16.360 --> 06:17.440 your application. 06:17.920 --> 06:25.130 This slide walks through the complete request lifecycle for an LM application, as shown on page six. 06:25.290 --> 06:32.090 Every request flows through multiple stages, each introducing latency considerations and failure risks. 06:32.690 --> 06:38.450 The flow begins with a client request followed by input validation using pydantic schemas. 06:38.890 --> 06:44.330 Context retrieval pulls relevant information from vector databases in Rag pipelines. 06:44.850 --> 06:47.970 The formatted prompt is then sent to the LM provider. 06:48.570 --> 06:52.410 Optional tool execution may follow depending on model output. 06:52.890 --> 06:58.050 Finally, responses are post-processed validated and returned to the client. 06:58.530 --> 07:03.650 The slide emphasizes that this flow must be deterministic, observable, and resilient. 07:04.130 --> 07:06.370 Each stage should emit metrics and logs. 07:06.810 --> 07:13.050 Distributed tracing tools like Opentelemetry help visualize where time is spent and where failures occur. 07:13.690 --> 07:19.130 A disciplined request life cycle is what makes debugging possible and performance optimizable. 07:19.570 --> 07:23.210 Without it, LM systems become opaque and fragile. 07:23.660 --> 07:28.980 This slide addresses one of the most challenging aspects of LMS systems latency. 07:29.380 --> 07:31.140 As explained on page seven. 07:31.380 --> 07:37.100 LM response times are inherently variable, ranging from milliseconds to tens of seconds. 07:37.820 --> 07:41.340 Traditional blocking request response patterns are inadequate. 07:41.940 --> 07:48.060 Streaming responses using server sent events dramatically improve perceived performance by delivering 07:48.100 --> 07:49.580 tokens incrementally. 07:50.180 --> 07:56.020 Aggressive timeouts, typically 30 to 60s prevent requests from hanging indefinitely. 07:56.540 --> 08:00.420 The slide also introduces partial responses and async execution. 08:00.940 --> 08:06.140 Returning partial output with clear error indicators is often better than failing completely. 08:06.740 --> 08:11.500 For complex workflows, long running tasks should be offloaded to background workers. 08:12.060 --> 08:15.140 The rule at the bottom is clear and uncompromising. 08:15.500 --> 08:19.340 Never block the request thread waiting for LM responses. 08:19.860 --> 08:24.380 Async execution is essential for system responsiveness and scalability. 08:24.860 --> 08:31.200 This slide explains why error handling is a defining feature of production ready LM systems. 08:31.880 --> 08:34.200 LM APIs fail in many ways. 08:34.520 --> 08:41.000 Timeouts, rate limits, malformed responses, content policy violations, and provider outages. 08:41.520 --> 08:45.120 Page outlines best practices for handling these failures. 08:45.560 --> 08:48.880 Structured errors with clear codes make debugging easier. 08:49.320 --> 08:54.040 Graceful degradation allows systems to fall back to cached or simpler responses. 08:54.640 --> 08:57.280 Clear client messaging prevents confusion. 08:57.680 --> 09:00.920 Circuit breakers stop repeated calls to failing services. 09:01.400 --> 09:04.160 Retries should be applied selectively. 09:04.640 --> 09:10.240 Transient errors like network timeouts and 503 responses are valid retry candidates. 09:10.760 --> 09:13.960 Exponential backoff prevents thundering herd effects. 09:14.560 --> 09:19.520 Idempotency keys avoid duplicate operations when retrying state changing requests. 09:20.080 --> 09:22.320 The warning at the bottom is critical. 09:22.640 --> 09:25.960 Retries without limits lead to cost explosion. 09:26.280 --> 09:30.660 Retry logic must always be bounded and carefully controlled. 09:30.660 --> 09:34.940 The final slide synthesizes the core lessons of this section. 09:35.340 --> 09:42.180 Back end architecture determines whether an LLM application scales, performs reliably, and stays within 09:42.180 --> 09:42.820 budget. 09:43.180 --> 09:47.020 It must be treated with the same rigor as any distributed system. 09:47.500 --> 09:54.500 Fast API enables the async first design required for streaming, concurrency and non-blocking I o. 09:54.940 --> 10:01.660 A disciplined request lifecycle ensures every interaction flows through validation, context retrieval, 10:01.700 --> 10:06.140 inference, and post-processing in a predictable and observable way. 10:06.700 --> 10:08.740 Error handling is not an afterthought. 10:09.020 --> 10:10.580 It is a core feature. 10:11.020 --> 10:13.940 Structured exceptions retry with backoff. 10:14.100 --> 10:18.660 Graceful degradation and clear messaging are essential for resilience. 10:19.180 --> 10:22.380 The closing statement captures the philosophy of this section. 10:22.780 --> 10:27.660 Great LLM applications are built like great software, not demos. 10:27.980 --> 10:34.060 Architectural maturity, not model size is what separates prototypes from production systems.