{ "cells": [ { "cell_type": "markdown", "id": "fd2212da-8e12-42d1-9ed7-abdf3eb0d8c9", "metadata": {}, "source": [ "## LangSmith in depth" ] }, { "cell_type": "markdown", "id": "dff9de66-547f-4df6-ae93-d646be9168c3", "metadata": {}, "source": [ "## LS definition, terminology and FAQs" ] }, { "cell_type": "markdown", "id": "846f3b89-9664-45f2-b4b1-5c985846b5c5", "metadata": {}, "source": [ "#### What is LangSmith in her own words?\n", "* A unified DevOps platform\n", "* for developing,\n", "* collaborating,\n", "* testing,\n", "* deploying,\n", "* and monitoring\n", "* LLM Applications." ] }, { "cell_type": "markdown", "id": "0632293e-8ae4-4634-b1fd-6392371ffa70", "metadata": {}, "source": [ "#### LangSmith pitch messages\n", "* Get your LLM App from prototype to production.\n", " * All-in-one platform for every step of the LLM-powered application cycle. \n", "* LangSmith turns LLM \"magic\" into enterprise-ready application.\n", " * no more \"guessing\", or \"development by vibes\", use testing to confirm that your LLM Application performs as desired. " ] }, { "cell_type": "markdown", "id": "25d69fe4-e4d9-4784-899c-c867e16a4749", "metadata": {}, "source": [ "#### LangSmith Terminology\n", "* **Traces**: record of interactions with the LLM Application. Can contain more info beyond just input and output.\n", "* **LLM Calls**: calls to the LLM model. Input and output.\n", "* **Run**: a single execution of the LLM App to process an input and generate an output.\n", "* **Annotation Queues**: to add human labels and feedback on traces.\n", "* **Datasets** for evaluation, few-shot prompting or fine-tuning: build datasets from examples, production data or existing sources.\n", "* **(LangSmith Prompt) Hub**: for prompt engineering experimentation.\n", "* **Collaboration**: between developers and subject matter experts.\n", "* **Auto-evaluation**:\n", " * use an LLM and prompt to score your applicatoin output,\n", " * or write your own functional evaluation tests to record different measures of effectiveness.\n", "* **Regression testing**: ensure that new features or updates do not adversely affect the existing functionalities of the LLM Application.\n", "* **Online evaluation** (coming soon):\n", " * continuously track qualitative characteristics of any live application\n", " * and spot issues in real-time with monitoring.\n", "* **Observability**: monitoring health and performance of the LLM Applications, insights into its behavior, performance of the LLM model and the interactions between the LLM Application and the LLM model.\n", "* **Deployment, One-Click deploy**: deployment to LangServe (in beta, only in Plus and Enterprise versions).\n", "* More LS Terminology (feedback, tags, metadata) in [LS Concepts](https://docs.smith.langchain.com/tracing/concepts)" ] }, { "cell_type": "markdown", "id": "f7e84e05-0912-47dd-98a1-8a53c52eea30", "metadata": {}, "source": [ "#### FAQ Section\n", "* You can use LS even if you don't use LC.\n", "* You can self-host LS (with Enterprise pricing).\n", "* LS traces are encrypted and stored securely.\n", "* You can specify the % of traces you send to LS (**this can reduce your cost of LS**).\n", "* LS does not make your app slower.\n", "* LS does not use your data." ] }, { "cell_type": "markdown", "id": "3bbae844-713a-43ec-9905-1d25b6d887d7", "metadata": {}, "source": [ "## LS: Initial Operations" ] }, { "cell_type": "markdown", "id": "8f3f4553-3bac-4a59-b30d-4eac9b715b06", "metadata": {}, "source": [ "#### How to create a free account in LangSmith\n", "* Go to [smith.langchain.com](https://smith.langchain.com)" ] }, { "cell_type": "markdown", "id": "618fbbd7-6311-44c2-8fea-d177ed1cf183", "metadata": {}, "source": [ "#### How to create a new LS project\n", "Normally one LLM App is associated with one Project, but in reality a Project can be associated with any collection of traces.\n", "* Click on New Project\n", "* In the Project page, you will see four tabs:\n", " * Traces\n", " * LLM Calls\n", " * Monitor\n", " * **Setup: log your first run**\n", " * create API key\n", " * select desired runtime environment\n", " * you can use LS with a LangChain app, using the SDKs (Pyton or Typescript) or via API.\n", " * Here we explain how to use LS with a LangChain app. To know about the other options, check the LS [Tracing Quickstart Guide](https://docs.smith.langchain.com/tracing/quick_start).\n", " * To use LS with a LangChain app. [See step by step guide here](https://docs.smith.langchain.com/tracing/quick_start):\n", " * (recommended) create a new virtual environment\n", " * install langchain\n", " * set the LS environment variables\n", " * via terminal\n", " * or via .env file (remove the export keyword)\n", " * run any LLM app \n", "* Add data to .env file\n", " * Create a new LangSmith API Key.\n", " * Copy your API Key in the .env file.\n", "* Use the .env file in your LC project." ] }, { "cell_type": "markdown", "id": "6baa9419-ad37-4976-95a9-b0a3f3946361", "metadata": {}, "source": [ "#### How to see the traces of your project\n", "* Go to Projecs > YourProjectName\n", "* Go to the Traces tab\n", "* Click on each trace to see the trace details\n", " * Input\n", " * Latency\n", " * In test dataset?\n", " * With annotation?" ] }, { "cell_type": "markdown", "id": "cc04f7a5-854c-4e59-a3bd-15d24458fb1c", "metadata": {}, "source": [ "## Prototyping phase: challenges solved by LS" ] }, { "cell_type": "markdown", "id": "e7e59661-54b0-469d-8e66-44c550daddb3", "metadata": {}, "source": [ "#### Have LS enabled since day 1" ] }, { "cell_type": "markdown", "id": "8c5ece9f-1045-4b0a-8cfd-7f9575b34fa2", "metadata": {}, "source": [ "#### How to use LS Playground to iterate and experiment: How to experiment with prompts in the LS Hub Prompt Playground Environment\n", "* You can use it just by going to Hub.\n", "* In the LS Hub you will find examples of prompts that other developers are using for many different cases (purpose, model, etc).\n", " * Clicking on a prompt example will open the Prompt Playground Environment.\n", " * One interesting feature here is prompt versioning. With this you can see how the initial prompt has evolved with the contributions of different people. This can be an interesting way of collaborating between developers and subject matter experts (product managers, marketing people, etc).\n", "* You can import a prompt from the LS Hub into your LangChain Application without having to copy the entire prompt. See detailed instructions on how to do this [here](https://docs.smith.langchain.com/hub/dev-setup#3-pull-an-object-from-the-hub-and-use-it). \n", "* But the interesting thing is that you can use it from any trace.\n", " * You can open any trace in this Prompt Playground Environment and change the prompt, the LLM model, or the LLM model features (temperature, etc).\n", " * There are 2 LLM models you can use here for free: GooglePalm and Fireworks.\n", " * To use chatGPT you will need to enter your OpenAI API key. When you are in the Playground, there is a button in the right top corner called \"Secrets & API Keys\" to do that. " ] }, { "cell_type": "markdown", "id": "aca54606-95d4-42dc-94dd-f9e775b5775a", "metadata": {}, "source": [ "#### How to use LS Comparison View to compare the performance of alternative approaches.\n", "In the Dataset dashboard, go to one dataset, select several tests that you performed using that dateset and click on the Compare button to see the Comparison View.\n", "* We can use this to compare several tests: compare outputs, compare performance, etc." ] }, { "cell_type": "markdown", "id": "8e08043c-7c49-4a73-88cf-09a1626cc122", "metadata": {}, "source": [ "#### How to use LS to create a Test Dataset and include trace examples in it\n", "* Database section > create Database.\n", "* After clicking on a particular trace, click on Add to Dataset." ] }, { "cell_type": "markdown", "id": "ae364759-c526-4b0a-ae00-d9a68373c2f0", "metadata": {}, "source": [ "## LS Datasets: Advanced Tips" ] }, { "cell_type": "markdown", "id": "94c1d152-daa5-4696-b609-3184b9bfd69a", "metadata": {}, "source": [ "#### How to evaluate your LLM Application with a Test Dataset\n", "In the prototyping phase, you will create your own Test Database. In the Beta Testing phase, you will add to that initial database examples of real feedback from your beta users (mostly, relevant cases when the user has labelled the llm answer as THUMBS UP or THUMBS DOWN)\n", "* You can use the test dataset to evaluate different versions of your LLM Application (with different LLM models, with different LLM Model features like temperature, with different prompts, etc) and compare the performance in terms of accuracy, latency, cost, etc.\n", "* It is very useful to use the Comparison View to compare the performance of different versions of the LLM Application with the Test Dataset." ] }, { "cell_type": "markdown", "id": "31d399b2-7688-493e-b359-4a0a65159dbd", "metadata": {}, "source": [ "#### How many examples should have the Test Dataset?\n", "* The LS team says that the average Test Dataset has around 20 examples when an LLM App Development team starts the Beta Testing phase, but the right number really depends on each project and how much time and effort they want/can invest on evaluation." ] }, { "cell_type": "markdown", "id": "6f263da3-271b-4c32-8588-1c58d1b48967", "metadata": {}, "source": [ "#### LS Datasets can be used for more things other than Evaluation\n", "The main use of LS Datasets is evaluation, but some teams have also used them for other purposes like:\n", "* Few-shot prompting,\n", "* Or even fine-tuning." ] }, { "cell_type": "markdown", "id": "e769a9e5-1c9e-4603-afa1-71d2dff4a012", "metadata": {}, "source": [ "#### Offline Evaluation vs. Online Evaluation\n", "* Offline Evaluation: current LS Evaluation. Your LLM App is tested against a test dataset.\n", "* Online Evaluation: next LS feature.\n", " * Evaluators will run on a sample of your traffic. For example, evaluate 20% of your down-voted traces with a particular evaluator in production, with real data." ] }, { "cell_type": "markdown", "id": "fe9366bc-52bd-46a7-a96b-7e7a163f96dc", "metadata": {}, "source": [ "## Beta testing phase: challenges solved by LS\n" ] }, { "cell_type": "markdown", "id": "5855c9fa-bbfb-43ef-93dd-004f793a79fd", "metadata": {}, "source": [ "#### How to use LS to filter traces with negative human feedback to understand the problems behind them.\n", "In the project > traces view, click on any of the tags displayed on the right sidebar. Those tags were created by you when you coded, for example, the THUMBS UP and THUBS DOWN buttons of your LLM App.\n", "* We can use this if we have something like thumbs up or down in the LLM App UI. Then we can filter with the tag that thumbs up or down button." ] }, { "cell_type": "markdown", "id": "51e9d247-2b26-4033-a33b-efc3de55792c", "metadata": {}, "source": [ "#### How to use LS to inspect interesting traces and how to add annotations (\"human feedback\") to one trace\n", "* Click on each trace to see the trace details\n", " * Input\n", " * Latency\n", " * In test dataset?\n", " * With annotation?\n", "* After clicking on a particular trace, click on send to annotation queue. There you can add human feedback to the trace like:\n", " * default LS tags:\n", " * correctnes.\n", " * faithfulness.\n", " * conciseness.\n", " * context relevancy.\n", " * etc.\n", " * custom tags.\n", " * notes: human feedback.\n", "* You can also annotate one individual run inside a trace.\n", " * For example, if you find out that the problem with a bad trace is in the retieval step, it could be interesting to annotate the Retriever run with feedback that can help you to fix it later.\n", "* Annotation is specially useful in a collaboration environment, when the LLM App Developer and the Subject Matter expert can both comment on the highlighted traces." ] }, { "cell_type": "markdown", "id": "ed84adc6-607c-4c2a-8c08-29056b578e95", "metadata": {}, "source": [ "#### How to use LS to expand the Test Dataset by adding runs as examples.\n", "* You can add traces as examples to the Test Database.\n", "* Runs are the different steps that compose a trace." ] }, { "cell_type": "markdown", "id": "c4fff192-4c8e-425c-8244-44d57d490b3c", "metadata": {}, "source": [ "## Production phase: challenges solved by LS" ] }, { "cell_type": "markdown", "id": "52d1e0d4-745b-4f06-b4a1-03562dcd57c5", "metadata": {}, "source": [ "#### Keep using LS as in the Beta Testing phase to keep processing and analyzing user feedback." ] }, { "cell_type": "markdown", "id": "e79ba18d-d86e-4a9a-9da2-591ad2aaa341", "metadata": {}, "source": [ "#### How to use LS to monitor key metrics.\n", "* With the Monitor tab of the project.\n", "* Key metrics: cost, number of tokens per trace, latency, etc.\n", "* LS allows for tag and metadata grouping, which allows users to mark different versions of their applications with different identifiers and view how they are performing side-by-side within each chart. This is helpful for A/B testing changes in prompt, model, or retrieval strategy.\n", " * After opening the Monitor tab, click on the tag or metadata buttons that are located on the top and select the metadata or tag that you want to use to display the monitoring data.\n", "* Apart from A/B Testing, monitoring is a great way to see if your application in production is performing better or worse with the time." ] }, { "cell_type": "markdown", "id": "d6dfff8e-e91e-4d37-81ed-774d4b864523", "metadata": {}, "source": [ "#### How to use LS to mark different versions for A/B Testing of prompts, models or retrieval strategies.\n", "* One of the most interesting uses of monitoring is to compare the performance of different versions of your LLM App.\n", " * Clicking on the Metadata button and selecting one particular metadata parameter will show you how the different versions of your app behave. It allows you to A/B Test different configurations (for example, using different LLM models, prompts or retrieval strategies) of your app and see how each of them impacts on performance metrics (cost, number of tokens per trace, latency, etc)." ] }, { "cell_type": "markdown", "id": "189b0d81-9c68-4764-95ab-99ebada210c3", "metadata": {}, "source": [ "## LS: Advanced Tips" ] }, { "cell_type": "markdown", "id": "d03f3683-86ed-4790-9b5b-3d671bd790eb", "metadata": {}, "source": [ "#### Advanced tip: Deploying multiple versions of the LLM app to production and monitor performance\n", "* You can route percentages of the traffic to different versions of your application in production.\n", "* Or you can route all the traffic to just one production app and clone that traffic in a \"shadow pipeline\" with a different version." ] }, { "cell_type": "markdown", "id": "e7c99fc1-87eb-453c-94dc-ccd512799ee8", "metadata": {}, "source": [ "#### Advanced tip: What LLM models can you use with LS?\n", "* LS can work with any LLM model." ] }, { "cell_type": "markdown", "id": "82bba319-8b9c-4b81-b2fe-750f77cfa9b9", "metadata": {}, "source": [ "#### Advanced tip: Can I use LS if my LLM App is not developed using LangChain?\n", "* Yes.\n", "* You have different ways to use LS: via API and via SDK (Python or Typescript)." ] }, { "cell_type": "markdown", "id": "06694cde-3179-4900-beb3-6ae771af1af8", "metadata": {}, "source": [ "#### Advanced tip: Can I use LS with multi-modal apps?\n", "* LS is starting to experiment with this. It currently works with ChatGPT-4 Vision." ] }, { "cell_type": "markdown", "id": "6352c5d6-0a34-4b07-8be1-af3cc8a05339", "metadata": {}, "source": [ "#### Advanced tip: How to deploy your project to LangServe\n", "* This option is in beta and only available for Plus and Enterprise versions." ] }, { "cell_type": "markdown", "id": "02b96490-2af8-4597-9a09-a93e618a62a7", "metadata": {}, "source": [ "## LS Guides and Recommendations" ] }, { "cell_type": "markdown", "id": "5346973b-2aba-447b-a6cd-027d934b1eac", "metadata": {}, "source": [ "#### LS Tracing: How-to Guides\n", "* **[How to specify the % of traces you send to LS](https://docs.smith.langchain.com/tracing/faq/logging_and_viewing#setting-a-sampling-rate-for-tracing)**\n", "* [How to add metadata and tags to taces](https://docs.smith.langchain.com/tracing/faq/customizing_trace_attributes#adding-metadata-and-tags-to-traces)\n", "* [How to add an annotation to one particular trace](https://docs.smith.langchain.com/tracing/faq/logging_feedback#annotating-traces-with-feedback)\n", "* [See all How-to Guides](https://docs.smith.langchain.com/tracing/faq)\n", "* [See the LangChain-specific guides](https://docs.smith.langchain.com/tracing/faq/langchain_specific_guides)" ] }, { "cell_type": "markdown", "id": "4a988397-3530-470d-954d-5376887a4ba8", "metadata": {}, "source": [ "#### LS Tracing: Use Cases Guides\n", "* [Monitor application sentiment](https://docs.smith.langchain.com/tracing/use_cases/track-sentiment)\n", "* [Summarize app usage](https://docs.smith.langchain.com/tracing/use_cases/summarize-usage)\n", "* [Few-shot prompting with LS datasets](https://docs.smith.langchain.com/tracing/use_cases/few-shot-datasets)" ] }, { "cell_type": "markdown", "id": "c1fbe886-d00c-4703-b65e-bdb7870ce1c9", "metadata": {}, "source": [ "#### LS Evaluation: Quickstart\n", "* [Quickstart](https://docs.smith.langchain.com/evaluation/quickstart)\n", "* [Concepts](https://docs.smith.langchain.com/evaluation/concepts)" ] }, { "cell_type": "markdown", "id": "71c11658-1d22-485f-9dfd-377e480098f4", "metadata": {}, "source": [ "#### LS Evaluation: How-To Guides\n", "* [How to create custom evaluators](https://docs.smith.langchain.com/evaluation/faq/custom-evaluators)\n", "* [How to paraphrase examples to expand your dataset](https://docs.smith.langchain.com/evaluation/faq/expand-datasets-paraphrase)\n", "* [How to create new examples using prompting](https://docs.smith.langchain.com/evaluation/faq/expand-datasets-prompting)\n", "* [Other How-To Guides](https://docs.smith.langchain.com/evaluation/faq)" ] }, { "cell_type": "markdown", "id": "54cbf8f6-84cc-47bf-b530-a34c01ee2b4f", "metadata": {}, "source": [ "#### LS Evaluation: Recommendations\n", "* Test early and often\n", "* Create domain-specific evaluators\n", "* Use labels where possible\n", "* Use aggregate evals\n", "* Measure model stability\n", "* Measure performance of subsets\n", "* Evaluate production data\n", "* Don't train on test datasets\n", "* Test the model yourself\n", "* Ask appropriate questions\n", "* Interesting resources to learn more\n", "* [See these recommendations in detail](https://docs.smith.langchain.com/evaluation/recommendations)" ] }, { "cell_type": "markdown", "id": "cb95dd5b-aa97-4158-a814-9cb5912226e9", "metadata": {}, "source": [ "#### LS Monitoring\n", "* [More details](https://docs.smith.langchain.com/monitoring)" ] }, { "cell_type": "markdown", "id": "86057553-7265-4fe3-9a4a-c2feb4ac101a", "metadata": {}, "source": [ "#### LS Prompt Hub\n", "* [Quickstart](https://docs.smith.langchain.com/hub/quickstart)" ] }, { "cell_type": "markdown", "id": "79e460e6-3ac2-4a7d-9b9f-f5d3caf1aab1", "metadata": {}, "source": [ "#### More info\n", "* Blog post [LangChain announces general availability of LangSmith and 25M Series A led by Sequoia Capital](https://blog.langchain.dev/langsmith-ga/).\n", "* Video [LangSmith In-Depth, by LangSmith Team](https://www.youtube.com/watch?v=3wAON0Lqviw).\n" ] }, { "cell_type": "markdown", "id": "0be28146-5b29-4a2d-976a-41a4efb1d2ec", "metadata": {}, "source": [ "## LS Next Features: Detailed View\n", "#### Support for regression testing.\n", "* Closer integration with CI/CD pipelines, so you can run LS tests in Github actions, or in Gitlab, etc.\n", "* Make it very simple for people to make corrections on scores that were submitted by LLM evaluators.\n", "#### Online evaluation on a sample of production data.\n", "* Be able to configure an online evaluator in a very simple manner.\n", "#### Better filtering and conversation support.\n", "* Now you can do it with metadata filters, but in the future LS will have a more simple and intuitive way of filtering traces from chat conversations. \n", "#### Easy deployment of applications with hosted LangServe.\n", "#### Enterprise features to support the administration and security needs for our largest customers.\n", "* Permissioning. For example: who in the team has permission to make annotations? \n" ] }, { "cell_type": "code", "execution_count": null, "id": "ae46df2c-b3ed-4797-9480-764370761344", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 5 }