{ "cells": [ { "cell_type": "markdown", "id": "81c5fc85-e4e5-470c-b2bd-b523aeb0d870", "metadata": {}, "source": [ "## SEC Insights" ] }, { "cell_type": "markdown", "id": "f634a184-4b6a-452f-81b2-c018fbe31929", "metadata": {}, "source": [ "## What it is?\n", "* Chat application\n", "* RAG technique\n", "* Answers questions about SEC 10k and 10Q documents\n", "* Production-ready\n", "* Full-stack repo\n", "* Ready for you to fork" ] }, { "cell_type": "markdown", "id": "2c542e4b-79ff-4968-b4e0-2c68719be020", "metadata": {}, "source": [ "## Try it out\n", "[secinsights.ai](secinsights.ai)" ] }, { "cell_type": "markdown", "id": "8274cfea-e621-4186-9d20-d1988e7dc1e1", "metadata": {}, "source": [ "## Product Features\n", "* QA chat grounded in source-of-truth SEC documents\n", "* PDF viewer\n", "* Token-level streaming of chat responses\n", "* Streaming of reasoning steps (sub-questions)\n", "* Citation of source data\n", "* Use of API-based tools (in addition to semantic search)" ] }, { "cell_type": "markdown", "id": "7dcb8dfd-df87-4e01-8524-09c05234b6f0", "metadata": {}, "source": [ "![alt text](images/architecture-v1.png)" ] }, { "cell_type": "markdown", "id": "9e117747-ad8f-492e-b46c-f64579859ba1", "metadata": {}, "source": [ "## Architecture\n", "* Backend\n", " * Render.com: hosting most of the backend.\n", " * Similar to AWS but easier to use.\n", " * FastAPI Backend Service.\n", " * Traditional load-balanced API Service.\n", " * Load balancing requests to service instances.\n", " * Auto-scaling\n", " * Postgres 15 database\n", " * Cron job service\n", " * All of the above prepared for us in the file render.yaml" ] }, { "cell_type": "markdown", "id": "279dec2b-716e-4286-8939-eb2e70833774", "metadata": {}, "source": [ "* AWS S3: **you will have to setup this yourself**\n", " * Private StorageContext Bucket: metadata from the llamaindex library\n", " * Public PDF Bucket" ] }, { "cell_type": "markdown", "id": "4a396a5b-c980-441f-9a22-cea47b722bec", "metadata": {}, "source": [ "* Frontend\n", " * NextJS\n", " * Hosted in Vercel\n", " * Interacts with the FastAPI backend for some of the chat endpoints" ] }, { "cell_type": "markdown", "id": "3d51b9bb-616b-4ebd-b5cd-c4e7fdeb6431", "metadata": {}, "source": [ "* External services:\n", " * Polygon.io (Financial Data API, Numeric Data). Good example on how to integrate tools in your chat agent.\n", " * SEC's Edgar API\n", " * OpenAI Service (LLM)\n", " * The cron service is the main worker that calls the OpenAI Embedding API to get the embeddings for the given SEC documents.\n", " * The cron service calls the Edgar API from the SEC, get the PDFs, store them in the AWS Public PDF Bucket, run the embeddings on them and store the embeddings in the Postgres database (we use the PG Vectorstore integration).\n", " * The cron job can run at whatever schedule you set in the render.yaml file.\n", " * Sentry.io (Production-level Monitoring Service. It will ping you whenever there is an error in the backend service, or threshold errors, etc. You can also do Performance Monitoring: what sections of the code are taking more time in your service)" ] }, { "cell_type": "markdown", "id": "16c1bcef-d58c-4a6a-8fc2-81e3eaa8ac13", "metadata": {}, "source": [ "All the setup is open source and is easy to deploy on Vercel and Render.com" ] }, { "cell_type": "markdown", "id": "cab00900-8a6a-4742-a221-77fd5c52c943", "metadata": {}, "source": [ "## Dev Environment" ] }, { "cell_type": "markdown", "id": "cb9da60b-bf8d-41e0-a23c-d7157c7b2176", "metadata": {}, "source": [ "* It's recommended to use the config included for a Github Codespace in the devcontiner.json file." ] }, { "cell_type": "markdown", "id": "2329e830-ddf6-4e41-b93e-7056be902df2", "metadata": {}, "source": [ "* Create a Github Codespace.\n", "* cd frontend\n", "* ls\n", "* This is a basic vercel app\n", "* npm install\n", "* source the .env.example folder, load the environment variables that are present here\n", "* the url there is the local one of the backend (localhost/8000), will have to be changed when we use the backend in the cloud.\n", "* set -a\n", "* source .env.example\n", "* npm run dev\n", "* that starts running the app in localhost/3000\n", "* It comes with live reload, so if you edit any UI file it will show immediately. For example, you can edit the title in components/landing-page/TitleAndDropdown.tsx\n", "* With the app open in one terminal, open a second terminal and cd backend\n", "* This is a fastAPI python backend app. Most of this app is based in the templates fastAPI offers." ] }, { "cell_type": "markdown", "id": "3ebf5b10-0253-434d-a027-00348bee0518", "metadata": {}, "source": [ "## Backend" ] }, { "cell_type": "markdown", "id": "d0f32411-7fd2-47e8-8953-18a25107cb94", "metadata": {}, "source": [ "* In the github folder, go to /backend and read the readMe file.\n", "* cd backend\n", "* No need to install pyenv nor docker if you are running from the devcontainer image in Github Codespaces\n", "* cat .python-version: confirms you have 3.11.3\n", "* poetry shell" ] }, { "cell_type": "markdown", "id": "ef79e823-455c-44b4-b7e3-a5e25ef11560", "metadata": {}, "source": [ "The command `poetry shell` is used in the context of Python programming, specifically when managing Python projects with the tool Poetry. Poetry is a tool for dependency management and packaging in Python, allowing developers to declare, manage, and install dependencies of Python projects.\n", "\n", "Here's what `poetry shell` does:\n", "\n", "1. **Activates the Virtual Environment**: When you run `poetry shell`, it activates the project's virtual environment. A virtual environment is a self-contained directory that contains a Python installation for a particular version of Python, along with a number of additional packages.\n", "\n", "2. **Isolation of Project Dependencies**: This isolation ensures that the dependencies of the project do not interfere with the system-wide Python installation or other Python projects. It's a key practice in Python development to avoid dependency conflicts and maintain project consistency.\n", "\n", "3. **Interactive Shell**: Once the virtual environment is activated, you are placed into an interactive shell (like bash or Command Prompt) that is configured to use the project's Python interpreter. This means any Python commands you run in this shell will use the project's Python version and have access to its dependencies.\n", "\n", "4. **Convenience for Development**: The `poetry shell` command is convenient for development purposes. You can run Python scripts, start a Python interactive session, or use command-line tools that are part of your project's dependencies without needing to manually activate the virtual environment or adjust your system's `PATH`.\n", "\n", "5. **Temporary Activation**: The activation of the virtual environment using `poetry shell` is temporary. Once you exit the shell, the environment is deactivated, and your terminal returns to its previous state.\n", "\n", "In summary, `poetry shell` is a command to activate the virtual environment associated with your Poetry-managed Python project, providing an isolated and consistent development environment for that project." ] }, { "cell_type": "markdown", "id": "2bbe912d-58f8-4bc0-8370-039db80e697d", "metadata": {}, "source": [ "* Now in the github codespaces terminal you see (llama-app-backend-py3.11) as virtual environment\n", "* poetry install\n", "* Now you have to create the backend/.env file\n", " * cp .env.development .env\n", " * set -a\n", " * source .env" ] }, { "cell_type": "markdown", "id": "ed21f464-9e05-444f-aa52-c656a71f1dd9", "metadata": {}, "source": [ "The command `set -a` in a Unix/Linux shell environment is used to change the behavior of the shell with respect to how it handles variables and their visibility (exporting) to child processes. Here's what it does:\n", "\n", "1. **Auto-Export Variables**: When you use `set -a`, any variable that you subsequently define or modify in your shell session will be automatically exported. This means that these variables become environment variables and are inherited by any child processes or sub-shells spawned from your shell.\n", "\n", "2. **Child Process Inheritance**: Normally, when you create a variable in a shell, it's only known to that particular shell session. Child processes or scripts invoked from that shell don't have access to those variables unless they are explicitly exported using the `export` command. However, with `set -a`, this export is implicit for all variables set after the command.\n", "\n", "3. **Use Cases**: This command is particularly useful in scripts where you need to ensure that all defined variables are available to sub-processes without having to explicitly export each one. It's often used in startup scripts or in scripts that configure environment variables for a particular application or service.\n", "\n", "4. **Reversing the Effect**: If you want to revert to the normal behavior where variables are not automatically exported, you can use `set +a`. This command will stop the automatic export of variables defined after it.\n", "\n", "5. **Scope of Effect**: It's important to note that the effect of `set -a` is limited to the current shell session or script in which it is run. It does not affect other shell sessions or globally change the behavior of the shell.\n", "\n", "In summary, `set -a` is a shell command used to automatically export all variables set in the current shell session, making them available to any child processes. This can be useful in scripting scenarios where environment variable propagation is desired." ] }, { "cell_type": "markdown", "id": "08f90b78-38fe-4afe-907b-7cfbf9fa4102", "metadata": {}, "source": [ "* source .env" ] }, { "cell_type": "markdown", "id": "435f0ed3-62f6-4bf7-8a1d-10b970eb7479", "metadata": {}, "source": [ "The command `source .env` in a Unix/Linux shell is used to execute the contents of a file (in this case, a file named `.env`) in the current shell session. Here's what this command specifically does:\n", "\n", "1. **Loads Environment Variables**: The `.env` file typically contains environment variables. These are often key-value pairs that are used to configure the behavior of an application or script. By using `source .env`, you are effectively loading these variables into your current shell environment.\n", "\n", "2. **Executes in Current Shell**: The `source` command (which can also be represented as a dot `.` in some shells) executes the file in the context of the current shell, rather than starting a new shell to run the script. This means any changes made to the environment, such as setting variables, changing directories, etc., will persist in the current shell after the script completes.\n", "\n", "3. **Use in Application Configuration**: This is a common practice in application development, especially in web development, where `.env` files are used to set configuration variables that should not be hard-coded into the application, such as database passwords, API keys, and other sensitive information.\n", "\n", "4. **Security Note**: It's important to be cautious with `.env` files, especially regarding sensitive information. These files should not be included in version control (like Git) if they contain sensitive data.\n", "\n", "5. **Portability and Convenience**: This approach allows for easy customization of application behavior in different environments (like development, testing, production) by simply changing the contents of the `.env` file, rather than altering the application code.\n", "\n", "In summary, `source .env` is used to execute the contents of the `.env` file in the current shell, typically for the purpose of setting environment variables that configure the behavior of an application or script. This allows for a flexible and secure way to manage configuration settings." ] }, { "cell_type": "markdown", "id": "533fd7ea-bb02-4149-9857-d7fd52516b68", "metadata": {}, "source": [ "## Start the backend server" ] }, { "cell_type": "markdown", "id": "57651ac7-8608-4fc5-91ae-83b7e1826444", "metadata": {}, "source": [ "* make migrate (Run the database migrations)\n", "* make run (start the server locally)\n", " * This spins up the Postgres 15 DB & Localstack in their own docker containers.\n", " * The server will not run in a container but will instead run directly on your OS.\n", " * This is to allow for use of debugging tools like pdb" ] }, { "cell_type": "markdown", "id": "6b1df44d-fce6-49f1-8ebf-a27ee6a74236", "metadata": {}, "source": [ "## Enter your OpenAI API Key in the .env file" ] }, { "cell_type": "markdown", "id": "0f77d1fc-3421-48ca-8ec6-ae121fa4d766", "metadata": {}, "source": [ "* open your .env file and replace the placeholder value for the OPENAI_API_KEY with your own OpenAI API key\n", "* At some point you will want to do the same for the other secret keys in here like POLYGON_IO_API_KEY, AWS_KEY, & AWS_SECRET\n", "* To follow the SEC's Internet Security Policy, make sure to also replace the SEC_EDGAR_COMPANY_NAME & SEC_EDGAR_EMAIL values in the .env file with your own values.\n", "* Source the file again with set -a then source .env" ] }, { "cell_type": "markdown", "id": "da76b2ef-bfab-436e-a366-038121b533e0", "metadata": {}, "source": [ "## Populate your local database with some sample SEC filings" ] }, { "cell_type": "markdown", "id": "a08d4b5c-daa8-4f37-89b3-50d6093cd07a", "metadata": {}, "source": [ "* Run make seed_db_local\n", " * If this step fails, you may find it helpful to run make refresh_db to wipe your local database and re-start with emptied tables.\n", "* Done 🏁! You can run make run again and you should see some documents loaded at [http://localhost:8000/api/document](http://localhost:8000/api/document)" ] }, { "cell_type": "markdown", "id": "e4459821-d126-4410-8dd3-4b0bdbda7c99", "metadata": {}, "source": [ "## For any issues\n", "* For any issues in setting up the above or during the rest of your development, you can check for solutions in the following places:\n", "\n", " * [backend/troubleshooting.md](https://github.com/run-llama/sec-insights/blob/main/backend/troubleshooting.md)\n", " * [Open & already closed Github Issues](https://github.com/run-llama/sec-insights/issues?q=is%3Aissue+is%3Aclosed)\n", " * [The #sec-insights discord channel](https://discord.com/channels/1059199217496772688/1150942525968879636)" ] }, { "cell_type": "markdown", "id": "1258a338-6ca3-4e61-9dea-0c7f4a33cda4", "metadata": {}, "source": [ "## SEC Document Downloader" ] }, { "cell_type": "markdown", "id": "fd1b8891-6c1f-4446-b107-9111a8eceb9d", "metadata": {}, "source": [ "We have a script to easily download SEC 10-K & 10-Q files! This is a single step of the larger seed script described in the next section. Unless you have some use for just running this step on it's own, you probably want to stick to the Seed script described in the section below. However, the setup instructions for this script are a pre-requisite for running the seed script.\n", "\n", "No API keys are needed to use this, it calls the SEC's free to use Edgar API.\n", "\n", "The instructions below explain a process to use the script to download the SEC filings, convert the to PDFs, and store them in an S3 bucket.\n", "\n", "## Setup / Usage Instructions\n", "Pre-requisite setup steps to use the downloader script to load the SEC PDFs directly into an S3 bucket.\n", "\n", "These steps assume you've already followed the steps above for setting up your dev workspace!\n", "\n", "#### Setup AWS CLI\n", "* Install AWS CLI\n", " * This step can be skipped if you're running from the devcontainer image in Github Codespaces\n", " * Steps:\n", " * curl \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"\n", " * unzip awscliv2.zip\n", " * sudo ./aws/install\n", "* Configure AWS CLI\n", " * This is mainly to set the AWS credentials that will later be used by s3fs\n", " * Run aws configure and enter the access key & secret key for a AWS IAM user that has access to the PDFs where you want to store the SEC files.\n", " * set the default AWS region to us-east-1 (what we're primarily using).\n", "\n", "#### Setup s3fs\n", "* Install s3fs\n", " * This step can be skipped if you're running from the devcontainer image in Github Codespaces\n", " * sudo apt install s3fs\n", "* Setup a s3fs mounted folder\n", " * Create the mounted folder locally mkdir ~/mounted_folder\n", " * s3fs llama-app-web-assets-preview ~/mounted_folder\n", " * You can replace llama-app-web-assets-preview with the name of the S3 bucket you want to upload the files to.\n", "\n", "#### Install wkhtmltopdf\n", "* This step can be skipped if you're running from the devcontainer image in Github Codespaces\n", "* Steps:\n", " * sudo apt-get update\n", " * sudo apt-get install wkhtmltopdf\n", "\n", "#### Get into your poetry shell with poetry shell from the project's root directory.\n", "\n", "#### Run the script! \n", "* python scripts/download_sec_pdf.py -o ~/mounted_folder --file-types=\"['10-Q','10-K']\"\n", "* Take a 🚽 break while it's running, it'll take a while!\n", "\n", "#### Go to AWS Console and verify you're seeing the SEC files in the S3 bucket." ] }, { "cell_type": "markdown", "id": "7a2b32a0-88b3-4547-b1ec-6ed9d74b938b", "metadata": {}, "source": [ "## Seed DB Script" ] }, { "cell_type": "markdown", "id": "cf33c50a-6320-41ee-b4d3-0b037f8db2f8", "metadata": {}, "source": [ "There are a collection of scripts we have for seeding the database with a set of documents. The script in scripts/seed_db.py is an attempt at consolidating those disparate scripts into one unified command.\n", "\n", "This script will:\n", "\n", "* Download a set of SEC 10-K & 10-Q documents to a local temp directory\n", "* Upload those SEC documents to the S3 folder specified by $S3_ASSET_BUCKET_NAME\n", "* Crawl through all the PDF files in the S3 folder and upsert a database row into the Document table based on the path of the file within the bucket\n", "\n", "#### Use Cases\n", "This is useful for times when:\n", "\n", "* You want to setup a local environment with your local Postgres DB to have a set of documents in the documents table\n", " * When running locally, this will use localstack to store the documents into a local S3 bucket instead of a real one.\n", "\n", "* You want to update the documents present in either Prod or Preview DBs\n", " * In fact, this is the very script that is run by the llama-app-cron cron job service that gets setup by the render.yaml blueprint when deploying this service to Render.com.\n", "\n", "#### Usage\n", "To run the script, make sure you've:\n", "\n", "* Activated your Python virtual environment using poetry shell\n", "* Installed all the pre-requisite dependencies for the SEC Document Downloader script.\n", "* Defined all the environment variables from .env.development within your shell environment according to the environment you want to execute the seed script (e.g. local, preview, prod environments)\n", "\n", "After that you can run python scripts/seed_db.py to start the seed process.\n", "\n", "To make things easier, the Makefile has some shorthand commands.\n", "\n", "* make seed_db\n", " * Just runs the seed_db.py script with no CLI args, so just based on what env vars you've set\n", "\n", "* make seed_db_preview\n", " * Same as make seed_db but only loads SEC documents from Amazon & Meta\n", " * We don't need to load that many company documents for Preview environments.\n", "\n", "* make seed_db_local\n", " * To be used for local database seeding\n", " * Runs seed_db.py just for `$AMZN & $META` documents\n", " * Sets up the localstack bucket to actually serve the documents locally as well, so you can load them in your local browser.\n", "\n", "* make seed_db_based_on_env\n", " * Automatically calls one of the above shorthands based on the RENDER & IS_PREVIEW_ENV environment variables" ] }, { "cell_type": "code", "execution_count": null, "id": "00751b93-68b3-4e0d-8106-3a56f7ef2d27", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 5 }