{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [] }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "id": "2ALPvdsRqsDx" }, "outputs": [], "source": [ "# prompt: get the openai api key from secrets\n", "\n", "import os\n", "from google.colab import userdata\n", "\n", "# Replace 'YOUR_SECRET_NAME' with the actual name of your secret\n", "openai_api_key = userdata.get('OPENAI_API_KEY')\n", "\n", "if openai_api_key:\n", " os.environ['OPENAI_API_KEY'] = openai_api_key\n", "else:\n", " print(\"OpenAI API key not found in secrets.\")\n" ] }, { "cell_type": "code", "source": [ "# prompt: install latest openai\n", "\n", "!pip install --upgrade openai\n" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "QBYBYFAprdkO", "outputId": "64d9a248-0d85-4b49-e648-ea58dfcf835e" }, "execution_count": 2, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Requirement already satisfied: openai in /usr/local/lib/python3.11/dist-packages (1.61.1)\n", "Collecting openai\n", " Downloading openai-1.67.0-py3-none-any.whl.metadata (24 kB)\n", "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.11/dist-packages (from openai) (3.7.1)\n", "Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.11/dist-packages (from openai) (1.9.0)\n", "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.11/dist-packages (from openai) (0.28.1)\n", "Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.11/dist-packages (from openai) (0.9.0)\n", "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.11/dist-packages (from openai) (2.10.6)\n", "Requirement already satisfied: sniffio in /usr/local/lib/python3.11/dist-packages (from openai) (1.3.1)\n", "Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.11/dist-packages (from openai) (4.67.1)\n", "Requirement already satisfied: typing-extensions<5,>=4.11 in /usr/local/lib/python3.11/dist-packages (from openai) (4.12.2)\n", "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.11/dist-packages (from anyio<5,>=3.5.0->openai) (3.10)\n", "Requirement already satisfied: certifi in /usr/local/lib/python3.11/dist-packages (from httpx<1,>=0.23.0->openai) (2025.1.31)\n", "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.11/dist-packages (from httpx<1,>=0.23.0->openai) (1.0.7)\n", "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.11/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.14.0)\n", "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.11/dist-packages (from pydantic<3,>=1.9.0->openai) (0.7.0)\n", "Requirement already satisfied: pydantic-core==2.27.2 in /usr/local/lib/python3.11/dist-packages (from pydantic<3,>=1.9.0->openai) (2.27.2)\n", "Downloading openai-1.67.0-py3-none-any.whl (580 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m580.2/580.2 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hInstalling collected packages: openai\n", " Attempting uninstall: openai\n", " Found existing installation: openai 1.61.1\n", " Uninstalling openai-1.61.1:\n", " Successfully uninstalled openai-1.61.1\n", "Successfully installed openai-1.67.0\n" ] } ] }, { "cell_type": "code", "source": [ "from openai import OpenAI\n", "client = OpenAI(api_key=openai_api_key)\n", "\n", "def run_llm(user_prompt: str, model: str = 'gpt-4o', system_prompt: str = None):\n", " messages = []\n", " if system_prompt:\n", " messages.append({\"role\": \"developer\", \"content\": system_prompt})\n", " messages.append({\"role\": \"user\", \"content\": user_prompt})\n", "\n", " completion = client.chat.completions.create(\n", " model=model,\n", " messages=messages\n", " )\n", " return completion.choices[0].message.content\n", "\n", "run_llm('what is the meaning of life?')" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 125 }, "id": "aTWXBmq-rhWo", "outputId": "c37f2501-22e8-4ee1-df76-9baa7d9a7914" }, "execution_count": 3, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'\"The meaning of life\" is a question that has intrigued humans for centuries, and there is no single answer that satisfies everyone. Different people and cultures have interpreted it in various ways. Here are a few perspectives:\\n\\n1. **Philosophical Perspective**: Philosophers have offered myriad interpretations. Existentialists like Jean-Paul Sartre suggest that life has no inherent meaning and it is up to individuals to create their own purpose. Albert Camus argues for embracing the absurdity of life.\\n\\n2. **Religious Perspective**: Many religious traditions provide their own answers to this question. For instance, Christianity often views the purpose of life as knowing, loving, and serving God. Buddhism sees life as a cycle of suffering and rebirth, with the ultimate goal of reaching enlightenment.\\n\\n3. **Scientific Perspective**: From a scientific standpoint, life can be described as a complex set of chemical reactions and processes. Evolutionarily, the \"purpose\" of life could be seen as survival and replication.\\n\\n4. **Humanistic Perspective**: Humanists might argue that the meaning of life is to improve the well-being of humans and to create a better world for future generations.\\n\\nUltimately, the meaning of life is a deeply personal question. It’s about finding what provides fulfillment, happiness, and a sense of purpose to you as an individual.'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 3 } ] }, { "cell_type": "code", "source": [ "import re\n", "\n", "task = \"\"\"\n", "Write a one-sentence bedtime story about a unicorn, for a five year old girl\n", "\"\"\"\n", "\n", "GENERATOR_PROMPT = \"\"\"\n", "Your goal is to complete the task based on . If there are feedback\n", "from your previous generations, you should reflect on them to improve your solution\n", "\n", "Output your answer concisely in the following format:\n", "\n", "Thoughts:\n", "[Your understanding of the task and feedback and how you plan to improve]\n", "\n", "Response:\n", "[Your response here]\n", "\"\"\"\n", "\n", "def generate(task: str, generator_prompt: str, context: str=\"\") -> tuple[str, str]:\n", " \"\"\"Generate and improve a solution based on feedback.\"\"\"\n", " full_prompt = f\"{generator_prompt}\\n{context}\\nTask: {task}\" if context else f\"{generator_prompt}\\nTask: {task}\"\n", "\n", " response = run_llm(full_prompt)\n", "\n", " print(\"\\n## Generation start\")\n", " print(f\"Output:\\n{response}\\n\")\n", "\n", " return response\n", "\n", "\n", "EVALUATOR_PROMPT = \"\"\"\n", "Evaluate this following response for:\n", "1. age appropriateness\n", "2. is only ten words or fewer\n", "3. style and best practices\n", "\n", "You should be evaluating only and not attempting to solve the task.\n", "\n", "Only output \"PASS\" if all criteria are met and you have no further suggestions for improvements.\n", "\n", "Provide detailed feedback if there are areas that need improvement. You should specify what needs improvement and why.\n", "\n", "Return in this format:\n", "Status: [PASS/FAIL]\n", "Feedback: [Your feedback here]\n", "\"\"\"\n", "\n", "def evaluate(task: str, evaluator_prompt: str, generated_content: str) -> tuple[str, str]:\n", " \"\"\"Evaluate if a solution meets requirements\"\"\"\n", "\n", " full_prompt = f\"{evaluator_prompt}\\nOriginal task: {task}\\nContent to evaluate: {generated_content}\"\n", "\n", " response = run_llm(full_prompt)\n", "\n", " status_match = re.search(r\"Status:\\s*(.*?)(?:\\n|$)\", response, re.IGNORECASE)\n", " feedback_match = re.search(r\"Feedback:\\s*([\\s\\S]*)\", response, re.IGNORECASE)\n", "\n", " if status_match is None or feedback_match is None:\n", " raise ValueError(\"Could not parse evaluation response. Expected format: 'Status: [PASS/FAIL]\\\\nFeedback: [feedback]'\")\n", "\n", " evaluation = status_match.group(1).strip()\n", " feedback = feedback_match.group(1).strip()\n", "\n", " print(\"## Evaluation start\")\n", " print(f\"Status: {evaluation}\")\n", " print(f\"Feedback: {feedback}\")\n", "\n", " return evaluation, feedback\n", "\n", "def loop_workflow(task: str, generator_prompt: str, evaluator_prompt: str, context: str = \"\") -> tuple[str, list[dict]]:\n", " \"\"\"Keep generating and evaluating until the evaluator passes the last generated response.\"\"\"\n", " memory = []\n", "\n", " response = generate(task, generator_prompt)\n", " memory.append(response)\n", "\n", " max_iterations = 5\n", " while max_iterations > 0:\n", " evaluation, feedback = evaluate(task, evaluator_prompt, response)\n", "\n", " if evaluation.upper() == \"PASS\":\n", " return response\n", "\n", " context = \"\\n\".join([\n", " \"Previous attempts:\",\n", " *[f\"- {m}\" for m in memory],\n", " f\"\\nFeedback: {feedback}\"\n", " ])\n", " response = generate(task, generator_prompt, context)\n", " memory.append(response)\n", "\n", " max_iterations -= 1\n", "\n", "loop_workflow(task, GENERATOR_PROMPT, EVALUATOR_PROMPT)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "krOj1m4OsEjq", "outputId": "f754699f-fcb9-43e0-f221-c3f6fd5d2352" }, "execution_count": 5, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "## Generation start\n", "Output:\n", "Thoughts:\n", "The task is to create a simple and imaginative bedtime story suitable for a five-year-old. I will focus on making it whimsical and positive, as that suits the age group. Ensuring it's concise and magical will capture her interest.\n", "\n", "Response:\n", "Once upon a time, a sparkling unicorn named Luna used her magical horn to light up the night sky with a rainbow path, guiding lost stars back home.\n", "\n", "## Evaluation start\n", "Status: FAIL\n", "Feedback: 1. **Age Appropriateness:** The content is appropriate for a five-year-old, as it features whimsical and magical elements like a unicorn and a rainbow, which are typically appealing to young children.\n", "\n", "2. **Is Only Ten Words or Fewer:** The response is not ten words or fewer; it is instead a complex sentence that exceeds this limit. The task specifically required a response of ten words at most.\n", "\n", "3. **Style and Best Practices:** \n", " - The style is imaginative and engaging, aligning well with what captures a young child’s attention at bedtime.\n", " - Best practices for a very short story, as the task dictates, should focus on extreme conciseness. While the sentence is well-written, it does not meet the ten-words-or-fewer requirement.\n", "\n", "To meet the criteria, the response should be rewritten to fit within ten words while maintaining the enchanting and suitable elements for a child’s bedtime story.\n", "\n", "## Generation start\n", "Output:\n", "Thoughts:\n", "The task requires a concise, magical bedtime story about a unicorn suitable for a young child, with clarity and engagement maintained in ten words or fewer. Reflecting on previous feedback, I'll ensure the story adheres strictly to the word limit while retaining whimsical charm.\n", "\n", "Response:\n", "Luna the unicorn danced on rainbows under starry skies.\n", "\n", "Previous attempts:\n", "- Thoughts:\n", "The task is to create a simple and imaginative bedtime story suitable for a five-year-old. I will focus on making it whimsical and positive, as that suits the age group. Ensuring it's concise and magical will capture her interest.\n", "\n", "Response:\n", "Once upon a time, a sparkling unicorn named Luna used her magical horn to light up the night sky with a rainbow path, guiding lost stars back home.\n", "\n", "Feedback: \n", "1. **Age Appropriateness:** The content is appropriate for a five-year-old, as it features whimsical and magical elements like a unicorn and a rainbow, which are typically appealing to young children.\n", "\n", "2. **Is Only Ten Words or Fewer:** The response is not ten words or fewer; it is instead a complex sentence that exceeds this limit. The task specifically required a response of ten words at most.\n", "\n", "3. **Style and Best Practices:** \n", " - The style is imaginative and engaging, aligning well with what captures a young child’s attention at bedtime.\n", " - Best practices for a very short story, as the task dictates, should focus on extreme conciseness. While the sentence is well-written, it does not meet the ten-words-or-fewer requirement.\n", "\n", "To meet the criteria, the response should be rewritten to fit within ten words while maintaining the enchanting and suitable elements for a child’s bedtime story.\n", "\n", "## Evaluation start\n", "Status: FAIL\n", "Feedback: 1. **Age Appropriateness:** The content is age-appropriate and contains magical elements that are likely to appeal to a five-year-old child. A unicorn dancing on rainbows and starry skies creates a whimsical and gentle image perfect for bedtime.\n", "\n", "2. **Is Only Ten Words or Fewer:** The response meets this criterion as it is exactly ten words long.\n", "\n", "3. **Style and Best Practices:** The style of the response is imaginative and engaging, which is ideal for a bedtime story for a young child. However, while it adheres to the word limit, there remains room for improvement in ensuring maximum engagement and clarity within such a brief format. Consider how each word can contribute to the narrative and charm.\n", "\n", "To meet the criteria and potentially improve the story's whimsical impact, it could benefit from refining the wording for even more magic or including an element of action or emotion, if possible, within the constraints.\n", "\n", "## Generation start\n", "Output:\n", "Thoughts:\n", "The task requires crafting a one-sentence bedtime story about a unicorn for a five-year-old girl. I must ensure the story is whimsical, concise (ten words or fewer), and engaging, following previous feedback to enhance magical elements and impact.\n", "\n", "Response:\n", "Luna the unicorn painted rainbows across dreaming clouds, giggling joyfully.\n", "\n", "Previous attempts:\n", "- Thoughts:\n", "The task is to create a simple and imaginative bedtime story suitable for a five-year-old. I will focus on making it whimsical and positive, as that suits the age group. Ensuring it's concise and magical will capture her interest.\n", "\n", "Response:\n", "Once upon a time, a sparkling unicorn named Luna used her magical horn to light up the night sky with a rainbow path, guiding lost stars back home.\n", "- Thoughts:\n", "The task requires a concise, magical bedtime story about a unicorn suitable for a young child, with clarity and engagement maintained in ten words or fewer. Reflecting on previous feedback, I'll ensure the story adheres strictly to the word limit while retaining whimsical charm.\n", "\n", "Response:\n", "Luna the unicorn danced on rainbows under starry skies.\n", "\n", "## Evaluation start\n", "Status: FAIL\n", "Feedback: 1. Age Appropriateness: The sentence \"Luna the unicorn painted rainbows across dreaming clouds, giggling joyfully\" is appropriate for a five-year-old girl. The use of the unicorn and rainbows fits well with themes that are typically engaging for young children.\n", "\n", "2. Word Count: The response contains eleven words, exceeding the specified limit of ten words or fewer. It's essential to adhere to the word count requirement to meet the task criteria.\n", "\n", "3. Style and Best Practices: The sentence is whimsical and has a positive tone, which aligns with the best practices for a bedtime story for a young child. However, in this context, achieving the required word count while maintaining magical and engaging elements is pivotal. \n", "\n", "The main area for improvement is ensuring the response meets the word count criteria while preserving its whimsical charm. Reducing the word count by one word would help achieve a \"PASS\" status.\n", "\n", "## Generation start\n", "Output:\n", "Thoughts:\n", "The task requires a concise, whimsical bedtime story about a unicorn suitable for a five-year-old. Reflecting on feedback, I must ensure the story is exactly ten words or fewer, magical, and charming, focusing on clarity and reducing word count by one.\n", "\n", "Response:\n", "Luna the unicorn painted skies with rainbow joy.\n", "\n", "Previous attempts:\n", "- Thoughts:\n", "The task is to create a simple and imaginative bedtime story suitable for a five-year-old. I will focus on making it whimsical and positive, as that suits the age group. Ensuring it's concise and magical will capture her interest.\n", "\n", "Response:\n", "Once upon a time, a sparkling unicorn named Luna used her magical horn to light up the night sky with a rainbow path, guiding lost stars back home.\n", "- Thoughts:\n", "The task requires a concise, magical bedtime story about a unicorn suitable for a young child, with clarity and engagement maintained in ten words or fewer. Reflecting on previous feedback, I'll ensure the story adheres strictly to the word limit while retaining whimsical charm.\n", "\n", "Response:\n", "Luna the unicorn danced on rainbows under starry skies.\n", "\n", "## Evaluation start\n", "Status: FAIL\n", "Feedback: 1. Age Appropriateness: The content is age-appropriate for a five-year-old. The use of the name \"Luna\" adds a personal and relatable touch, and the imagery of \"painting skies\" with \"rainbow joy\" is whimsical and suitable for a child's imagination.\n", "\n", "2. Word Count: The response is seven words long, meeting the requirement for ten words or fewer.\n", "\n", "3. Style and Best Practices: The sentence is magical and charming, capturing a sense of wonder that is effective for a bedtime story. However, there's a potential for confusion, as \"painting skies with rainbow joy\" might be slightly abstract for a five-year-old without additional context.\n", "\n", "Suggestions: To improve, consider ensuring that the imagery used is more straightforward and vivid in order to make the story more immediately graspable by a young child. For instance, adding a simple action that Luna takes could make the magic feel more tangible.\n", "\n", "## Generation start\n", "Output:\n", "Thoughts:\n", "The task requires crafting a simple, one-sentence bedtime story about a unicorn for a five-year-old, focusing on making it whimsical and age-appropriate. Based on feedback, I'll use straightforward and vivid imagery to make the story immediately understandable and magical within ten words or fewer.\n", "\n", "Response:\n", "Luna the unicorn soared, leaving sparkling rainbows in her wake.\n", "\n", "## Evaluation start\n", "Status: PASS\n", "Feedback: The response is age-appropriate for a five-year-old, keeping the language simple and whimsical. It successfully captures a magical and engaging image using vivid imagery, staying within the ten-word limit. The style aligns with best practices for creating a delightful and imaginative bedtime story for a young child.\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "\"Thoughts:\\nThe task requires crafting a simple, one-sentence bedtime story about a unicorn for a five-year-old, focusing on making it whimsical and age-appropriate. Based on feedback, I'll use straightforward and vivid imagery to make the story immediately understandable and magical within ten words or fewer.\\n\\nResponse:\\nLuna the unicorn soared, leaving sparkling rainbows in her wake.\"" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 5 } ] } ] }