diff --git a/README.md b/README.md index 71ee112..2578c70 100644 --- a/README.md +++ b/README.md @@ -19,9 +19,6 @@ > **PromptWizard: Task-Aware Prompt Optimization Framework**
> Eshaan Agarwal, Joykirat Singh, Vivek Dani, Raghav Magazine, Tanuja Ganu, Akshay Nambi
->**Abstract**:
-> Large language models (LLMs) have transformed AI across diverse domains, with prompting being central to their success in guiding model outputs. However, manual prompt engineering is both labor-intensive and domain-specific, necessitating the need for automated solutions. We introduce PromptWizard, a novel, fully automated framework for discrete prompt optimization, utilizing a self-evolving, self-adapting mechanism. Through a feedback-driven critique and synthesis process, PromptWizard achieves an effective balance between exploration and exploitation, iteratively refining both prompt instructions and in-context examples to generate human-readable, task-specific prompts. This guided approach systematically improves prompt quality, resulting in superior performance across 45 tasks. PromptWizard excels even with limited training data, smaller LLMs, and various LLM architectures. Additionally, our cost analysis reveals a substantial reduction in API calls, token usage, and overall cost, demonstrating PromptWizard's efficiency, scalability, and advantages over existing prompt optimization strategies. - ## Overview 🌟

Overview of the PromptWizard framework

@@ -36,6 +33,14 @@

+PromptWizard is a discrete prompt optimization framework that employs a self-evolving mechanism where the LLM generates, critiques, and refines its own prompts and examples, continuously improving through iterative feedback and synthesis. This self-adaptive approach ensures holistic optimization by evolving both the instructions and in-context learning examples for better task performance. + +Three key components of PromptWizard are te following : + +- Feedback-driven Refinement: LLM generates, critiques, and refines its own prompts and examples, continuously improving through iterative feedback and synthesis​ +- Critique and Synthesize diverse examples: Generates synthetic examples that are robust, diverse and task-aware. Also it optimizes both prompt and examples in tandem​ +- Self generated Chain of Thought (CoT) steps with combination of positive, negative and synthetic examples + ## Installation ⬇️ @@ -66,12 +71,15 @@ Follow these steps to set up the development environment and install the package ## Quickstart 🏃 -- We support [GSM8k](https://huggingface.co/datasets/openai/gsm8k), [SVAMP](https://huggingface.co/datasets/ChilleD/SVAMP), [AQUARAT](https://huggingface.co/datasets/deepmind/aqua_rat) and [Instruction_Induction(BBII)](https://github.com/xqlin98/INSTINCT/tree/main/Induction/experiments/data/instruction_induction/raw) datasets -- Please note that time taken for prompt optimzation is dependent on the dataset. In our experiments for the above mentioned datasets, it took around 20 - 30 minutes on average. +There are three main ways to use PromptWizard: +- Scenario 1 : Optimizing prompts without examples +- Scenario 2 : Generating synthetic examples and using them to optimize prompts +- Scenario 3 : Optimizing prompts with training data + +**NOTE** : Refer this [notebook](demos/scenarios/dataset_scenarios_demo.ipynb) to get a detailed understanding of the usage for each of the scenarios. **This servers as a starting point to understand the usage of PromptWizard** #### High level overview of using PromptWizard -- Load your dataset - - Follow steps mentioned [here](#create-custom-dataset) +- Decide your scenario - Fix the configuration and environmental varibles for API calling - Use ```promptopt_config.yaml``` to set configurations. For example for GSM8k this [file](demos/gsm8k/configs/promptopt_config.yaml) can be used - Use ```.env``` to set environmental varibles. For GSM8k this [file](demos/gsm8k/.env) can be used @@ -88,6 +96,10 @@ Follow these steps to set up the development environment and install the package - Run the code - To run PromptWizard on your custom dataset please jump [here](#run-on-custom-dataset) +#### Running PromptWizard with training data (Scenario 3) +- We support [GSM8k](https://huggingface.co/datasets/openai/gsm8k), [SVAMP](https://huggingface.co/datasets/ChilleD/SVAMP), [AQUARAT](https://huggingface.co/datasets/deepmind/aqua_rat) and [Instruction_Induction(BBII)](https://github.com/xqlin98/INSTINCT/tree/main/Induction/experiments/data/instruction_induction/raw) datasets +- Please note that time taken for prompt optimzation is dependent on the dataset. In our experiments for the above mentioned datasets, it took around 20 - 30 minutes on average. + #### Running on GSM8k (AQUARAT/SVAMP) - Please note that this code requires access to LLMs via API calling, we use AZURE endpoints for this @@ -101,32 +113,6 @@ Follow these steps to set up the development environment and install the package - A demo is presented in [demo.ipynb](demos/bbh/demo.ipynb) -## PromptWizard usage based on data availability 💻 - -Following are the global parameters which can be set based on the availability of the training data - - - ```run_without_train_examples``` is a global hyperparameter which can be used when there are no training samples and in-context examples are not required in the final prompt - - ```use_only_synthetic_examples``` is a global hyperparameter which can be used when there are no training samples but in-context examples are required in the final prompt - - ```use_synthetic_examples``` is a global hyperparameter which can be used to set the type of in-context examples in the final prompt, i.e. it can be synthetic examples or examples from train data or mixture of both. In the next section we provide further details for this scenario - - Scenario based usage is given below, refer this [notebook](demos/dataset_scenarios_demo.ipynb) for the cells to be replaced with the below: - - If there is no training data and in-context examples are not required in the final prompt, then run - ``` - best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=False, - run_without_train_examples=True, - use_only_synthetic_examples=False) - ``` - - If there is no training data and synthetic in-context examples are required in the final prompt, then run - ``` - best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=False, - run_without_train_examples=False, - use_only_synthetic_examples=True) - ``` - - If there is training data, then run - ``` - best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=True, - run_without_train_examples=False, - use_only_synthetic_examples=False) - ``` ## Run on Custom Datasets 🗃️ @@ -179,6 +165,10 @@ NOTE : Refer to [demos](demos) folder for examples of folders for four datasets. - In our experiments we found it to improve the prompt overall as it provides a step-by-step approach to reach the final answer. However if there is a constraint on the prompt length or number of prompt tokens, it can be turned off to get smaller sized prompts - ```generate_expert_identity``` and ```generate_intent_keywords``` : Having these helped improve the prompt as they help making the prompt relevant to the task - Refer ```promptopt_config.yaml``` files in folders present [here](demos) for the descriptions used for AQUARAT, SVAMP and GSM8k. For BBII refer [description.py](demos/bbh/description.py) which has the meta instructions for each of the datasets + - Following are the global parameters which can be set based on the availability of the training data + - ```run_without_train_examples``` is a global hyperparameter which can be used when there are no training samples and in-context examples are not required in the final prompt + - ```generate_synthetic_examples``` is a global hyperparameter which can be used when there are no training samples and we want to generate synthetic data for training + - ```use_examples``` is a global hyperparameter which can be used to optimize prompts using training data 3) Create a dataset specific class which inherits ```class DatasetSpecificProcessing``` similar to ```GSM8k(DatasetSpecificProcessing)``` in [demo.ipynb](demos/gsm8k/demo.ipynb) and define the following functions in it 1) In ```def extract_answer_from_output()``` : This is a dataset specific function, given the ```answer``` from the dataset it should extract and return a consize form of the answer. Note that based on the dataset it can also simply return the ```answer``` as it is like in case of SVAMP and AQUARAT datasets 2) ```def extract_final_answer()``` : This is a LLM output specific function, given the verbose answer from the LLM it should extract and return the consize final answer diff --git a/demos/aquarat/demo.ipynb b/demos/aquarat/demo.ipynb index f5fdcb3..c7a5c9d 100644 --- a/demos/aquarat/demo.ipynb +++ b/demos/aquarat/demo.ipynb @@ -4,6 +4,14 @@ "cell_type": "markdown", "id": "695a1a9b", "metadata": {}, + "source": [ + "#### Set environment variables in [.env](.env) for LLM API calling" + ] + }, + { + "cell_type": "markdown", + "id": "8042a9cc", + "metadata": {}, "source": [ "### Import Dependencies" ] @@ -189,8 +197,8 @@ "metadata": {}, "source": [ "### Call prompt optmization function\n", - "1. ```use_synthetic_examples``` can be used when there are training samples and a mixture of real and synthetic in-context examples are required in the final prompt. When set to ```False``` all the in-context examples will be real\n", - "2. ```use_only_synthetic_examples``` can be used when there are no training samples but in-context examples are required in the final prompt \n", + "1. ```use_examples``` can be used when there are training samples and a mixture of real and synthetic in-context examples are required in the final prompt. When set to ```False``` all the in-context examples will be real\n", + "2. ```generate_synthetic_examples``` can be used when there are no training samples and we want to generate synthetic examples \n", "3. ```run_without_train_examples``` can be used when there are no training samples and in-context examples are not required in the final prompt " ] }, @@ -204,7 +212,7 @@ "outputs": [], "source": [ "# Function call to generate optimal prompt and expert profile \n", - "best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=False,run_without_train_examples=False,use_only_synthetic_examples=False)" + "best_prompt, expert_profile = gp.get_best_prompt(use_examples=True,run_without_train_examples=False,generate_synthetic_examples=False)" ] }, { diff --git a/demos/bbh/demo.ipynb b/demos/bbh/demo.ipynb index ed10119..00017b6 100644 --- a/demos/bbh/demo.ipynb +++ b/demos/bbh/demo.ipynb @@ -4,6 +4,14 @@ "cell_type": "markdown", "id": "ece8514e", "metadata": {}, + "source": [ + "#### Set environment variables in [.env](.env) for LLM API calling" + ] + }, + { + "cell_type": "markdown", + "id": "678ed8db", + "metadata": {}, "source": [ "### Import Dependencies" ] @@ -322,8 +330,8 @@ "metadata": {}, "source": [ "### Call prompt optmization function\n", - "1. ```use_synthetic_examples``` can be used when there are training samples and a mixture of real and synthetic in-context examples are required in the final prompt. When set to ```False``` all the in-context examples will be real\n", - "2. ```use_only_synthetic_examples``` can be used when there are no training samples but in-context examples are required in the final prompt \n", + "1. ```use_examples``` can be used when there are training samples and a mixture of real and synthetic in-context examples are required in the final prompt. When set to ```False``` all the in-context examples will be real\n", + "2. ```generate_synthetic_examples``` can be used when there are no training samples and we want to generate synthetic examples \n", "3. ```run_without_train_examples``` can be used when there are no training samples and in-context examples are not required in the final prompt " ] }, @@ -337,7 +345,7 @@ "outputs": [], "source": [ "# Function call to generate optimal prompt and expert profile \n", - "best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=False,run_without_train_examples=False,use_only_synthetic_examples=False)" + "best_prompt, expert_profile = gp.get_best_prompt(use_examples=True,run_without_train_examples=False,generate_synthetic_examples=False)" ] }, { diff --git a/demos/dataset_scenarios_demo.ipynb b/demos/dataset_scenarios_demo.ipynb deleted file mode 100644 index 761b6ec..0000000 --- a/demos/dataset_scenarios_demo.ipynb +++ /dev/null @@ -1,996 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "6eb94b72", - "metadata": {}, - "source": [ - "### Following is a demo on running PromptWizard under different scenarios of dataset availability" - ] - }, - { - "cell_type": "markdown", - "id": "3cffa5ef", - "metadata": {}, - "source": [ - "### Import Dependencies" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "11efa138", - "metadata": {}, - "outputs": [], - "source": [ - "import sys\n", - "sys.path.insert(0, \"../../\")\n", - "import promptwizard\n", - "from promptwizard.glue.promptopt.instantiate import GluePromptOpt\n", - "from promptwizard.glue.promptopt.techniques.common_logic import DatasetSpecificProcessing\n", - "from promptwizard.glue.common.utils.file import save_jsonlist\n", - "from typing import Any\n", - "from tqdm import tqdm\n", - "from re import compile, findall\n", - "import os\n", - "from datasets import load_dataset\n", - "\n", - "from dotenv import load_dotenv\n", - "load_dotenv(override = True)" - ] - }, - { - "cell_type": "markdown", - "id": "beb14821", - "metadata": {}, - "source": [ - "### Create a dataset specific class and define the required functions " - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "5f325d33", - "metadata": {}, - "outputs": [], - "source": [ - "class GSM8k(DatasetSpecificProcessing):\n", - "\n", - " def dataset_to_jsonl(self, dataset_jsonl: str, **kwargs: Any) -> None:\n", - " def extract_answer_from_output(completion):\n", - " # Your functions for metrics and prompt building\n", - " ans_re = compile(r\"#### (\\-?[0-9\\.\\,]+)\")\n", - " self.INVALID_ANS = \"[invalid]\"\n", - "\n", - " match = ans_re.search(completion)\n", - " if match:\n", - " match_str = match.group(1).strip()\n", - " match_str = match_str.replace(\",\", \"\")\n", - " return match_str\n", - " else:\n", - " return self.INVALID_ANS\n", - "\n", - " examples_set = []\n", - "\n", - " for _, sample in tqdm(enumerate(kwargs[\"dataset\"]), desc=\"Evaluating samples\"):\n", - " example = {\n", - " DatasetSpecificProcessing.QUESTION_LITERAL: sample['question'],\n", - " DatasetSpecificProcessing.ANSWER_WITH_REASON_LITERAL: sample['answer'],\n", - " DatasetSpecificProcessing.FINAL_ANSWER_LITERAL: extract_answer_from_output(sample[\"answer\"])\n", - " }\n", - " examples_set.append(example)\n", - "\n", - " save_jsonlist(dataset_jsonl, examples_set, \"w\")\n", - "\n", - " def extract_final_answer(self, answer: str):\n", - " \n", - " if not answer:\n", - " return self.INVALID_ANS\n", - "\n", - " model_pred = answer.lower()\n", - " preds = model_pred.split(self.ANSWER_START.lower())\n", - " answer_flag = True if len(preds) > 1 else False\n", - "\n", - " pred = preds[-1].replace(\",\", \"\")\n", - " pred = [s for s in findall(r'-?\\d+\\.?\\d*', pred)]\n", - "\n", - " if len(pred) == 0:\n", - " return self.INVALID_ANS\n", - "\n", - " if answer_flag:\n", - " # choose the first element in list\n", - " pred = pred[0]\n", - " else:\n", - " # choose the last element in list\n", - " pred = pred[-1]\n", - "\n", - " # (For arithmetic tasks) if a word ends with period, it will be omitted ...\n", - " if pred[-1] == \".\":\n", - " pred = pred[:-1]\n", - " return pred" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "f384eb57", - "metadata": {}, - "outputs": [], - "source": [ - "gsm8k_processor = GSM8k()" - ] - }, - { - "cell_type": "markdown", - "id": "11d2de75", - "metadata": {}, - "source": [ - "### Load and save the dataset " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "976681bd-4f43-4dbc-947e-cdb94d4824f0", - "metadata": {}, - "outputs": [], - "source": [ - "if not os.path.exists(\"data\"):\n", - " os.mkdir(\"data\")\n", - " \n", - "dataset = load_dataset(\"openai/gsm8k\", \"main\")\n", - "num_samples = 0\n", - "for dataset_type in ['train','test']:\n", - " data_list = []\n", - " for data in dataset[dataset_type]:\n", - " data_list.append({\"question\": data['question'], \"answer\": data['answer']})\n", - " if num_samples == 100 and dataset_type == 'train': # We sample only 100 train examples and use 25 out them for training randomly\n", - " break\n", - " num_samples += 1\n", - " gsm8k_processor.dataset_to_jsonl(\"data/\"+ dataset_type+'.jsonl', dataset=data_list)" - ] - }, - { - "cell_type": "markdown", - "id": "ac30e74f", - "metadata": {}, - "source": [ - "### Set paths" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "f43482f1-3e10-4cf7-8ea6-ff42c04067a6", - "metadata": {}, - "outputs": [], - "source": [ - "train_file_name = os.path.join(\"data\", \"train.jsonl\")\n", - "test_file_name = os.path.join(\"data\", \"test.jsonl\")\n", - "path_to_config = \"configs\"\n", - "promptopt_config_path = os.path.join(path_to_config, \"promptopt_config.yaml\")\n", - "setup_config_path = os.path.join(path_to_config, \"setup_config.yaml\")" - ] - }, - { - "cell_type": "markdown", - "id": "3392594d", - "metadata": {}, - "source": [ - "### Create an object for calling prompt optimization and inference functionalities" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "8af4246f-db32-4b37-a73a-f9e2e5125d09", - "metadata": {}, - "outputs": [], - "source": [ - "gp = GluePromptOpt(promptopt_config_path,\n", - " setup_config_path,\n", - " train_file_name,\n", - " gsm8k_processor)" - ] - }, - { - "cell_type": "markdown", - "id": "1784648c", - "metadata": {}, - "source": [ - "### Call prompt optmization function\n", - "1. ```use_synthetic_examples``` can be used when there are training samples and a mixture of real and synthetic in-context examples are required in the final prompt. When set to ```False``` all the in-context examples will be real\n", - "2. ```use_only_synthetic_examples``` can be used when there are no training samples but in-context examples are required in the final prompt \n", - "3. ```run_without_train_examples``` can be used when there are no training samples and in-context examples are not required in the final prompt " - ] - }, - { - "cell_type": "markdown", - "id": "33f9a557", - "metadata": {}, - "source": [ - "### Now let us consider the three scenarios with respect to availability of training data" - ] - }, - { - "cell_type": "markdown", - "id": "11ce2c44", - "metadata": {}, - "source": [ - "Scenario 1 : We have no training data , but we also don't want in-context examples in final prompt" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "701d8b4b", - "metadata": {}, - "outputs": [], - "source": [ - "best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=False,run_without_train_examples=True,use_only_synthetic_examples=False)" - ] - }, - { - "cell_type": "markdown", - "id": "52e25e8b", - "metadata": {}, - "source": [ - "Output : Several mutated prompts with minor differences are printed on the termial as shown below :" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "e42551ff", - "metadata": {}, - "outputs": [], - "source": [ - "OUTPUT = \"\"\"\n", - "_______________________________________________________________________\n", - "\n", - "Variations 1:\n", - "Expert Profile:\n", - "You are a mathematician with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach mathematical problems methodically, breaking them down into manageable steps and applying appropriate techniques to find solutions. You are familiar with both theoretical and applied mathematics, and you can explain your reasoning and solutions in a clear and concise manner. Your ability to solve mathematical problems efficiently and accurately makes you an invaluable resource for anyone seeking help with mathematics.:\n", - "Prompt:\n", - "You are a mathematics expert. You will be given a mathematics problem which you need to solve\n", - "Lets think step by step.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 2:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach problems methodically, breaking them down into manageable steps and applying appropriate mathematical techniques to find solutions. You are also adept at explaining your reasoning and methods in a clear and concise manner, making it easy for others to follow your thought process. Whether the problem involves solving equations, proving theorems, or analyzing data, you are well-equipped to provide accurate and insightful solutions. Your proficiency in mathematics is complemented by your ability to communicate complex concepts effectively, making you an invaluable resource for tackling challenging mathematical problems.:\n", - "Prompt:\n", - "Let's break this problem down step by step and devise an experiment to help solve it.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 3:\n", - "Expert Profile:\n", - "You are a mathematician with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach mathematical problems methodically, breaking them down into manageable steps and applying appropriate techniques to find solutions. You are familiar with both theoretical and applied mathematics, and you can explain your reasoning and solutions in a clear and concise manner. Your ability to solve mathematical problems efficiently and accurately makes you an invaluable resource for anyone seeking help with mathematics.:\n", - "Prompt:\n", - "Let's list out our ideas for solving this problem and apply them one by one to see if we make any progress.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 4:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down complicated problems into manageable steps, making it easier for others to follow your reasoning and understand the solution. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find accurate solutions. Your ability to explain your thought process clearly and logically makes you an excellent resource for anyone seeking help with mathematics problems. Whether the problem involves basic arithmetic or advanced calculus, you are well-equipped to provide a thorough and correct solution.:\n", - "Prompt:\n", - "Let's measure our progress on this problem by tackling it step by step.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 5:\n", - "Expert Profile:\n", - "You are a mathematician with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach mathematical problems methodically, breaking them down into manageable steps and applying appropriate techniques to find solutions. You are familiar with both theoretical and applied mathematics, and you can explain your reasoning and solutions in a clear and concise manner. Your ability to solve mathematical problems efficiently and accurately makes you an invaluable resource for anyone seeking help with mathematics.:\n", - "Prompt:\n", - "Let's simplify the problem step by step to make it easier to solve.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 6:\n", - "Expert Profile:\n", - "You are a mathematician with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach mathematical problems methodically, breaking them down into manageable steps and applying appropriate techniques to find solutions. You are familiar with both theoretical and applied mathematics, and you can explain your reasoning and solutions in a clear and concise manner. Your ability to solve mathematical problems efficiently and accurately makes you an invaluable resource for anyone seeking help with mathematics.:\n", - "Prompt:\n", - "Let's identify the key assumptions underlying this problem and address them step by step.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 7:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down complicated problems into manageable steps, making it easier for others to follow your reasoning. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find solutions. Whether the problem involves solving equations, proving theorems, or analyzing data, you can provide a clear, accurate, and well-explained solution. Your ability to communicate complex mathematical concepts in an understandable way makes you an invaluable resource for anyone seeking to solve mathematical problems.:\n", - "Prompt:\n", - "Let's break this problem down step by step and devise an experiment to solve it.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 8:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down intricate problems into manageable steps, making it easier for others to follow your reasoning and understand the solution. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find accurate solutions. Your ability to explain mathematical concepts in a clear and concise manner makes you an excellent resource for anyone seeking to solve a mathematics problem. Whether the problem involves basic arithmetic or advanced calculus, you are well-equipped to provide a thorough and correct solution.:\n", - "Prompt:\n", - "Let's list out the key assumptions and simplify the problem step by step.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 9:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down intricate problems into manageable steps, making it easier for others to follow your reasoning and understand the solution. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find accurate solutions. Your ability to explain mathematical concepts in a clear and concise manner makes you an excellent resource for anyone seeking to solve a mathematics problem. Whether the problem involves basic arithmetic or advanced calculus, you are well-equipped to provide a thorough and correct solution.:\n", - "Prompt:\n", - "Let's think through each step methodically and measure our progress as we go.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 10:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down complicated problems into manageable steps, making it easier for others to follow your reasoning and understand the solution. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find accurate solutions. Your ability to explain your thought process clearly and logically makes you an excellent resource for anyone seeking help with mathematics problems. Whether the problem involves basic arithmetic or advanced calculus, you are well-equipped to provide a thorough and correct solution.:\n", - "Prompt:\n", - "Let's identify the key assumptions and tackle the problem step by step.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 11:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down complicated problems into manageable steps, making it easier for others to follow your reasoning. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find solutions. Whether the problem involves solving equations, proving theorems, or analyzing data, you can provide a clear, accurate, and well-explained solution. Your ability to communicate complex mathematical concepts in an understandable way makes you an invaluable resource for anyone seeking help with mathematics.:\n", - "Prompt:\n", - "Let's simplify the problem and solve it step by step, measuring our progress along the way.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 12:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down intricate problems into manageable steps, making it easier for others to follow your reasoning. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find solutions. Whether the problem involves solving equations, proving theorems, or analyzing data, you can provide a clear, accurate, and well-explained solution. Your ability to communicate complex mathematical concepts in an understandable way makes you an invaluable resource for anyone seeking to solve mathematical problems.:\n", - "Prompt:\n", - "Let's break this problem down step by step and devise an experiment to solve it.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 13:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down complicated mathematical problems into manageable steps, making it easier for others to follow your reasoning and understand the solution. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find accurate solutions. Your ability to explain mathematical concepts in a clear and concise manner makes you an excellent resource for anyone seeking help with mathematics problems. Whether it's solving equations, proving theorems, or analyzing data, your proficiency in mathematics ensures that you can provide the correct solution efficiently and effectively.:\n", - "Prompt:\n", - "Let's list out our ideas for solving this problem and apply them one by one to see if we make any progress.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 14:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach mathematical problems methodically, breaking them down into manageable steps and applying appropriate techniques to find solutions. You are familiar with both theoretical and applied mathematics, and you can explain your reasoning and solutions in a clear and concise manner. Your ability to solve mathematical problems efficiently and accurately makes you an invaluable resource for anyone seeking help with mathematics.:\n", - "Prompt:\n", - "Let's measure our progress on this problem by tackling it step by step.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: [Intent]: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\n", - "Variations 15:\n", - "Expert Profile:\n", - "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach problems methodically, breaking them down into manageable steps and applying appropriate mathematical techniques to find solutions. You are also adept at explaining your reasoning and methods in a clear and concise manner, making it easy for others to follow your thought process. Whether the problem involves solving equations, proving theorems, or analyzing data, you are well-equipped to provide accurate and insightful solutions. Your proficiency in mathematics is complemented by your ability to communicate complex concepts effectively, making you an invaluable resource for tackling challenging mathematical problems.:\n", - "Prompt:\n", - "Let's simplify the problem and solve it step by step.\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", - "_______________________________________________________________________\n", - "\"\"\"\n" - ] - }, - { - "cell_type": "markdown", - "id": "3092e1ce", - "metadata": {}, - "source": [ - "Scenario 2 : We have no training data , but we also want in-context examples in final prompt" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "22cb4e37", - "metadata": {}, - "outputs": [], - "source": [ - "best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=False,run_without_train_examples=False,use_only_synthetic_examples=True)" - ] - }, - { - "cell_type": "markdown", - "id": "c64f361d", - "metadata": {}, - "source": [ - "Output : Following Prompt and Expert Profile are generated " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "0f4b1ab1", - "metadata": {}, - "outputs": [], - "source": [ - "OUTPUT = \"\"\"\n", - "Expert Identity: You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach problems methodically, breaking them down into manageable steps and applying appropriate mathematical techniques to find solutions. You are also adept at explaining your reasoning and methods in a clear and concise manner, making it easy for others to follow your thought process. Whether the problem involves solving equations, proving theorems, or analyzing data, you are well-equipped to provide accurate and insightful solutions. Your proficiency in mathematics is complemented by your ability to communicate complex concepts effectively, making you an invaluable resource for tackling challenging mathematical problems.\n", - "Final best prompt: \n", - "Apply the relevant mathematical principles and formulas to solve the problem step by step. Ensure that the final answer is clearly marked within and tags. Verify the final answer for correctness by rechecking each step of the calculation, and ensure that the solutions are correctly identified and presented. Simplify the final answer where possible, and use proper mathematical notation and symbols. Tailor your approach to the specific type of mathematical problem being solved, and ensure that all arithmetic and algebraic operations are accurate.\n", - "\n", - "\n", - "[Question] Solve the quadratic equation \\(3x^2 - 12x + 9 = 0\\).\n", - "[Answer] To solve the quadratic equation \\(3x^2 - 12x + 9 = 0\\), we will follow these steps:\n", - "\n", - "1. **Identify the coefficients**: The given quadratic equation is in the form \\(ax^2 + bx + c = 0\\), where \\(a = 3\\), \\(b = -12\\), and \\(c = 9\\).\n", - "\n", - "2. **Simplify the equation**: Notice that all the coefficients are divisible by 3. We can simplify the equation by dividing every term by 3:\n", - " \\[\n", - " \\frac{3x^2}{3} - \\frac{12x}{3} + \\frac{9}{3} = 0 \\implies x^2 - 4x + 3 = 0\n", - " \\]\n", - "\n", - "3. **Factor the quadratic equation**: We need to find two numbers that multiply to \\(c = 3\\) and add up to \\(b = -4\\). These numbers are -1 and -3. Thus, we can factor the quadratic equation as:\n", - " \\[\n", - " (x - 1)(x - 3) = 0\n", - " \\]\n", - "\n", - "4. **Solve for \\(x\\)**: Set each factor equal to zero and solve for \\(x\\):\n", - " \\[\n", - " x - 1 = 0 \\implies x = 1\n", - " \\]\n", - " \\[\n", - " x - 3 = 0 \\implies x = 3\n", - " \\]\n", - "\n", - "5. **Verify the solutions**: Substitute \\(x = 1\\) and \\(x = 3\\) back into the original equation to ensure they satisfy it:\n", - " \\[\n", - " 3(1)^2 - 12(1) + 9 = 3 - 12 + 9 = 0\n", - " \\]\n", - " \\[\n", - " 3(3)^2 - 12(3) + 9 = 27 - 36 + 9 = 0\n", - " \\]\n", - "\n", - "Both solutions satisfy the original equation.\n", - "\n", - "Therefore, the solutions to the quadratic equation \\(3x^2 - 12x + 9 = 0\\) are:\n", - "1 and 3 3\n", - "\n", - "[Question] Find the area of a triangle with a base of 8 units and a height of 5 units.\n", - "[Answer] To find the area of a triangle, we use the formula:\n", - "\n", - "\\[ \\text{Area} = \\frac{1}{2} \\times \\text{base} \\times \\text{height} \\]\n", - "\n", - "Given:\n", - "- Base (\\( b \\)) = 8 units\n", - "- Height (\\( h \\)) = 5 units\n", - "\n", - "Step-by-step solution:\n", - "\n", - "1. Identify the formula for the area of a triangle:\n", - " \\[ \\text{Area} = \\frac{1}{2} \\times \\text{base} \\times \\text{height} \\]\n", - "\n", - "2. Substitute the given values into the formula:\n", - " \\[ \\text{Area} = \\frac{1}{2} \\times 8 \\times 5 \\]\n", - "\n", - "3. Perform the multiplication inside the formula:\n", - " \\[ \\text{Area} = \\frac{1}{2} \\times 40 \\]\n", - "\n", - "4. Simplify the expression by multiplying by \\(\\frac{1}{2}\\):\n", - " \\[ \\text{Area} = 20 \\]\n", - "\n", - "Therefore, the area of the triangle is:\n", - "\n", - "20 20\n", - "\n", - "[Question] Solve the system of equations:\n", - "\\[\n", - "\\begin{cases}\n", - "3x + 4y = 10 \\\\\n", - "5x - 2y = 3\n", - "\\end{cases}\n", - "\\]\n", - "[Answer] To solve the system of equations:\n", - "\\[\n", - "\\begin{cases}\n", - "3x + 4y = 10 \\\\\n", - "5x - 2y = 3\n", - "\\end{cases}\n", - "\\]\n", - "\n", - "we can use the method of elimination to find the values of \\(x\\) and \\(y\\).\n", - "\n", - "Step 1: Multiply the second equation by 2 to make the coefficients of \\(y\\) in both equations equal in magnitude but opposite in sign:\n", - "\\[\n", - "2(5x - 2y) = 2(3)\n", - "\\]\n", - "This simplifies to:\n", - "\\[\n", - "10x - 4y = 6\n", - "\\]\n", - "\n", - "Step 2: Add the modified second equation to the first equation to eliminate \\(y\\):\n", - "\\[\n", - "(3x + 4y) + (10x - 4y) = 10 + 6\n", - "\\]\n", - "This simplifies to:\n", - "\\[\n", - "13x = 16\n", - "\\]\n", - "\n", - "Step 3: Solve for \\(x\\):\n", - "\\[\n", - "x = \\frac{16}{13}\n", - "\\]\n", - "\n", - "Step 4: Substitute \\(x = \\frac{16}{13}\\) back into the first original equation to solve for \\(y\\):\n", - "\\[\n", - "3\\left(\\frac{16}{13}\\right) + 4y = 10\n", - "\\]\n", - "This simplifies to:\n", - "\\[\n", - "\\frac{48}{13} + 4y = 10\n", - "\\]\n", - "\n", - "Step 5: Isolate \\(4y\\) by subtracting \\(\\frac{48}{13}\\) from both sides:\n", - "\\[\n", - "4y = 10 - \\frac{48}{13}\n", - "\\]\n", - "Convert 10 to a fraction with a denominator of 13:\n", - "\\[\n", - "10 = \\frac{130}{13}\n", - "\\]\n", - "So,\n", - "\\[\n", - "4y = \\frac{130}{13} - \\frac{48}{13}\n", - "\\]\n", - "This simplifies to:\n", - "\\[\n", - "4y = \\frac{82}{13}\n", - "\\]\n", - "\n", - "Step 6: Solve for \\(y\\):\n", - "\\[\n", - "y = \\frac{82}{13 \\cdot 4} = \\frac{82}{52} = \\frac{41}{26}\n", - "\\]\n", - "\n", - "Thus, the solution to the system of equations is:\n", - "\\[\n", - "x = \\frac{16}{13}, \\quad y = \\frac{41}{26}\n", - "\\]\n", - "\n", - "The final answer is:\n", - "\\(\\left( \\frac{16}{13}, \\frac{41}{26} \\right)\\) 16\n", - "\n", - "[Question] Find the derivative of the function \\(f(x) = 4x^4 - 3x^3 + 2x^2 - x + 5\\).\n", - "[Answer] To find the derivative of the function \\(f(x) = 4x^4 - 3x^3 + 2x^2 - x + 5\\), we will apply the power rule of differentiation. The power rule states that if \\(f(x) = ax^n\\), then \\(f'(x) = anx^{n-1}\\).\n", - "\n", - "Let's differentiate each term of the function \\(f(x)\\) step by step:\n", - "\n", - "1. Differentiate \\(4x^4\\):\n", - " \\[\n", - " \\frac{d}{dx}(4x^4) = 4 \\cdot 4x^{4-1} = 16x^3\n", - " \\]\n", - "\n", - "2. Differentiate \\(-3x^3\\):\n", - " \\[\n", - " \\frac{d}{dx}(-3x^3) = -3 \\cdot 3x^{3-1} = -9x^2\n", - " \\]\n", - "\n", - "3. Differentiate \\(2x^2\\):\n", - " \\[\n", - " \\frac{d}{dx}(2x^2) = 2 \\cdot 2x^{2-1} = 4x\n", - " \\]\n", - "\n", - "4. Differentiate \\(-x\\):\n", - " \\[\n", - " \\frac{d}{dx}(-x) = -1 \\cdot x^{1-1} = -1\n", - " \\]\n", - "\n", - "5. Differentiate the constant term \\(5\\):\n", - " \\[\n", - " \\frac{d}{dx}(5) = 0\n", - " \\]\n", - "\n", - "Now, combine all the differentiated terms to get the derivative of the function \\(f(x)\\):\n", - "\\[\n", - "f'(x) = 16x^3 - 9x^2 + 4x - 1\n", - "\\]\n", - "\n", - "Thus, the derivative of the function \\(f(x) = 4x^4 - 3x^3 + 2x^2 - x + 5\\) is:\n", - "\\[\n", - "16x^3 - 9x^2 + 4x - 1\n", - "\\] 16\n", - "\n", - "[Question] Evaluate the definite integral \\(\\int_{1}^{3} (2x^2 - 3x + 1) \\, dx\\).\n", - "[Answer] To evaluate the definite integral \\(\\int_{1}^{3} (2x^2 - 3x + 1) \\, dx\\), we will follow these steps:\n", - "\n", - "1. **Find the antiderivative**: First, we need to find the antiderivative (indefinite integral) of the integrand \\(2x^2 - 3x + 1\\).\n", - "\n", - " \\[\n", - " \\int (2x^2 - 3x + 1) \\, dx\n", - " \\]\n", - "\n", - " We integrate each term separately:\n", - "\n", - " \\[\n", - " \\int 2x^2 \\, dx = \\frac{2x^3}{3}\n", - " \\]\n", - "\n", - " \\[\n", - " \\int -3x \\, dx = -\\frac{3x^2}{2}\n", - " \\]\n", - "\n", - " \\[\n", - " \\int 1 \\, dx = x\n", - " \\]\n", - "\n", - " Therefore, the antiderivative of \\(2x^2 - 3x + 1\\) is:\n", - "\n", - " \\[\n", - " \\frac{2x^3}{3} - \\frac{3x^2}{2} + x + C\n", - " \\]\n", - "\n", - "2. **Evaluate the definite integral**: Now, we need to evaluate this antiderivative at the upper and lower limits of the integral, and then subtract the results.\n", - "\n", - " \\[\n", - " \\left[ \\frac{2x^3}{3} - \\frac{3x^2}{2} + x \\right]_{1}^{3}\n", - " \\]\n", - "\n", - " First, evaluate at the upper limit \\(x = 3\\):\n", - "\n", - " \\[\n", - " \\frac{2(3)^3}{3} - \\frac{3(3)^2}{2} + 3 = \\frac{2 \\cdot 27}{3} - \\frac{3 \\cdot 9}{2} + 3 = 18 - 13.5 + 3 = 7.5\n", - " \\]\n", - "\n", - " Next, evaluate at the lower limit \\(x = 1\\):\n", - "\n", - " \\[\n", - " \\frac{2(1)^3}{3} - \\frac{3(1)^2}{2} + 1 = \\frac{2 \\cdot 1}{3} - \\frac{3 \\cdot 1}{2} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - 1.5 + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\frac{2}{3} - \\frac{3}{2} + 1 = \\frac{2}{3} - \\frac{9}{6} + 1 = \\ 7.3333\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\n", - "\"\"\"" - ] - }, - { - "cell_type": "markdown", - "id": "65a605da", - "metadata": {}, - "source": [ - "Scenario 3 : We have training data and also want in-context examples in final prompt" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "457951b0", - "metadata": {}, - "outputs": [], - "source": [ - "best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=True,run_without_train_examples=False,use_only_synthetic_examples=False)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "696e6612", - "metadata": {}, - "outputs": [], - "source": [ - "OUTPUT = \"\"\"Expert Identity: You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down intricate problems into manageable steps, making it easier for others to follow your reasoning. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find solutions. Whether the problem involves solving equations, proving theorems, or analyzing data, you can provide a clear, accurate, and well-explained solution. Your ability to communicate complex mathematical concepts in an understandable way makes you an invaluable resource for anyone seeking help with mathematics.\n", - "\n", - "Final best prompt: \n", - "\n", - "You are a mathematics expert. Your task is to solve a given mathematics problem accurately and provide a clear, detailed explanation of your solution process. Follow these steps to ensure a comprehensive and well-structured solution:\n", - "\n", - "1. **Understand the Problem**: Carefully read and comprehend the problem statement. Identify the key components and what is being asked.\n", - "\n", - "2. **Identify Components**: Break down the problem into its fundamental components, such as variables, constants, and relevant quantities (e.g., base pay, overtime pay, distances, speeds, etc.).\n", - "\n", - "3. **Apply Relevant Principles**: Use appropriate mathematical principles, formulas, and methods to solve the problem step by step.\n", - "\n", - "4. **Logical Reasoning**: Employ logical reasoning to explain each step of your solution process. Ensure that each step follows logically from the previous one.\n", - "\n", - "5. **Detailed Explanations**: Provide detailed explanations for each step to ensure clarity and understanding. Include intermediate results and how they contribute to the final solution.\n", - "\n", - "6. **Explicit Calculation Steps**: Show each calculation step in detail, including intermediate results. Use proper mathematical notation and symbols.\n", - "\n", - "7. **Verify Each Step**: Recheck each intermediate step of your calculation to verify the correctness of the final answer. Ensure that all arithmetic and algebraic operations are accurate.\n", - "\n", - "8. **Combine Results**: Clearly combine different components of the problem (e.g., base pay and overtime pay) before arriving at the final answer.\n", - "\n", - "9. **Simplify and Notate**: Simplify the final answer where possible, and use proper mathematical notation and symbols.\n", - "\n", - "10. **Mark the Final Answer**: Clearly mark the final answer within and tags.\n", - "\n", - "Ensure that your approach is tailored to the specific type of mathematical problem being solved, whether it involves arithmetic, algebra, geometry, calculus, or any other area of mathematics. Present the solutions in a clear and organized manner.\n", - "\n", - "**Additional Guidelines:**\n", - "- **Contextual Understanding**: Pay close attention to the context of the problem to ensure that all relationships and quantities are correctly interpreted.\n", - "- **Correct Application of Arithmetic Operations**: Double-check that all arithmetic operations are applied correctly and align with the problem's requirements.\n", - "- **Verification of Final Answer**: Dedicate a step to verify the final answer by rechecking all intermediate steps and ensuring they logically lead to the correct final result.\n", - "- **Clarity in Marking Final Answer**: Use the and tags to clearly mark the final answer.\n", - "\n", - "By following these steps and additional guidelines, you will ensure that the solution is accurate, well-explained, and clearly presented.\n", - "\n", - "\n", - "[Question] Bella bought stamps at the post office. Some of the stamps had a snowflake design, some had a truck design, and some had a rose design. Bella bought 11 snowflake stamps. She bought 9 more truck stamps than snowflake stamps, and 13 fewer rose stamps than truck stamps. How many stamps did Bella buy in all?\n", - "[Answer] 1. **Understand the Problem**: Bella bought three types of stamps: snowflake, truck, and rose. We need to determine the total number of stamps she bought, given the relationships between the quantities of each type.\n", - "\n", - "2. **Identify Components**:\n", - " - Number of snowflake stamps: 11.\n", - " - Number of truck stamps: 9 more than the number of snowflake stamps.\n", - " - Number of rose stamps: 13 fewer than the number of truck stamps.\n", - "\n", - "3. **Apply Relevant Principles**: Use basic arithmetic operations to find the quantities of truck and rose stamps, and then sum all the quantities to find the total number of stamps.\n", - "\n", - "4. **Logical Reasoning**:\n", - " - Number of snowflake stamps: 11.\n", - " - Number of truck stamps: 11 (snowflake stamps) + 9 = 20.\n", - " - Number of rose stamps: 20 (truck stamps) - 13 = 7.\n", - "\n", - "5. **Detailed Explanations**:\n", - " - Calculate the number of truck stamps: 11 (snowflake stamps) + 9 = 20.\n", - " - Calculate the number of rose stamps: 20 (truck stamps) - 13 = 7.\n", - " - Calculate the total number of stamps: 11 (snowflake) + 20 (truck) + 7 (rose) = 38.\n", - "\n", - "6. **Explicit Calculation Steps**:\n", - " - Truck stamps: 11 + 9 = $<11+9=20>20.\n", - " - Rose stamps: 20 - 13 = $<20-13=7>7.\n", - " - Total stamps: 11 + 20 + 7 = $<11+20+7=38>38.\n", - "\n", - "7. **Verify Each Step**: Recheck each calculation step to ensure correctness:\n", - " - Truck stamps: 11 + 9 = 20.\n", - " - Rose stamps: 20 - 13 = 7.\n", - " - Total stamps: 11 + 20 + 7 = 38.\n", - "\n", - "8. **Combine Results**: Combine the number of each type of stamp correctly to find the total number of stamps.\n", - "\n", - "9. **Simplify and Notate**: The final answer is already simplified.\n", - "\n", - "10. **Mark the Final Answer**: 38\n", - "\n", - "By following these steps, we ensure that the solution is accurate, well-explained, and clearly presented. 38\n", - "\n", - "[Question] It takes Roque two hours to walk to work and one hour to ride his bike to work. Roque walks to and from work three times a week and rides his bike to and from work twice a week. How many hours in total does he take to get to and from work a week with walking and biking?\n", - "[Answer] 1. **Understand the Problem**: Roque has two modes of transportation to work: walking and biking. We need to calculate the total time he spends traveling to and from work in a week, considering the different times and frequencies for each mode.\n", - "\n", - "2. **Identify Components**:\n", - " - Time to walk to work: 2 hours (one way).\n", - " - Time to bike to work: 1 hour (one way).\n", - " - Frequency of walking: 3 times a week (to and from work).\n", - " - Frequency of biking: 2 times a week (to and from work).\n", - "\n", - "3. **Apply Relevant Principles**: Use basic arithmetic to calculate the total time spent walking and biking separately, then sum these times to get the total weekly travel time.\n", - "\n", - "4. **Logical Reasoning**:\n", - " - Calculate the total walking time for a week:\n", - " - One round trip (to and from work) by walking takes 2 hours (to work) + 2 hours (from work) = 4 hours.\n", - " - Roque walks to and from work 3 times a week, so the total walking time is 4 hours per round trip * 3 round trips = 12 hours.\n", - " - Calculate the total biking time for a week:\n", - " - One round trip (to and from work) by biking takes 1 hour (to work) + 1 hour (from work) = 2 hours.\n", - " - Roque bikes to and from work 2 times a week, so the total biking time is 2 hours per round trip * 2 round trips = 4 hours.\n", - "\n", - "5. **Detailed Explanations**:\n", - " - Walking time calculation:\n", - " - One round trip walking: 2 hours (to work) + 2 hours (from work) = 4 hours.\n", - " - Total walking time for the week: 4 hours per round trip * 3 round trips = 12 hours.\n", - " - Biking time calculation:\n", - " - One round trip biking: 1 hour (to work) + 1 hour (from work) = 2 hours.\n", - " - Total biking time for the week: 2 hours per round trip * 2 round trips = 4 hours.\n", - " - Combine the total walking and biking times to get the total weekly travel time:\n", - " - Total weekly travel time: 12 hours (walking) + 4 hours (biking) = 16 hours.\n", - "\n", - "6. **Explicit Calculation Steps**:\n", - " - Walking time: 2 hours (one way) * 2 (round trip) * 3 (times a week) = $<2*2*3=12>12 hours.\n", - " - Biking time: 1 hour (one way) * 2 (round trip) * 2 (times a week) = $<1*2*2=4>4 hours.\n", - " - Total time: 12 hours (walking) + 4 hours (biking) = $<12+4=16>16 hours.\n", - "\n", - "7. **Verify Each Step**: Recheck each calculation step to ensure correctness. Confirm that the arithmetic operations and logic used are accurate.\n", - "\n", - "8. **Combine Results**: Combine the total walking and biking times correctly to ensure the final answer is accurate.\n", - "\n", - "9. **Simplify and Notate**: The final answer is already simplified and clearly presented.\n", - "\n", - "10. **Mark the Final Answer**: 16\n", - "\n", - "By following these steps, we ensure that the solution is accurate, well-explained, and clearly presented. 16\n", - "\n", - "[Question] Betty is saving money for a new wallet which costs $100. Betty has only half of the money she needs. Her parents decided to give her $15 for that purpose, and her grandparents twice as much as her parents. How much more money does Betty need to buy the wallet?\n", - "[Answer] 1. **Understand the Problem**: Betty is saving money for a wallet that costs $100. She currently has half of the money she needs. Her parents and grandparents are contributing additional amounts to help her reach her goal. We need to determine how much more money Betty needs to buy the wallet.\n", - "\n", - "2. **Identify Components**:\n", - " - Total cost of the wallet: $100.\n", - " - Amount Betty currently has: half of $100.\n", - " - Contribution from parents: $15.\n", - " - Contribution from grandparents: twice the amount given by parents.\n", - "\n", - "3. **Apply Relevant Principles**: Use basic arithmetic to calculate the total amount of money Betty will have after receiving contributions from her parents and grandparents, and then determine how much more she needs to reach $100.\n", - "\n", - "4. **Logical Reasoning**:\n", - " - Calculate the amount Betty currently has: $100 / 2 = $50.\n", - " - Calculate the contribution from grandparents: 2 * $15 = $30.\n", - " - Calculate the total amount of money Betty will have: $50 (current amount) + $15 (parents' contribution) + $30 (grandparents' contribution).\n", - "\n", - "5. **Detailed Explanations**:\n", - " - Betty currently has $50 because half of $100 is $50.\n", - " - Her parents give her $15.\n", - " - Her grandparents give her twice the amount her parents give, which is 2 * $15 = $30.\n", - " - The total amount of money Betty will have is $50 (current amount) + $15 (parents' contribution) + $30 (grandparents' contribution) = $95.\n", - "\n", - "6. **Explicit Calculation Steps**:\n", - " - Current amount: $100 / 2 = $<100/2=50>50.\n", - " - Grandparents' contribution: 2 * $15 = $<2*15=30>30.\n", - " - Total amount: $50 + $15 + $30 = $<50+15+30=95>95.\n", - "\n", - "7. **Verify Each Step**: Recheck each calculation step to ensure correctness.\n", - " - Current amount: $100 / 2 = $50.\n", - " - Grandparents' contribution: 2 * $15 = $30.\n", - " - Total amount: $50 + $15 + $30 = $95.\n", - "\n", - "8. **Combine Results**: Combine the total amount of money Betty will have correctly.\n", - " - Total amount: $50 (current amount) + $15 (parents' contribution) + $30 (grandparents' contribution) = $95.\n", - "\n", - "9. **Simplify and Notate**: The final answer is already simplified.\n", - "\n", - "10. **Mark the Final Answer**: \n", - " - Amount Betty needs to buy the wallet: $100 - $95 = $<100-95=5>5.\n", - "\n", - "5 5\n", - "\n", - "[Question] A rectangle has a length of 10 cm and a width of 5 cm. What is the area and perimeter of the rectangle?\n", - "[Answer] 1. **Understand the Problem**: We need to find both the area and the perimeter of a rectangle given its length and width.\n", - "\n", - "2. **Identify Components**: \n", - " - Length of the rectangle (L) = 10 cm\n", - " - Width of the rectangle (W) = 5 cm\n", - "\n", - "3. **Apply Relevant Principles**: \n", - " - The formula for the area of a rectangle is \\( \\text{Area} = \\text{Length} \\times \\text{Width} \\).\n", - " - The formula for the perimeter of a rectangle is \\( \\text{Perimeter} = 2 \\times (\\text{Length} + \\text{Width}) \\).\n", - "\n", - "4. **Logical Reasoning**:\n", - " - To find the area, multiply the length by the width.\n", - " - To find the perimeter, add the length and the width, then multiply the result by 2.\n", - "\n", - "5. **Detailed Explanations**:\n", - " - Calculate the area: \\( \\text{Area} = 10 \\, \\text{cm} \\times 5 \\, \\text{cm} \\).\n", - " - Calculate the perimeter: \\( \\text{Perimeter} = 2 \\times (10 \\, \\text{cm} + 5 \\, \\text{cm}) \\).\n", - "\n", - "6. **Explicit Calculation Steps**:\n", - " - Area: \\( 10 \\times 5 = 50 \\, \\text{cm}^2 \\).\n", - " - Perimeter: \\( 2 \\times (10 + 5) = 2 \\times 15 = 30 \\, \\text{cm} \\).\n", - "\n", - "7. **Verify Each Step**: \n", - " - Recheck the area calculation: \\( 10 \\times 5 = 50 \\, \\text{cm}^2 \\).\n", - " - Recheck the perimeter calculation: \\( 2 \\times 15 = 30 \\, \\text{cm} \\).\n", - "\n", - "8. **Combine Results**: \n", - " - The area of the rectangle is \\( 50 \\, \\text{cm}^2 \\).\n", - " - The perimeter of the rectangle is \\( 30 \\, \\text{cm} \\).\n", - "\n", - "9. **Simplify and Notate**: \n", - " - The final answers are already simplified.\n", - "\n", - "10. **Mark the Final Answer**: \n", - " - Area: 50 \\, \\text{cm}^2\n", - " - Perimeter: 30 \\, \\text{cm}\n", - "\n", - "By following these steps, we ensure that the solution is accurate, well-explained, and clearly presented. 50\n", - "\n", - "[Question] Solve for x in the equation 2x + 3 = 11.\n", - "[Answer] **Understand the Problem**: We need to solve for the variable \\( x \\) in the given linear equation \\( 2x + 3 = 11 \\).\n", - "\n", - "**Identify Components**: \n", - "- The equation is \\( 2x + 3 = 11 \\).\n", - "- We need to isolate \\( x \\) on one side of the equation.\n", - "\n", - "**Apply Relevant Principles**: \n", - "- Use basic algebraic principles to isolate \\( x \\).\n", - "\n", - "**Logical Reasoning**:\n", - "1. Start with the given equation: \\( 2x + 3 = 11 \\).\n", - "2. Subtract 3 from both sides of the equation to isolate the term with \\( x \\):\n", - " \\[\n", - " 2x + 3 - 3 = 11 - 3\n", - " \\]\n", - "3. Simplify both sides:\n", - " \\[\n", - " 2x = 8\n", - " \\]\n", - "4. Divide both sides by 2 to solve for \\( x \\):\n", - " \\[\n", - " \\frac{2x}{2} = \\frac{8}{2}\n", - " \\]\n", - "5. Simplify the division:\n", - " \\[\n", - " x = 4\n", - " \\]\n", - "\n", - "**Detailed Explanations**:\n", - "- Subtracting 3 from both sides removes the constant term on the left side, leaving \\( 2x \\) isolated.\n", - "- Dividing both sides by 2 isolates \\( x \\) by removing the coefficient of 2.\n", - "\n", - "**Explicit Calculation Steps**:\n", - "1. \\( 2x + 3 = 11 \\)\n", - "2. \\( 2x + 3 - 3 = 11 - 3 \\)\n", - "3. \\( 2x = 8 \\)\n", - "4. \\( \\frac{2x}{2} = \\frac{8}{2} \\)\n", - "5. \\( x = 4 \\)\n", - "\n", - "**Verify Each Step**:\n", - "- Recheck each step to ensure no arithmetic errors:\n", - " - Subtracting 3 from 11 gives 8.\n", - " - Dividing 8 by 2 gives 4.\n", - "\n", - "**Combine Results**: The final value of \\( x \\) is correctly isolated and calculated.\n", - "\n", - "**Simplify and Notate**: The final answer is already simplified.\n", - "\n", - "**Mark the Final Answer**: 4\n", - "\n", - "By following these steps, we ensure that the solution is accurate, well-explained, and clearly presented. 4\n", - "\n", - "\n", - "For each question present the reasoning followed by the correct answer.\"\"\"" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "general", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.4" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/demos/gsm8k/demo.ipynb b/demos/gsm8k/demo.ipynb index 97a3c85..e979ee7 100644 --- a/demos/gsm8k/demo.ipynb +++ b/demos/gsm8k/demo.ipynb @@ -4,6 +4,14 @@ "cell_type": "markdown", "id": "6eb94b72", "metadata": {}, + "source": [ + "#### Set environment variables in [.env](.env) for LLM API calling" + ] + }, + { + "cell_type": "markdown", + "id": "388020c6", + "metadata": {}, "source": [ "### Import Dependencies" ] @@ -191,8 +199,8 @@ "metadata": {}, "source": [ "### Call prompt optmization function\n", - "1. ```use_synthetic_examples``` can be used when there are training samples and a mixture of real and synthetic in-context examples are required in the final prompt. When set to ```False``` all the in-context examples will be real\n", - "2. ```use_only_synthetic_examples``` can be used when there are no training samples but in-context examples are required in the final prompt \n", + "1. ```use_examples``` can be used when there are training samples and a mixture of real and synthetic in-context examples are required in the final prompt. When set to ```False``` all the in-context examples will be real\n", + "2. ```generate_synthetic_examples``` can be used when there are no training samples and we want to generate synthetic examples \n", "3. ```run_without_train_examples``` can be used when there are no training samples and in-context examples are not required in the final prompt " ] }, @@ -206,7 +214,7 @@ "outputs": [], "source": [ "# Function call to generate optimal prompt and expert profile \n", - "best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=False,run_without_train_examples=False,use_only_synthetic_examples=False)" + "best_prompt, expert_profile = gp.get_best_prompt(use_examples=True,run_without_train_examples=False,generate_synthetic_examples=False)" ] }, { diff --git a/demos/scenarios/.env b/demos/scenarios/.env new file mode 100644 index 0000000..0ddcaf3 --- /dev/null +++ b/demos/scenarios/.env @@ -0,0 +1,3 @@ +OPENAI_API_VERSION="2024-02-15-preview" +AZURE_OPENAI_ENDPOINT="https://vellm-openai3.openai.azure.com/" +AZURE_OPENAI_DEPLOYMENT_NAME="gpt-4o" \ No newline at end of file diff --git a/demos/scenarios/configs/prompt_library.yaml b/demos/scenarios/configs/prompt_library.yaml new file mode 100644 index 0000000..81a2a5f --- /dev/null +++ b/demos/scenarios/configs/prompt_library.yaml @@ -0,0 +1,36 @@ +system_prompts: | + You are a helpful assistant that assists research students in understanding research papers. +system_guidelines: | + Guidelines + - Your role must always be a helpful assistant that assists students in understanding research papers. + - Only answer questions that are directly or indirectly related to the referenced paper(s). + +mode: + chat: + - name: CHAT-FIRST-MESSAGE + llm_request_type: rag-query + prompt_template: | + {user_msg} + emb_model_id: text embedding ada 002 [vellm-openai2] + llm_model_id: gpt 35 Turbo [vellm-openai2] + prepend_system_prompts: False + prepend_system_guidelines: False + + - name: CHAT-NEXT-MESSAGES + llm_request_type: rag-query + prompt_template: | + {user_msg} + emb_model_id: text embedding ada 002 [vellm-openai2] + llm_model_id: gpt 35 Turbo [vellm-openai2] + prepend_system_prompts: False + prepend_system_guidelines: False + + generation: + - name: FLASH_PROFILE + prompt_template: | + {user_msg} + prepend_system_prompts: False + prepend_system_guidelines: False + llm_request_type: rag-query + emb_model_id: text embedding ada 002 [vellm-openai2] + llm_model_id: gpt 35 Turbo [vellm-openai2] \ No newline at end of file diff --git a/demos/scenarios/configs/promptopt_config.yaml b/demos/scenarios/configs/promptopt_config.yaml new file mode 100644 index 0000000..eacaf5f --- /dev/null +++ b/demos/scenarios/configs/promptopt_config.yaml @@ -0,0 +1,21 @@ +answer_format: For each question present the reasoning followed by the correct answer. +base_instruction: Lets think step by step. +few_shot_count: 5 +generate_expert_identity: true +generate_intent_keywords: true +generate_reasoning: true +max_eval_batches: 6 +min_correct_count: 3 +mutate_refine_iterations: 3 +mutation_rounds: 2 +num_train_examples: 40 +prompt_technique_name: critique_n_refine +questions_batch_size: 1 +refine_instruction: true +refine_task_eg_iterations: 3 +seen_set_size: 20 +style_variation: 5 +task_description: You are a mathematics expert. You will be given a mathematics problem + which you need to solve +top_n: 1 +unique_model_id: gpt-4o diff --git a/demos/scenarios/configs/setup_config.yaml b/demos/scenarios/configs/setup_config.yaml new file mode 100644 index 0000000..79773bf --- /dev/null +++ b/demos/scenarios/configs/setup_config.yaml @@ -0,0 +1,14 @@ +assistant_llm: + # put the unique_model_id that you specified in llm_config.yaml + prompt_opt: gpt-4o +dir_info: + # Base directory for everything + base_dir: logs + log_dir_name: glue_logs +experiment_name: gsm8k +# Many features are different for mode: online/offline. For eg +# 1) Print of logs happens on console for offline mode +# 2) LLM Queue gets instantiated only in online mode +mode: offline +# Full length description of the experiment. This would be logged. +description: diff --git a/demos/scenarios/dataset_scenarios_demo.ipynb b/demos/scenarios/dataset_scenarios_demo.ipynb new file mode 100644 index 0000000..e918489 --- /dev/null +++ b/demos/scenarios/dataset_scenarios_demo.ipynb @@ -0,0 +1,1119 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "6eb94b72", + "metadata": {}, + "source": [ + "## Following is a demo on running PromptWizard under different scenarios " + ] + }, + { + "cell_type": "markdown", + "id": "52c7ee0a", + "metadata": {}, + "source": [ + "#### Set environment variables in [.env](.env) for LLM API calling" + ] + }, + { + "cell_type": "markdown", + "id": "3cffa5ef", + "metadata": {}, + "source": [ + "#### Import Dependencies" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "11efa138", + "metadata": {}, + "outputs": [], + "source": [ + "import sys\n", + "sys.path.insert(0, \"../../\")\n", + "import promptwizard\n", + "from promptwizard.glue.promptopt.instantiate import GluePromptOpt\n", + "from promptwizard.glue.promptopt.techniques.common_logic import DatasetSpecificProcessing\n", + "from promptwizard.glue.common.utils.file import save_jsonlist\n", + "from typing import Any\n", + "from tqdm import tqdm\n", + "from re import compile, findall\n", + "import os\n", + "from datasets import load_dataset\n", + "import yaml\n", + "from dotenv import load_dotenv\n", + "load_dotenv(override = True)" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "9be22d5d", + "metadata": {}, + "outputs": [], + "source": [ + "def update_yaml_file(file_path,config_dict):\n", + "\n", + " with open(file_path, 'r') as file:\n", + " data = yaml.safe_load(file)\n", + "\n", + "\n", + " for field,value in config_dict.items():\n", + " data[field] = value\n", + "\n", + " with open(file_path, 'w') as file:\n", + " yaml.dump(data, file, default_flow_style=False)\n", + "\n", + " print(\"YAML file updated successfully!\")" + ] + }, + { + "cell_type": "markdown", + "id": "78abb34a", + "metadata": {}, + "source": [ + "Set the paths" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "14399d47", + "metadata": {}, + "outputs": [], + "source": [ + "path_to_config = \"configs\"\n", + "promptopt_config_path = os.path.join(path_to_config, \"promptopt_config.yaml\")\n", + "setup_config_path = os.path.join(path_to_config, \"setup_config.yaml\")" + ] + }, + { + "cell_type": "markdown", + "id": "0f274af9", + "metadata": {}, + "source": [ + "### Now let us consider the three scenarios with respect to availability of training data" + ] + }, + { + "cell_type": "markdown", + "id": "5aaed236", + "metadata": {}, + "source": [ + "#### Scenario 1 : We have no training data , but we also don't want in-context examples in final prompt" + ] + }, + { + "cell_type": "markdown", + "id": "4c34423d", + "metadata": {}, + "source": [ + "Set the configurations" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ec4e7607", + "metadata": {}, + "outputs": [], + "source": [ + "file_path = 'configs/promptopt_config.yaml' \n", + "# Set the following based on the use case\n", + "config_dict = {\n", + " \"task_description\": \"You are a mathematics expert. You will be given a mathematics problem which you need to solve\",\n", + " \"base_instruction\": \"Lets think step by step.\",\n", + " \"mutation_rounds\": 2\n", + " }\n", + "update_yaml_file(file_path,config_dict)" + ] + }, + { + "cell_type": "markdown", + "id": "d984e84e", + "metadata": {}, + "source": [ + "Create an object for calling prompt optimization and inference functionalities" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c7aa4ccb", + "metadata": {}, + "outputs": [], + "source": [ + "gp = GluePromptOpt(promptopt_config_path,\n", + " setup_config_path,\n", + " dataset_jsonl=None,\n", + " data_processor=None)" + ] + }, + { + "cell_type": "markdown", + "id": "8d587065", + "metadata": {}, + "source": [ + "Call the optimization function" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "afe8de4f", + "metadata": {}, + "outputs": [], + "source": [ + "best_prompt, expert_profile = gp.get_best_prompt(use_examples=False,run_without_train_examples=True,generate_synthetic_examples=False)" + ] + }, + { + "cell_type": "markdown", + "id": "a30db274", + "metadata": {}, + "source": [ + "Output : Five mutated prompts are printed on the termial as shown below :" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "e5cb1a65", + "metadata": {}, + "outputs": [], + "source": [ + "OUTPUT = \"\"\"\n", + "Variations 1:\n", + "Expert Profile:\n", + "You are a mathematician with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach mathematical problems methodically, breaking them down into manageable steps and applying appropriate techniques to find solutions. You are familiar with both theoretical and applied mathematics, and you can explain your reasoning and solutions in a clear and concise manner. Your ability to solve mathematical problems efficiently and accurately makes you an invaluable resource for anyone seeking help with mathematics.:\n", + "Prompt:\n", + "You are a mathematics expert. You will be given a mathematics problem which you need to solve\n", + "Lets think step by step.\n", + "\n", + "\n", + "For each question present the reasoning followed by the correct answer.\n", + "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", + "_______________________________________________________________________\n", + "\n", + "Variations 2:\n", + "Expert Profile:\n", + "You are a mathematician with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to approach mathematical problems methodically, breaking them down into manageable steps and applying appropriate techniques to find solutions. You are familiar with both theoretical and applied mathematics, and you can explain your reasoning and solutions in a clear and concise manner. Your ability to solve mathematical problems efficiently and accurately makes you an invaluable resource for anyone seeking help with mathematics.:\n", + "Prompt:\n", + "Let's break this problem down step by step and devise an experiment to help solve it.\n", + "\n", + "\n", + "For each question present the reasoning followed by the correct answer.\n", + "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", + "_______________________________________________________________________\n", + "\n", + "Variations 3:\n", + "Expert Profile:\n", + "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down intricate problems into manageable steps, making it easier to find solutions. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to solve problems. Whether the problem involves solving equations, proving theorems, or analyzing data, you can provide a clear and accurate solution. Your ability to explain your reasoning and methodology ensures that others can follow and understand your approach, making you an invaluable resource for tackling challenging mathematical problems.:\n", + "Prompt:\n", + "Let's think through this problem step by step and make a list of ideas to solve it.\n", + "\n", + "\n", + "For each question present the reasoning followed by the correct answer.\n", + "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", + "_______________________________________________________________________\n", + "\n", + "Variations 4:\n", + "Expert Profile:\n", + "You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down intricate problems into manageable steps, making it easier for others to follow your reasoning. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find solutions. Whether the problem involves solving equations, proving theorems, or analyzing data, you can provide a clear, accurate, and well-explained solution. Your ability to communicate complex mathematical concepts in an understandable way makes you an invaluable resource for anyone seeking to solve mathematical problems.:\n", + "Prompt:\n", + "Let's approach this problem step by step and measure our progress as we go.\n", + "\n", + "\n", + "For each question present the reasoning followed by the correct answer.\n", + "Keywords: mathematics, problem-solving, step-by-step, logical reasoning, expert\n", + "Iterations completed: 0%| | 0/3 [00:24 None:\n", + " def extract_answer_from_output(completion):\n", + " # Your functions for metrics and prompt building\n", + " ans_re = compile(r\"#### (\\-?[0-9\\.\\,]+)\")\n", + " self.INVALID_ANS = \"[invalid]\"\n", + "\n", + " match = ans_re.search(completion)\n", + " if match:\n", + " match_str = match.group(1).strip()\n", + " match_str = match_str.replace(\",\", \"\")\n", + " return match_str\n", + " else:\n", + " return self.INVALID_ANS\n", + "\n", + " examples_set = []\n", + "\n", + " for _, sample in tqdm(enumerate(kwargs[\"dataset\"]), desc=\"Evaluating samples\"):\n", + " example = {\n", + " DatasetSpecificProcessing.QUESTION_LITERAL: sample['question'],\n", + " DatasetSpecificProcessing.ANSWER_WITH_REASON_LITERAL: sample['answer'],\n", + " DatasetSpecificProcessing.FINAL_ANSWER_LITERAL: extract_answer_from_output(sample[\"answer\"])\n", + " }\n", + " examples_set.append(example)\n", + "\n", + " save_jsonlist(dataset_jsonl, examples_set, \"w\")\n", + "\n", + " def extract_final_answer(self, answer: str):\n", + " \n", + " if not answer:\n", + " return self.INVALID_ANS\n", + "\n", + " model_pred = answer.lower()\n", + " preds = model_pred.split(self.ANSWER_START.lower())\n", + " answer_flag = True if len(preds) > 1 else False\n", + "\n", + " pred = preds[-1].replace(\",\", \"\")\n", + " pred = [s for s in findall(r'-?\\d+\\.?\\d*', pred)]\n", + "\n", + " if len(pred) == 0:\n", + " return self.INVALID_ANS\n", + "\n", + " if answer_flag:\n", + " # choose the first element in list\n", + " pred = pred[0]\n", + " else:\n", + " # choose the last element in list\n", + " pred = pred[-1]\n", + "\n", + " # (For arithmetic tasks) if a word ends with period, it will be omitted ...\n", + " if pred[-1] == \".\":\n", + " pred = pred[:-1]\n", + " return pred" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "212bea42", + "metadata": {}, + "outputs": [], + "source": [ + "gsm8k_processor = GSM8k()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "67db60b0", + "metadata": {}, + "outputs": [], + "source": [ + "file_path = 'configs/promptopt_config.yaml' \n", + "config_dict = {\n", + " \"task_description\": \"You are a mathematics expert. You will be given a mathematics problem which you need to solve\",\n", + " \"base_instruction\": \"Lets think step by step.\",\n", + " \"mutation_rounds\": 2,\n", + " \"few_shot_count\": 5,\n", + " \"generate_reasoning\": True,\n", + " \"mutate_refine_iterations\" : 3,\n", + " \"seen_set_size\":20\n", + " }\n", + "update_yaml_file(file_path,config_dict)" + ] + }, + { + "cell_type": "markdown", + "id": "fc8eb2c5", + "metadata": {}, + "source": [ + "Call the optimization function " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e53934e6", + "metadata": {}, + "outputs": [], + "source": [ + "gp = GluePromptOpt(promptopt_config_path,\n", + " setup_config_path,\n", + " dataset_jsonl = \"train_synthetic.jsonl\",\n", + " data_processor=gsm8k_processor)\n", + "best_prompt, expert_profile = gp.get_best_prompt(use_examples=True,run_without_train_examples=False,generate_synthetic_examples=False)" + ] + }, + { + "cell_type": "markdown", + "id": "b4bcd46b", + "metadata": {}, + "source": [ + "Output : Following Prompt and Expert Profile are generated " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ee6006f0", + "metadata": {}, + "outputs": [], + "source": [ + "OUTPUT = \"\"\"\n", + "Generating Expert Identity....\n", + "Expert Identity: You are a mathematician with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your analytical skills and logical reasoning enable you to break down problems into manageable steps and find accurate solutions efficiently. You are familiar with a wide range of mathematical techniques and tools, and you can apply them to solve problems in both theoretical and applied contexts. Your expertise allows you to explain your solutions clearly and concisely, making complex concepts accessible to others. Whether the problem involves solving equations, proving theorems, or analyzing data, you are well-equipped to provide a thorough and correct solution.\n", + "Final best prompt: Provide a clear and detailed solution, breaking down all necessary steps. Ensure that the final answer is clearly marked and separated from the solution steps. Use proper mathematical notation and formatting throughout. Verify the final answer by checking the solution steps for accuracy. Simplify all expressions and fractions where possible. Handle special cases or edge cases appropriately, and clearly state any assumptions or conditions applied during the solution process. Finally, review the entire solution to ensure logical consistency and correct formatting.\n", + "\n", + "[Question] Solve for \\( x \\) in the equation \\( 2x + 3 = 11 \\).\n", + "[Answer] To solve for \\( x \\) in the equation \\( 2x + 3 = 11 \\), we will follow these steps:\n", + "\n", + "1. **Isolate the term containing \\( x \\)**:\n", + " We start by isolating the term with \\( x \\) on one side of the equation. To do this, we need to eliminate the constant term on the left side of the equation.\n", + "\n", + " \\[\n", + " 2x + 3 = 11\n", + " \\]\n", + "\n", + " Subtract 3 from both sides of the equation:\n", + "\n", + " \\[\n", + " 2x + 3 - 3 = 11 - 3\n", + " \\]\n", + "\n", + " Simplifying this, we get:\n", + "\n", + " \\[\n", + " 2x = 8\n", + " \\]\n", + "\n", + "2. **Solve for \\( x \\)**:\n", + " Now, we need to solve for \\( x \\) by isolating \\( x \\) itself. Since \\( x \\) is multiplied by 2, we will divide both sides of the equation by 2 to solve for \\( x \\).\n", + "\n", + " \\[\n", + " \\frac{2x}{2} = \\frac{8}{2}\n", + " \\]\n", + "\n", + " Simplifying this, we get:\n", + "\n", + " \\[\n", + " x = 4\n", + " \\]\n", + "\n", + "3. **Verify the solution**:\n", + " To ensure our solution is correct, we substitute \\( x = 4 \\) back into the original equation and check if both sides are equal.\n", + "\n", + " Original equation:\n", + "\n", + " \\[\n", + " 2x + 3 = 11\n", + " \\]\n", + "\n", + " Substitute \\( x = 4 \\):\n", + "\n", + " \\[\n", + " 2(4) + 3 = 11\n", + " \\]\n", + "\n", + " Simplifying this, we get:\n", + "\n", + " \\[\n", + " 8 + 3 = 11\n", + " \\]\n", + "\n", + " \\[\n", + " 11 = 11\n", + " \\]\n", + "\n", + " Since both sides of the equation are equal, our solution is verified to be correct.\n", + "\n", + "**Final Answer**: \\( x = 4 \\) \\( x = 4 \\) \n", + "\n", + "[Question] Solve for \\( x \\) in the equation \\( x^2 - 4x + 4 = 0 \\).\n", + "[Answer] To solve the quadratic equation \\( x^2 - 4x + 4 = 0 \\), we will follow these steps:\n", + "\n", + "1. **Identify the quadratic equation**: The given equation is \\( x^2 - 4x + 4 = 0 \\).\n", + "\n", + "2. **Recognize the standard form**: The standard form of a quadratic equation is \\( ax^2 + bx + c = 0 \\). Here, \\( a = 1 \\), \\( b = -4 \\), and \\( c = 4 \\).\n", + "\n", + "3. **Factor the quadratic expression**: We need to factor the quadratic expression on the left-hand side of the equation. We look for two numbers that multiply to \\( c \\) (which is 4) and add up to \\( b \\) (which is -4). These numbers are -2 and -2.\n", + "\n", + "4. **Write the factored form**: The quadratic expression \\( x^2 - 4x + 4 \\) can be factored as \\( (x - 2)(x - 2) \\) or \\( (x - 2)^2 \\).\n", + "\n", + "5. **Set the factored form equal to zero**: We now have \\( (x - 2)^2 = 0 \\).\n", + "\n", + "6. **Solve for \\( x \\)**: To find the value of \\( x \\), we take the square root of both sides of the equation:\n", + " \\[\n", + " \\sqrt{(x - 2)^2} = \\sqrt{0}\n", + " \\]\n", + " This simplifies to:\n", + " \\[\n", + " x - 2 = 0\n", + " \\]\n", + "\n", + "7. **Isolate \\( x \\)**: Add 2 to both sides of the equation to solve for \\( x \\):\n", + " \\[\n", + " x = 2\n", + " \\]\n", + "\n", + "8. **Verify the solution**: Substitute \\( x = 2 \\) back into the original equation to ensure it satisfies the equation:\n", + " \\[\n", + " (2)^2 - 4(2) + 4 = 4 - 8 + 4 = 0\n", + " \\]\n", + " Since the left-hand side equals the right-hand side (0), the solution \\( x = 2 \\) is verified.\n", + "\n", + "**Final Answer**: \\( x = 2 \\) \\( x = 2 \\) \n", + "\n", + "[Question] Find the derivative of \\( f(x) = 3x^2 \\cdot \\sin(x) \\).\n", + "[Answer] To find the derivative of the function \\( f(x) = 3x^2 \\cdot \\sin(x) \\), we will use the product rule of differentiation. The product rule states that if we have a function \\( f(x) = u(x) \\cdot v(x) \\), then its derivative \\( f'(x) \\) is given by:\n", + "\n", + "\\[ f'(x) = u'(x) \\cdot v(x) + u(x) \\cdot v'(x) \\]\n", + "\n", + "Here, we identify \\( u(x) = 3x^2 \\) and \\( v(x) = \\sin(x) \\).\n", + "\n", + "Step 1: Differentiate \\( u(x) = 3x^2 \\)\n", + "\\[ u'(x) = \\frac{d}{dx}(3x^2) = 3 \\cdot 2x = 6x \\]\n", + "\n", + "Step 2: Differentiate \\( v(x) = \\sin(x) \\)\n", + "\\[ v'(x) = \\frac{d}{dx}(\\sin(x)) = \\cos(x) \\]\n", + "\n", + "Step 3: Apply the product rule\n", + "\\[ f'(x) = u'(x) \\cdot v(x) + u(x) \\cdot v'(x) \\]\n", + "\\[ f'(x) = (6x) \\cdot \\sin(x) + (3x^2) \\cdot \\cos(x) \\]\n", + "\n", + "Step 4: Simplify the expression\n", + "\\[ f'(x) = 6x \\sin(x) + 3x^2 \\cos(x) \\]\n", + "\n", + "Thus, the derivative of the function \\( f(x) = 3x^2 \\cdot \\sin(x) \\) is:\n", + "\n", + "\\[ \\boxed{f'(x) = 6x \\sin(x) + 3x^2 \\cos(x)} \\]\n", + "\n", + "To verify the final answer, we can recheck each step to ensure accuracy:\n", + "- The derivative of \\( 3x^2 \\) is correctly calculated as \\( 6x \\).\n", + "- The derivative of \\( \\sin(x) \\) is correctly calculated as \\( \\cos(x) \\).\n", + "- The product rule is correctly applied, and the terms are correctly combined and simplified.\n", + "\n", + "Therefore, the final answer is confirmed to be correct. \\( f'(x) = 3x^2 \\cos(x) + 6x \\sin(x) \\) \n", + "\n", + "[Question] Evaluate the definite integral \\( \\int_{0}^{1} (4x^3 - 2x + 1) \\, dx \\).\n", + "[Answer] To evaluate the definite integral \\( \\int_{0}^{1} (4x^3 - 2x + 1) \\, dx \\), we will follow these steps:\n", + "\n", + "1. **Find the antiderivative** of the integrand \\( 4x^3 - 2x + 1 \\).\n", + "2. **Evaluate the antiderivative** at the upper limit of integration (1).\n", + "3. **Evaluate the antiderivative** at the lower limit of integration (0).\n", + "4. **Subtract the value** of the antiderivative at the lower limit from the value at the upper limit to find the definite integral.\n", + "\n", + "### Step-by-Step Solution:\n", + "\n", + "1. **Find the antiderivative**:\n", + " - The antiderivative of \\( 4x^3 \\) is \\( \\frac{4x^4}{4} = x^4 \\).\n", + " - The antiderivative of \\( -2x \\) is \\( -\\frac{2x^2}{2} = -x^2 \\).\n", + " - The antiderivative of \\( 1 \\) is \\( x \\).\n", + "\n", + " Therefore, the antiderivative of \\( 4x^3 - 2x + 1 \\) is:\n", + " \\[\n", + " F(x) = x^4 - x^2 + x\n", + " \\]\n", + "\n", + "2. **Evaluate the antiderivative at the upper limit (1)**:\n", + " \\[\n", + " F(1) = 1^4 - 1^2 + 1 = 1 - 1 + 1 = 1\n", + " \\]\n", + "\n", + "3. **Evaluate the antiderivative at the lower limit (0)**:\n", + " \\[\n", + " F(0) = 0^4 - 0^2 + 0 = 0\n", + " \\]\n", + "\n", + "4. **Subtract the value at the lower limit from the value at the upper limit**:\n", + " \\[\n", + " \\int_{0}^{1} (4x^3 - 2x + 1) \\, dx = F(1) - F(0) = 1 - 0 = 1\n", + " \\]\n", + "\n", + "### Final Answer:\n", + "\\[\n", + "\\boxed{1}\n", + "\\] \\( 1 \\) \n", + "\n", + "[Question] Solve the system of equations:\n", + "\\[ \\begin{cases} \n", + "x + 2y + z = 6 \\\\\n", + "2x - y + 3z = 14 \\\\\n", + "3x + y - z = 2 \n", + "\\end{cases} \\]\n", + "[Answer] To solve the system of equations:\n", + "\\[ \\begin{cases} \n", + "x + 2y + z = 6 \\\\\n", + "2x - y + 3z = 14 \\\\\n", + "3x + y - z = 2 \n", + "\\end{cases} \\]\n", + "\n", + "we will use the method of elimination and substitution to find the values of \\(x\\), \\(y\\), and \\(z\\).\n", + "\n", + "**Step 1: Eliminate \\(z\\) from the first two equations.**\n", + "\n", + "First, we multiply the first equation by 3 to align the coefficients of \\(z\\):\n", + "\\[ 3(x + 2y + z) = 3 \\cdot 6 \\]\n", + "\\[ 3x + 6y + 3z = 18 \\]\n", + "\n", + "Now, we subtract the second equation from this result:\n", + "\\[ (3x + 6y + 3z) - (2x - y + 3z) = 18 - 14 \\]\n", + "\\[ 3x + 6y + 3z - 2x + y - 3z = 4 \\]\n", + "\\[ x + 7y = 4 \\]\n", + "\\[ \\text{(Equation 4)} \\]\n", + "\n", + "**Step 2: Eliminate \\(z\\) from the first and third equations.**\n", + "\n", + "Next, we multiply the first equation by 1 and the third equation by 1 to align the coefficients of \\(z\\):\n", + "\\[ 1(x + 2y + z) = 1 \\cdot 6 \\]\n", + "\\[ x + 2y + z = 6 \\]\n", + "\n", + "\\[ 1(3x + y - z) = 1 \\cdot 2 \\]\n", + "\\[ 3x + y - z = 2 \\]\n", + "\n", + "Now, we add these two equations:\n", + "\\[ (x + 2y + z) + (3x + y - z) = 6 + 2 \\]\n", + "\\[ x + 2y + z + 3x + y - z = 8 \\]\n", + "\\[ 4x + 3y = 8 \\]\n", + "\\[ \\text{(Equation 5)} \\]\n", + "\n", + "**Step 3: Solve the system of equations formed by Equation 4 and Equation 5.**\n", + "\n", + "We now have:\n", + "\\[ \\begin{cases} \n", + "x + 7y = 4 \\\\\n", + "4x + 3y = 8 \n", + "\\end{cases} \\]\n", + "\n", + "First, we solve Equation 4 for \\(x\\):\n", + "\\[ x = 4 - 7y \\]\n", + "\n", + "Substitute \\(x = 4 - 7y\\) into Equation 5:\n", + "\\[ 4(4 - 7y) + 3y = 8 \\]\n", + "\\[ 16 - 28y + 3y = 8 \\]\n", + "\\[ 16 - 25y = 8 \\]\n", + "\\[ -25y = 8 - 16 \\]\n", + "\\[ -25y = -8 \\]\n", + "\\[ y = \\frac{8}{25} \\]\n", + "\n", + "**Step 4: Substitute \\(y\\) back into Equation 4 to find \\(x\\).**\n", + "\n", + "\\[ x + 7\\left(\\frac{8}{25}\\right) = 4 \\]\n", + "\\[ x + \\frac{56}{25} = 4 \\]\n", + "\\[ x = 4 - \\frac{56}{25} \\]\n", + "\\[ x = \\frac{100}{25} - \\frac{56}{25} \\]\n", + "\\[ x = \\frac{44}{25} \\]\n", + "\n", + "**Step 5: Substitute \\(x\\) and \\(y\\) back into the first original equation to find \\(z\\).**\n", + "\n", + "\\[ x + 2y + z = 6 \\]\n", + "\\[ \\frac{44}{25} + 2\\left(\\frac{8}{25}\\right) + z = 6 \\]\n", + "\\[ \\frac{44}{25} + \\frac{16}{25} + z = 6 \\]\n", + "\\[ \\frac{60}{25} + z = 6 \\]\n", + "\\[ \\frac{60}{25} = 2.4 \\]\n", + "\\[ 2.4 + z = 6 \\]\n", + "\\[ z = 6 - 2.4 \\]\n", + "\\[ z = 3.6 \\]\n", + "\n", + "**Final Answer:**\n", + "\\[ x = \\frac{44}{25}, y = \\frac{8}{25}, z = 3.6 \\]\n", + "\n", + "We have verified each step and simplified all expressions. The solution is logically consistent and correctly formatted. \\( x = \\frac{44}{25}, y = \\frac{8}{25}, z = 3.6 \\) \n", + "\n", + "\n", + "For each question present the reasoning followed by the correct answer.\n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "id": "c61c2f84", + "metadata": {}, + "source": [ + "#### Scenario 3 : We have training data and also want in-context examples in final prompt" + ] + }, + { + "cell_type": "markdown", + "id": "11d2de75", + "metadata": {}, + "source": [ + "Load and save the dataset " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "976681bd-4f43-4dbc-947e-cdb94d4824f0", + "metadata": {}, + "outputs": [], + "source": [ + "if not os.path.exists(\"data\"):\n", + " os.mkdir(\"data\")\n", + " \n", + "dataset = load_dataset(\"openai/gsm8k\", \"main\")\n", + "num_samples = 0\n", + "for dataset_type in ['train','test']:\n", + " data_list = []\n", + " for data in dataset[dataset_type]:\n", + " data_list.append({\"question\": data['question'], \"answer\": data['answer']})\n", + " if num_samples == 100 and dataset_type == 'train': # We sample only 100 train examples and use 25 out them for training randomly\n", + " break\n", + " num_samples += 1\n", + " gsm8k_processor.dataset_to_jsonl(\"data/\"+ dataset_type+'.jsonl', dataset=data_list)" + ] + }, + { + "cell_type": "markdown", + "id": "abf1671a", + "metadata": {}, + "source": [ + "Set the configurations" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cc841576", + "metadata": {}, + "outputs": [], + "source": [ + "file_path = 'configs/promptopt_config.yaml' \n", + "config_dict = {\n", + " \"task_description\": \"You are a mathematics expert. You will be given a mathematics problem which you need to solve\",\n", + " \"base_instruction\": \"Lets think step by step.\",\n", + " \"mutation_rounds\": 2,\n", + " \"few_shot_count\": 5,\n", + " \"generate_reasoning\": True,\n", + " \"mutate_refine_iterations\" : 3,\n", + " \"seen_set_size\":20\n", + " }\n", + "update_yaml_file(file_path,config_dict)" + ] + }, + { + "cell_type": "markdown", + "id": "3392594d", + "metadata": {}, + "source": [ + "Create an object for calling prompt optimization and inference functionalities" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8af4246f-db32-4b37-a73a-f9e2e5125d09", + "metadata": {}, + "outputs": [], + "source": [ + "gp = GluePromptOpt(promptopt_config_path,\n", + " setup_config_path,\n", + " dataset_jsonl = os.path.join(\"data\", \"train.jsonl\"),\n", + " data_processor = gsm8k_processor)" + ] + }, + { + "cell_type": "markdown", + "id": "6f421ce9", + "metadata": {}, + "source": [ + "Call the optimization function " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "09e3e6e1", + "metadata": {}, + "outputs": [], + "source": [ + "best_prompt, expert_profile = gp.get_best_prompt(use_examples=True,run_without_train_examples=False,generate_synthetic_examples=False)" + ] + }, + { + "cell_type": "markdown", + "id": "15bb0e80", + "metadata": {}, + "source": [ + "Output : Following Prompt and Expert Profile are generated " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "696e6612", + "metadata": {}, + "outputs": [], + "source": [ + "OUTPUT = \"\"\"Expert Identity: You are a mathematics expert with a strong background in various fields of mathematics, including algebra, calculus, geometry, and statistics. You have a deep understanding of mathematical theories and principles, and you are skilled at solving complex problems with precision and clarity. Your expertise allows you to break down intricate problems into manageable steps, making it easier for others to follow your reasoning. You are familiar with a wide range of mathematical techniques and tools, and you can apply them effectively to find solutions. Whether the problem involves solving equations, proving theorems, or analyzing data, you can provide a clear, accurate, and well-explained solution. Your ability to communicate complex mathematical concepts in an understandable way makes you an invaluable resource for anyone seeking help with mathematics.\n", + "\n", + "Final best prompt: \n", + "\n", + "You are a mathematics expert. Your task is to solve a given mathematics problem accurately and provide a clear, detailed explanation of your solution process. Follow these steps to ensure a comprehensive and well-structured solution:\n", + "\n", + "1. **Understand the Problem**: Carefully read and comprehend the problem statement. Identify the key components and what is being asked.\n", + "\n", + "2. **Identify Components**: Break down the problem into its fundamental components, such as variables, constants, and relevant quantities (e.g., base pay, overtime pay, distances, speeds, etc.).\n", + "\n", + "3. **Apply Relevant Principles**: Use appropriate mathematical principles, formulas, and methods to solve the problem step by step.\n", + "\n", + "4. **Logical Reasoning**: Employ logical reasoning to explain each step of your solution process. Ensure that each step follows logically from the previous one.\n", + "\n", + "5. **Detailed Explanations**: Provide detailed explanations for each step to ensure clarity and understanding. Include intermediate results and how they contribute to the final solution.\n", + "\n", + "6. **Explicit Calculation Steps**: Show each calculation step in detail, including intermediate results. Use proper mathematical notation and symbols.\n", + "\n", + "7. **Verify Each Step**: Recheck each intermediate step of your calculation to verify the correctness of the final answer. Ensure that all arithmetic and algebraic operations are accurate.\n", + "\n", + "8. **Combine Results**: Clearly combine different components of the problem (e.g., base pay and overtime pay) before arriving at the final answer.\n", + "\n", + "9. **Simplify and Notate**: Simplify the final answer where possible, and use proper mathematical notation and symbols.\n", + "\n", + "10. **Mark the Final Answer**: Clearly mark the final answer within and tags.\n", + "\n", + "Ensure that your approach is tailored to the specific type of mathematical problem being solved, whether it involves arithmetic, algebra, geometry, calculus, or any other area of mathematics. Present the solutions in a clear and organized manner.\n", + "\n", + "**Additional Guidelines:**\n", + "- **Contextual Understanding**: Pay close attention to the context of the problem to ensure that all relationships and quantities are correctly interpreted.\n", + "- **Correct Application of Arithmetic Operations**: Double-check that all arithmetic operations are applied correctly and align with the problem's requirements.\n", + "- **Verification of Final Answer**: Dedicate a step to verify the final answer by rechecking all intermediate steps and ensuring they logically lead to the correct final result.\n", + "- **Clarity in Marking Final Answer**: Use the and tags to clearly mark the final answer.\n", + "\n", + "By following these steps and additional guidelines, you will ensure that the solution is accurate, well-explained, and clearly presented.\n", + "\n", + "\n", + "[Question] Bella bought stamps at the post office. Some of the stamps had a snowflake design, some had a truck design, and some had a rose design. Bella bought 11 snowflake stamps. She bought 9 more truck stamps than snowflake stamps, and 13 fewer rose stamps than truck stamps. How many stamps did Bella buy in all?\n", + "[Answer] 1. **Understand the Problem**: Bella bought three types of stamps: snowflake, truck, and rose. We need to determine the total number of stamps she bought, given the relationships between the quantities of each type.\n", + "\n", + "2. **Identify Components**:\n", + " - Number of snowflake stamps: 11.\n", + " - Number of truck stamps: 9 more than the number of snowflake stamps.\n", + " - Number of rose stamps: 13 fewer than the number of truck stamps.\n", + "\n", + "3. **Apply Relevant Principles**: Use basic arithmetic operations to find the quantities of truck and rose stamps, and then sum all the quantities to find the total number of stamps.\n", + "\n", + "4. **Logical Reasoning**:\n", + " - Number of snowflake stamps: 11.\n", + " - Number of truck stamps: 11 (snowflake stamps) + 9 = 20.\n", + " - Number of rose stamps: 20 (truck stamps) - 13 = 7.\n", + "\n", + "5. **Detailed Explanations**:\n", + " - Calculate the number of truck stamps: 11 (snowflake stamps) + 9 = 20.\n", + " - Calculate the number of rose stamps: 20 (truck stamps) - 13 = 7.\n", + " - Calculate the total number of stamps: 11 (snowflake) + 20 (truck) + 7 (rose) = 38.\n", + "\n", + "6. **Explicit Calculation Steps**:\n", + " - Truck stamps: 11 + 9 = $<11+9=20>20.\n", + " - Rose stamps: 20 - 13 = $<20-13=7>7.\n", + " - Total stamps: 11 + 20 + 7 = $<11+20+7=38>38.\n", + "\n", + "7. **Verify Each Step**: Recheck each calculation step to ensure correctness:\n", + " - Truck stamps: 11 + 9 = 20.\n", + " - Rose stamps: 20 - 13 = 7.\n", + " - Total stamps: 11 + 20 + 7 = 38.\n", + "\n", + "8. **Combine Results**: Combine the number of each type of stamp correctly to find the total number of stamps.\n", + "\n", + "9. **Simplify and Notate**: The final answer is already simplified.\n", + "\n", + "10. **Mark the Final Answer**: 38\n", + "\n", + "By following these steps, we ensure that the solution is accurate, well-explained, and clearly presented. 38\n", + "\n", + "[Question] It takes Roque two hours to walk to work and one hour to ride his bike to work. Roque walks to and from work three times a week and rides his bike to and from work twice a week. How many hours in total does he take to get to and from work a week with walking and biking?\n", + "[Answer] 1. **Understand the Problem**: Roque has two modes of transportation to work: walking and biking. We need to calculate the total time he spends traveling to and from work in a week, considering the different times and frequencies for each mode.\n", + "\n", + "2. **Identify Components**:\n", + " - Time to walk to work: 2 hours (one way).\n", + " - Time to bike to work: 1 hour (one way).\n", + " - Frequency of walking: 3 times a week (to and from work).\n", + " - Frequency of biking: 2 times a week (to and from work).\n", + "\n", + "3. **Apply Relevant Principles**: Use basic arithmetic to calculate the total time spent walking and biking separately, then sum these times to get the total weekly travel time.\n", + "\n", + "4. **Logical Reasoning**:\n", + " - Calculate the total walking time for a week:\n", + " - One round trip (to and from work) by walking takes 2 hours (to work) + 2 hours (from work) = 4 hours.\n", + " - Roque walks to and from work 3 times a week, so the total walking time is 4 hours per round trip * 3 round trips = 12 hours.\n", + " - Calculate the total biking time for a week:\n", + " - One round trip (to and from work) by biking takes 1 hour (to work) + 1 hour (from work) = 2 hours.\n", + " - Roque bikes to and from work 2 times a week, so the total biking time is 2 hours per round trip * 2 round trips = 4 hours.\n", + "\n", + "5. **Detailed Explanations**:\n", + " - Walking time calculation:\n", + " - One round trip walking: 2 hours (to work) + 2 hours (from work) = 4 hours.\n", + " - Total walking time for the week: 4 hours per round trip * 3 round trips = 12 hours.\n", + " - Biking time calculation:\n", + " - One round trip biking: 1 hour (to work) + 1 hour (from work) = 2 hours.\n", + " - Total biking time for the week: 2 hours per round trip * 2 round trips = 4 hours.\n", + " - Combine the total walking and biking times to get the total weekly travel time:\n", + " - Total weekly travel time: 12 hours (walking) + 4 hours (biking) = 16 hours.\n", + "\n", + "6. **Explicit Calculation Steps**:\n", + " - Walking time: 2 hours (one way) * 2 (round trip) * 3 (times a week) = $<2*2*3=12>12 hours.\n", + " - Biking time: 1 hour (one way) * 2 (round trip) * 2 (times a week) = $<1*2*2=4>4 hours.\n", + " - Total time: 12 hours (walking) + 4 hours (biking) = $<12+4=16>16 hours.\n", + "\n", + "7. **Verify Each Step**: Recheck each calculation step to ensure correctness. Confirm that the arithmetic operations and logic used are accurate.\n", + "\n", + "8. **Combine Results**: Combine the total walking and biking times correctly to ensure the final answer is accurate.\n", + "\n", + "9. **Simplify and Notate**: The final answer is already simplified and clearly presented.\n", + "\n", + "10. **Mark the Final Answer**: 16\n", + "\n", + "By following these steps, we ensure that the solution is accurate, well-explained, and clearly presented. 16\n", + "\n", + "[Question] Betty is saving money for a new wallet which costs $100. Betty has only half of the money she needs. Her parents decided to give her $15 for that purpose, and her grandparents twice as much as her parents. How much more money does Betty need to buy the wallet?\n", + "[Answer] 1. **Understand the Problem**: Betty is saving money for a wallet that costs $100. She currently has half of the money she needs. Her parents and grandparents are contributing additional amounts to help her reach her goal. We need to determine how much more money Betty needs to buy the wallet.\n", + "\n", + "2. **Identify Components**:\n", + " - Total cost of the wallet: $100.\n", + " - Amount Betty currently has: half of $100.\n", + " - Contribution from parents: $15.\n", + " - Contribution from grandparents: twice the amount given by parents.\n", + "\n", + "3. **Apply Relevant Principles**: Use basic arithmetic to calculate the total amount of money Betty will have after receiving contributions from her parents and grandparents, and then determine how much more she needs to reach $100.\n", + "\n", + "4. **Logical Reasoning**:\n", + " - Calculate the amount Betty currently has: $100 / 2 = $50.\n", + " - Calculate the contribution from grandparents: 2 * $15 = $30.\n", + " - Calculate the total amount of money Betty will have: $50 (current amount) + $15 (parents' contribution) + $30 (grandparents' contribution).\n", + "\n", + "5. **Detailed Explanations**:\n", + " - Betty currently has $50 because half of $100 is $50.\n", + " - Her parents give her $15.\n", + " - Her grandparents give her twice the amount her parents give, which is 2 * $15 = $30.\n", + " - The total amount of money Betty will have is $50 (current amount) + $15 (parents' contribution) + $30 (grandparents' contribution) = $95.\n", + "\n", + "6. **Explicit Calculation Steps**:\n", + " - Current amount: $100 / 2 = $<100/2=50>50.\n", + " - Grandparents' contribution: 2 * $15 = $<2*15=30>30.\n", + " - Total amount: $50 + $15 + $30 = $<50+15+30=95>95.\n", + "\n", + "7. **Verify Each Step**: Recheck each calculation step to ensure correctness.\n", + " - Current amount: $100 / 2 = $50.\n", + " - Grandparents' contribution: 2 * $15 = $30.\n", + " - Total amount: $50 + $15 + $30 = $95.\n", + "\n", + "8. **Combine Results**: Combine the total amount of money Betty will have correctly.\n", + " - Total amount: $50 (current amount) + $15 (parents' contribution) + $30 (grandparents' contribution) = $95.\n", + "\n", + "9. **Simplify and Notate**: The final answer is already simplified.\n", + "\n", + "10. **Mark the Final Answer**: \n", + " - Amount Betty needs to buy the wallet: $100 - $95 = $<100-95=5>5.\n", + "\n", + "5 5\n", + "\n", + "[Question] A rectangle has a length of 10 cm and a width of 5 cm. What is the area and perimeter of the rectangle?\n", + "[Answer] 1. **Understand the Problem**: We need to find both the area and the perimeter of a rectangle given its length and width.\n", + "\n", + "2. **Identify Components**: \n", + " - Length of the rectangle (L) = 10 cm\n", + " - Width of the rectangle (W) = 5 cm\n", + "\n", + "3. **Apply Relevant Principles**: \n", + " - The formula for the area of a rectangle is \\( \\text{Area} = \\text{Length} \\times \\text{Width} \\).\n", + " - The formula for the perimeter of a rectangle is \\( \\text{Perimeter} = 2 \\times (\\text{Length} + \\text{Width}) \\).\n", + "\n", + "4. **Logical Reasoning**:\n", + " - To find the area, multiply the length by the width.\n", + " - To find the perimeter, add the length and the width, then multiply the result by 2.\n", + "\n", + "5. **Detailed Explanations**:\n", + " - Calculate the area: \\( \\text{Area} = 10 \\, \\text{cm} \\times 5 \\, \\text{cm} \\).\n", + " - Calculate the perimeter: \\( \\text{Perimeter} = 2 \\times (10 \\, \\text{cm} + 5 \\, \\text{cm}) \\).\n", + "\n", + "6. **Explicit Calculation Steps**:\n", + " - Area: \\( 10 \\times 5 = 50 \\, \\text{cm}^2 \\).\n", + " - Perimeter: \\( 2 \\times (10 + 5) = 2 \\times 15 = 30 \\, \\text{cm} \\).\n", + "\n", + "7. **Verify Each Step**: \n", + " - Recheck the area calculation: \\( 10 \\times 5 = 50 \\, \\text{cm}^2 \\).\n", + " - Recheck the perimeter calculation: \\( 2 \\times 15 = 30 \\, \\text{cm} \\).\n", + "\n", + "8. **Combine Results**: \n", + " - The area of the rectangle is \\( 50 \\, \\text{cm}^2 \\).\n", + " - The perimeter of the rectangle is \\( 30 \\, \\text{cm} \\).\n", + "\n", + "9. **Simplify and Notate**: \n", + " - The final answers are already simplified.\n", + "\n", + "10. **Mark the Final Answer**: \n", + " - Area: 50 \\, \\text{cm}^2\n", + " - Perimeter: 30 \\, \\text{cm}\n", + "\n", + "By following these steps, we ensure that the solution is accurate, well-explained, and clearly presented. 50\n", + "\n", + "[Question] Solve for x in the equation 2x + 3 = 11.\n", + "[Answer] **Understand the Problem**: We need to solve for the variable \\( x \\) in the given linear equation \\( 2x + 3 = 11 \\).\n", + "\n", + "**Identify Components**: \n", + "- The equation is \\( 2x + 3 = 11 \\).\n", + "- We need to isolate \\( x \\) on one side of the equation.\n", + "\n", + "**Apply Relevant Principles**: \n", + "- Use basic algebraic principles to isolate \\( x \\).\n", + "\n", + "**Logical Reasoning**:\n", + "1. Start with the given equation: \\( 2x + 3 = 11 \\).\n", + "2. Subtract 3 from both sides of the equation to isolate the term with \\( x \\):\n", + " \\[\n", + " 2x + 3 - 3 = 11 - 3\n", + " \\]\n", + "3. Simplify both sides:\n", + " \\[\n", + " 2x = 8\n", + " \\]\n", + "4. Divide both sides by 2 to solve for \\( x \\):\n", + " \\[\n", + " \\frac{2x}{2} = \\frac{8}{2}\n", + " \\]\n", + "5. Simplify the division:\n", + " \\[\n", + " x = 4\n", + " \\]\n", + "\n", + "**Detailed Explanations**:\n", + "- Subtracting 3 from both sides removes the constant term on the left side, leaving \\( 2x \\) isolated.\n", + "- Dividing both sides by 2 isolates \\( x \\) by removing the coefficient of 2.\n", + "\n", + "**Explicit Calculation Steps**:\n", + "1. \\( 2x + 3 = 11 \\)\n", + "2. \\( 2x + 3 - 3 = 11 - 3 \\)\n", + "3. \\( 2x = 8 \\)\n", + "4. \\( \\frac{2x}{2} = \\frac{8}{2} \\)\n", + "5. \\( x = 4 \\)\n", + "\n", + "**Verify Each Step**:\n", + "- Recheck each step to ensure no arithmetic errors:\n", + " - Subtracting 3 from 11 gives 8.\n", + " - Dividing 8 by 2 gives 4.\n", + "\n", + "**Combine Results**: The final value of \\( x \\) is correctly isolated and calculated.\n", + "\n", + "**Simplify and Notate**: The final answer is already simplified.\n", + "\n", + "**Mark the Final Answer**: 4\n", + "\n", + "By following these steps, we ensure that the solution is accurate, well-explained, and clearly presented. 4\n", + "\n", + "\n", + "For each question present the reasoning followed by the correct answer.\"\"\"" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "general", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.4" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/demos/svamp/demo.ipynb b/demos/svamp/demo.ipynb index 96b277b..1da0a24 100644 --- a/demos/svamp/demo.ipynb +++ b/demos/svamp/demo.ipynb @@ -4,6 +4,14 @@ "cell_type": "markdown", "id": "14360485", "metadata": {}, + "source": [ + "#### Set environment variables in [.env](.env) for LLM API calling" + ] + }, + { + "cell_type": "markdown", + "id": "6bd95c11", + "metadata": {}, "source": [ "### Import Dependencies" ] @@ -188,8 +196,8 @@ "metadata": {}, "source": [ "### Call prompt optmization function\n", - "1. ```use_synthetic_examples``` can be used when there are training samples and a mixture of real and synthetic in-context examples are required in the final prompt. When set to ```False``` all the in-context examples will be real\n", - "2. ```use_only_synthetic_examples``` can be used when there are no training samples but in-context examples are required in the final prompt \n", + "1. ```use_examples``` can be used when there are training samples and a mixture of real and synthetic in-context examples are required in the final prompt. When set to ```False``` all the in-context examples will be real\n", + "2. ```generate_synthetic_examples``` can be used when there are no training samples and we want to generate synthetic examples \n", "3. ```run_without_train_examples``` can be used when there are no training samples and in-context examples are not required in the final prompt " ] }, @@ -203,7 +211,7 @@ "outputs": [], "source": [ "# Function call to generate optimal prompt and expert profile \n", - "best_prompt, expert_profile = gp.get_best_prompt(use_synthetic_examples=False,run_without_train_examples=False,use_only_synthetic_examples=False)" + "best_prompt, expert_profile = gp.get_best_prompt(use_examples=True,run_without_train_examples=False,generate_synthetic_examples=False)" ] }, { diff --git a/docs/index.html b/docs/index.html index bb99e1d..26ff184 100644 --- a/docs/index.html +++ b/docs/index.html @@ -388,7 +388,7 @@

🧙 PromptWizard

Overview

- Large language models (LLMs) like GPT-4 have achieved remarkable performance across diverse tasks. At the core of this success is prompting—the process of providing input instructions to guide models toward desired outputs. Studies have shown that prompting significantly influences LLM performance, making prompt engineering—the design and refinement of prompts—critical for maximizing accuracy. However, crafting effective prompts remains a labor-intensive and domain-specific task, requiring human expertise and subjective judgment. As models evolve and tasks vary, the need to repeatedly design prompts raises an important question: + Large language models (LLMs) like GPT-4 have achieved remarkable performance across diverse tasks. At the core of this success is prompting—the process of providing input instructions to guide models toward desired outputs. Studies have shown that prompting significantly influences LLM performance, making prompt engineering—the design and refinement of prompts—critical for maximizing accuracy. However, crafting effective prompts remains a labor-intensive and domain-specific task, requiring human expertise and subjective judgment. As models evolve and tasks vary, the need to repeatedly design prompts raises an important question:
Can prompt engineering be automated to streamline this process and enhance scalability?
diff --git a/promptwizard/glue/promptopt/instantiate.py b/promptwizard/glue/promptopt/instantiate.py index 332ff26..9d2a105 100644 --- a/promptwizard/glue/promptopt/instantiate.py +++ b/promptwizard/glue/promptopt/instantiate.py @@ -51,11 +51,12 @@ def __init__(self, serialized. :param prompt_pool_path: Path to yaml file that has prompts """ - if data_processor: - self.data_processor = data_processor - else: - with open(dataset_processor_pkl_path, "rb") as file: - self.data_processor = pickle.load(file) # datatype: class DatasetSpecificProcessing + if dataset_jsonl != None: + if data_processor: + self.data_processor = data_processor + else: + with open(dataset_processor_pkl_path, "rb") as file: + self.data_processor = pickle.load(file) # datatype: class DatasetSpecificProcessing prompt_config_dict = yaml_to_dict(prompt_config_path) prompt_opt_cls, prompt_opt_hyperparam_cls, promptpool_cls = get_promptopt_class( @@ -71,18 +72,20 @@ def __init__(self, self.prompt_pool = yaml_to_class(prompt_pool_path, promptpool_cls, default_yaml_path) - dataset = read_jsonl(dataset_jsonl) + if dataset_jsonl != None: + dataset = read_jsonl(dataset_jsonl) self.prompt_opt_param.answer_format += self.prompt_pool.ans_delimiter_instruction base_path = join(self.setup_config.dir_info.base_dir, self.setup_config.experiment_name) set_logging_config(join(base_path, self.setup_config.dir_info.log_dir_name), self.setup_config.mode) self.logger = get_glue_logger(__name__) - if len(dataset) < self.prompt_opt_param.seen_set_size: - self.prompt_opt_param.seen_set_size = len(dataset) - self.logger.info(f"Dataset has {len(dataset)} samples. However values for seen_set_size is " - f"{self.prompt_opt_param.seen_set_size}. Hence resetting seen_set_size" - f" to {len(dataset)}") + if dataset_jsonl != None: + if len(dataset) < self.prompt_opt_param.seen_set_size: + self.prompt_opt_param.seen_set_size = len(dataset) + self.logger.info(f"Dataset has {len(dataset)} samples. However values for seen_set_size is " + f"{self.prompt_opt_param.seen_set_size}. Hence resetting seen_set_size" + f" to {len(dataset)}") if self.prompt_opt_param.few_shot_count > self.prompt_opt_param.seen_set_size: self.prompt_opt_param.few_shot_count = self.prompt_opt_param.seen_set_size @@ -90,7 +93,10 @@ def __init__(self, f"However values for seen_set_size is {self.prompt_opt_param.seen_set_size}. " f"Hence resetting few_shot_count to {self.prompt_opt_param.few_shot_count}") - training_dataset = dataset[:self.prompt_opt_param.seen_set_size] + if dataset_jsonl != None: + training_dataset = dataset[:self.prompt_opt_param.seen_set_size] + else: + training_dataset = None self.logger.info(f"Setup configurations parameters: {self.setup_config} \n{CommonLogsStr.LOG_SEPERATOR}") self.logger.info(f"Prompt Optimization parameters: {self.prompt_opt_param} \n{CommonLogsStr.LOG_SEPERATOR}") @@ -100,7 +106,7 @@ def __init__(self, self.prompt_opt = prompt_opt_cls(training_dataset, base_path, self.setup_config, self.prompt_pool, self.data_processor, self.logger) - def get_best_prompt(self,use_synthetic_examples=False,run_without_train_examples=False,use_only_synthetic_examples=False) -> (str, Any): + def get_best_prompt(self,use_examples=False,run_without_train_examples=False,generate_synthetic_examples=False) -> (str, Any): """ Call get_best_prompt() method of class PromptOptimizer & return its value. :return: (best_prompt, expert_profile) @@ -109,7 +115,7 @@ def get_best_prompt(self,use_synthetic_examples=False,run_without_train_examples identity of described in expert_profile. """ start_time = time.time() - self.BEST_PROMPT, self.EXPERT_PROFILE = self.prompt_opt.get_best_prompt(self.prompt_opt_param,use_synthetic_examples=use_synthetic_examples,run_without_train_examples=run_without_train_examples,use_only_synthetic_examples=use_only_synthetic_examples) + self.BEST_PROMPT, self.EXPERT_PROFILE = self.prompt_opt.get_best_prompt(self.prompt_opt_param,use_examples=use_examples,run_without_train_examples=run_without_train_examples,generate_synthetic_examples=generate_synthetic_examples) self.logger.info(f"Time taken to find best prompt: {(time.time() - start_time)} sec") return self.BEST_PROMPT, self.EXPERT_PROFILE diff --git a/promptwizard/glue/promptopt/techniques/critique_n_refine/base_classes.py b/promptwizard/glue/promptopt/techniques/critique_n_refine/base_classes.py index c5dcacc..9f0259e 100644 --- a/promptwizard/glue/promptopt/techniques/critique_n_refine/base_classes.py +++ b/promptwizard/glue/promptopt/techniques/critique_n_refine/base_classes.py @@ -65,3 +65,5 @@ class CritiqueNRefineParams(PromptOptimizationParams, UniversalBaseClass): generate_expert_identity: bool # Generate keywords that describe the intent of the task generate_intent_keywords: bool + # number of synthetic training examples to be generated + num_train_examples: int diff --git a/promptwizard/glue/promptopt/techniques/critique_n_refine/core_logic.py b/promptwizard/glue/promptopt/techniques/critique_n_refine/core_logic.py index 8a524a8..f176f57 100644 --- a/promptwizard/glue/promptopt/techniques/critique_n_refine/core_logic.py +++ b/promptwizard/glue/promptopt/techniques/critique_n_refine/core_logic.py @@ -3,6 +3,7 @@ from os.path import join from tqdm import tqdm from typing import Any, Dict, List +import json from ....paramlogger import ParamLogger from ....paramlogger.constants import LogLiterals @@ -14,6 +15,30 @@ from ...techniques.critique_n_refine.base_classes import CritiqueNRefinePromptPool +def extract_between(start, end, text): + """ + Extracts the substring from 'text' that is between 'start' and 'end' strings. + + Parameters: + - start (str): The starting delimiter string. + - end (str): The ending delimiter string. + - text (str): The text to search within. + + Returns: + - str: The extracted substring between the start and end delimiters. + """ + start_index = text.find(start) + if start_index == -1: + return '' + + start_index += len(start) + + end_index = text.find(end, start_index) + if end_index == -1: + return '' + return text[start_index:end_index] + + class CritiqueNRefine(PromptOptimizer, UniversalBaseClass): """ TODO: Explain this method @@ -281,7 +306,11 @@ def extract_examples_frm_response(self, response_with_examples: str) -> List: answer_with_reason = text[text.find(DatasetSpecificProcessing.ANSWER_KEY_IN_PROMPT) + len(DatasetSpecificProcessing.ANSWER_KEY_IN_PROMPT):].strip() - final_answer = self.data_processor.extract_final_answer(answer_with_reason) + if self.data_processor != None: + final_answer = self.data_processor.extract_final_answer(answer_with_reason) + else: + final_answer = extract_between(text=answer_with_reason,start="",end="") + formatted_data = { DatasetSpecificProcessing.QUESTION_LITERAL: question, @@ -375,7 +404,7 @@ def generate_best_examples_zero_shot(self,params: PromptOptimizationParams) -> L few_shot_critique_prompt = self.prompt_pool.examples_critique_template_zero_shot.\ format(prompt=params.base_instruction, task_description=params.task_description, - num_examples=params.few_shot_count) + num_examples=params.num_train_examples) critique = self.chat_completion(few_shot_critique_prompt, self.prompt_pool.expert_profile) @@ -385,15 +414,27 @@ def generate_best_examples_zero_shot(self,params: PromptOptimizationParams) -> L gt_example="", critique=critique, task_description=params.task_description, - num_examples=params.few_shot_count) + num_examples=params.num_train_examples) synthetic_examples = self.chat_completion(few_shot_opt_prompt, self.prompt_pool.expert_profile) synthetic_examples = self.extract_examples_frm_response(synthetic_examples) return synthetic_examples @iolog.append_to_chained_log def get_best_instr_by_critique(self, examples: List, params: PromptOptimizationParams): - example_string = self.data_processor.collate_to_str(examples, + + if self.data_processor != None: + example_string = self.data_processor.collate_to_str(examples, self.prompt_pool.quest_reason_ans) + else: + example_string = "" + for example in examples: + answer = example[DatasetSpecificProcessing.FINAL_ANSWER_LITERAL] + if DatasetSpecificProcessing.ANSWER_WITH_REASON_LITERAL in example: + answer = example[DatasetSpecificProcessing.ANSWER_WITH_REASON_LITERAL] + + example_string += self.prompt_pool.quest_reason_ans.format(question=example[DatasetSpecificProcessing.QUESTION_LITERAL], + answer=answer) + meta_critique_prompt = self.prompt_pool.meta_critique_template.format(instruction=params.base_instruction, examples=example_string) critique_text = self.chat_completion(meta_critique_prompt, self.prompt_pool.expert_profile) @@ -402,11 +443,15 @@ def get_best_instr_by_critique(self, examples: List, params: PromptOptimizationP critique=critique_text, steps_per_sample=1) refined_prompts = self.chat_completion(critique_refine_prompt) - refined_instructions = re.findall(self.data_processor.TEXT_DELIMITER_PATTERN, refined_prompts) + + if self.data_processor != None: + refined_instructions = re.findall(self.data_processor.TEXT_DELIMITER_PATTERN, refined_prompts) + else: + refined_instructions = re.findall(DatasetSpecificProcessing.TEXT_DELIMITER_PATTERN, refined_prompts) return refined_instructions[0] if refined_instructions else None - def get_best_prompt(self, params: PromptOptimizationParams,use_synthetic_examples=False,run_without_train_examples=False,use_only_synthetic_examples=False) -> (str, Any): + def get_best_prompt(self, params: PromptOptimizationParams,use_examples=False,run_without_train_examples=False,generate_synthetic_examples=False) -> (str, Any): """ Perform `params.max_iterations` iterations for optimizing your prompt. And return the best prompt found so far. @@ -416,8 +461,8 @@ def get_best_prompt(self, params: PromptOptimizationParams,use_synthetic_example current_base_instruction = params.base_instruction - if not use_only_synthetic_examples: - print("\nMutating and Refining Task Description....") + if not generate_synthetic_examples: + print("\nMutating Task Description....") # Mutate and refine task description for round_num in tqdm(range(1, params.mutate_refine_iterations+1), desc="Iterations completed: "): self.logger.info(f"{CommonLogsStr.LOG_SEPERATOR} + Starting iteration: {round_num} \n " @@ -431,7 +476,7 @@ def get_best_prompt(self, params: PromptOptimizationParams,use_synthetic_example prompt_index = 1 print("\nOptimization Finished...") print("\nPossible prompt variations:") - for candidate in candidate_prompts: + for candidate in candidate_prompts[:5]: final_best_prompt = self.prompt_pool.final_prompt.format( instruction=candidate, answer_format=params.answer_format, @@ -491,15 +536,18 @@ def get_best_prompt(self, params: PromptOptimizationParams,use_synthetic_example if refined_instruction: params.base_instruction = refined_instruction # comment this to turn off synthetic examples - elif use_synthetic_examples: + elif use_examples: examples = self.generate_best_examples(examples, params) else: - examples = [] - print("\nRefining Task Description and Examples iteratively...") - for i in tqdm(range(params.refine_task_eg_iterations)): - examples = self.generate_best_examples_zero_shot(params) - refined_instruction = self.get_best_instr_by_critique(examples, params) - params.base_instruction = refined_instruction + print("Generating Sythetic Examples....") + train_examples = self.generate_best_examples_zero_shot(params) + with open("train_synthetic.jsonl", 'w') as file: + for record in train_examples: + json.dump(record, file) + file.write('\n') + + print("Synthetic examples saved at train.jsonl....") + return "","" if params.generate_reasoning: @@ -514,8 +562,17 @@ def get_best_prompt(self, params: PromptOptimizationParams,use_synthetic_example f"{DatasetSpecificProcessing.ANSWER_START}" + \ f"{example[DatasetSpecificProcessing.FINAL_ANSWER_LITERAL]}" + \ f"{DatasetSpecificProcessing.ANSWER_END}" - - example_string = self.data_processor.collate_to_str(examples, self.prompt_pool.quest_reason_ans) + if self.data_processor != None: + example_string = self.data_processor.collate_to_str(examples, self.prompt_pool.quest_reason_ans) + else: + example_string = "" + for example in examples: + answer = example[DatasetSpecificProcessing.FINAL_ANSWER_LITERAL] + if DatasetSpecificProcessing.ANSWER_WITH_REASON_LITERAL in example: + answer = example[DatasetSpecificProcessing.ANSWER_WITH_REASON_LITERAL] + + example_string += self.prompt_pool.quest_reason_ans.format(question=example[DatasetSpecificProcessing.QUESTION_LITERAL], + answer=answer) if params.few_shot_count == 0: final_best_prompt = self.prompt_pool.final_prompt.format(