Automated System Prompt Optimization (OPRO)

Find your optimized system prompt automatically; removes the need for manual prompt iteration

Elevate your LLM application development process with Farsight OPRO: automated system prompt optimization. This library intelligently iterates through prompts to efficiently find the optimal one for any given LLM system built on OpenAI - the approach is based on the Google DeepMind paper 'Large Language Models as Optimizers'.

Simply provide a dataset with inputs and target outputs (min 3 pairs, though recommend at least 50), and Farsight OPRO will converge to the optimal prompt in one line of code.

Want to integrate quickly? Try out Farsight OPRO on a Colab notebook here.

Installation​

Install our library by running:

pip install farsight-opro

Instantiation

Begin using the SDK with the following few lines of code:

# with openai credentials 
client = FarsightOPRO(openai_key="<openai_key>")

# with azure credentials
client = FarsightOPRO(
    openai_key="<azure_openai_key>",
    azure_endpoint="<azure_endpoint",
    api_version="<api_version>",
    model="<model_name>"
)

Dataset Configuration

The Farsight OPRO library requires only a dataset from the user. It expects datasets in the form of a list of dictionaries, as illustrated below:

[{"input": "x", "target": "y"}, ... , {"input": "a", "target": "z"}]

For those users that may not have a dataset easily available in this format, we recommend generating a synthetic dataset from ChatGPT to get started. Below is a suggested prompt that will quickly generate a dataset that meets the Farsight OPRO spec:

'''Create an input-output test set in the form of a python list of dictionaries with 10 samples, reflecting question-answer pairs that might be asked during [describe your use case]. Each item in the list should be a dictionary with keys "input" and "target".'''

Example Usage

import json
import random
from sklearn.model_selection import train_test_split
from opro import FarsightOPRO

# replace with your openAI credentials
OPEN_AI_KEY = "<openai_key>"
farsight = FarsightOPRO(openai_key=OPEN_AI_KEY)

# load dataset
dataset_path = "/content/movie_recommendation.json"
with open(dataset_path, "r") as file:
    data = json.load(file)

# split dataset
dataset, test_set = train_test_split(
    data["examples"],
    train_size=0.4
)

##################### For a short test run, try this #####################

# dataset = [
#     {'input': 'Find a movie similar to Batman, The Mask, The Fugitive, Pretty Woman:\nOptions:\n(A) The Front Page\n(B) Maelstrom\n(C) The Lion King\n(D) Lamerica','target': '(C)'},
#     {'input': 'Find a movie similar to The Sixth Sense, The Matrix, Forrest Gump, The Shawshank Redemption:\nOptions:\n(A) Street Fighter II The Animated Movie\n(B) The Sheltering Sky\n(C) The Boy Who Could Fly\n(D) Terminator 2 Judgment Day', 'target': '(D)'},
#     {'input': "Find a movie similar to Schindler's List, Braveheart, The Silence of the Lambs, Tombstone:\nOptions:\n(A) Orlando\n(B) Guilty of Romance\n(C) Forrest Gump\n(D) All the Real Girls", 'target': '(C)'},
#  ]
# prompts_and_scores = farsight.generate_optimized_prompts(dataset, prompts_generated_per_iteration=2, num_iterations=3)
# print(prompts_and_scores)

########################################################################


# get optimized prompts
prompts_and_scores = farsight.generate_optimized_prompts(dataset, test_set)
print(prompts_and_scores) 
#  [{
#        "prompt": "Choose the movie option that aligns with the given movies' genres, popularity, critical acclaim, and overall quality to provide the most accurate and comprehensive recommendation."
#        "score": 0.94,
#        "test_score": 0.88
#
#   },
#   {
#        "prompt": "Choose the movie option that aligns with the genres, themes, popularity, critical acclaim, and overall quality of the given movies to provide the most accurate and comprehensive recommendation."
#        "score": 0.9,
#        "test_score": 0.86
#   }, ...