# Automated System Prompt Optimization (OPRO)

Elevate your LLM application development process with Farsight OPRO: automated system prompt optimization. This library intelligently iterates through prompts to efficiently find the optimal one for any given LLM system built on OpenAI - the approach is based on the Google DeepMind paper  ['Large Language Models as Optimizers'](https://arxiv.org/abs/2309.03409).

Simply provide a dataset with inputs and target outputs *(min 3 pairs, though recommend at least 50)*, and Farsight OPRO will converge to the optimal prompt in one line of code. 

{% hint style="info" %}
Want to integrate quickly? Try out Farsight OPRO on a Colab notebook [here.](https://colab.research.google.com/drive/1QRdrQhgyX4bDmCBBahJg78woeZQcFVTZ#scrollTo=sfzimoOEr0_5)
{% endhint %}

### Installation[​](https://docs.confident-ai.com/docs/getting-started#installation) <a href="#installation" id="installation"></a>

Install our library by running:&#x20;

```
pip install farsight-opro
```

**Instantiation**

Begin using the SDK with the following few lines of code:

```python
# with openai credentials 
client = FarsightOPRO(openai_key="<openai_key>")

# with azure credentials
client = FarsightOPRO(
    openai_key="<azure_openai_key>",
    azure_endpoint="<azure_endpoint",
    api_version="<api_version>",
    model="<model_name>"
)
```

**Dataset Configuration**

The Farsight OPRO library requires only a dataset from the user. It expects datasets in the form of a list of dictionaries, as illustrated below:

```python
[{"input": "x", "target": "y"}, ... , {"input": "a", "target": "z"}]
```

For those users that may not have a dataset easily available in this format, we recommend generating a synthetic dataset from ChatGPT to get started. Below is a suggested prompt that will quickly generate a dataset that meets the Farsight OPRO spec:

{% code overflow="wrap" %}

```
'''Create an input-output test set in the form of a python list of dictionaries with 10 samples, reflecting question-answer pairs that might be asked during [describe your use case]. Each item in the list should be a dictionary with keys "input" and "target".'''
```

{% endcode %}

### Example Usage <a href="#installation" id="installation"></a>

```python
import json
import random
from sklearn.model_selection import train_test_split
from opro import FarsightOPRO

# replace with your openAI credentials
OPEN_AI_KEY = "<openai_key>"
farsight = FarsightOPRO(openai_key=OPEN_AI_KEY)

# load dataset
dataset_path = "/content/movie_recommendation.json"
with open(dataset_path, "r") as file:
    data = json.load(file)

# split dataset
dataset, test_set = train_test_split(
    data["examples"],
    train_size=0.4
)

##################### For a short test run, try this #####################

# dataset = [
#     {'input': 'Find a movie similar to Batman, The Mask, The Fugitive, Pretty Woman:\nOptions:\n(A) The Front Page\n(B) Maelstrom\n(C) The Lion King\n(D) Lamerica','target': '(C)'},
#     {'input': 'Find a movie similar to The Sixth Sense, The Matrix, Forrest Gump, The Shawshank Redemption:\nOptions:\n(A) Street Fighter II The Animated Movie\n(B) The Sheltering Sky\n(C) The Boy Who Could Fly\n(D) Terminator 2 Judgment Day', 'target': '(D)'},
#     {'input': "Find a movie similar to Schindler's List, Braveheart, The Silence of the Lambs, Tombstone:\nOptions:\n(A) Orlando\n(B) Guilty of Romance\n(C) Forrest Gump\n(D) All the Real Girls", 'target': '(C)'},
#  ]
# prompts_and_scores = farsight.generate_optimized_prompts(dataset, prompts_generated_per_iteration=2, num_iterations=3)
# print(prompts_and_scores)

########################################################################


# get optimized prompts
prompts_and_scores = farsight.generate_optimized_prompts(dataset, test_set)
print(prompts_and_scores) 
#  [{
#        "prompt": "Choose the movie option that aligns with the given movies' genres, popularity, critical acclaim, and overall quality to provide the most accurate and comprehensive recommendation."
#        "score": 0.94,
#        "test_score": 0.88
#
#   },
#   {
#        "prompt": "Choose the movie option that aligns with the genres, themes, popularity, critical acclaim, and overall quality of the given movies to provide the most accurate and comprehensive recommendation."
#        "score": 0.9,
#        "test_score": 0.86
#   }, ...
```