# Generation and Evaluation

Given shadow traffic of inputs you expect your system to handle, Farsight will automatically generate your use case information and iteratively generate and converge to an optimal system prompt.

{% hint style="info" %}
Make sure you have your [OpenAI API Key](https://platform.openai.com/account/api-keys) before you begin. 
{% endhint %}

Best Prompt

`get_best_system_prompts()`

Generates, evaluates, and iteratively improves system prompts to produce the optimized system prompt based our your shadow traffic. 

{% hint style="success" %}
Not happy with your results? Set `gpt_optimized=True` for better results utilizing `gpt-4`for prompt generation calls although it increases some costs. 
{% endhint %}

<table><thead><tr><th width="259">Param</th><th width="105">Type</th><th>Description</th></tr></thead><tbody><tr><td>shadow_traffic</td><td>list[str]</td><td>list of shadow traffic inputs length must be greater or equal to 5</td></tr><tr><td>gpt_optimized</td><td>boolean</td><td><em>optional</em>: if set to True, we optimized open-ai calls to generate the best prompts while minimizing costs. We use gpt-4 for <strong>num_iterations*num_prompts_per_iteration calls</strong></td></tr><tr><td>num_prompts</td><td>int</td><td><em>optional</em>: number of best prompts you would like to be returned (default: 3)</td></tr><tr><td>num_iterations</td><td>int</td><td><em>optional</em>: number of total prompt optimizing iterations following the ORPO method (default: 3)</td></tr><tr><td>num_prompts_per_iteration</td><td>int</td><td><em>optional</em>: number of prompts to be generated per iteration (default: 2)</td></tr><tr><td>num_inputs_to_test</td><td>int</td><td><em>optional</em>: number of inputs to evaluate per generated prompt (default: 3)</td></tr><tr><td>rubric</td><td>str</td><td><em>optional</em>: description of rubric, use our rubric development suggestions to easily create a rubric</td></tr><tr><td>reference_answer</td><td>str</td><td><em>optional:</em> reference answer to insert into the prompt with a score of 5, use our rubric development suggestions to easily generate a reference answer</td></tr></tbody></table>

<table><thead><tr><th width="259">Output Type</th><th>Output Definition</th></tr></thead><tbody><tr><td>list[PromptEvaluation]</td><td>includes the top system prompts with the highest score of the generated prompts with all of the information in the PromptEvaluation object shown below. </td></tr></tbody></table>

### Prompt Evaluation Response Format

```
PromptEvaluation Object: 
    score: average score accross tested inputs
    system_prompt: generated system prompt
    test_results: TestResults Object:
        score: score of test input:
        input: inputed message
        output: response generated from gpt-3.5-turbo 
```

<pre class="language-python"><code class="lang-python">from farsightai import FarsightAI
<strong>
</strong><strong># Replace with your openAI credentials
</strong>OPEN_AI_KEY = "&#x3C;openai_key>"

farsight = FarsightAI(openai_key=OPEN_AI_KEY)

shadow_traffic = [
    "What are the current job openings in the company?",
    "How can I apply for a specific position?",
    "What is the status of my job application?",
    "Can you provide information about the company's benefits and perks?",
    "What is the company's policy on remote work or flexible schedules?",
    "How do I update my personal information in the HR system?",
    "Can you explain the process for employee onboarding?",
    "What training and development opportunities are available for employees?",
    "How is performance evaluation conducted in the company?",
    "Can you assist with information about employee assistance programs or wellness initiatives?",
]

best_prompts = farsight.get_best_system_prompts(shadow_traffic, gpt_optimized=True)
print (best_prompts)
# Result:
# [
#     PromptEvaluation(
#         score=4.666666666666667, 
#         system_prompt="&#x3C;SYS> Thank you for reaching out to our HR chatbot. How can I assist you with your HR-related queries while ensuring the protection ...""
#         test_results=[
#               TestResult(
#                     score=5, 
#                     input="What is the company's policy on remote work or flexible schedules?   "
#                     output="Our company recognizes the importance of work-life balance and understands that remote work or flexible schedules can contribute ..."
#               ), 
#               TestResult( ... ),
#               TestResult( ... ),
#           ]),
#     PromptEvaluation( ... ),
#     PromptEvaluation( ... )
# ]
</code></pre>