Generation and Evaluation

Find your optimized system prompt automatically

Given shadow traffic of inputs you expect your system to handle, Farsight will automatically generate your use case information and iteratively generate and converge to an optimal system prompt.

Make sure you have your OpenAI API Key before you begin.

Best Prompt

get_best_system_prompts()

Generates, evaluates, and iteratively improves system prompts to produce the optimized system prompt based our your shadow traffic.

Not happy with your results? Set gpt_optimized=True for better results utilizing gpt-4for prompt generation calls although it increases some costs.

Param	Type	Description
shadow_traffic	list[str]	list of shadow traffic inputs length must be greater or equal to 5
gpt_optimized	boolean	optional: if set to True, we optimized open-ai calls to generate the best prompts while minimizing costs. We use gpt-4 for num_iterations*num_prompts_per_iteration calls
num_prompts	int	optional: number of best prompts you would like to be returned (default: 3)
num_iterations	int	optional: number of total prompt optimizing iterations following the ORPO method (default: 3)
num_prompts_per_iteration	int	optional: number of prompts to be generated per iteration (default: 2)
num_inputs_to_test	int	optional: number of inputs to evaluate per generated prompt (default: 3)
rubric	str	optional: description of rubric, use our rubric development suggestions to easily create a rubric
reference_answer	str	optional: reference answer to insert into the prompt with a score of 5, use our rubric development suggestions to easily generate a reference answer

Output Type	Output Definition
list[PromptEvaluation]	includes the top system prompts with the highest score of the generated prompts with all of the information in the PromptEvaluation object shown below.

Prompt Evaluation Response Format

PromptEvaluation Object: 
    score: average score accross tested inputs
    system_prompt: generated system prompt
    test_results: TestResults Object:
        score: score of test input:
        input: inputed message
        output: response generated from gpt-3.5-turbo 

from farsightai import FarsightAI

# Replace with your openAI credentials
OPEN_AI_KEY = "<openai_key>"

farsight = FarsightAI(openai_key=OPEN_AI_KEY)

shadow_traffic = [
    "What are the current job openings in the company?",
    "How can I apply for a specific position?",
    "What is the status of my job application?",
    "Can you provide information about the company's benefits and perks?",
    "What is the company's policy on remote work or flexible schedules?",
    "How do I update my personal information in the HR system?",
    "Can you explain the process for employee onboarding?",
    "What training and development opportunities are available for employees?",
    "How is performance evaluation conducted in the company?",
    "Can you assist with information about employee assistance programs or wellness initiatives?",
]

best_prompts = farsight.get_best_system_prompts(shadow_traffic, gpt_optimized=True)
print (best_prompts)
# Result:
# [
#     PromptEvaluation(
#         score=4.666666666666667, 
#         system_prompt="<SYS> Thank you for reaching out to our HR chatbot. How can I assist you with your HR-related queries while ensuring the protection ...""
#         test_results=[
#               TestResult(
#                     score=5, 
#                     input="What is the company's policy on remote work or flexible schedules?   "
#                     output="Our company recognizes the importance of work-life balance and understands that remote work or flexible schedules can contribute ..."
#               ), 
#               TestResult( ... ),
#               TestResult( ... ),
#           ]),
#     PromptEvaluation( ... ),
#     PromptEvaluation( ... )
# ]