Given shadow traffic of inputs you expect your system to handle, Farsight will automatically generate your use case information and iteratively generate and converge to an optimal system prompt.
Make sure you have your OpenAI API Key before you begin.
Best Prompt
get_best_system_prompts()
Generates, evaluates, and iteratively improves system prompts to produce the optimized system prompt based our your shadow traffic.
Not happy with your results? Set gpt_optimized=True for better results utilizing gpt-4for prompt generation calls although it increases some costs.
Prompt Evaluation Response Format
PromptEvaluation Object:
score: average score accross tested inputs
system_prompt: generated system prompt
test_results: TestResults Object:
score: score of test input:
input: inputed message
output: response generated from gpt-3.5-turbo
from farsightai import FarsightAI# Replace with your openAI credentialsOPEN_AI_KEY ="<openai_key>"farsight =FarsightAI(openai_key=OPEN_AI_KEY)shadow_traffic = ["What are the current job openings in the company?","How can I apply for a specific position?","What is the status of my job application?","Can you provide information about the company's benefits and perks?","What is the company's policy on remote work or flexible schedules?","How do I update my personal information in the HR system?","Can you explain the process for employee onboarding?","What training and development opportunities are available for employees?","How is performance evaluation conducted in the company?","Can you assist with information about employee assistance programs or wellness initiatives?",]best_prompts = farsight.get_best_system_prompts(shadow_traffic, gpt_optimized=True)print (best_prompts)# Result:# [# PromptEvaluation(# score=4.666666666666667, # system_prompt="<SYS> Thank you for reaching out to our HR chatbot. How can I assist you with your HR-related queries while ensuring the protection ...""
# test_results=[# TestResult(# score=5, # input="What is the company's policy on remote work or flexible schedules? "# output="Our company recognizes the importance of work-life balance and understands that remote work or flexible schedules can contribute ..."
# ), # TestResult( ... ),# TestResult( ... ),# ]),# PromptEvaluation( ... ),# PromptEvaluation( ... )# ]
list of shadow traffic inputs length must be greater or equal to 5
gpt_optimized
boolean
optional: if set to True, we optimized open-ai calls to generate the best prompts while minimizing costs. We use gpt-4 for num_iterations*num_prompts_per_iteration calls
num_prompts
int
optional: number of best prompts you would like to be returned (default: 3)
num_iterations
int
optional: number of total prompt optimizing iterations following the ORPO method (default: 3)
num_prompts_per_iteration
int
optional: number of prompts to be generated per iteration (default: 2)
num_inputs_to_test
int
optional: number of inputs to evaluate per generated prompt (default: 3)
rubric
str
optional: description of rubric, use our rubric development suggestions to easily create a rubric
reference_answer
str
optional: reference answer to insert into the prompt with a score of 5, use our rubric development suggestions to easily generate a reference answer
list[PromptEvaluation]
includes the top system prompts with the highest score of the generated prompts with all of the information in the PromptEvaluation object shown below.
