Prompt Evaluation

Find your optimized system prompt automatically

Farsight AI utilizes the leading approach for prompt optimization by incorporating the Prometheus prompt evaluation rubric to evaluate systems prompts.

We suggest you follow our rubric development process to create the rubric that will then be used to evaluate the performance of potential system prompts. If no rubric is provided, Farsight will auto-generate one for you based on your shadow input traffic.

Make sure you have your OpenAI API Key before you begin.

Here is an example reference answer and rubric.

Reference Answer (Score 5):

Photosynthesis is the process by which plants make their own 
food using sunlight. In simple terms, they take in carbon dioxide from the air and 
water from the soil, and with the help of sunlight, they transform these into sugars, 
which the plant uses as energy. In the process, oxygen is released into the air, 
benefiting the environment. So, photosynthesis is like the plant's way of cooking up 
its own food using sunlight and a few basic ingredients.

Score Rubric:

Can the model's response be understood by a non-expert in the subject?
Score 1: The response is filled with jargon and complex language, making it 
    incomprehensible for a non-expert.
Score 2: the response includes some explanations, but still relies heavily on 
    jargon and complex language. 
Score 3: The response is somewhat clear, but could still be challenging for a 
    non-expert to fully understand. 
Score 4: the response is mostly comprehensible to a non-expert, with only a few 
    complex terms or concepts
Score 5: the response is completely clear and understandable for a non-expert, with 
    no reliance on jargon or complex language.

Best Prompt

get_best_prompt()

Determine which prompt optimizes for your use case with our best_prompt function.

Param	Type	Description
criteria_description	str	description of criteria for the rubric
rubric	str	description of rubric
reference_answer	str	reference answer to insert into the prompt with a score of 5
prompts	list(str)	system prompts to evaluate
outputs	list(str)	optional: we generate outputs using gpt-3.5-turbo. To evaluate on other LLM outputs, generate your outputs and include them here.

Output Type	Output Definition
str	prompt with the highest score of the given prompts according to the given rubric and reference answer
int	score of the best prompt on a scale of 1-5, 1 being low according to the given rubric and 5 being high

from farsightai import FarsightAI

# Replace with your openAI credentials
OPEN_AI_KEY = "<openai_key>"

query = "Can you describe the carbon cycle?"
farsight = FarsightAI(openai_key=OPEN_AI_KEY)

criteria_description = """Can the model's response be understood by a non-expert 
in the subject"""

rubric="""
Score 1: The response is filled with jargon and complex language, making it incomprehensible for a non-expert.
Score 2: the response includes some explanations, but still relies heavily on jargon and complex language. 
Score 3: The response is somewhat clear, but could still be challenging for a non-expert to fully unsterdtnad. 
Score 4: the response is mostly comprehensible to a non-expert, with only a few complex terms or concepts
Score 5: the response is completely clear and understandable for a non-expert, with no reliance on jargon or complex language.
"""

reference_answer="""Photosynthesis is the process by which plants make their own 
food using sunlight. In simple terms, they take in carbon dioxide from the air and 
water from the soil, and with the help of sunlight, they transform these into sugars, 
which the plant uses as energy. In the process, oxygen is released into the air, 
benefiting the environment. So, photosynthesis is like the plant's way of cooking up 
its own food using sunlight and a few basic ingredients."""

best_prompt = farsight.best_prompt(
    criteria_description, 
    rubric,
    reference_answer, 
    prompts)
print("best_prompt: ", best_prompt)