Farsight AI Starter Docs
  • Get Started
    • Welcome to the Farsight AI Starter Library
      • Overview
      • Getting Started
  • Metrics
    • Standard Metrics
    • Custom Metrics
  • Fully Automated Prompt Optimization
    • Introduction
      • Tutorial
    • Generation and Evaluation
  • Step by Step Prompt Optimization
    • Introduction
      • Tutorial
    • Prompt Generation
    • Prompt Evaluation
      • Rubric Development
Powered by GitBook
On this page
  • Overview
  • Prompt Optimization
  • Prompt Generation
  • Prompt Evaluation
  1. Get Started
  2. Welcome to the Farsight AI Starter Library

Overview

Check out Farsight's Capabilities

PreviousWelcome to the Farsight AI Starter LibraryNextGetting Started

Last updated 1 year ago

Get evaluating now! Follow a few simple steps to improve your LLMs.

Want to integrate even quicker? Try out Farsight AI on a Colab notebook .

Note: While you have the flexibility to assess the results of any Language Model (LLM), we specifically leverage OpenAI for some of the evaluation functions in Farsight AI. To utilize our package, you must have access to an .

Overview


Prompt Optimization

For prompt optimization, we offer two distinct approaches - one with manual oversight and one with full automation. Choose the one that aligns best with your use case, workflows and anticipated functionality:

  1. : Generate multiple system prompts for evaluation and testing purposes. Tailor them based on context and optional system guidelines.

  2. : Leverage our comprehensive automated prompt optimization function. This feature not only generates prompts but also evaluates and iteratively improves them. It operates based on your provided shadow traffic, evaluation rubric, and optional ground truth outputs.


Prompt Generation

Our platform also includes a straightforward prompt generation function designed to assist you in creating system prompts tailored to your specific use case and other relevant information.


Prompt Evaluation

For prompt evaluation, we offer two alternative methods:

Define your rubric, and our system will assess your prompts, identifying the optimal one for your needs. This method involves generating multiple system prompts for evaluation and testing, considering context and optional system guidelines.

We provide both standard, off-the-shelf metrics (consistency, conciseness, factuality, quality) and customizable metrics. These metrics enable you to thoroughly evaluate your system prompts and ensure they meet your desired criteria.

Metrics Evaluation:
here
OpenAI API Key
Step by Step Approach
Fully Automated Approach
Prompt Generation:
Rubric Evaluation: