Table of Contents
Open Table of Contents
Introduction
In recent years, the landscape of AI has undergone dramatic transformation. While ML or AI-powered products were once the domain of a select few companies and teams, the advent of Large Language Models (LLMs) has democratized intelligence, enabling everyone to build AI powered products.
However, AI responses or products are not deterministic and measuring efficacy of these products on various tasks could be a new & challenging territory to navigate.
As someone building LLM-powered products, I am developing mental models and frameworks to help in development and evaluation of AI-powered products
Framework for effective evaluation of AI powered products
Step 1 : Define your task
- Clearly articulate the specific task your LLM-powered product is designed to perform.
- Task : “Generate code snippets as per context, Generate text, classification, sentiment analysis, QnA, conversations bot, etc”
Step 2 : Define quantitative metrics to evaluate response
These aspects of LLM responses can be measured :
- Task Fidelity : How well the model is performing the task :
- F1 score, Precision, Recall, (Typically classification tasks, etc)
- BLEU Score : To evaluate quality of translated text against human reference
- Perplexity : How well a model predicts a sample
- Custom metrics like “responses contain so and so keywords”
- Task Consistency : Models responses are consistent across similar inputs :
- Cosine similarity of responses
- Context Utilisation : Provide the conversations context and define a likert scale (1-5)
- Latency : Prompt response
- Cost : Cost per LLM call based on request & response tokens utilised and your production volume!
Step 3 : Define an evaluation dataset
Design principles :
- Good indicator of real-world task distribution! Add edge cases as well
- Have good volume of test prompts
- For each prompt their is golden answer or golden keywords/claims
Step 4 : Define automated grading approach
- Code based for simple metrics that can be calculated like for classifier F1 score, accuracy, precision, BLEU score for translation tasks, etc
- LLM based : Custom metrics like “responses contain so and so keywords or claims”, Ensure to structure your output to make it easier to calculate scores! (like a series of true/false on claims)
Step 5 : Report scores & compare against baseline scores
- Have baseline scores to check the efficacy of your solution pipeline (LLM + your custom components)