Testing Prompts Systematically Before Real-World Use

How to evaluate prompts: define success criteria, build a test set, choose grading methods, and retest every time the prompt changes.

You tweak your chatbot's prompt to make it more polite. You try three questions, it looks great, and you ship it. Two days later the sales team reports that the bot is quoting wrong prices, because that edit made it ignore the attached price table. The new prompt was not bad. The problem was "I tried it and it felt fine" instead of testing.

What is prompt evaluation?

Prompt evaluation, or evals, means measuring a prompt's quality with a predefined test set, clear criteria, and a run you can repeat every time you make a change, just like unit tests in software. OpenAI's prompting guide recommends treating prompts as application code and running prompt tests and evaluation cases on every publish.

Step 1: Define success criteria

Anthropic's "Define success criteria and build evaluations" documentation says good criteria are:

• Specific: not "performs well" but "classifies review sentiment accurately".
• Measurable: mostly numbers, with qualitative measures alongside if applied consistently.
• Achievable: based on benchmarks or prior experiments, not beyond what the model can do.
• Relevant: aligned with what users actually need.

Anthropic also notes that most use cases need multidimensional evaluation, such as accuracy, consistency, tone, privacy preservation, latency and cost.

Step 2: Build a test set

Mirror the real task, include edge cases

Your test set should look like real inputs and include hard cases. Anthropic's examples include irrelevant or missing input, overly long input, sarcasm, mixed sentiment, typos, and cases where even humans would disagree.

Volume matters

Anthropic advises that more questions with slightly lower-signal automated grading beat fewer questions with careful human grading.

Step 3: Choose grading methods

• Code-based grading: such as exact match for classification. Fastest and most reliable, and works well when output is Structured Output.
• Similarity measures: such as cosine similarity to check that differently worded questions with the same meaning get consistent answers, or ROUGE-L for summaries.
• LLM as grader: have a model score 1 to 5 against criteria or answer yes/no. Anthropic recommends using a different model from the one that generated the answer.
• Human grading: most flexible but slow and costly. Use it for random samples or critical cases.

Step 4: Rerun whenever something changes

Evals are only valuable when run consistently. Run the full test set every time you:

• Edit the prompt wording, even a single sentence.
• Change the model or model version.
• Change reference data, such as a price table or policy.
• Add or modify tools the AI calls.

Example: a restaurant review classifier

A team uses AI to sort reviews into positive, negative and neutral. The criteria: classify correctly according to the team's labels, and always answer in a format the system can read. The test set contains real reviews with personal details removed, including sarcastic ones like "so delicious I had to wait 50 minutes". Every time the prompt changes, the team runs the full set and compares scores with the previous version. If the score drops, it does not ship.

If you cannot measure improvement, you do not know whether your prompt edit made things better or only felt like it did.

Things to watch out for

• LLM graders have biases and make mistakes. Periodically compare them with human scores on a sample.
• A test set that never updates falls behind real use. Keep adding cases that failed in production.
• Do not put real personal data in test sets.

How to start

1. Write 2 to 3 success criteria that can be measured as numbers.
2. Collect 30 to 50 real inputs with correct answers, with edge cases making up at least one in five.
3. Write a script or use a spreadsheet to run every case and grade automatically wherever possible.
4. Record each prompt version's score and rerun whenever you edit the prompt or change the model.

Systematic prompt testing starts with a well-designed System Prompt, and it is a skill teams building AI products need. To see where your AI skills stand, try a quiz on Plants.work.