Evaluating AI outputs: building a small test set and scoring rubric

Most teams judge AI output the same way: someone tries a few examples, the answers look good, and the tool goes live. A month later a customer receives a reply with the wrong refund policy, and nobody can say whether the prompt change last week caused it. OpenAI’s guide to evaluation calls this approach “vibe-based evals”, and lists it as an anti-pattern.

You do not need a data science team to do better. A spreadsheet with 30 to 100 real examples and a short scoring rubric will tell you whether a prompt, model or tool is good enough, and whether a change made it better or worse. This guide shows how to build both.

Why a test set beats spot checks

AI output varies from run to run, and a prompt that handles your three favourite examples may fail on the awkward fourth. OpenAI’s evaluation guide describes evals as “a way to test your AI system despite this variability.” A fixed set of test cases lets you compare like with like: the same inputs, scored the same way, before and after every change.

It also turns arguments about quality into numbers. “The new prompt feels better” becomes “the new prompt passed 41 of 50 cases, up from 33.”

Define success first

Anthropic’s guide to defining success criteria and building evaluations says good criteria are specific, measurable, achievable and relevant. “Good performance” is not a criterion. “Uses the correct returns window for the product category” is.

The guide also notes that “most use cases need multidimensional evaluation.” For a customer email assistant, you might care about:

• Accuracy: facts and policies are correct.
• Completeness: every question in the customer’s message is answered.
• Tone: polite, plain and on-brand.
• Safety: no promises outside policy and no personal data exposed.
• Length: within the agreed limit.

Pick three to five. More than that and scoring becomes slow and inconsistent.

Build the test set

OpenAI suggests several sources of test data: real production data, historical logs, cases written by experts and synthetic examples generated by a model. For a small business, the best start is usually your own history, such as past emails, tickets, invoices or briefs, with personal details removed.

Include a mix:

1. Typical cases that represent everyday work, in roughly the proportions you see them.
2. Edge cases: Anthropic lists irrelevant or missing input, overly long input, poor or harmful user input, and ambiguous cases where “even humans would find it hard to reach an assessment consensus.”
3. Known failures: every time the tool gets something wrong in real use, add that case.

For each case, record the input, the expected answer or the key facts a good answer must contain, and any notes. Anthropic’s guide favours volume: more cases with simpler automated grading beat a handful of perfectly hand-graded ones. Start with 30 and grow towards 100.

Write a scoring rubric

A rubric turns each criterion into a clear judgement that two people would score the same way. Anthropic advises detailed, clear rubrics and outputs that are “empirical and specific”, such as correct or incorrect, or a 1 to 5 scale, because purely qualitative feedback is hard to compare at scale.

Write anchors for each score. For example, for accuracy:

• Pass: every fact and policy matches the source documents.
• Fail: any wrong fact, policy, price or date.

For tone, a three-point scale might read: 3, sounds like our best agent and needs no edits; 2, acceptable with light edits; 1, would need rewriting. Pass or fail is easier to apply consistently than a five-point scale, so use it wherever you can.

Choose how to grade

Anthropic describes three grading methods:

If you use an AI model to grade, follow a few rules from both guides. Use a different model from the one that produced the output. Ask the grader to reason before giving its score. Prefer pass or fail and side-by-side comparisons over open-ended scores. Watch for known biases: OpenAI names position bias (favouring the first answer shown) and verbosity bias (favouring longer answers). Then have a person grade a sample too, and only trust the AI grader once it agrees with human scores.

A worked example

An example (illustration only): a small travel agency uses AI to draft replies to booking enquiries. It builds a test set of 40 past enquiries, including eight edge cases such as missing travel dates and a request that breaks its cancellation policy.

Its rubric has four pass or fail checks: correct policy, all questions answered, no unauthorised promises, under 180 words. A simple script checks length. A different AI model grades the other three, and the manager hand-grades ten replies to confirm the grader agrees with her.

Results, all invented for illustration: the current prompt passes all four checks on 26 of 40 cases. A revised prompt that adds the cancellation policy text and two example replies passes on 35 of 40. The five remaining failures all involve group bookings, so the agency adds its group terms to the prompt and five more group cases to the test set.

The whole exercise took the manager about a day to set up. Each rerun now takes under an hour, most of it spent reading the ten replies she grades by hand. When the provider later announces a model update, she reruns the set before switching and can see straight away whether anything slipped.

Keep running it

A test set is most useful as a habit. OpenAI’s guide recommends running evals on every change and growing the set over time. Rerun yours whenever you edit the prompt, change tools or the provider updates the model. OpenAI also recommends pinning production apps to a specific model snapshot so behaviour does not shift without warning. NIST’s generative AI risk profile similarly suggests comparing outputs against known ground truth as an ongoing check, not a one-off.

Common mistakes:

• Testing only easy, typical cases.
• Vague criteria such as “high quality”.
• Changing the test set and the prompt at the same time.
• Trusting an AI grader without checking it against people.
• Building the test set once and never adding new failures.