Evaluation guide · 10 min

Build an evaluation set for a business AI system

A successful demonstration says little about ordinary, ambiguous or difficult requests. A small, carefully designed test set makes quality observable and helps catch regressions after a change.

A team reviews documents, charts and notes to evaluate answers.
AI-generated illustration
Key points

Where to start

Collect anonymized real cases, add edge cases and requests the AI should decline. Define an acceptable output, then compare versions with the same rubric and human review.

  • Representative and difficult cases
  • Observable criteria
  • Regular comparison

Build a reference set in six steps

  1. AI-generated illustration

    1. Bound the task

    List permitted tasks, users, response formats and the most costly errors. Sourced research needs different measures from customer support.

  2. AI-generated illustration

    2. Collect examples

    Use common requests and rare but important cases. Remove personal data and keep the origin and date of each example.

  3. Sorting documents with folders and an e-reader
    AI-generated illustration

    3. Add adverse cases

    Include ambiguous requests, conflicting documents, empty inputs, different languages and malicious instructions where these situations can occur.

  4. AI-generated illustration

    4. Write a rubric

    Score accuracy, sources, format, appropriate refusal and correction time. Define observable levels rather than relying on a general impression.

  5. Editorial illustration

    5. Establish a baseline

    Run every case on the current version. Ask two people to review a sample and resolve scoring disagreements before comparing a new version.

  6. AI-generated illustration

    6. Track regressions

    Rerun the set after changing a model, prompt, corpus or tool. Add real incidents and retire outdated examples.

Put the method to work

Document research with an atlas and books in a library
AI-generated illustration

Practical case

Assemble twelve anonymised cases: six common, three ambiguous, two critical and one the system should decline.

Wooden desk with a laptop and monitor
AI-generated illustration

Evidence to keep

For each case, define expected outcome, observable rubric, tested version and any disagreement between reviewers.

Two screens show queued tasks and their results in a nighttime workspace.
AI-generated illustration

Make the decision

Compare versions on this stable set; then add real incidents without retrospectively rewriting baseline results.

Four qualities of a useful set

Creative studio with a camera and drawing tablet
AI-generated illustration

Representative

Cases reflect real uses and their frequency.

Library with books and a research notebook
AI-generated illustration

Risk coverage

Costly errors and expected refusals are included.

AI-generated illustration

Repeatable

Inputs, versions and scoring rules are retained.

AI-generated illustration

Current

Incidents and business changes feed the set.

4 starting points

Compare services on identical cases

Shortlist services suited to the task; the test set reveals differences in your own context.

How is this selection produced?

These profiles are linked to the guide’s subject in our editorial catalogue. Inclusion is neither a ranking nor a performance measurement on your documents. Methodology.

Explore the full category

Related tool families

Frequently asked questions

How many cases are needed?

Start with a small, diverse and stable set, then add incidents and missing cases. Coverage matters more than an arbitrary count.

Is automatic scoring enough?

No. For important outcomes, review a sample manually and check disagreement between evaluators.

When should tests be rerun?

After changes to the model, instructions, data or tools, and regularly for critical cases.

The references below expand on the concepts and checks discussed. Scenarios and trial frameworks remain editorial proposals; provider documentation describes its own product rather than an independent benchmark.

Official sources

Does a search retrieve the right documents?

Evaluate retrieval before judging the generated answer. In a bounded corpus, identify relevant documents for a question, then count relevant hits, irrelevant hits and missed relevant documents. Count each document once and use the same maximum result count when comparing two systems.

Relevant documents and retrieved documents The overlap represents relevant documents retrieved. Missed relevant documents are on the left; irrelevant results are on the right. Areas are schematic and not proportional to counts. Missed Relevant Irrelevant 231
Counts follow your inputs; areas remain schematic. Fictional initial values: five relevant documents, three retrieved, plus one irrelevant result.
Precision: 75% · Recall: 60% · F1: 66.7%

Calculation runs locally in your browser without storage. Without JavaScript, use the formulas below; displayed values remain those of the example.

Development workstation with a code screen
AI-generated illustration
Precision: avoid noise
Relevant retrieved ÷ all retrieved documents. Here: 3 ÷ (3 + 1) = 75%.
AI-generated illustration
Recall: avoid omissions
Relevant retrieved ÷ all relevant documents in the corpus. Here: 3 ÷ (3 + 2) = 60%.
AI-generated illustration
F1
2 × relevant retrieved ÷ (2 × relevant retrieved + irrelevant retrieved + relevant missed). Here: 6 ÷ 9 ≈ 66.7%.

A zero denominator produces “undefined”. Without an inventory of relevant documents, recall is unknown: do not infer it across the entire web. These measures assess neither answer truthfulness nor the severity of a missed document; omitting a decisive exception remains a blocking error.

Metric definitions: scikit-learn documentation

Continue with another guide

Keep evidence from this trial

Connect this material to a decision and a retest condition. Prepare a decision record →