Skip to content
v1·1 version

Build an eval set from real production failures before touching a prompt

Turn a pile of logged bad outputs into a scored eval set, so the next prompt change is measured instead of guessed.

by Chisom collins · published Sep 6, 2026

Reported, not yet verifiedGPT-4o · a month ago · 1 report

Almost everyone edits a prompt, eyeballs three outputs, and ships. Then a model updates and nobody can tell whether the regression is new or was always there. This builds the eval set first, from failures that actually happened rather than cases someone imagined. The discipline that matters is refusing to write a test you cannot score automatically, because a rubric that needs a human to judge it will never be run twice.

Target models
claude-opus-4.5, claude-sonnet-4.5, gpt-4o
Time
45 min
Difficulty
Hard
Cost
20,000 input tokens across

Tools / stack

Claude APIyour logs or a CSV exportPython or a spreadsheet

Inputs required

Twenty or more logged production outputs you consider wrong, with their inputs, plus the current prompt

Steps

  1. 1.Group the failures by what went wrong, not by which customer hit them. Three or four real categories usually cover the lot.
  2. 2.For each group, write the shortest input that reproduces the failure. If you cannot reproduce it, drop it and say so rather than guessing.
  3. 3.Define a pass condition per case that a script can check: a regex, a JSON schema, an exact value, a length bound.
  4. 4.Reject any case whose pass condition needs human judgement. Those belong in a separate list to look at by hand, not in the eval.

Expected output

A scored eval set with input, pass condition, and baseline result per case, plus a separate list of failures that could not be reduced to an automatic check

Golden Example

Input
34 logged bad outputs from a document extraction prompt, spanning four months
Output
21 cases with automatic pass conditions across three groups (missing field, wrong date format, hallucinated value). Nine dropped as unreproducible. Four moved to the manual list because “reads oddly” is not scoreable. Baseline 13 of 21 on the current prompt and model.

Sign in to verify this playbook: run it and log what happened.

Reports

3 reports · 2 worked · 1 changed · 0 broke

Raw reproduction reports: evidence, not a verified badge yet.

  • Worked· claude-opus-4.5
    v1·Sep 6, 2026

    Ran it on 60 failures from a support classifier. Four clean groups came out.

    by Kelv

  • Worked· claude-opus-4.5
    v1·Sep 6, 2026

    Shortest reproducing input per case cut my eval runtime by about 70 percent.

    by The Je’don

  • Worked with changes· gpt-4o
    v1·Sep 6, 2026

    Grouped well but two pass conditions needed judgement despite the instruction.

    by TEKAnology