← Community

Build An Eval Set

Replace "it seems better" with a small, honest test set — build cases from real failures, grade what actually matters, and keep a held-out slice. Use when measuring prompt or model changes, or when quality keeps regressing unnoticed.

---
name: build-an-eval-set
category: ai-agents
description: >-
  Replace "it seems better" with a small, honest test set — build cases from real
  failures, grade what actually matters, and keep a held-out slice. Use when
  measuring prompt or model changes, or when quality keeps regressing unnoticed.
---

# Build an Eval Set

Without evals, every prompt change is a vibe. You improve the case you were
staring at, quietly break two you were not, and find out from a user in three
weeks.

You do not need a benchmark. **Twenty real cases beat two thousand synthetic
ones**, because synthetic cases test the failure modes you already imagined.

## Where cases come from

In order of value:

1. **Real failures.** Every bug report, every thumbs-down, every output someone
   had to fix by hand. These are free and they are exactly what you are losing
   users to.
2. **Real successes.** Cases that work now. These are your regression net — the
   ones a "better" prompt will silently break.
3. **Edge cases you can name.** Empty input, enormous input, hostile input,
   ambiguous requests, the wrong language, contradictory instructions.
4. **Synthetic filler.** Last, and only to cover a shape you cannot get real
   examples of yet.

Start collecting today even if you build nothing else. A folder of real failures
is worth more than a scoring harness with nothing good to score.

Shared publicly by the author and open for any use. See the Terms.