← Community

Eval after you ship

Sample live traces, grade a slice on a schedule, and catch drift the offline eval set never saw — instead of trusting a spreadsheet you ran once. Use when a model is in production, after a prompt or model change, or when quality complaints do not match your eval score.

---
name: eval-after-you-ship
category: ai-agents
description: >-
  Sample live traces, grade a slice on a schedule, and catch drift the
  offline eval set never saw — instead of trusting a spreadsheet you ran
  once. Use when a model is in production, after a prompt or model change, or
  when quality complaints do not match your eval score.
---

# Eval After You Ship

An eval set is a snapshot of the problems you already knew about. Production
is where users ask the questions you did not write down, in the words you
did not think of, against the data that arrived this week.

If the only grade you have is from last quarter's thirty examples, you
are not measuring the product. You are measuring your memory of the
product.

## Sample the traces you actually serve

Log enough to **replay**: input (redacted), model id, prompt version, tools
called, output, and whether a human later fixed it. If you cannot replay,
you cannot grade.

Then sample, on a schedule, not when someone is angry:

- A **random slice** of traffic (so you see the ordinary cases).
- A **tail slice**: the slow, the retried, the ones users edited, the
  ones that hit a fallback. Failures hide in the tail. An eval that only
  looks at happy JSON is a demo.

Twenty to fifty graded traces a week beats a heroic 500 once a quarter.
The point is a trend, not a paper.

Hold the grader to a rubric you wrote when you were calm. Grade **blind**
to which prompt version produced the output, or you will prefer the one
you just shipped.

Shared publicly by the author and open for any use. See the Terms.