Build · Step 10 of 14

Evals: know if it got better

You changed the prompt and the one example you tried looks better. Is the feature better? Without a test set you are guessing. An eval is that test set — and it is smaller and easier to build than you think.

Why ordinary tests are not enough

Normal code gives the same output for the same input, so one passing test means it works. A model doesn't: the output varies, there are many acceptable answers, and a prompt change that fixes one case quietly breaks another. Trying three examples by hand and feeling good about them is how regressions ship.

An eval is three things: a set of inputs, a way to grade each output, and a score you can compare between runs.

1. Collect the cases

Start with twenty. Real inputs beat invented ones — take them from your logs, your tickets, your own use of the feature. Make sure the set includes:

  • The ordinary cases that must keep working.
  • The awkward ones: empty input, very long input, another language, missing data.
  • Every failure you have actually seen. Each bug becomes a case, so it never comes back unnoticed.

For each case write down what a good answer must contain — not the exact text, the facts or properties that make it right.

2. Decide how to grade

  • By code, whenever you can. Is it valid against the schema? Does it contain the order number? Is the category one of the five allowed? Fast, free, and never wrong about what it checks.
  • By a model, for qualities code can't see: is the summary faithful, is the tone right. Give the grading model a specific rubric and ask for pass or fail per criterion — not "rate this 1 to 10", which produces numbers that mean nothing.
  • By you, at first. Read every output for the first few runs. You will find that some of your cases and your rubric are wrong; fix those before trusting the score.

3. Write the runner

Here is an eval for the release-notes tool from Projects. Each case is a commit log plus the things the notes must and must not mention:

// eval.ts
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();
const MODEL = process.argv[2] ?? "claude-opus-5-5";
const SYSTEM = "You turn raw git commit messages into clean release notes. ...";

const cases = [
  {
    name: "drops noise",
    commits: "- add CSV export\n- wip\n- fix typo in readme",
    mustInclude: ["CSV"],
    mustNotInclude: ["wip", "typo"],
  },
  {
    name: "groups a fix under Fixes",
    commits: "- fix crash when cart is empty",
    mustInclude: ["Fixes", "cart"],
    mustNotInclude: [],
  },
  // ...eighteen more, from real commit logs
];

let passed = 0;
for (const c of cases) {
  const res = await client.messages.create({
    model: MODEL,
    max_tokens: 16000,
    system: SYSTEM,
    messages: [{ role: "user", content: c.commits }],
  });
  const text = res.content
    .map((b) => (b.type === "text" ? b.text : ""))
    .join("")
    .toLowerCase();

  const missing = c.mustInclude.filter((s) => !text.includes(s.toLowerCase()));
  const leaked = c.mustNotInclude.filter((s) => text.includes(s.toLowerCase()));
  const ok = missing.length === 0 && leaked.length === 0;
  if (ok) passed++;
  console.log(ok ? "PASS" : "FAIL", c.name, { missing, leaked });
}
console.log(`${passed}/${cases.length} passed on ${MODEL}`);
npx tsx eval.ts                     # the current setup
npx tsx eval.ts claude-haiku-5-5    # same cases, cheaper model

That is a real eval. It is crude — matching words is a blunt check — and it is still far better than none, because it runs the same way every time.

4. Use it for every decision

  • Changing the prompt? Run before and after. Keep the change only if the score holds or improves — and read the cases that flipped.
  • Picking a model? This is how you answer the AI Picker's question with evidence. If the cheap model passes the same cases, use the cheap model.
  • A new model came out? One command tells you whether to switch.
  • A user reports a bad answer? Add it as a case first, then fix it.
Don't tune until every case passes. A prompt bent to pass twenty specific cases can get worse on everything else. Keep a few cases aside that you never look at while tuning, and check them at the end. If the score on those drops, you have fitted the prompt to the test.

Evaluating an agent

The same idea, one level up: give the agent a task in a prepared folder and check the outcome — do the tests pass, does the file exist, is the answer right — rather than the steps it took. Also record how many steps and tokens it used; an agent that gets there in twice the cost has not improved.

Each run costs real API money, though with twenty cases it is pennies. Run the eval on changes to the prompt or model, not on every commit.