Know that it works, and keep knowing

Traditional software tests check that the same input gives the same output. AI systems are probabilistic, so "it worked in the demo" is not evidence. Chapter 4 shows how to measure quality systematically, before release and in production.

Test at four levels

Test each of these separately, because each fails in its own way:

• components: single prompts and tools, such as whether extraction returns the right fields;
• retrieval: whether the right passages are found (recall, precision);
• end-to-end results: whether the final answer is correct, grounded and useful;
• agent trajectories: whether the agent took sensible steps, called the right tools and stayed within its limits.

Build the golden dataset in week one

A golden dataset is a set of realistic inputs with expected outputs or grading rubrics. It is the most valuable asset of an AI project. Build it early and grow it continuously:

• typical cases that cover the everyday workload;
• edge cases: missing data, unusual formats, ambiguous requests;
• adversarial cases: prompt injection and attempts to extract data;
• regression cases: every production failure becomes a new test.

LLMs can help generate candidates, but a domain expert must verify the labels.

Grade with the cheapest reliable method

Use, in this order:

1. deterministic checks: exact match, schema validation, numeric tolerance, required citations;
2. LLM-as-judge for open-ended quality such as tone, completeness and faithfulness, calibrated against human ratings;
3. human review for calibration, audits and high-stakes cases.

A judge that is never compared with humans is an opinion, not a measurement.

Gate releases on evaluations

Every change to a prompt, model, retrieval setting or agent logic runs the evaluation suite in CI. The release passes only if quality, cost and latency stay within agreed thresholds. This makes prompt changes as safe as code changes (see also the DevOps chapter).

Keep evaluating in production

Production traffic is different from your test set. Online evaluators score a sample of live runs, user feedback (thumbs, edits, overrides) is captured, and negative cases flow back into the golden dataset. Evaluation is a loop, not a phase.

Key takeaways

• Build the golden dataset in week one; grow it with every failure.
• Grade with the cheapest reliable method.
• Gate releases on evaluations, then keep evaluating in production.