Evaluation, Testing & Optimization

"The output is good" is not a metric. Before optimizing anything you need numbers across four axes, and they pull against each other, which is the point.

• Quality: task-specific and the hard one. Exact match for extraction, an F-score for classification, a rubric score for generation. Define it per task; there is no general answer.
• Latency: report percentiles, not the mean. P50 tells you the typical experience; P95 and P99 tell you what your unhappy users see, and they are what an SLA is actually written against.
• Cost: per unit of work, not per token. Cost per resolved ticket is a number a business can act on; cost per million tokens is not.
• Safety: rate of harmful, non-compliant, or out-of-policy output. Usually rare, which means you need deliberate adversarial cases rather than waiting for production traffic to produce them.

The reason to name all four before tuning is that improving one silently degrades another. A verification pass raises quality and raises latency and cost. Lowering effort cuts cost and may cut quality. If you only measure the one you were trying to improve, you will ship a regression you never saw.

Three grading methods, each with a distinct cost and blind spot. Production systems use all three.

Programmatic checks: exact match, schema validation, regex, "does this parse". Cheap, deterministic, run on every change. Limited to properties you can express as code, which excludes most of what "good" means for generated prose.

Model-graded (LLM-as-judge): a model scores the output against a rubric. Scales to properties code cannot express. Two cautions worth knowing: the rubric must state gradeable criteria ("the summary names every party in the contract"), not vibes ("the summary is good"); and a judge sharing the generator's blind spots will confidently approve the errors it would have made itself.

Human review: the ground truth, and the bottleneck. Use it to calibrate the other two rather than as the primary loop: sample enough to know whether your model-grader agrees with people, then trust the grader within that measured agreement.

For the eval set itself: build it from real inputs, including the ones that failed. A set of cases you invented reflects what you imagined going wrong. Cover the ordinary path, the known edge cases, and deliberate adversarial input. Freeze a portion as a regression set that never changes, so you can tell whether today's improvement broke last month's fix.

On A/B testing: change one variable at a time, decide the sample size before looking at results, and be willing to read "no measurable difference" as the answer. LLM output is noisy enough that a handful of examples proves nothing in either direction, and the pressure to declare a winner from an underpowered test is exactly the failure this objective is testing.

The competence being tested is localizing the fault before changing anything. Four candidate causes, and the observable that distinguishes them:

Retrieved context. Look here first for any RAG system. Read what was actually retrieved. If the right material wasn't in the context, the prompt is not your problem and editing it will not help.

The prompt. Suspect it when the retrieved context was correct and the model still went wrong: an ambiguous instruction, a missing constraint, a format the model has to guess at.

Model choice or effort. Suspect when the task genuinely requires reasoning the tier isn't delivering, and failures cluster on the harder cases rather than appearing uniformly.

Surrounding code. Parsing, truncation, a tool returning something unexpected, a silent exception. Easy to overlook precisely because it's the part you're confident about.

The highest-value diagnostic pattern to recognize: what changed, and when? A system that was fine and is now confidently wrong, with model version and latency unchanged, points at whatever did change, usually the data. Confident-but-wrong answers appearing right after a content refresh is the canonical retrieval-side signature: a broken re-index, a mismatched embedding model, or stale chunks that no longer reflect the source.

Sudden versus gradual is the other useful axis. Sudden degradation implicates a discrete change: a deploy, a re-index, a config edit. Gradual degradation implicates drift: the input distribution moving away from what your prompt and eval set assumed.

Cost and latency levers, in rough order of "free" to "costly":

Prompt caching. The largest win available when a stable prefix repeats across requests, and it costs nothing in quality, cache reads run at roughly a tenth of base input price. Always the first thing to check.

Lower effort. Often produces comparable quality at meaningfully lower token spend and latency. Sweep it against your eval set rather than assuming a direction; the relationship isn't linear, and on agentic work higher effort sometimes reduces total cost by needing fewer turns.

Trim the context. Retrieve five relevant chunks instead of twenty marginal ones. This usually improves quality as well as cost, because irrelevant context dilutes attention.

Batch what isn't interactive. Work with no user waiting on it can run asynchronously at a substantial discount.

Route by difficulty. Send easy cases to a cheaper tier and escalate the hard ones. This requires a reliable difficulty signal, without one you have added a failure mode rather than removed a cost.

Change model tier. The blunt instrument, and the one to reach for last, because it changes quality across the board rather than at a chosen point.

The discipline that makes this an architect-level skill rather than a knob-turning exercise: measure quality on your eval set after every change. A cost reduction with no measured quality number is not an optimization, it is an undetected regression. Use the token-counting endpoint against the specific model you're deploying rather than estimating, token counts are model-specific, and a count measured on a different model can be off by a third or more.