Running a two-week bake-off that ends in a decision

Two vendors, or two approaches, or one vendor and the status quo. You have a fortnight and a small budget, and at the end of it you need to be able to say yes or no in a way that survives someone asking why.

The failure mode is well established. The pilot runs, everybody forms an impression, and the final meeting is an argument about whether it felt better — settled by whoever is most senior or most enthusiastic. The way to prevent that is unglamorous and takes about ninety minutes: decide what would count as success before you have seen anything.

Before you start: write the criteria down

Anthropic's guidance on defining success criteria is a good checklist precisely because it is boring. Criteria should be specific ("correctly extracts the renewal date" rather than "handles contracts well"), measurable, achievable against some external reference rather than invented optimism, and relevant to what the thing is actually for — citation accuracy matters enormously for a clinical tool and barely at all for a brainstorming aid.

Write down, and circulate before day one:

• The threshold. Not "as good as possible". A number, with a rationale. "90% on field extraction, because below that the checking costs more than the typing it replaced."
• The floor for the thing you can't compromise on. Usually one item. If it fabricates a citation even once in 200 cases, that is disqualifying regardless of everything else.
• The budget per transaction, and what it may not exceed at projected volume.
• The latency you can live with. Users abandon things. Price is a named criterion and so is latency, for the same reason: both are invisible in a pilot and decisive in production.
• Who decides. One name. Agreed in advance, so the decision isn't relitigated by whoever is unhappiest with it.

The point of writing these first is not rigour for its own sake. It's that thresholds chosen after you have seen the results tend to land conveniently near the results.

Days one to three: build the test set

This is the whole exercise. Anthropic's own framing is unambiguous — a good evaluation set is the most important step, and you test with your actual prompts and your actual data. Everything else here is scaffolding around that sentence.

Pull real cases from your own history. Aim for more than feels necessary: the guidance is explicit that many cases graded automatically beats a few graded painstakingly by hand, and the published examples run to hundreds — a thousand items for a classification task, two hundred documents for summarisation, a hundred conversations for judging context use. If you can only manage sixty, do sixty, but know that the number is the limit on what you can conclude.

Make sure the set contains the cases nobody wants to put in it:

• Input that is irrelevant, or missing entirely.
• Input far longer than typical — the forty-page contract, the customer who wrote six paragraphs.
• Badly formed, hostile, or nonsense input, if members of the public can reach it.
• Cases where two of your own experts would disagree about the right answer. These are not a distraction; they tell you the ceiling. If your own people agree only 80% of the time, no system is going to score 95% against a single "correct" answer, and a vendor claiming otherwise is measuring something else.

Hold some back. Anything the vendors get to iterate against stops being a test.

Days four to five: decide how you'll grade

Match the method to the task and automate what you can.

Exact match for categories, dates, amounts, yes/no. Cheap, unambiguous, and a spreadsheet formula does it.

Overlap metrics for summaries where you have reference versions. Useful for spotting when a change makes things worse; weak as evidence that the output is good. A fluent summary can score well while getting a fact wrong, which is why this cannot be your only measure.

Model-graded scoring for anything subjective — tone, whether a claim is supported by the source, whether the reply actually answered the question. A 1-to-5 scale or a binary judgment, applied by a second model against explicit criteria. It scales past what your team can read, and it needs calibrating: have a human grade forty of the same cases and check the two agree. If they don't, fix the rubric before trusting the machine's version.

Human review on a sample, plus every failure. Not on everything — you don't have the budget and it isn't the best use of the attention you do have.

Week two: run it blind

Strip the vendor names off the outputs before anyone grades them. This is the single cheapest intervention in the entire process and it removes an enormous amount of unconscious tilt, particularly if one supplier has been more charming than the other.

Run the same test set through every option including, if there is one, the current manual process. The status quo's score is the most useful number you will produce, and the one most often skipped. Plenty of pilots have discovered a 91% system replacing a process that was running at 96%.

Record for each option: the score against threshold, the disqualifying-failure count, cost per transaction at projected volume, and observed latency. Four columns.

The final meeting

If the criteria were written down first, this meeting is short, and it is one of three outcomes.

Everything clears. Proceed, and carry the test set forward — it becomes your regression suite, and the thing that tells you six months from now whether quality has drifted.

Nothing clears. This is a successful pilot. You spent a fortnight instead of two quarters, and you have specific evidence about why. Keep the test set; the market moves and you can re-run it in six months for almost nothing.

Something clears on quality but fails on cost or latency. The most common result and the most interesting. It is usually an engineering conversation rather than a dead end — a smaller model for the easy majority, better request structuring, caching the parts that repeat. Worth one more week, with the same thresholds and no new ones.

What you must not do is pass something that missed the threshold because the demo was impressive and the team is keen. You wrote the number down for exactly this moment. That was the point of writing it down.