How to build an LLM evaluation harness before production.
A model that answers well in a demo has shown that a good answer is possible. Before it takes part in real work you need something else: how often it is wrong on the inputs that actually arrive, which of those errors matter, and what each result costs. An evaluation harness is the code and data that answer those questions the same way every time something changes.
Published
Start from the task, not the model
Write down the decision the output supports and what happens next. A classification that routes a ticket to a queue has a different cost of error than an extraction that updates a customer record. Define the output format precisely enough to check automatically, and name the person who decides what counts as correct when an answer is disputed.
Then rank the errors. Most workflows have a few mistakes that are expensive and many that are merely inconvenient. A harness that reports one accuracy number hides that difference, so the acceptance criteria should say which errors are tolerable, at what rate, and which ones must go to a person.
Build the case set from real inputs
Take cases from the work as it runs today, not from examples written for the project. Cover the ordinary inputs, the awkward ones, the ambiguous ones and the known failures of the current process. Label them with the people who own the task, and keep a record of cases where they disagreed; those are often the cases a model struggles with too.
- Count cases per category, not in total. A large set with three examples of the case that matters most tells you little about that case.
- Keep a held-out portion that is not used while prompts are tuned, so improvements are not simply fitted to the examples.
- Version the set. When a label is corrected, earlier results should still be comparable.
- Remove or mask data that the evaluation environment is not allowed to hold.
Score what the next step depends on
Choose metrics that match how the output is used. Structured fields can be compared exactly or after normalisation. Routing decisions are better read as a confusion matrix than as a single percentage, because it shows which wrong destinations occur. Where the workflow lets the model abstain or send a case to review, measure how often it does so and how accurate the remaining results are.
Check format validity separately from correctness. An output that fails schema validation is a different operational problem from one that is well formed and wrong. If a model grades free-text answers, compare its grades with human labels on a sample first; an unchecked grader adds a second unknown error rate.
Measure cost and latency in the same run
Record tokens, latency and cost for every case alongside its score. Include retries, fallback calls and the review time that uncertain results create, because those are part of the cost of the workflow rather than of the model. Look at tail latency as well as the average when a person is waiting for the result.
Make every change rerunnable
Treat the prompt, model identifier, retrieval configuration, output parser and dataset version as inputs to a run, and store results per run. A change to any of them then becomes a comparison against the previous run rather than an impression. Because model output can vary between calls, repeat a sample of cases and report the spread.
Use the harness as a gate. A prompt change that improves the average but increases the expensive error should not reach production on the strength of the average.
Compare with the current process
The useful question is rarely whether the model is perfect. It is whether the workflow with the model is better than the workflow without it on the measures that matter: errors on the important cases, handling time, review volume and cost per item. Measure the current process on the same cases where you can, so the comparison is like for like.
What to hand over
An evaluation that only its author can run stops being used when the author moves on. The handover should let the team that owns the workflow repeat it without help.
- The case set, its labelling guidelines and the record of disputed cases.
- The harness code, the acceptance criteria and the results of each run.
- Review thresholds and the measurements behind them.
- A procedure for rerunning the evaluation when the model, prompt or input data changes.
Where Landvex fits
In our production AI work the evaluation harness is built before the workflow is widened, and it is handed over with the code so your team can rerun it after every change. Read how we scope and deliver production ai.
Bring one concrete workflow
Describe the work as it happens today: who does it, how often, and what breaks. You will hear back from a founder.