LLM Evals and Guardrails in 2026: Building a Battery You Can Trust

Table of Contents

1. Why LLM evals are not unit tests
2. The 15-question trap: statistical power
3. Designing a sealed battery
4. Choosing a judge: never let a model grade itself
5. Layered scoring: separate transport from behaviour from correctness
6. The statistics you actually need
7. Security evals and guardrails
8. Four eval bugs that produced confident, wrong verdicts
9. A worked result: RAG versus fine-tuning
10. Running evals in CI without going broke
11. Checklist before you trust a score

Why LLM evals are not unit tests

A unit test asserts one deterministic output. An LLM eval estimates a distribution: the same prompt can produce a right answer today and a subtly wrong one after a temperature change, a model upgrade, a prompt tweak or a retrieval index rebuild. That difference drives every design decision in this guide.

The practical consequence: a passing eval is a measurement with error bars, not a green check. If you cannot state the uncertainty of your score, you do not yet have an eval — you have an anecdote.

• Non-determinism — the same input can score differently across runs, so a single run proves little.
• Graded, not binary — most answers are partially right; you need a scoring rubric, not an equality assertion.
• The grader is a system too — if a model grades the output, the grader has its own failure modes and biases.
• Contamination — the thing under test may have memorised your test set, which looks like excellent performance.

The 15-question trap: statistical power

The most common eval in the wild is a hand-written list of about fifteen questions. It feels responsible and it is nearly worthless for deciding anything, because the confidence interval on a 15-item proportion is roughly plus or minus 25 percentage points. A system scoring 9/15 and one scoring 12/15 are statistically indistinguishable.

This is why the first real decision is not which model to use — it is how many items you need to detect the effect size you care about. If you want to detect a 5-point difference, a few dozen items will never do it.

Designing a sealed battery

A battery is not one bag of questions. It is several pools, each answering a different question about the system. The battery described here was generated from 82 facts, each verified against two independent sources, expanded into 830 graded items.

• Held-out core — facts the system should know, never seen during training or indexing in that exact form.
• Held-out paraphrase — the same facts asked many different ways. This is the largest pool, because robustness to phrasing is where most systems quietly fail.
• Seen — items the system did train on. Scoring near-perfect here while failing held-out is the signature of memorisation.
• Contamination — deliberately corrupted facts (a swapped digit, a wrong name) that the system must refuse or correct. A system that confidently agrees is hallucinating, not recalling.
• Fresh — facts created after the training cutoff, to test whether retrieval is actually live.
• Security — prompt injection, jailbreak and data-exfiltration probes.
• Reliability — the same item repeated, to measure run-to-run variance and punt rate.

Then seal it: hash the manifest (SHA-256) and freeze it. An eval you can edit after seeing the results is not a measurement, it is a negotiation.

Choosing a judge: never let a model grade itself

LLM-as-judge is the only scalable way to grade open-ended answers, and it has one dominant failure mode: self-preference. A model family tends to rate its own outputs higher. If you fine-tune a Qwen model and grade it with a Qwen judge, you have measured family loyalty, not quality.

• Cross-family judge — grade a Qwen system with a Gemma or Mistral judge, and vice versa.
• Collect, then judge — run the system under test to completion and store raw outputs; grade in a separate phase. Never let the judge and the system share a GPU or a run.
• Typed verdicts — make the judge emit a structured verdict, not prose, so scoring is mechanical and auditable.
• Spot-check the judge — hand-label a sample and measure judge-vs-human agreement before you trust any number it produces.

Layered scoring: separate transport from behaviour from correctness

Most eval harnesses collapse every failure into one number, which makes debugging impossible. Score in three independent layers instead:

1. Transport — did the request succeed at all? HTTP 429, a timeout or a truncated stream is an infrastructure failure, not a wrong answer. Counting these as wrong is the single most common way to slander your own model.
2. Behaviour — did it answer, refuse, or abstain? Refusal is a valid and sometimes correct behaviour; it must be a distinct category, never silently folded into wrong.
3. Correctness — and only for answers that actually arrived, typed by what kind of claim it is: an exact identifier, a number, a date, or a free-text assertion each need different matching rules.

With layers, "accuracy dropped 8 points" becomes answerable: was it more refusals, more rate limits, or genuinely worse answers? Without layers, you are guessing.

The statistics you actually need

• Bootstrap confidence intervals — resample your item scores to get an interval around the headline number. Report the interval, always.
• McNemar test for paired comparisons — when comparing two systems on the same items, the right question is not "which score is higher" but "how many items did A get right that B got wrong, and vice versa".
• Generalization gap — seen-pool score minus held-out score. A gap of more than a couple of points means memorisation, not learning.
• Wilson interval for attack success rate — security results are proportions near zero, where the naive normal interval is badly wrong.

Security evals and guardrails

Capability and safety are separate measurements and must be scored separately. A system can be more accurate and more leaky at the same time.

• Prompt injection — instructions hidden in retrieved documents, tool output or user content that try to override the system prompt.
• Exfiltration — attempts to make the model reveal its prompt, its context, or another user data.
• Scope — off-domain questions the system should refuse. Grounded systems need a distance threshold so an out-of-corpus question is refused rather than hallucinated.
• Refusal parity — if you fine-tune, you must explicitly train refusal examples, or the tuned model will lose the base model safety behaviour and score worse on security while scoring better on accuracy.

Four eval bugs that produced confident, wrong verdicts

These are real defects found in a working harness, each of which had already changed a conclusion before it was caught. Your eval is a system: it has bugs, and its bugs look exactly like model quality changes.

1. Rate limits counted as failures. HTTP 429 responses were scored as wrong answers, so the system looked worse purely because the eval ran too fast. Fixed by the transport layer above.
2. A null field read as a refusal. A missing value in the response was interpreted as "the model refused", inflating the refusal rate and depressing accuracy.
3. Substring matching produced phantom leaks. A naive contains() check flagged a secret because its letters appeared inside an ordinary longer word. Security scoring must use word boundaries or exact tokens.
4. The answer key was stale. The live system returned newer, correct values than the frozen expected answers — so the eval punished the system for being more current than its grader. Version your answer key alongside your data.

A worked result: RAG versus fine-tuning

The method above was used to settle a concrete question: for a knowledge assistant over a private corpus, is retrieval-augmented generation or a fine-tuned small model the right answer? Measured properly, at each approach’s own ceiling, the result was counter-intuitive.

They tie on the axes people argue about and separate on the axes people forget. The decision is therefore not "which is smarter" but which structural property the product needs: freshness and citations point to retrieval; single-shot latency and offline operation point to fine-tuning; a hybrid gives both at the cost of two systems to maintain.

Running evals in CI without going broke

• Tiered batteries — a fast smoke tier on every commit, the full sealed battery nightly or before a release.
• Cache aggressively — outputs are deterministic enough per (model, prompt, seed) to cache; only re-run what changed.
• Gate on the interval, not the point estimate — fail the build when the lower bound of the interval crosses your threshold, not when the raw number wobbles.
• Keep the battery in version control, sealed — with its hash, so a diff in results is always attributable to the system, never to a quietly edited question.

Checklist before you trust a score

1. The battery is large enough to detect the difference you care about, and you can state the confidence interval.
2. It has held-out, seen, contamination and security pools — not just questions you hope it answers.
3. It is sealed and hashed, and was not edited after you saw results.
4. The judge is from a different model family than the system under test, and you measured judge-human agreement.
5. Transport failures, refusals and wrong answers are three different numbers.
6. The generalization gap between seen and held-out is small.
7. The harness itself has unit tests, and you have re-read the answer key recently.

Related Guides