---
title: "Evaluating LLM outputs in production: 22 benchmarks, a 90% gate"
description: "How I evaluate LLM outputs in production: a 22-benchmark eval harness with tiered assertions. A hard 90% CI gate fails the build on regressions."
date: 2026-07-18
updated: 2026-10-24
canonical: https://pabloalbaladejo.com/blog/evaluating-llm-outputs-in-production
---

# Evaluating LLM outputs in production: 22 benchmarks, a 90% gate

An evaluation harness for production LLM output has three parts. It has a set of fixed input cases and a set of assertions that check each output against expectations a machine can verify. And it has one pass/fail gate wired into CI. The gate is what makes it worth building: a regression fails the build instead of reaching users. Without a hard gate, "the model feels worse since we changed the prompt" is an argument. With one, it is a red build that points at a specific failing case and a specific error string. Below is how one of these works in a system I ship, and why the gate is set at 90% and not 100%. I also say what I would keep if I built it again.

## The real problem: structured output you cannot check by eye

Kaiord turns a workout description in natural language into a structured `Workout` object that a device can execute. A typical input is "45min sweet spot cycling: 10min warmup, 3x10min at 88-93% FTP with 2min recovery, 5min cooldown". The model does not return prose. It returns JSON that must satisfy a Zod schema. The schema asks for a `sport`, an ordered list of `steps`, and repeat blocks with a `repeatCount` and nested children. Each step can carry a target, such as a power range in watts or a heart-rate zone in bpm.

This is where quiet LLM regressions hide. The output *reads* fine, and it is valid JSON. But the sport flipped from cycling to running. Or the model flattened a `3x` interval block into three plain steps. Or it emitted a power target 15% too high. You cannot see any of that by skimming, and none of it works on real hardware. So the harness checks properties that must hold no matter how the model phrases the result.

## The harness: fixtures, tiered assertions, one gate

**Benchmarks are fixtures in version control.** There are 22 of them in `benchmarks.json`. Each is a small object: an input `text`, an `expectedSport`, a `minSteps`/`maxSteps` band, a `category`, a `language`, and an optional `zoneCheck`. I spread them on purpose. They cover four sports (cycling, running, swimming, generic) and several levels of complexity (simple sessions, interval sets, repetition blocks, mixed workouts). They cover three language settings (English, Spanish, and mixed). And they cover edge cases: a 5-minute sprint, a 3-hour ride, and one input with no expected sport at all. That one is `edge-ambiguous`, and it reads "workout for tomorrow morning, something moderate". The spread is the point. It is a fixed, diverse slice of the input distribution. I can run it again against any model, any prompt or any provider, and compare the results directly.

**Assertions are tiered, cheapest and hardest first.** For every benchmark, `evaluateBenchmark` runs the output through four checks:

1. **Schema validity.** The output must pass `workoutSchema.safeParse`. If it does not, the benchmark fails at once and the other checks never run: a workout with a broken structure has nothing worth measuring. This is the one check I never relax. The AGENTS.md for the suite documents it as a 100% target, and in the code it is a hard short-circuit.
2. **Sport correctness.** If the benchmark declares an `expectedSport`, `workout.sport` must match it. Documented target: ≥95%.
3. **Step count.** The number of steps must fall inside the benchmark's `[minSteps, maxSteps]` band. The count includes the children inside repeat blocks, not only the top level. This is what catches a `3x` block that got expanded or collapsed.
4. **Zone accuracy.** When a benchmark carries a `zoneCheck`, the active steps of the right target type must have bounds within ±5% of the expected range. The tolerance is deliberate. The code comment says it plainly: it exists "to absorb AI rounding." A model that answers "88-93% FTP" with "87.5-93.2%" is correct for a human and should be correct for the gate.

Each benchmark passes only if *all* of its applicable assertions pass. Errors collect in a list, so one failing case can report both a wrong sport and a wrong step count.

**One numeric gate on the exit code.** The runner scores the whole suite and ends with a single line: `process.exit(report.passRate >= 90 ? 0 : 1)`. That is the gate. Ninety percent of 22 benchmarks means at least 20 must pass: 19 of 22 is 86% and fails, 20 of 22 is 91% and passes. In plain terms, **at most two benchmarks may fail.** The reporter also groups results by category and by language. So a red build does not only say "84%". It says the Spanish cases, or the zone cases, are where it broke.

> **Correction, 2026-10-24.** This runner never executed. It needs a provider API key (`ANTHROPIC_API_KEY`) that the project does not have, so the 90% threshold never ran in CI. On 2026-09-07 kaiord removed the threshold ([#1232](https://github.com/pablo-albaladejo/kaiord/pull/1232)): a comparison that cannot run is not a gate. The source link is pinned to the version quoted here. The argument about what one aggregate number hides still stands.

## Why 90%, not 100%

Because the harness runs against a live model. A 100% gate on a non-deterministic system with ambiguous inputs is a trap. `edge-ambiguous` ("something moderate") has no single correct sport. If I demand that it passes every time, I will weaken the assertion until it does, and a weak assertion protects nothing. The ±5% zone tolerance already absorbs rounding on the well-specified cases. The 90% line absorbs the remaining variance on the two hardest cases without hiding a real regression. Real regressions are rarely subtle: a bad schema change or a broken prompt breaks many benchmarks at once and goes straight through the threshold. A threshold you picked on purpose, with a failure budget you can state ("two of twenty-two"), is worth far more than a round 100%. You will erode that 100% the first time it flakes.

## What the gate is for

The live eval is not a per-commit check. It costs tokens and needs an API key, so it lives in a workflow I trigger by hand. I run it, for example, before I switch the model or the provider. What runs on every commit is the *assertion logic itself*. That is ordinary code with unit tests and mocked workouts: the step counter, the zone-tolerance math and the schema short-circuit each have tests. The split matters. The cheap, deterministic part guards the harness. The expensive, non-deterministic part guards the model, and you decide when to pay for it.

The value shows up at the moments where you would otherwise have no information. A prompt rewrite, a model version bump, a provider swap, a schema migration. Any of those can move quality without a visible sign, and "it still looks right in the demo" is not evidence. The gate turns that worry into a number and a diff. When the number drops below 90, the grouped report tells me whether schema, sport, step count or zones broke, and in which language. That is usually enough to know whether I broke the prompt or the provider changed under me.

## A checklist you can lift

If you are wiring evals around your own LLM output, this is the shape I would use again:

- **Pin inputs as fixtures in version control.** A diverse, fixed set beats a large random one. You want results you can reproduce more than you want volume.
- **Assert on properties a machine can verify, never on exact strings.** Does it parse, is the class right, is the number inside a band. "Does it read well" is not an assertion.
- **Layer assertions hardest first, and short-circuit.** If the schema fails, do not score the rest. It is noise.
- **Put a single numeric gate on the exit code and wire it to CI.** The red or green build is the whole product.
- **Choose a threshold below 100% on purpose, and know its failure budget.** "At most two of twenty-two" is a decision. "Usually passes" is not.
- **Group results so a red build says *where*.** By category, by language, and by any other axis your inputs vary along.
- **Keep the expensive live run on demand, and unit-test the assertions themselves.** Guard the harness cheaply on every commit. Pay for the model run when a change justifies it.
- **Include ambiguous and adversarial cases, and accept that they will flip.** They keep the suite honest, and they are the reason the gate is not 100%.

None of this is exotic. It is a fixtures file, a hundred lines of assertions, a reporter, and one comparison against a number. The discipline is not in the code. It is in the refusal to ship structured LLM output that nothing checks.

## Related writing

- [Shipping production software with AI agents](https://pabloalbaladejo.com/blog/shipping-production-software-with-ai-agents) — the operating system around the agents that built this harness.
- [Observability for LLM pipelines](https://pabloalbaladejo.com/blog/observability-for-llm-pipelines) — evals catch regressions before deploy; telemetry catches them after.
- [GEO: how AI agents find you](https://pabloalbaladejo.com/blog/geo-how-ai-agents-find-you) — the discoverability side of shipping AI products.
- [A tolerant OAuth server validates your broken client](https://pabloalbaladejo.com/blog/tolerant-oauth-server-validates-your-broken-client) — the same fixture discipline applied to a protocol instead of a model.
- [A 90% pass rate is not a gate](https://pabloalbaladejo.com/blog/a-pass-rate-is-not-a-gate) — why the 90% gate here cannot say which of four capabilities broke, and the criteria table that can.
- [A gate is only as real as its run](https://pabloalbaladejo.com/blog/a-gate-is-only-as-real-as-its-run) — why the 90% gate in this harness never executed, and how to check that a gate runs.
- [AI-DLC: the only gates that count are the ones code enforces](https://pabloalbaladejo.com/blog/ai-dlc-gates-only-count-when-code-enforces-them) — why a gate needs an exit code, shown by two agent methods whose prose gates lost.

## FAQ

### What is an LLM eval harness?

An eval harness has three parts. It has fixed input cases, assertions that a machine can check against each output, and one pass/fail gate wired into CI. It turns 'the model feels worse today' into a red build with a specific failing case and error message.

### How do you evaluate LLM outputs that aren't deterministic?

Assert on structural properties that must hold whatever the wording. The JSON parses against the schema, the classification is correct, the number sits inside a tolerance band. Do not compare exact strings. Then set the pass threshold below 100%, so normal variance on a few hard cases does not flip the build. Keep the expensive live run on demand, and unit-test the assertions themselves.

### Why a 90% gate instead of requiring 100% of cases to pass?

The harness runs against a live model, and some benchmarks are ambiguous on purpose. A 100% gate on ambiguous inputs trains you to weaken your assertions until they pass, which defeats the point. 90% of 22 benchmarks means at least 20 must pass, so at most 2 may fail. That is tight enough to catch a real regression and loose enough not to flip on noise.

### Should LLM evals run on every commit?

The assertion logic should run on every commit: it is ordinary code with unit tests. The live model run usually should not, because it costs money and needs an API key. I keep it as a CI job that I trigger by hand, for example before I swap the model or the provider. I read its exit code as the gate.

## Sources

- [kaiord/packages/ai/src/evals: the eval suite](https://github.com/pablo-albaladejo/kaiord/tree/main/packages/ai/src/evals)
- [benchmarks.json: the 22 curated benchmark fixtures](https://github.com/pablo-albaladejo/kaiord/blob/main/packages/ai/src/evals/benchmarks.json)
- [assertions.ts: schema, sport, step-count and ±5% zone checks](https://github.com/pablo-albaladejo/kaiord/blob/main/packages/ai/src/evals/assertions.ts)
- [run-evals.ts: the exit(passRate >= 90 ? 0 : 1) gate](https://github.com/pablo-albaladejo/kaiord/blob/23c788cd/packages/ai/src/evals/run-evals.ts)
- [eval.yml: the manually triggered CI job](https://github.com/pablo-albaladejo/kaiord/blob/main/.github/workflows/eval.yml)
- [Kaiord documentation](https://kaiord.com/docs/)