An eval gate is a threshold on a metric, wired to CI, so a regression fails the build instead of reaching users. I shipped one. My workout parser scores 22 fixed benchmarks and ends on process.exit(report.passRate >= 90 ? 0 : 1). Ninety percent of 22 means at most two may fail. That is a real gate and I still stand behind it. It also cannot tell me which two failed, or whether they broke for the same reason. Below is why one number cannot gate a system that fails in four different ways, the shape that can, and the scorer tier I had been skipping.
The gate I shipped, and what it averages away
My suite runs four assertions on every benchmark. The output must parse against a Zod schema, the sport must match, the step count must fall inside a band, and the power zones must land within 5% of the expected range. A benchmark passes only if all of its applicable assertions pass, and the suite passes if 90% of benchmarks do.
Read that as a measurement problem. Schema validity is a binary fact about a string. Sport correctness is a classification. Step count is arithmetic. Zone accuracy is a tolerance band. Those four fail for different reasons, cost different amounts to check, and have different fixes. I collapse them into one percentage and compare it to one number.
The arithmetic hides the rest. Twenty of 22 passing is 91%, so the build is green. If both failures are the schema check, my parser is emitting output no device can execute and the gate says fine. The reporter groups results by category and by language, which helps a human read a red build, but grouping is not gating: no group has a threshold of its own, so none can turn the build red on its own.
An aggregate gate hides which cases failed, so a green lane can carry a specific broken behaviour indefinitely. That is not a subtle risk. It is the direct consequence of averaging things that are not alike.
Four buckets, because “quality” is four questions
The taxonomy that fixed this for me is in Chip Huyen’s AI Engineering, chapter 4, page 161. Every criterion falls into one of four buckets:
- Domain-specific capability. Does the model have the underlying skill at all? No prompt fixes a capability that is not there. A model that never saw Latin will not translate Latin.
- Generation capability. Is the output itself good? Factual consistency and safety live here.
- Instruction-following capability. Did it do what you told it to do, in the shape you asked for?
- Cost and latency. Can you afford to ship it?
These are not four kinds of eval. They are four sources of truth, which is why each needs a different method. Huyen’s example is summarising a legal contract: understanding contracts is domain-specific, a faithful summary is generation, the requested length and format are instruction-following, and the price and the wait are the fourth.
Map my four assertions onto that and the collapse is obvious. Schema validity and step count are instruction-following: the model was told to return a shape and either did or did not. Sport correctness is domain capability. Zone accuracy is generation. The fourth bucket is not in my suite at all. I have never measured what a parse costs or how long it takes, and no threshold would notice if either doubled.
That is the part I did not see until I had the taxonomy. A missing criterion contributes no failures, so the bucket you lack is invisible from inside a pass rate.
There is a sharper reason to sort criteria this way, and page 173 puts it in a sidebar: “When a model performs poorly, it can either be because the model is bad or the instruction is bad.” Her clean case is on the page before: a sentiment classifier that reads sentiment perfectly and returns HAPPY and ANGRY instead of the three labels it was given. Domain capability is fine, instruction-following is broken, and a pass rate cannot tell you which. Only naming the bucket does. Test capability with a task that needs no unusual instruction, and test instruction-following with content the model certainly knows. One coordinate at a time.
I will add my own axis, because the buckets are not the only cut. A pipeline also fails by stage: retrieval, generation, tool calls. A stage is not a capability, and an actionable finding needs both coordinates.
Two thresholds, because one number cannot both block and steer
The shape that replaces the single gate is on page 178, in a table Huyen uses for picking a model. Five columns: criterion, metric, benchmark, hard requirement, ideal.
The two threshold columns are the point. A hard requirement is the floor that blocks a release. An ideal is what you are aiming at. One number in that space either blocks work it was never meant to block, or steers nothing because it sits where you already are. My 90% does both jobs badly at once: it is the floor and the aspiration, so I cannot raise the aspiration without making the build flakier.
The benchmark column is the one I would have skipped, and the most uncomfortable. It forces each row to say where its number comes from. In Huyen’s own five-row example, two rows point at an internal prompt dataset rather than a public benchmark. A row that cannot name its source is not ready.
There is a matching rule for setting a floor honestly: ask what breaks if the number is missed. If nothing breaks, it is an ideal. Page 178 makes the point about latency and it generalises. Ask users whether they want it faster and every one says yes, but high latency is usually an annoyance rather than a deal breaker. Preference cannot set a floor.
The same table gives the selection rule: filter on the axis you cannot compromise, then rank the survivors. A weighted average lets a candidate buy back a hard failure with a soft win, which is what a hard requirement exists to prevent.
The scorer tier in the middle
Filling a row means choosing how to score it, and here is the thing I changed my mind about. I used to treat scoring as two options: deterministic where possible, an LLM judge where not.
There is a tier between them. Factual consistency can be framed as textual entailment, a three-way classification of premise against hypothesis: entailment, contradiction, or neutral (page 168). Page 169 names a model that answers it, DeBERTa-v3-base-mnli-fever-anli, at 184 million parameters. Page 171 makes the same argument for safety, where specialised toxicity classifiers are much smaller, faster and cheaper than a general-purpose judge. Instruction-following has the same shape: IFEval defines 25 instruction types a program can verify with no model at all, and InFoBench decomposes the rest into yes/no questions.
So the gradient has three steps, not two: deterministic, then a small specialised model, then an LLM judge. The middle step matters because evals run on every change, so cost per call compounds. And the third has a cost that is easy to miss: change the judge’s prompt and you have changed the metric. The series breaks, and comparisons across the change are invalid. A judge is a versioned artifact, not an instrument.
One detail worth keeping from that three-way classification: neutral means consistency cannot be determined, which is not the same as false. A “cannot tell” scored as a failure punishes the model for the eval’s own gaps.
What this does not fix
The floors have no principled source. Chapter 4 offers one test, what breaks if you miss it, and nothing beyond that. A hard requirement is a judgement call wearing a number’s clothes, so write down who set it and why, next to the number.
A criteria table makes measurability bias efficient. Huyen states this against her own method on page 161: focusing only on what you can measure is looking for the lost key under the lamppost, because it is easier, not because the key is there. The counter is not to skip the table. It is to treat “we cannot measure this yet” as a research task with a budget rather than a reason to drop the feature.
A public benchmark cannot fill the benchmark column for you. In April 2024, 75% of the tasks in Eleuther’s evaluation harness were multiple-choice, which tests the ability to tell a good answer from a bad one: classification, not generation (page 163). Curate your own set, or the column names a source that measures a different skill from the one you ship.
A checklist you can lift
- Label every criterion with its bucket. A criterion that fits none of the four is a symptom, not a criterion.
- Add the bucket you do not have. It is almost always cost and latency, and a pass rate will never show you its absence.
- Give every row two thresholds. A hard requirement that blocks, an ideal that steers. Never one number doing both.
- Fill the
benchmarkcolumn first. A row that cannot say where its number comes from is not ready. - Test a floor by asking what breaks. If nothing breaks, you have written an ideal, and filter on it rather than averaging it away.
- Try the middle scorer tier before reaching for a judge, and version the judge prompt like code when you do reach for one. Changing it resets the metric.
I am keeping my 90% gate. What changes is that it stops being the whole answer and becomes one row in a table, next to rows that today have a metric and no threshold, and rows that have neither. The number was never the problem. Treating one number as if it could answer four different questions was.
Related writing
- Evaluating LLM outputs in production — the harness and the 90% gate this essay argues with; read it first for the mechanics.
- Observability for LLM pipelines — the same split after deploy: telemetry answers which stage, criteria answer which capability.
- Shipping production software with AI agents — the operating system around the gates, and why a check nobody reads is not a gate.
- A tolerant OAuth server validates your broken client — a hard requirement in protocol form: pin the spec, not the first partner who accepts you.